Well, this is different: A benchmark that doesn’t measure performance. The newest benchmark to come out of EEMBC (the Embedded Microprocessor Benchmark Consortium) isn’t designed to measure MIPS or megahertz, but instead is geared toward quantifying a chip’s power-saving modes. And Volkswagen and other automakers are deeply interested in the outcome.

Here’s what happened. VW, like all automakers, consumes a bunch of microcontrollers in its cars. In addition to the familiar VW brand, VW also owns Audi, Lamborghini, Bentley, Porsche, Bugatti, the Spanish brand SEAT, the Eastern European Skoda nameplate, Scania trucks, and Ducati motorcycles. In short, the company makes a lot of vehicles, from very high-end exotics to very low-end commuter boxes.

And you know what problem VW’s engineers face? They can’t make sense of microcontrollers’ low-power modes.

I can see their point. Every MCU seems to have two or three different power-saving modes, from “sleep” and “snooze” to “hibernate” or “comatose.” No two sleep modes are the same, and there’s no standardization whatsoever on what they’re called or how they work. An engineer can easily spend hours or days scrutinizing the datasheet to figure out what power-saving modes will work best for them – and that’s assuming all the claims made in the datasheet are correct and relevant to your application. Factor in another few weeks for testing the various modes, and you might develop enough confidence to actually use the chip in your system.

Then, of course, all that research goes out the window when you upgrade to the next chip.

Back to VW. The company scatters microcontrollers all over its vehicles, from engine and powertrain controllers to MCUs that handle the rear view mirror (seriously). The average car has over a dozen MCUs in it, and a high-end Audi, Lamborghini, or Bugatti can easily top 100. Most of those MCUs sleep most of the time. The chip that monitors the trunk to see if it’s latched, for example, might ping the sensor only once per second. The rest of the time, it’s monitoring the car’s battery temperature, or checking that the taillights aren’t burnt out, or – more than likely – just asleep. With dozens of similar MCUs dotted all over the car, their power consumption starts to add up. This is especially troubling with the increased emphasis on gas mileage and fuel efficiency, and doubly troubling when you get to electric vehicles (which VW is also developing). It’s not hard to eat up a couple of horsepower just generating DC current for all those microcontrollers.

Performance isn’t a big deal for most of these MCUs. After all, how hard is it to check the trunk latch? But sleep-mode power consumption is, and so is wakeup time. Does the chip retain all of its internal state information in sleep mode, or just the SRAM contents? Can I wake it with a timer, or does it require an interrupt? Will it awaken fast enough to catch traffic on the CAN bus, or do I need to keep it awake more often? All these questions, and more, were troubling VW’s engineers.

In a reversal of roles, VW approached EEMBC to ask for help developing a suite of benchmarks that would help the German automaker make sense of rival MCU vendors’ competing claims. VW supplied some of its own sensor data gleaned from in-house testing, while EEMBC came up with a vendor-neutral way to take that input and provide some sort of score for grading power modes.

Here’s how the test works: An MCU under scrutiny will have its inputs (say, CAN or an ADC) connected to a signal generator that mimics VW’s real-world data. The MCU vendor (Microchip, Renesas, Freescale, et al) gets to decide which of its many available power-saving modes is most suitable for that application. EEMBC randomizes the input data a little bit to prevent gaming the system, and checks to see if the MCU wakes up in time to perform all the necessary tasks, and, if so, how much energy it consumed. Assuming the chip passes, VW might (or might not) award the win to the most power-efficient chip. If the MCU fails for any reason, it’s hasta la vista, baby and on to the next MCU.

Why wouldn’t VW always pick the most power-efficient chip? For the same reasons you don’t. Sometimes it’s better to stick with one particular chip family or architecture because you prefer their development tools, or you get a break on pricing, or you just like the salesperson. Benchmarks don’t dictate behavior, they just inform our decisions. And VW has to make a lot of these decisions.

Remarkably, the company often clings to older MCUs and avoids upgrading simply because of the drama involved in qualifying a new chip. New silicon inevitably has newer, better power-saving modes, but it’s such a nuisance to document and test them that VW often doesn’t bother. They’re leaving energy efficiency on the table because it’s too much trouble to quantify it. That’s the real value of EEMBC’s new benchmark suite. It makes new chips interesting again, instead of an irritation. And that should, uh, drive more innovation.