feature article
Subscribe Now

My Ruler Must Be Broken

Measuring Power Consumption Can Drive You Crazy

“The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny…’” – Isaac Asimov

According to Adam Savage, the difference between science and just screwing around is writing it down. It’s the measurement – the annotation, the calibration, the methodical note-taking – that separates good science (and engineering) from mere hacking and tinkering. Without good measurements there can be no good science.

So raise a caliper and spare a thought for the measurers in our industry – the ones wielding the oscilloscope probes, the voltmeters, the electron microscopes. For they are the ones who enable us to produce better, faster, and more reliable electronics.

They’re also the ones who get to say, “That’s funny…”

It’s that single unexpected measurement that marks the difference between dreary routine and technological breakthrough. When you find the outlier you begin to discover that things aren’t as they seem. And you start down the path of discovery.

Okay, enough with the bromides. What we’re here to discuss is the work that EEMBC does in benchmarking low-power MCUs. That’s a tough task – tougher than I would have thought at first – but it’s made even tougher by the pesky nature of reality. As we mentioned a few weeks ago, today’s microprocessors and microcontrollers aren’t as stable and predictable as we like to think. Even mass-produced chips can vary from batch to batch, week to week, or chip to chip. Things aren’t always what they seem.

That’s why we need benchmarks: calibrated tests that tell us exactly what the chip is doing and how it compares to other chips doing the same work. Normally, benchmarks measure performance, but EEMBC does power-consumption benchmarks, too. Very useful stuff when you’re trying to decide which low-power MCU to put in your next wearable device or battery-powered gizmo.

EEMBC’s ULPbench (ultra-low-power benchmark) is designed for exactly that purpose. Take a bunch of competing MCUs and run them all through the same suite of tests, then measure both their active-mode and their sleep-mode energy usage. Voila! At the end you’ve got a big table of numbers to help you pick the best and most power-efficient chip for your design. How hard can that be?

Well… EEMBC’s lab technicians noticed a funny thing. The benchmark results weren’t very repeatable. Which sort of violates the Prime Directive of benchmark testing: all results should be objective, verifiable, and repeatable. Otherwise, you might as well be picking random lottery numbers. But the EEMBC folks are smart. These guys design and run benchmarks all day long. They know a thing or two about calibrating instruments, isolating variables, and generating repeatable results. So what gives?

In some cases, the exact same benchmark test running on the exact same hardware produced results that were 5% or 10% or even 15% different from test to test. That’s not supposed to happen.

One hint was that the sleep-mode power varied more than the active-mode power. ULPbench exercises both characteristics separately, on the theory that many MCU applications are asleep most of the time, waiting for an interrupt or an outside stimulus. For these designs, sleep-mode power affects battery life more than active-mode current consumption. And the sleep-mode numbers were all over the map, even on the same hardware.

CMOS circuity leaks. It leaks a lot. Electrons fairly fall out of your chips when they’re not working. It’s not unusual for a CPU or MCU to leak more current when it’s not running than it uses when it is running. Your MCU is like the old SR-71 spy plane, which was said to leak fuel on the runway because it was designed for supersonic speeds, where the air friction would heat the plane and cause it to expand, sealing the gaps. It was great at moving fast but leaked badly when sitting still.

And since CMOS circuits leak when they’re sitting still, the passive-mode power measurements were like trying to catch rain in a thimble. There’s no way to control leakage current. At least, not if you aren’t controlling the fab line that manufactures the chip. And even then, a difference of a few atoms here or there can make a measureable difference to a chip’s power characteristics. Particularly in sleep mode.

According to EEMBC’s tests, the biggest determinants to power consumption are temperature and wafer lot. Temperature makes a big difference, as many of us have probably already noticed. The hotter a chip gets, the more power it consumes – even if it’s sleeping. Especially if it’s sleeping. The correlation is not linear, either, it’s exponential. At room temperature (say, 25°C), a difference of a few degrees will barely make a detectable difference in power. But at the upper end of the chip’s rated temperature range, a few degrees can make a big difference. Good to know if you’re designing monitoring equipment that’s going to sit outside in the sun or be buried inside a fanless enclosure.

The other big variable is wafer lot, and there’s nothing we can do about that. If two chips are “sisters” – that is, they’re cut from the same silicon wafer and processed simultaneously – they’ll probably vary by no more than 5%, according to EEMBC’s observations. But if the two chips were processed on different wafers on different days, the variation can be as much as 15%. For supposedly identical chips. Same part number, same manufacturer, same software, same everything. It’s a total crap shoot.

Again, the difference is most noticeable in sleep mode. The more your chip sleeps (or hibernates, or dozes, or snoozes, etc.) the more its power will vary from the norm. For some very-low-duty-cycle applications like wireless utility meters, this is a big deal. Those devices are supposed to last 10 years on a single battery because they spend 99.99% of their time in sleep mode. Their active duty cycle is something like 1 in 10,000. Paradoxically, their power consumption would be more stable if they ran more often. Battery life might be shorter, but it would be more predictable.

Sadly, this problem only gets worse over time. CMOS circuits leak, and small-geometry CMOS leaks more than the older-generations stuff. So the more “advanced” your chip manufacturing is, the more it’s going to leak and the more unpredictable it’s going to be. A chip built in 90-nanometer silicon is going to leak more than one built in 180nm, and so on. Granted, newer process geometries lead to all sorts of advantages, including lower active-mode current consumption, but deteriorating passive-mode leakage is the price we pay. 

That means you’re sometimes better off using a chip built with “outdated” technology. A 0.35-micron device won’t leak nearly as much as one built on a newer fab. It might be physically larger (okay, it will definitely be larger), and it might not run as fast, but it’ll leak less and have a more stable and predictable power profile. If you’re in the 99% sleep camp, that’s probably the direction you want to be heading. 

There are a lot of variables in play here, but one good rule of thumb at EEMBC is that the crossover point is around the 1:5000 duty cycle. In other words, if your device is active less than 0.02% of the time, you’re probably better off with a chip built on older technology (180nm vs. 90nm, for instance). If your device is active more often than that, sleep-mode power starts to get overwhelmed by active power, so you’ll want a newer device.

Tests and benchmarks are supposed to be about repeatability. They’re supposed to be stable, reliable, and trustworthy. And normally, they are. Performance benchmarks are generally rock-solid. If you test a Core i7-2270K or a Cortex-A25 over and over, you get the same performance scores. It’s digital. Deterministic. Repeatable. It’s when someone gets different performance numbers that you suspect cheating.

But power benchmarks aren’t like that. Unlike traditional performance benchmarks, they’re measuring an analog quantity, not a quantized one. You expect some variation just because of the instruments and the nature of power itself. But not big 10% and 15% differences. And not because of the humidity in the lab.

EEMBC set itself a daunting task when it undertook ULPbench. But the results are actually better than many people expected. Their numbers don’t vary because of any flaw or limitation in the benchmark. On the contrary: they vary precisely because the benchmark is so accurate and well designed. The variations they’re seeing in ULPbench scores are real. The chips really are exhibiting random variations in behavior. It’s not measurement error; it’s physics.

Going forward, users of EEMBC’s ULPbench will have to calibrate their own expectations to align with reality. You’re simply not going to be able to duplicate EEMBC’s results – or anyone’s results – with any degree of accuracy. Want to drive your lab partner nuts? Ask him to benchmark the power consumption of the same MCU development board every day for a week. See how long it takes before he starts to say, “That’s funny…” 

Leave a Reply

featured blogs
Nov 25, 2020
It constantly amazes me how there are always multiple ways of doing things. The problem is that sometimes it'€™s hard to decide which option is best....
Nov 25, 2020
[From the last episode: We looked at what it takes to generate data that can be used to train machine-learning .] We take a break from learning how IoT technology works for one of our occasional posts on how IoT technology is used. In this case, we look at trucking fleet mana...
Nov 25, 2020
It might seem simple, but database units and accuracy directly relate to the artwork generated, and it is possible to misunderstand the artwork format as it relates to the board setup. Thirty years... [[ Click on the title to access the full blog on the Cadence Community sit...
Nov 23, 2020
Readers of the Samtec blog know we are always talking about next-gen speed. Current channels rates are running at 56 Gbps PAM4. However, system designers are starting to look at 112 Gbps PAM4 data rates. Intuition would say that bleeding edge data rates like 112 Gbps PAM4 onl...

featured video

Introduction to the fundamental technologies of power density

Sponsored by Texas Instruments

The need for power density is clear, but what are the critical components that enable higher power density? In this overview video, we will provide a deeper understanding of the fundamental principles of high-power-density designs, and demonstrate how partnering with TI, and our advanced technological capabilities can help improve your efforts to achieve those high-power-density figures.

featured paper

Learn how designing small is easier than you think

Sponsored by Texas Instruments

Designing with small-package ICs is easier than you think. Find out how our collection of industry's smallest signal-chain products can help you optimize board space without sacrificing features, cost, simplicity, or reliability in your system.

Click here to download the whitepaper

featured chalk talk

Minitek Microspace

Sponsored by Mouser Electronics and Amphenol ICC

With the incredible pace of automotive innovation these days, it’s important to choose the right connectors for the job. With everything from high-speed data to lighting, connectors have a huge impact on reliability, cost, and design. In this episode of Chalk Talk, Amelia Dalton chats with Glenn Heath from Amphenol ICC about the Minitek MicroSpace line of automotive- and industrial-grade connectors.

Click here for more information about Amphenol FCI Minitek MicroSpace™ Connector System