I wish you could visit me in Max’s World, where everything is bigger, brighter, and more colorful. The birds sing sweeter (and in harmony), the flowers are more fragrant, the butterflies are more brillacious—a neologism of brilliant and bodacious that I just invented—and the beer flows plentifully and cold. Most importantly, everyone is nice, kind, honorable, and trustworthy. No one would even dream of doing anything naughty.
Unfortunately, like you, I am forced to spend my waking hours in the real world where naughtiness is the order of the day. Take printed circuit board assembly (PCBA), for example, which is where all of the electronic components (like resistors, capacitors, ICs, connectors, etc.) are mounted and soldered onto the board.
You’d think that the hard part of a new electronic product development was creating and verifying the design. When it comes to actually manufacturing the circuit boards, for example, what could possibly go wrong? Well, obviously, there are myriad technical issues to be addressed, but today’s automation and PCB assembly processes are spectacularly awesome. Of more concern is the possibilities that stem from human incompetence and maleficence.
Sometimes we may be talking about honest mistakes, like an inadvertent component substitution. For example, I heard tell of a board intended for automotive applications where one of the components was the correct function but wasn’t of automotive grade, resulting in a multimillion-dollar recall event when the issue eventually came to light.
Other times, we may be talking about a contract manufacturer (CM) deliberately substituting a specified part for something that’s supposed to be equivalent (perhaps because it’s cheaper), but that turns out to be not as good. There’s also the possibility of the CM knowingly or unknowingly using cloned, counterfeit, or reclaimed parts. Another concern is when a bad actor, like a nation state, arranges for additional components to be added to a board in order to compromise its cybersecurity.
As one example of this latter case, I’m thinking of the Bloomberg Businessweek Report circa 2018 (see also the Dialogue China Column). The gist of this column is that a company called Elemental Technologies designed a high-end server. In addition to commercial companies like Apple and Amazon, these servers ended up in Department of Defense data centers, Navy warships, the CIA, and… the list goes on.
The servers in question were assembled for Elemental by a San Jose-based company called Supermicro (more formally, Super Micro Computer Inc.). According to Bloomberg, Chinese operatives infiltrated Supermicro’s supply chain and embedded tiny microchips into the server motherboards. These chips were said to be about the size of a grain of rice and capable of creating backdoors into affected systems.
Apple, Amazon, Supermicro, the US Department of Homeland Security, and the UK’s National Cyber Security Centre have all stated (and keep on stating, even if you don’t ask them anything) that they have no knowledge of any of this… so it must be true.
Why am I waffling on about this? Well, I was just chatting with Eyal Weiss, the CTO at Cybord. Eyal informed me that the folks at Cybord have just announced an advanced visual-AI PCBA inspection solution called ShieldScan that delivers unprecedented visibility and security in electronics manufacturing. In a crunchy nutshell, ShieldScan offers the following:
- Detects malicious hardware implants to prevent rogue chips from compromising security.
- Detects missing components.
- Detects unauthorized replacements.
- Verifies component origins to meet strict regulations, including US country-of-origin bans.
- Achieves unmatched traceability with 100% visual verification of every component.
Before we talk about ShieldScan in more detail, I think you’ll be interested in hearing the backstory.
Eyal spent 15 years working on a big project with a team of 25 engineers and scientists. He says this was a new technology he’d invented, but not something he’s free to discuss in detail. Suffice it to say that the team did the development, got the project working, performed the testing and qualification, went to production, and finally deployed.
After a few months in the field, they started to see failures. Initially, everybody blamed the new technology. Eventually, the problem was tracked down to faulty capacitors. It turned out that the parts used on the boards were supposed to be new but were in fact old stock that had been sitting “on the shelf” for around 10 years, resulting in bad contacts and reliability issues. As Eyal says, “A billion-dollar 15-year project was almost canceled because of some bad 1- or 2-cent capacitors.” (Eyal has since discovered that between 70% and 80% of the failures in electronics come from the components used on the boards, not from the process, and not from the design.)
Although the team had to rework thousands of boards, this story has a happy ending, because the product ended up being wildly successful, ultimately receiving something called the Israel Security Prize, which is sort of equivalent to the Presidential Medal of Honor in the USA.
Eyal says that when he went to the folks at the manufacturer and asked why they used old parts, they said they thought they were using new parts, and they showed him all the documentation that showed that the capacitors were supposed to be new. They also said that there was no process in place to test such components. That they identified whitelist and blacklist suppliers, that they only acquired parts from trusted sources, and thereafter they assumed that the parts were good.
As Eyal told me, “Once we had successfully finished that project, I knew what my next project was going to be, which is why I founded Cybord.”
The overall idea here is easy to wrap one’s brain around, which is fortunate because my brain is not as limber as it used to be. Let’s start with the fact that PCBA facilities already have production lines that boast sophisticated pick-and-place machines. These machines, which pick and place all of the components on the board, have high-resolution cameras that take pictures of the bottom of each component to verify alignment.
PCBA facilities also have automated optical inspection (AOI) machines that take high-resolution pictures of the top of the board. Pre-reflow AOI checks the solder paste application before the soldering process takes place, but we don’t care about that here. We’re interested in post-reflow AOI, which inspects completed boards for assembly defects after components are soldered. This uses image processing algorithms to examine things like
- Solder joints
- Bridging or tombstoning
- Component placement
- Polarity
- Missing components
Two interesting points leap out from the above. First, although there are checks that the components have been soldered the right way, there are no checks that the right components have been used. Second, as part of the existing production line, we already have high-resolution photos of the bottom and top of all the components.
This means that no changes are required to the existing production line. All that is required is an additional server that is granted access to all the pictures as they are taken. Cybord’s software is so fast and unintrusive that it can recognize a problematic component from its bottom image and instruct the pick-and-place machine to discard that component in real-time without disrupting the flow.
In turn, this leads us to the fact that Cybord is at heart a software company. They don’t sell servers or anything like that, although they do recommend configurations of appropriate CPU and GPU hardware. What they do sell is sophisticated AI software that has been trained on countless billions of components.
How good is this? Well, suffice it to say that I’m impressed. In the case of the bottom-side component photos, ShieldScan can detect things like tarnished contacts and other things indicating age. It can use a combination of clues to tell you, “This is the right part from the right manufacturer, but it’s supposed to be new, and if it’s less than ten years old, then I’m a monkey’s uncle!”
In the case of the top-side photos, ShieldScan’s AI-driven optical character recognition (OCR) and natural language processing (NLP) ensure unmatched accuracy in analyzing manufacturer markings, lot codes, date-codes, and compliance data.
So much tasty information to digest (Source: Cybord)
Right from the get-go, this will prevent problems like the non-automotive grade part problem we discussed earlier in this column. This data will also allow ShieldScan to determine if the part is from the desired manufacturer or from an acceptable substitute.
But what about a counterfeit component where the markings look almost identical? Well, the “almost” part of the question is the first giveaway, because ShieldScan can detect tiny flaws in fonts and suchlike. Also, in addition to any textual data, each packaging machine has tiny differences from its cohorts. This means that each machine essentially leaves its “fingerprints” on every package it touches, so even if the markings look good to the untrained eye, ShieldScan can still detect a “bad egg” as it were.
You don’t believe me? Well, what about things like surface mount capacitors and resistors that don’t have any markings at all? Believe it or not, ShieldScan can identify the manufacturer from the package alone, as illustrated below. (Color me impressed!)
Could you tell the difference? (Source: Cybord)
ShieldScan doesn’t rely on BOM lists or CAD files. All it needs for each type of board is a brief training session that involves scanning a batch of approved boards, after which it’s ready to rock and roll.
We’ve all seen those “Spot the Difference ” puzzles featuring a pair of side-by-side images that look almost identical at first glance. The challenge is to look closely to find small differences between them. This is just like that. Look at the images below. Can you spot any differences between the reference (left) and analyzed (right) versions?
Can you spot the difference? (Source: Cybord)
This is hard enough to do once. Now imagine trying to do it on a production line running at full pelt. Just to put you out of your misery, the customer added a component to test the system to see if ShieldScan could spot an additional device (the upper red bounding box indicates this in the image below). ShieldScan also detected foreign object debris (the lower red bounding box in the image below), which could turn into a performance issue downstream.
Who put that there? (Source: Cybord)
As one final thought, even reputable manufacturers can get a little desperate when supply chain issues rear their ugly heads, which is when slithy toves start to crawl out of their lairs. We all remember the pandemic-era supply chain problems as if they were yesterday. We can but hope we aren’t currently heading willy-nilly into a tariff-driven supply chain storm.
All I can say is that, were I in charge of building mission-critical or safety-critical boards (well, almost any board now that I come to think about it), ShieldScan would be at the top of my shopping list. What say you? Do you have any thoughts you’d care to share on anything you’ve read here?
