“All I was doing was trying to get home from work.” – Rosa Parks
Bugs are an inevitability, but they’re especially embarrassing when your customer finds them before you do. All things being equal, it would be better if your devices notified you of a bug instead of your customers calling to complain that their shiny and expensive new device just tipped over.
Once you confirm that the bug is real, the next steps are finding and fixing it, and then somehow updating all the affected devices in the field. No small task, but one that many of us have lived through.
Given that many new products are connected to “the cloud” in some manner, it seems like we could use that connection for remote diagnosis and notification, right? Wish granted. Tiny software vendor Percepio has created just such a toolkit that allows remote devices in the field to send us text alerts when they hiccup, and then follow up with diagnostic traces that we can analyze at our desks. It’s about the most helpful and cooperative that a failing device can be.
Sweden-based Percepio is just now rolling out DevAlert, an add-on to its existing debugging tools that creates a back channel from devices to their developers. The idea is that developers get notified the moment a device detects that something’s wrong. What you do after that depends on how serious or widespread you judge the problem to be. Information is power, and DevAlert aims to put that control in developers’ hands.
DevAlert consists of three parts. First, there’s the firmware on the device itself, along with a RAM buffer for trace data. This is an internal health monitor that maintains a ring buffer of relevant event data. You get to decide the size of the buffer and what qualifies as an “event.” The buffer can be as small as 4KB, and Percepio says even that is enough to store “a few hundred to a few thousand” events, which might translate into milliseconds or seconds of real time.
The second part is Amazon’s AWS IoT Core cloud service. This is where the magic happens. Afflicted devices send their alerts and trace data up to AWS, where Percepio’s own software analyzes and classifies the reports. If the reported event is deemed anomalous – that is, it’s a potential bug – it sends a notification directly to the contact person, usually someone on the development team. The total flight time is just a few seconds, about the same as for any text message or email.
Just as important, DevAlert filters out redundant reports of the same (or substantially the same) bug, so that you’re not inundated with reports from devices in the field all complaining of the same thing. It still stores those reports. They’re not lost. They’re just not all forwarded directly to you in a flurry of redundant “help me!” pleas.
The third leg of DevAlert’s stool is the client-side diagnostic GUI, called Tracealyzer. Percepio has been selling Tracealyzer for years and, naturally, considers it an ideal debugging tool for real-time tracing. Tracealyzer is very graphically rich and relies on visualization rather than tables of numbers or source code listings to help make anomalous behavior stand out from mundane routines.
For now, DevAlert is specific to AWS and two operating systems, FreeRTOS (owned by Amazon) and ThreadX (now owned by Microsoft). “Support for other RTOSes is available on demand,” says Percepio, and could be spun up on request “in a matter of weeks.”
DevAlert is sold as an annual subscription service rather than a one-off product purchase. Pricing varies with the number of notifications you expect to receive.
Shipping with bugs – or rather, knowingly shipping with known bugs – is the new normal. “We’ll fix it in the next firmware release” seems to be the first thing new product managers learn to say. With that in mind, an early-warning system like DevAlert seems prudent. Its communication is all one-way, but by sending helpful diagnostic data nearly in real time, developers can get a head start on analyzing problems as they occur. Sending out over-the-air firmware updates and appeasing cranky customers are different problems.
