As the Internet of Trouble (IoT) continues to evolve, most of us designing electronic systems are working to make our devices “smart.” By adding a microcontroller and some snazzy firmware, we can create products that take care of themselves – monitoring critical operational parameters and taking proactive steps to keep everything in line. One goal is to reduce the burden of responsibility on the user, which is really a release-note euphemism for “prevent the stupid customer from breaking our well-designed hardware.”
We gain a measure of post-release control as well, as we can release firmware updates that alter the behavior of the product in the field, even after the customer has bought it and placed it in service. And, by taking advantage of agile software development practices, our system can continue to improve and evolve long after the initial sale. In fact, customers have come to expect this sort of behavior from products, eagerly awaiting software and firmware updates that will give their product new capabilities and fix existing annoyances.
Unfortunately, there is a very real downside to this arrangement. What happens if a new firmware bug creeps in that can actually damage or destroy the hardware? Then, we run the risk of alienating customers by breaking the things they already own. This may sound far-fetched, but I just experienced exactly that scenario with a product from DJI.
Like most engineers, I have some nerdy hobbies. One of those is building and flying RC aircraft. For the past several years, I’ve been building and flying multicopters designed for aerial photography and videography. It’s a fun and challenging hobby. Last summer, while on a boat in Alaska, my favorite copter met with an untimely demise. Since there were many things I still wanted to film during the trip, I opted to buy a ready-made replacement and have it shipped to a port where I could pick it up. (There was no reasonable way to build a new drone myself on the boat.) I chose the DJI Inspire 1 – a very capable and well-designed platform for aerial photography.
The Inspire 1 worked well for the rest of the season, and then I stored it for the winter. It features “smart” Lithium Polymer (LiPo) batteries that monitor their condition, manage charging, and even automatically optimize the cell voltage for long-term storage. The algorithm is simple: When the battery detects that it has not been used for a specified amount of time (10 days by default) it begins to slowly discharge the cells back to an appropriate storage voltage – around 3.8V per cell. This improves battery longevity compared with storing at a “full” 4.2V per cell charge. This “smart” device stuff is cool!
I prepared my batteries for storage, installed the latest firmware update, and put them away for the winter.
Fast forward a few months to springtime.
Last week, I got a notice that a new firmware release was available for the Inspire 1. DJI releases firmware bundles that can include updates for the Inspire 1 itself, the batteries, and the controller. I was excited to read the release notes, as it was about time to get my stuff out of storage to begin the new season. I went to see what new features they’d added.
I got worried on the second item of the release notes:
1. Improved encryption to enhance transmission security. Aircraft and Remote Controller must be upgraded to prevent unlinking.
2. Fixed issue of batteries over-discharging when stored for extended periods of time (over 90 days).”
Uh, oh. My batteries had been stored for well over 90 days, and over-discharging LiPos can damage or destroy them.
I retrieved my batteries from storage and tried powering them up… Nada. Every single pack seemed stone cold dead. These are not cheap batteries, by the way. Each pack costs upward of $150 USD. Mine were less than a year old and had been flown about six times each before storage. I plugged each pack into a charger and watched them for a couple hours – nothing. Still no sign of life whatsoever.
Reading forums online, I found that there was a “stuck in hibernation” condition that these battery controllers could fall into, and that you could force a hard reset by opening the case and powering down the controller board for awhile, then reconnecting it.
I then opened the case of each pack farther and manually measured the cell voltage. Bad news. 0.7-0.9V per cell (LiPo Cells are supposed to be over 3V.) Mine were toast. I manually applied a charging voltage to the cells and brought them carefully and slowly up to 4.1V (fire extinguisher at the ready). Nope. As soon as the charging source was removed, the cells trundled right back down to sub-1V level.
I then emailed DJI customer support, because occasionally one needs a humorous escape from reality – courtesy of a privately-held Chinese electronics company. I gave a detailed explanation of my problem. DJI replied about 24 hours later:
Thank you for the information. Please also reply with a video of the problem for verification.
They wanted a video of dead batteries?
Hmmm.. I replied asking them what exactly they’d like to see in this video.
“Dear Kevin Morris
Thank you for your understanding.
Please send us a video for verification purposes, of the problem you described in the previous email.
OK, great. That clears things right up. Not a video of unicorns frolicking in the field or eagles soaring lazily overhead. Just a nice, action-packed video of some dead battery packs. I created a cinematic masterpiece, showing my hand as I repeatedly pushed the power buttons on each pack with no LED response, connected each pack to a charger with no response, and installed the pack into the aircraft with no response. I considered an hour-long video of the packs sitting idle and dead connected to a charger, but bandwidth got the better of that idea.
A couple days later, DJI responded again:
“Dear Kevin Morris
Thank you for your email,
Can you kindly provide a little more information? It would be greatly appreciated and allow us to greater assist you in getting your aircraft back in the air as soon as possible.
Hmmm… Wonder what kind of “more information” they want? I decided to reply with a comprehensive recap of everything I had told them so far, in case they hadn’t read the entire thread.
DJI then replied again:
“Thank you for contacting DJI Customer Support.
We’d like to ensure that your battery is communicating properly with the unit. We would recommend that you attempt to refresh the latest available firmware update for the battery. The firmware can be found on the DJIWebsite.
Please be aware that the information provided indicates that the batteries are outside of their warranty period. If the issue does require you to send the battery in for repair, it would not qualify for a warranty replacement. To review the warranty period for all parts related to our products, please review the After-Sales Service Policy, located at the following link:
You may contact us Monday-Friday, 9:00am-5:00pm PST at 818-235-0789.
DJI Customer Support”
I responded that I could not update the firmware on completely dead batteries. Updating the firmware requires, you know, power. My batteries had none. And the battery controller could not be updated even if activated even with the batteries connected to power, because the update mechanism requires the batteries to be installed in the aircraft.
I also explained that, although the batteries were out of warranty, it was not the batteries that were defective. It was firmware that was installed much later (a mandatory firmware update, by the way).
DJI replied again:
“Dear Kevin Morris
Thank you for your response. As your batteries are out of warranty, it would require for you to purchase new batteries. Please check the online store: http://store.dji.com/?site=brandsite”
Well, that’s very kind of them. I replied with a plea, explaining that my batteries were destroyed by a firmware update issued after the battery warranty had expired. And, in a touch of wonderful irony, that the manifestation of the problem actually required more time than the battery warranty period, since the batteries had to be stored for several months before the bug would do the damage.
DJI replied one final time:
“Dear Kevin Morris
Thank you for your response. As your batteries are out of warranty, it would require for you to purchase new batteries. Please click on this link to learn more on this point:
For quicker assistance please feel free to utilize our Live Chat option by following the link provided: http://www.dji.com/support
While this is a bit of a pathological worst case for firmware updates damaging stable hardware, it is a wake-up call for all of us that our system is only as good as its weakest link, and quite often that weakest link will be software or firmware. Furthermore, continual updating of software on hardware that is deployed in the field runs the risk of high-cost failures that could not be detected when the initial product was released. In my case, the loss was just a few hundred bucks worth of batteries, but even in this kind of system, firmware bugs could easily be in a position to cause fires or in-flight failures of the aircraft.
There is often a temptation to issue a quick software patch to repair a problem found in the field. Our engineering discipline should stop us from doing that until we have been able to do adequate testing of the entire system to be sure we are not introducing a worse problem than the one we’re fixing.
Dear Kevin Morris,
Our Support Rep has indicated that your ticket has been Resolved.
If you believe that the ticket has not been resolved, please reply to this email to automatically reopen the ticket.
If there is no response from you, we will assume that the ticket has been resolved and the ticket will be automatically closed after 48 hours.
DJI TechnologySupport Team
I’m not quite sure what they mean by “Resolved” here.
One thought on “Rock, Paper, Firmware”
Resolved – a final state of a customer service event where the “customer stops complaining at us” goal has been achieved.
Apparently has nothing to do with customer satisfaction. I’ll bet someone got a bonus for closing tickets at no cost to DJI. They will soon come out with a book (in Chinese only) “How to improve profits by hiding disgust for customers behind language translation and cultural differences.”
I’ve been looking at aerial photography for farm inspection. I now know to cross DJI off my list. I’d been on the phone, plowing my way up the management chain until I got satisfaction. I don’t have time and my blood pressure doesn’t need that.