editor's blog
Subscribe Now

Is Exactly-Once Delivery Possible with MQTT?

We looked at MQTT, along with various other messaging protocols, not too long ago. Included in the discussion was a brief mention of MQTT having quality-of-service (QoS) features – and one of those is debatable.

MQTT’s optional QoS levels are to guarantee delivery no more than once (in which case you may miss a message), at least once (in which case you might get a duplicate), or exactly once. This latter one is the subject of debate: various threads in various discussions argue why it’s impossible in practice.

This can be discussed on a number of levels. Within the protocol itself, these QoS levels are implemented through acknowledgment sequences.

For at-least-once service, the sender has to keep a copy of the message until it receives a PUBACK from the receiver. In the absence of a PUBACK, the message must be resent. There are two possible causes for a resend: the original message may indeed not have been received, or the PUBACK may not have been received. In the latter case, the original was received and sent on for processing, but the sender doesn’t know that. So it sends a duplicate, but the receiver will have no record that this is a duplicate message; to it this is simply a new message. So it will process it just as it did the first one.

 Figure_1.png

Figure 1. At-least-once delivery. Note that in the third scenario, the message is sent for processing (the dashed arrow) twice.

Exactly-once is slightly more involved. And there are a couple choices the receiver has in how it handles the message once received, and I don’t want to get lost in that, so I’ll abstract the receiver behavior (and the drawing picks one option). As above, the sender sends the message and keeps a copy. When the receiver gets the message, it sends a PUBREC back to the sender, but it keeps a record of the receipt for the moment. Once the sender gets the PUBREC, it can delete its copy of the message, but it keeps a record of the PUBREC for the moment and sends a PUBREL back to the receiver, letting it know that it knows that the receiver got its copy. Once the receiver gets the PUBREL, then it sends back a PUBCOMP and discards its record of the message; when the sender gets the PUBCOMP, then it discards its record of the PUBREC and the exchange terminates.

 Figure_2.png

Figure 2, Exactly-once delivery. One message processing option shown. Note that in no scenario is the message sent for processing (dashed arrow) more than once.

The basic idea is that both sender and receiver keep a record of the state until they’re both convinced that the message was delivered. If, during the process, one of the ACKs gets lost and a duplicate message is sent, the receiver is still tracking that message, so it can ignore the duplicate.

That’s all well and good from the limited standpoint of message delivery, but messages have a purpose; presumably their intent is that something happen as a result of the message. These are quaintly referred to as “side effects” in the discussions, which they are in a computer science sense, but they may really be the “main effect” desired as a result of the message.

If the message instructs an oven to change its temperature, then, from a message-sending standpoint, the fact that the temperature changed after the message is a side effect. But from a system standpoint, it’s the main effect; that’s the purpose of the message. So there’s a question of liability here: if the message doesn’t arrive, then clearly there’s a messaging problem. But if the message arrives, but then something goes wrong in the process of changing the temperature, who’s responsible?

MQTT washes its hands of the issue as soon as the receiving end confirms that it got the message. If the process of sending the message further on in the receiving system breaks down, or if some other system element fails, then, in reality, you still need a message resend. But if a “reduce by 10 degrees” message is received and processed, but something in an internal acknowledgment loop fails, then the message might be processed twice with the net result of reducing the temperature by 20 degrees.

The basic problem is that there are many steps along the way where failure can occur. Even in cases of redundancy or failover, there are typically delays either in detecting that there’s a problem or in performing a switch, and things can go wrong in those timing gaps.

The solution here comes back to “idempotence”: to quote Wikipedia, this “is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.” You have to do some work on the receiving side to make this happen.

Essentially, the receiving side has to take full ownership of state; it can’t count on the sender to share in storage of the state in case state changes get lost between sender and receiver. This is the principle behind the REST architecture, and it may be familiar in how the HTTP protocol works.

You could also argue that, instead of sending the instruction, “reduce by 10 degrees,” the message should say, “Set to this new temperature” (which happens to be 10 degrees lower). But that shifts state responsibility to the sender. If the receiver’s true state changed and the sender didn’t find out, then the state is effectively undefined.

Bringing things back to MQTT, the problem is that the protocol considers itself successful simply when it delivers its message exactly once. That’s a parochial viewpoint. If reliability is important enough to use that QoS level, then you can’t count on the QoS, because it gives only message, not system, guarantees.

(At least, that’s my take on the whole debate…)

Leave a Reply

featured blogs
Apr 11, 2021
https://youtu.be/D29rGqkkf80 Made in "Hawaii" (camera Ziyue Zhang) Monday: Dynamic Duo 2: The Sequel Tuesday: Gall's Law and Big Ball of Mud Wednesday: Benedict Evans on Tech in 2021... [[ Click on the title to access the full blog on the Cadence Community sit...
Apr 8, 2021
We all know the widespread havoc that Covid-19 wreaked in 2020. While the electronics industry in general, and connectors in particular, took an initial hit, the industry rebounded in the second half of 2020 and is rolling into 2021. Travel came to an almost stand-still in 20...
Apr 7, 2021
We explore how EDA tools enable hyper-convergent IC designs, supporting the PPA and yield targets required by advanced 3DICs and SoCs used in AI and HPC. The post Why Hyper-Convergent Chip Designs Call for a New Approach to Circuit Simulation appeared first on From Silicon T...
Apr 5, 2021
Back in November 2019, just a few short months before we all began an enforced… The post Collaboration and innovation thrive on diversity appeared first on Design with Calibre....

featured video

Learn the basics of Hall Effect sensors

Sponsored by Texas Instruments

This video introduces Hall Effect, permanent magnets and various magnetic properties. It'll walk through the benefits of Hall Effect sensors, how Hall ICs compare to discrete Hall elements and the different types of Hall Effect sensors.

Click here for more information

featured paper

From Chips to Ships, Solve Them All With HFSS

Sponsored by Ansys

There are virtually no limits to the design challenges that can be solved with Ansys HFSS and the new HFSS Mesh Fusion technology! Check out this blog to know what the latest innovation in HFSS 2021 can do for you.

Click here to read the blog post

Featured Chalk Talk

Electronic Fuses (eFuses)

Sponsored by Mouser Electronics and ON Semiconductor

Today’s advanced designs demand advanced circuit protection. The days of replacing old-school fuses are long gone, and we need solutions that provide more robust protection and improved failure modes. In this episode of Chalk Talk, Amelia Dalton chats with Pramit Nandy of ON Semiconductor about the latest advances in electronic fuses, and how they can protect against overcurrent, thermal, and overvoltage.

More information about ON Semiconductor Electronic Fuses