editor's blog
Subscribe Now

Is Exactly-Once Delivery Possible with MQTT?

We looked at MQTT, along with various other messaging protocols, not too long ago. Included in the discussion was a brief mention of MQTT having quality-of-service (QoS) features – and one of those is debatable.

MQTT’s optional QoS levels are to guarantee delivery no more than once (in which case you may miss a message), at least once (in which case you might get a duplicate), or exactly once. This latter one is the subject of debate: various threads in various discussions argue why it’s impossible in practice.

This can be discussed on a number of levels. Within the protocol itself, these QoS levels are implemented through acknowledgment sequences.

For at-least-once service, the sender has to keep a copy of the message until it receives a PUBACK from the receiver. In the absence of a PUBACK, the message must be resent. There are two possible causes for a resend: the original message may indeed not have been received, or the PUBACK may not have been received. In the latter case, the original was received and sent on for processing, but the sender doesn’t know that. So it sends a duplicate, but the receiver will have no record that this is a duplicate message; to it this is simply a new message. So it will process it just as it did the first one.

 Figure_1.png

Figure 1. At-least-once delivery. Note that in the third scenario, the message is sent for processing (the dashed arrow) twice.

Exactly-once is slightly more involved. And there are a couple choices the receiver has in how it handles the message once received, and I don’t want to get lost in that, so I’ll abstract the receiver behavior (and the drawing picks one option). As above, the sender sends the message and keeps a copy. When the receiver gets the message, it sends a PUBREC back to the sender, but it keeps a record of the receipt for the moment. Once the sender gets the PUBREC, it can delete its copy of the message, but it keeps a record of the PUBREC for the moment and sends a PUBREL back to the receiver, letting it know that it knows that the receiver got its copy. Once the receiver gets the PUBREL, then it sends back a PUBCOMP and discards its record of the message; when the sender gets the PUBCOMP, then it discards its record of the PUBREC and the exchange terminates.

 Figure_2.png

Figure 2, Exactly-once delivery. One message processing option shown. Note that in no scenario is the message sent for processing (dashed arrow) more than once.

The basic idea is that both sender and receiver keep a record of the state until they’re both convinced that the message was delivered. If, during the process, one of the ACKs gets lost and a duplicate message is sent, the receiver is still tracking that message, so it can ignore the duplicate.

That’s all well and good from the limited standpoint of message delivery, but messages have a purpose; presumably their intent is that something happen as a result of the message. These are quaintly referred to as “side effects” in the discussions, which they are in a computer science sense, but they may really be the “main effect” desired as a result of the message.

If the message instructs an oven to change its temperature, then, from a message-sending standpoint, the fact that the temperature changed after the message is a side effect. But from a system standpoint, it’s the main effect; that’s the purpose of the message. So there’s a question of liability here: if the message doesn’t arrive, then clearly there’s a messaging problem. But if the message arrives, but then something goes wrong in the process of changing the temperature, who’s responsible?

MQTT washes its hands of the issue as soon as the receiving end confirms that it got the message. If the process of sending the message further on in the receiving system breaks down, or if some other system element fails, then, in reality, you still need a message resend. But if a “reduce by 10 degrees” message is received and processed, but something in an internal acknowledgment loop fails, then the message might be processed twice with the net result of reducing the temperature by 20 degrees.

The basic problem is that there are many steps along the way where failure can occur. Even in cases of redundancy or failover, there are typically delays either in detecting that there’s a problem or in performing a switch, and things can go wrong in those timing gaps.

The solution here comes back to “idempotence”: to quote Wikipedia, this “is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.” You have to do some work on the receiving side to make this happen.

Essentially, the receiving side has to take full ownership of state; it can’t count on the sender to share in storage of the state in case state changes get lost between sender and receiver. This is the principle behind the REST architecture, and it may be familiar in how the HTTP protocol works.

You could also argue that, instead of sending the instruction, “reduce by 10 degrees,” the message should say, “Set to this new temperature” (which happens to be 10 degrees lower). But that shifts state responsibility to the sender. If the receiver’s true state changed and the sender didn’t find out, then the state is effectively undefined.

Bringing things back to MQTT, the problem is that the protocol considers itself successful simply when it delivers its message exactly once. That’s a parochial viewpoint. If reliability is important enough to use that QoS level, then you can’t count on the QoS, because it gives only message, not system, guarantees.

(At least, that’s my take on the whole debate…)

Leave a Reply

featured blogs
Jul 3, 2020
[From the last episode: We looked at CNNs for vision as well as other neural networks for other applications.] We'€™re going to take a quick detour into math today. For those of you that have done advanced math, this may be a review, or it might even seem to be talking down...
Jul 2, 2020
Using the bitwise operators in general -- and employing them to perform masking, bit testing, and bit setting/clearing operations in particular -- can be extremely efficacious....
Jul 2, 2020
In June, we continued to upgrade several key pieces of content across the website, including more interactive product explorers on several pages and a homepage refresh. We also made a significant update to our product pages which allows logged-in users to see customer-specifi...

Featured Video

Product Update: New DesignWare® IOs

Sponsored by Synopsys

Join Faisal Goriawalla for an update on Synopsys’ DesignWare GPIO and Specialty IO IP, including LVDS, I2C and I3C. The IO portfolio is silicon-proven across a range of foundries and process nodes, and is ready for your next SoC design.

Click here for more information about DesignWare Embedded Memories, Logic Libraries and Test Videos

Featured Paper

Cryptography: How It Helps in Our Digital World

Sponsored by Maxim Integrated

Gain a basic understanding of how cryptography works and how cryptography can help you protect your designs from security threats.

Click here to download the whitepaper

Featured Chalk Talk

Power Supply Design

Sponsored by Mouser Electronics and KEMET

There is a bewildering range of choices for components for power supply design. Considering EMI protection, surge protection, transformers, rectifiers - the list goes on and on. In this episode of Chalk Talk, Amelia Dalton chats with Nick Stephen of KEMET to sort out the puzzle of power supply component selection, and to look at the latest trends and best practices in power supply design.

Click here for more information about KEMET Electronics METCOM MPX1 Metal Composite Power Inductors