editor's blog
Subscribe Now

Is Exactly-Once Delivery Possible with MQTT?

We looked at MQTT, along with various other messaging protocols, not too long ago. Included in the discussion was a brief mention of MQTT having quality-of-service (QoS) features – and one of those is debatable.

MQTT’s optional QoS levels are to guarantee delivery no more than once (in which case you may miss a message), at least once (in which case you might get a duplicate), or exactly once. This latter one is the subject of debate: various threads in various discussions argue why it’s impossible in practice.

This can be discussed on a number of levels. Within the protocol itself, these QoS levels are implemented through acknowledgment sequences.

For at-least-once service, the sender has to keep a copy of the message until it receives a PUBACK from the receiver. In the absence of a PUBACK, the message must be resent. There are two possible causes for a resend: the original message may indeed not have been received, or the PUBACK may not have been received. In the latter case, the original was received and sent on for processing, but the sender doesn’t know that. So it sends a duplicate, but the receiver will have no record that this is a duplicate message; to it this is simply a new message. So it will process it just as it did the first one.

 Figure_1.png

Figure 1. At-least-once delivery. Note that in the third scenario, the message is sent for processing (the dashed arrow) twice.

Exactly-once is slightly more involved. And there are a couple choices the receiver has in how it handles the message once received, and I don’t want to get lost in that, so I’ll abstract the receiver behavior (and the drawing picks one option). As above, the sender sends the message and keeps a copy. When the receiver gets the message, it sends a PUBREC back to the sender, but it keeps a record of the receipt for the moment. Once the sender gets the PUBREC, it can delete its copy of the message, but it keeps a record of the PUBREC for the moment and sends a PUBREL back to the receiver, letting it know that it knows that the receiver got its copy. Once the receiver gets the PUBREL, then it sends back a PUBCOMP and discards its record of the message; when the sender gets the PUBCOMP, then it discards its record of the PUBREC and the exchange terminates.

 Figure_2.png

Figure 2, Exactly-once delivery. One message processing option shown. Note that in no scenario is the message sent for processing (dashed arrow) more than once.

The basic idea is that both sender and receiver keep a record of the state until they’re both convinced that the message was delivered. If, during the process, one of the ACKs gets lost and a duplicate message is sent, the receiver is still tracking that message, so it can ignore the duplicate.

That’s all well and good from the limited standpoint of message delivery, but messages have a purpose; presumably their intent is that something happen as a result of the message. These are quaintly referred to as “side effects” in the discussions, which they are in a computer science sense, but they may really be the “main effect” desired as a result of the message.

If the message instructs an oven to change its temperature, then, from a message-sending standpoint, the fact that the temperature changed after the message is a side effect. But from a system standpoint, it’s the main effect; that’s the purpose of the message. So there’s a question of liability here: if the message doesn’t arrive, then clearly there’s a messaging problem. But if the message arrives, but then something goes wrong in the process of changing the temperature, who’s responsible?

MQTT washes its hands of the issue as soon as the receiving end confirms that it got the message. If the process of sending the message further on in the receiving system breaks down, or if some other system element fails, then, in reality, you still need a message resend. But if a “reduce by 10 degrees” message is received and processed, but something in an internal acknowledgment loop fails, then the message might be processed twice with the net result of reducing the temperature by 20 degrees.

The basic problem is that there are many steps along the way where failure can occur. Even in cases of redundancy or failover, there are typically delays either in detecting that there’s a problem or in performing a switch, and things can go wrong in those timing gaps.

The solution here comes back to “idempotence”: to quote Wikipedia, this “is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.” You have to do some work on the receiving side to make this happen.

Essentially, the receiving side has to take full ownership of state; it can’t count on the sender to share in storage of the state in case state changes get lost between sender and receiver. This is the principle behind the REST architecture, and it may be familiar in how the HTTP protocol works.

You could also argue that, instead of sending the instruction, “reduce by 10 degrees,” the message should say, “Set to this new temperature” (which happens to be 10 degrees lower). But that shifts state responsibility to the sender. If the receiver’s true state changed and the sender didn’t find out, then the state is effectively undefined.

Bringing things back to MQTT, the problem is that the protocol considers itself successful simply when it delivers its message exactly once. That’s a parochial viewpoint. If reliability is important enough to use that QoS level, then you can’t count on the QoS, because it gives only message, not system, guarantees.

(At least, that’s my take on the whole debate…)

Leave a Reply

featured blogs
May 24, 2022
Today is going to be my monthly update. This normally runs on the last Friday of the month, but that's a Cadence Global Recharge Day, so we will all be off. For various other reasons, I need to... ...
May 20, 2022
I'm very happy with my new OMTech 40W CO2 laser engraver/cutter, but only because the folks from Makers Local 256 helped me get it up and running....
May 19, 2022
Learn about the AI chip design breakthroughs and case studies discussed at SNUG Silicon Valley 2022, including autonomous PPA optimization using DSO.ai. The post Key Highlights from SNUG 2022: AI Is Fast Forwarding Chip Design appeared first on From Silicon To Software....
May 12, 2022
By Shelly Stalnaker Every year, the editors of Elektronik in Germany compile a list of the most interesting and innovative… ...

featured video

EdgeQ Creates Big Connections with a Small Chip

Sponsored by Cadence Design Systems

Find out how EdgeQ delivered the world’s first 5G base station on a chip using Cadence’s logic simulation, digital implementation, timing and power signoff, synthesis, and physical verification signoff tools.

Click here for more information

featured paper

Intel Agilex FPGAs Deliver Game-Changing Flexibility & Agility for the Data-Centric World

Sponsored by Intel

The new Intel® Agilex™ FPGA is more than the latest programmable logic offering—it brings together revolutionary innovation in multiple areas of Intel technology leadership to create new opportunities to derive value and meaning from this transformation from edge to data center. Want to know more? Start with this white paper.

Click to read more

featured chalk talk

"Scalable Power Delivery" for High-Performance ASICs, SoCs, and xPUs

Sponsored by Infineon

Today’s AI and Networking applications are driving an exponential increase in compute power. When it comes to scaling power for these kinds of applications with next generation chipsets, we need to keep in mind package size constraints, dynamic current balancing, and output capacitance. In this episode of Chalk Talk, Mark Rodrigues from Infineon joins Amelia Dalton to discuss the system design challenges with increasing power density for next generation chipsets, the benefits that phase paralleling brings to the table, and why Infineon’s best in class transient performance with XDP architecture and Trans Inductor Voltage Regulator can help power  your next high performance ASIC, SoC or xPU design.

Click here for more information about computing and data storage from Infineon