Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Pohl's law: Nothing is so good that somebody, somewhere, will not hate it.


devel / comp.arch.embedded / Re: MQTT QoS vs TCP

SubjectAuthor
* MQTT QoS vs TCPpozz
+* Re: MQTT QoS vs TCPEd Prochak
|`* Re: MQTT QoS vs TCPpozz
| `* Re: MQTT QoS vs TCPDavid Brown
|  `* Re: MQTT QoS vs TCPpozz
|   `* Re: MQTT QoS vs TCPDavid Brown
|    `* Re: MQTT QoS vs TCPpozz
|     `* Re: MQTT QoS vs TCPDavid Brown
|      `- Re: MQTT QoS vs TCPpozz
`- Re: MQTT QoS vs TCPIvan Shmakov

1
MQTT QoS vs TCP

<u47g53$jgfj$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1599&group=comp.arch.embedded#1599

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: pozzu...@gmail.com (pozz)
Newsgroups: comp.arch.embedded
Subject: MQTT QoS vs TCP
Date: Fri, 19 May 2023 11:39:47 +0200
Organization: A noiseless patient Spider
Lines: 20
Message-ID: <u47g53$jgfj$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 19 May 2023 09:39:47 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="58910ab3dbcebee3424783b084f4b3b9";
logging-data="639475"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19eAspgE7wxYvTzR3SnyFHMGlm4rnnLEnk="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Cancel-Lock: sha1:5H9yIK18WrsVUZm/42XCNJUsZeo=
 by: pozz - Fri, 19 May 2023 09:39 UTC

I know TCP is able to guarantee the delivery of messages from sender to
receiver, without corruption (thanks to checksums), in order and without
duplication (thanks to sequence numbers).

So I'm confused when I read something about MQTT QoS. For example, QoS 1
(at least once) uses ack (PUBACK) and retransmissions (with packet ids)
to guarantee that the message is delivered at least one time.

I'm wondering how this is possible over a TCP connection that uses the
same technique of ack and sequence numbers.

From what I know about TCP, if some data isn't well acknowledged by the
receiver, the sender automatically resends the packets not acked. This
is performed by TCP/IP stack or kernel, without the application knows
anything about this.

It seems to me it's impossible that a MQTT client needs to resend a MQTT
message because it wasn't received by the broker. If this happens, TCP
should signal the error to the application that should close and try to
reopen the connection.

Re: MQTT QoS vs TCP

<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1600&group=comp.arch.embedded#1600

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:622a:1826:b0:3e3:8bbd:b367 with SMTP id t38-20020a05622a182600b003e38bbdb367mr832467qtc.7.1684509272738;
Fri, 19 May 2023 08:14:32 -0700 (PDT)
X-Received: by 2002:a05:620a:4010:b0:758:898b:8973 with SMTP id
h16-20020a05620a401000b00758898b8973mr565618qko.3.1684509272562; Fri, 19 May
2023 08:14:32 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.misty.com!border-2.nntp.ord.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Fri, 19 May 2023 08:14:32 -0700 (PDT)
In-Reply-To: <u47g53$jgfj$1@dont-email.me>
Injection-Info: google-groups.googlegroups.com; posting-host=2600:1700:5f82:5000:6496:ba6c:ea1d:ad8f;
posting-account=3ty6FAkAAACYEfch20jQ1ZACFatw-Vdx
NNTP-Posting-Host: 2600:1700:5f82:5000:6496:ba6c:ea1d:ad8f
References: <u47g53$jgfj$1@dont-email.me>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
Subject: Re: MQTT QoS vs TCP
From: edproc...@gmail.com (Ed Prochak)
Injection-Date: Fri, 19 May 2023 15:14:32 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Lines: 37
 by: Ed Prochak - Fri, 19 May 2023 15:14 UTC

On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
> I know TCP is able to guarantee the delivery of messages from sender to
> receiver, without corruption (thanks to checksums), in order and without
> duplication (thanks to sequence numbers).
>
> So I'm confused when I read something about MQTT QoS. For example, QoS 1
> (at least once) uses ack (PUBACK) and retransmissions (with packet ids)
> to guarantee that the message is delivered at least one time.
>
> I'm wondering how this is possible over a TCP connection that uses the
> same technique of ack and sequence numbers.
>
> From what I know about TCP, if some data isn't well acknowledged by the
> receiver, the sender automatically resends the packets not acked. This
> is performed by TCP/IP stack or kernel, without the application knows
> anything about this.
>
> It seems to me it's impossible that a MQTT client needs to resend a MQTT
> message because it wasn't received by the broker. If this happens, TCP
> should signal the error to the application that should close and try to
> reopen the connection.

TCP is only one of the lower levels of the protocol stack.
Data can sometimes be lost in the higher levels.

Secondly, there is the issue of resend timeouts. If TCP fails to deliver
the message past the MQTT retry time limit, then MQTT will resend the
message.

HTH,
Ed

Re: MQTT QoS vs TCP

<u489dl$nqcl$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1601&group=comp.arch.embedded#1601

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: iva...@siamics.netNOSPAM.invalid (Ivan Shmakov)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Fri, 19 May 2023 16:51:01 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 25
Message-ID: <u489dl$nqcl$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
Injection-Date: Fri, 19 May 2023 16:51:01 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="6448ed01eecce87e2af15cf3313ef1c2";
logging-data="780693"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+zi7JS/oG5ZkM3YUu1jygN"
User-Agent: slrn/1.0.3 (Linux)
Cancel-Lock: sha1:aagLKPP+5fpPyXqX4QkoohfvQ18=
License: CC0-1.0 (original contributions only)
 by: Ivan Shmakov - Fri, 19 May 2023 16:51 UTC

>>>>> On 2023-05-19, pozz <pozzugno@gmail.com> wrote:

> It seems to me it's impossible that a MQTT client needs to resend
> a MQTT message because it wasn't received by the broker. If this
> happens, TCP should signal the error to the application that
> should close and try to reopen the connection.

After which an MQTT client will need to retransmit its
message, no? The difference between QoS 0 and QoS 1 boils
down to whether the sender of the message is actually
bothered to do that.

(I /think/ QoS 1 also allows for reliable delivery along the
entire client-to-server-to-another-client path, but I'm not
sure about that.)

TCP will also signal an error when the message has
successfully reached its destination, but the respective
acknowledgement has not; as such, an application level
protocol running over TCP generally needs the means to weed
out the duplicates that are bound to happen in this case.
Which is what MQTT QoS 2 does.

--
FSF associate member #7257 http://am-1.org/~ivan/

Re: MQTT QoS vs TCP

<u4f4if$23mii$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1603&group=comp.arch.embedded#1603

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: pozzu...@gmail.com (pozz)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Mon, 22 May 2023 09:11:11 +0200
Organization: A noiseless patient Spider
Lines: 54
Message-ID: <u4f4if$23mii$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 22 May 2023 07:11:11 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="c8db7f232f2717aed271abb9ef7c734b";
logging-data="2218578"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19531p9JY0KXacrpVS/k7bbZCJ2r6FCHwE="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Cancel-Lock: sha1:3FA0wbrnEabqLuzycxo3gs/GO6Q=
In-Reply-To: <2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
 by: pozz - Mon, 22 May 2023 07:11 UTC

Il 19/05/2023 17:14, Ed Prochak ha scritto:
> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>> I know TCP is able to guarantee the delivery of messages from sender to
>> receiver, without corruption (thanks to checksums), in order and without
>> duplication (thanks to sequence numbers).
>>
>> So I'm confused when I read something about MQTT QoS. For example, QoS 1
>> (at least once) uses ack (PUBACK) and retransmissions (with packet ids)
>> to guarantee that the message is delivered at least one time.
>>
>> I'm wondering how this is possible over a TCP connection that uses the
>> same technique of ack and sequence numbers.
>>
>> From what I know about TCP, if some data isn't well acknowledged by the
>> receiver, the sender automatically resends the packets not acked. This
>> is performed by TCP/IP stack or kernel, without the application knows
>> anything about this.
>>
>> It seems to me it's impossible that a MQTT client needs to resend a MQTT
>> message because it wasn't received by the broker. If this happens, TCP
>> should signal the error to the application that should close and try to
>> reopen the connection.
>
> TCP is only one of the lower levels of the protocol stack.
> Data can sometimes be lost in the higher levels.

In this case, there's only one higher level, that is MQTT application.
How an application running on a machine could lost something? Network
links aren't reliable, but applicaions running on a processor are reliable.
Do you think about application crash or an entire machine crash that
needs a reboot? In this case, after the reboot, the MQTT application
usually doesn't know anything about the previous connection, timeout and
lost messages... except it saved something on a non volatile memory.

> Secondly, there is the issue of resend timeouts. If TCP fails to deliver
> the message past the MQTT retry time limit, then MQTT will resend the
> message.

What happens in this case? Suppose one TCP fragment with a single MQTT
message (just for simplicity) sent by a client to the server (the
broker) was lost. After a TCP timeout, the network stack autonomously
resend the fragment until an ACK is received. Even if the MQTT
application resend the MQTT message *before* TCP timeout, it will not be
sent by TCP layer until the previous fragment is acked.
Maybe, more exactly, on the receiver machine, the TCP layer will not
pass the resent message to the application (the MQTT broker) before the
lost TCP segment is received as well. When the lost TCP fragment is
received, the broker will receive two MQTT messages: the "original" and
the resent ones. I think it's impossible for the broker to receive the
second transmission without receiving the first.

So it seems to me the retransmission made at the MQTT level is
completely useless... but I think I didn't get the real point here.

Re: MQTT QoS vs TCP

<u4f7v3$243vd$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1604&group=comp.arch.embedded#1604

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Mon, 22 May 2023 10:09:07 +0200
Organization: A noiseless patient Spider
Lines: 70
Message-ID: <u4f7v3$243vd$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 22 May 2023 08:09:07 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="59249502c4cb254068db32687b4528ab";
logging-data="2232301"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/jHV4ifKca3kIrcbDKMYzf8q4coSlgp8E="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:uAuDoWQJKQdO9AFYpBZtMWKFJQI=
In-Reply-To: <u4f4if$23mii$1@dont-email.me>
Content-Language: en-GB
 by: David Brown - Mon, 22 May 2023 08:09 UTC

On 22/05/2023 09:11, pozz wrote:
> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>> I know TCP is able to guarantee the delivery of messages from sender to
>>> receiver, without corruption (thanks to checksums), in order and without
>>> duplication (thanks to sequence numbers).
>>>
>>> So I'm confused when I read something about MQTT QoS. For example, QoS 1
>>> (at least once) uses ack (PUBACK) and retransmissions (with packet ids)
>>> to guarantee that the message is delivered at least one time.
>>>
>>> I'm wondering how this is possible over a TCP connection that uses the
>>> same technique of ack and sequence numbers.
>>>
>>>  From what I know about TCP, if some data isn't well acknowledged by the
>>> receiver, the sender automatically resends the packets not acked. This
>>> is performed by TCP/IP stack or kernel, without the application knows
>>> anything about this.
>>>
>>> It seems to me it's impossible that a MQTT client needs to resend a MQTT
>>> message because it wasn't received by the broker. If this happens, TCP
>>> should signal the error to the application that should close and try to
>>> reopen the connection.
>>
>> TCP is only one of the lower levels of the protocol stack.
>> Data can sometimes be lost in the higher levels.
>
> In this case, there's only one higher level, that is MQTT application.
> How an application running on a machine could lost something? Network
> links aren't reliable, but applicaions running on a processor are reliable.
> Do you think about application crash or an entire machine crash that
> needs a reboot? In this case, after the reboot, the MQTT application
> usually doesn't know anything about the previous connection, timeout and
> lost messages... except it saved something on a non volatile memory.
>
>
>> Secondly, there is the issue of resend timeouts. If TCP fails to deliver
>> the message past the MQTT retry time limit, then MQTT will resend the
>> message.
>
> What happens in this case? Suppose one TCP fragment with a single MQTT
> message (just for simplicity) sent by a client to the server (the
> broker) was lost. After a TCP timeout, the network stack autonomously
> resend the fragment until an ACK is received. Even if the MQTT
> application resend the MQTT message *before* TCP timeout, it will not be
> sent by TCP layer until the previous fragment is acked.
> Maybe, more exactly, on the receiver machine, the TCP layer will not
> pass the resent message to the application (the MQTT broker) before the
> lost TCP segment is received as well. When the lost TCP fragment is
> received, the broker will receive two MQTT messages: the "original" and
> the resent ones. I think it's impossible for the broker to receive the
> second transmission without receiving the first.
>
> So it seems to me the retransmission made at the MQTT level is
> completely useless... but I think I didn't get the real point here.

I haven't used MQTT much, but generally if an application gets a timeout
and wants to retry, it will close the TCP/IP connection and open a new
one. (Or rather, open a new one while the old one is closing - closing
a failing TCP/IP connection can be slow.)

I would actually have thought that UDP was a more natural choice for
MQTT, rather than TCP - although older versions of MQTT did not have QoS
and were therefore reliant on TCP's acknowledges and retries.

(I always think its a shame that SCTP never caught on - among its many
benefits, you don't have this head-of-line blocking issue.)

Re: MQTT QoS vs TCP

<u4fbef$24f2n$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1605&group=comp.arch.embedded#1605

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: pozzu...@gmail.com (pozz)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Mon, 22 May 2023 11:08:31 +0200
Organization: A noiseless patient Spider
Lines: 80
Message-ID: <u4fbef$24f2n$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me> <u4f7v3$243vd$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 22 May 2023 09:08:31 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="c8db7f232f2717aed271abb9ef7c734b";
logging-data="2243671"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+h6ZsLIjb3Ug2UW53A7Mnsn+lV9prM+wg="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Cancel-Lock: sha1:89BCbZmF3Wx/z80Go51drCP8BN8=
In-Reply-To: <u4f7v3$243vd$1@dont-email.me>
 by: pozz - Mon, 22 May 2023 09:08 UTC

Il 22/05/2023 10:09, David Brown ha scritto:
> On 22/05/2023 09:11, pozz wrote:
>> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>>> I know TCP is able to guarantee the delivery of messages from sender to
>>>> receiver, without corruption (thanks to checksums), in order and
>>>> without
>>>> duplication (thanks to sequence numbers).
>>>>
>>>> So I'm confused when I read something about MQTT QoS. For example,
>>>> QoS 1
>>>> (at least once) uses ack (PUBACK) and retransmissions (with packet ids)
>>>> to guarantee that the message is delivered at least one time.
>>>>
>>>> I'm wondering how this is possible over a TCP connection that uses the
>>>> same technique of ack and sequence numbers.
>>>>
>>>>  From what I know about TCP, if some data isn't well acknowledged by
>>>> the
>>>> receiver, the sender automatically resends the packets not acked. This
>>>> is performed by TCP/IP stack or kernel, without the application knows
>>>> anything about this.
>>>>
>>>> It seems to me it's impossible that a MQTT client needs to resend a
>>>> MQTT
>>>> message because it wasn't received by the broker. If this happens, TCP
>>>> should signal the error to the application that should close and try to
>>>> reopen the connection.
>>>
>>> TCP is only one of the lower levels of the protocol stack.
>>> Data can sometimes be lost in the higher levels.
>>
>> In this case, there's only one higher level, that is MQTT application.
>> How an application running on a machine could lost something? Network
>> links aren't reliable, but applicaions running on a processor are
>> reliable.
>> Do you think about application crash or an entire machine crash that
>> needs a reboot? In this case, after the reboot, the MQTT application
>> usually doesn't know anything about the previous connection, timeout
>> and lost messages... except it saved something on a non volatile memory.
>>
>>
>>> Secondly, there is the issue of resend timeouts. If TCP fails to deliver
>>> the message past the MQTT retry time limit, then MQTT will resend the
>>> message.
>>
>> What happens in this case? Suppose one TCP fragment with a single MQTT
>> message (just for simplicity) sent by a client to the server (the
>> broker) was lost. After a TCP timeout, the network stack autonomously
>> resend the fragment until an ACK is received. Even if the MQTT
>> application resend the MQTT message *before* TCP timeout, it will not
>> be sent by TCP layer until the previous fragment is acked.
>> Maybe, more exactly, on the receiver machine, the TCP layer will not
>> pass the resent message to the application (the MQTT broker) before
>> the lost TCP segment is received as well. When the lost TCP fragment
>> is received, the broker will receive two MQTT messages: the "original"
>> and the resent ones. I think it's impossible for the broker to receive
>> the second transmission without receiving the first.
>>
>> So it seems to me the retransmission made at the MQTT level is
>> completely useless... but I think I didn't get the real point here.
>
> I haven't used MQTT much, but generally if an application gets a timeout
> and wants to retry, it will close the TCP/IP connection and open a new
> one.  (Or rather, open a new one while the old one is closing - closing
> a failing TCP/IP connection can be slow.)

I'm quite sure that MQTT retransmission mechanism is *not* based on a
new TCP connection. In MQTT, the TCP connection is persistent. It can
stay open for days without exchanging any real data. In this case, the
keepalive facility is there to detect a broken link.

> I would actually have thought that UDP was a more natural choice for
> MQTT, rather than TCP - although older versions of MQTT did not have QoS
> and were therefore reliant on TCP's acknowledges and retries.
>
> (I always think its a shame that SCTP never caught on - among its many
> benefits, you don't have this head-of-line blocking issue.)

Re: MQTT QoS vs TCP

<u4fhl7$259bh$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1606&group=comp.arch.embedded#1606

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Mon, 22 May 2023 12:54:31 +0200
Organization: A noiseless patient Spider
Lines: 106
Message-ID: <u4fhl7$259bh$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me> <u4f7v3$243vd$1@dont-email.me>
<u4fbef$24f2n$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 22 May 2023 10:54:31 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="59249502c4cb254068db32687b4528ab";
logging-data="2270577"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18CiSU68RnhYSGj2+Dy9dvpovWXMc2ocoQ="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:ErK+4JNq5NBABi282KtKVTZwvx4=
Content-Language: en-GB
In-Reply-To: <u4fbef$24f2n$1@dont-email.me>
 by: David Brown - Mon, 22 May 2023 10:54 UTC

On 22/05/2023 11:08, pozz wrote:
> Il 22/05/2023 10:09, David Brown ha scritto:
>> On 22/05/2023 09:11, pozz wrote:
>>> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>>>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>>>> I know TCP is able to guarantee the delivery of messages from
>>>>> sender to
>>>>> receiver, without corruption (thanks to checksums), in order and
>>>>> without
>>>>> duplication (thanks to sequence numbers).
>>>>>
>>>>> So I'm confused when I read something about MQTT QoS. For example,
>>>>> QoS 1
>>>>> (at least once) uses ack (PUBACK) and retransmissions (with packet
>>>>> ids)
>>>>> to guarantee that the message is delivered at least one time.
>>>>>
>>>>> I'm wondering how this is possible over a TCP connection that uses the
>>>>> same technique of ack and sequence numbers.
>>>>>
>>>>>  From what I know about TCP, if some data isn't well acknowledged
>>>>> by the
>>>>> receiver, the sender automatically resends the packets not acked. This
>>>>> is performed by TCP/IP stack or kernel, without the application knows
>>>>> anything about this.
>>>>>
>>>>> It seems to me it's impossible that a MQTT client needs to resend a
>>>>> MQTT
>>>>> message because it wasn't received by the broker. If this happens, TCP
>>>>> should signal the error to the application that should close and
>>>>> try to
>>>>> reopen the connection.
>>>>
>>>> TCP is only one of the lower levels of the protocol stack.
>>>> Data can sometimes be lost in the higher levels.
>>>
>>> In this case, there's only one higher level, that is MQTT
>>> application. How an application running on a machine could lost
>>> something? Network links aren't reliable, but applicaions running on
>>> a processor are reliable.
>>> Do you think about application crash or an entire machine crash that
>>> needs a reboot? In this case, after the reboot, the MQTT application
>>> usually doesn't know anything about the previous connection, timeout
>>> and lost messages... except it saved something on a non volatile memory.
>>>
>>>
>>>> Secondly, there is the issue of resend timeouts. If TCP fails to
>>>> deliver
>>>> the message past the MQTT retry time limit, then MQTT will resend the
>>>> message.
>>>
>>> What happens in this case? Suppose one TCP fragment with a single
>>> MQTT message (just for simplicity) sent by a client to the server
>>> (the broker) was lost. After a TCP timeout, the network stack
>>> autonomously resend the fragment until an ACK is received. Even if
>>> the MQTT application resend the MQTT message *before* TCP timeout, it
>>> will not be sent by TCP layer until the previous fragment is acked.
>>> Maybe, more exactly, on the receiver machine, the TCP layer will not
>>> pass the resent message to the application (the MQTT broker) before
>>> the lost TCP segment is received as well. When the lost TCP fragment
>>> is received, the broker will receive two MQTT messages: the
>>> "original" and the resent ones. I think it's impossible for the
>>> broker to receive the second transmission without receiving the first.
>>>
>>> So it seems to me the retransmission made at the MQTT level is
>>> completely useless... but I think I didn't get the real point here.
>>
>> I haven't used MQTT much, but generally if an application gets a
>> timeout and wants to retry, it will close the TCP/IP connection and
>> open a new one.  (Or rather, open a new one while the old one is
>> closing - closing a failing TCP/IP connection can be slow.)
>
> I'm quite sure that MQTT retransmission mechanism is *not* based on a
> new TCP connection. In MQTT, the TCP connection is persistent. It can
> stay open for days without exchanging any real data. In this case, the
> keepalive facility is there to detect a broken link.

If the TCP/IP connection is working correctly, messages will be
transmitted correctly to the broker. If a QoS message fails to be
transmitted - the MQTT client or server does not receive an acknowledge
in time - then there are two possible issues. One is that the
server/broker application is in trouble. The other is that there is an
issue with the network. In most cases, I would suspect the network
first. TCP/IP already has acknowledges and timeouts, so if it is a
temporary problem then it is likely to be handled there. By the time it
reaches the attention of the application protocol's QoS handling, you
are definitely at the point where a new TCP/IP connection is the right
way to go - perhaps targeting a different IP address or via a different
route.

The MQTT application already has to handle dropping and making new
TCP/IP connections - even if the norm is for the connection to last for
weeks at a time or more. So creating a new TCP/IP link has a lot to
gain, and very little to lose, and it is the standard way to handle such
issues.

>
>
>> I would actually have thought that UDP was a more natural choice for
>> MQTT, rather than TCP - although older versions of MQTT did not have
>> QoS and were therefore reliant on TCP's acknowledges and retries.
>>
>> (I always think its a shame that SCTP never caught on - among its many
>> benefits, you don't have this head-of-line blocking issue.)
>

Re: MQTT QoS vs TCP

<u4hnsq$2fj32$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1607&group=comp.arch.embedded#1607

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: pozzu...@gmail.com (pozz)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Tue, 23 May 2023 08:53:14 +0200
Organization: A noiseless patient Spider
Lines: 126
Message-ID: <u4hnsq$2fj32$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me> <u4f7v3$243vd$1@dont-email.me>
<u4fbef$24f2n$1@dont-email.me> <u4fhl7$259bh$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 23 May 2023 06:53:14 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="3193bf130cce34bf139585509c2da88d";
logging-data="2608226"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18Hvqqd2rzEcOxB/dotKdHZNa4cOijPGMI="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Cancel-Lock: sha1:LOrM85XJZGFzQeYMR4hS1uTizZw=
In-Reply-To: <u4fhl7$259bh$1@dont-email.me>
 by: pozz - Tue, 23 May 2023 06:53 UTC

Il 22/05/2023 12:54, David Brown ha scritto:
> On 22/05/2023 11:08, pozz wrote:
>> Il 22/05/2023 10:09, David Brown ha scritto:
>>> On 22/05/2023 09:11, pozz wrote:
>>>> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>>>>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>>>>> I know TCP is able to guarantee the delivery of messages from
>>>>>> sender to
>>>>>> receiver, without corruption (thanks to checksums), in order and
>>>>>> without
>>>>>> duplication (thanks to sequence numbers).
>>>>>>
>>>>>> So I'm confused when I read something about MQTT QoS. For example,
>>>>>> QoS 1
>>>>>> (at least once) uses ack (PUBACK) and retransmissions (with packet
>>>>>> ids)
>>>>>> to guarantee that the message is delivered at least one time.
>>>>>>
>>>>>> I'm wondering how this is possible over a TCP connection that uses
>>>>>> the
>>>>>> same technique of ack and sequence numbers.
>>>>>>
>>>>>>  From what I know about TCP, if some data isn't well acknowledged
>>>>>> by the
>>>>>> receiver, the sender automatically resends the packets not acked.
>>>>>> This
>>>>>> is performed by TCP/IP stack or kernel, without the application knows
>>>>>> anything about this.
>>>>>>
>>>>>> It seems to me it's impossible that a MQTT client needs to resend
>>>>>> a MQTT
>>>>>> message because it wasn't received by the broker. If this happens,
>>>>>> TCP
>>>>>> should signal the error to the application that should close and
>>>>>> try to
>>>>>> reopen the connection.
>>>>>
>>>>> TCP is only one of the lower levels of the protocol stack.
>>>>> Data can sometimes be lost in the higher levels.
>>>>
>>>> In this case, there's only one higher level, that is MQTT
>>>> application. How an application running on a machine could lost
>>>> something? Network links aren't reliable, but applicaions running on
>>>> a processor are reliable.
>>>> Do you think about application crash or an entire machine crash that
>>>> needs a reboot? In this case, after the reboot, the MQTT application
>>>> usually doesn't know anything about the previous connection, timeout
>>>> and lost messages... except it saved something on a non volatile
>>>> memory.
>>>>
>>>>
>>>>> Secondly, there is the issue of resend timeouts. If TCP fails to
>>>>> deliver
>>>>> the message past the MQTT retry time limit, then MQTT will resend the
>>>>> message.
>>>>
>>>> What happens in this case? Suppose one TCP fragment with a single
>>>> MQTT message (just for simplicity) sent by a client to the server
>>>> (the broker) was lost. After a TCP timeout, the network stack
>>>> autonomously resend the fragment until an ACK is received. Even if
>>>> the MQTT application resend the MQTT message *before* TCP timeout,
>>>> it will not be sent by TCP layer until the previous fragment is acked.
>>>> Maybe, more exactly, on the receiver machine, the TCP layer will not
>>>> pass the resent message to the application (the MQTT broker) before
>>>> the lost TCP segment is received as well. When the lost TCP fragment
>>>> is received, the broker will receive two MQTT messages: the
>>>> "original" and the resent ones. I think it's impossible for the
>>>> broker to receive the second transmission without receiving the first.
>>>>
>>>> So it seems to me the retransmission made at the MQTT level is
>>>> completely useless... but I think I didn't get the real point here.
>>>
>>> I haven't used MQTT much, but generally if an application gets a
>>> timeout and wants to retry, it will close the TCP/IP connection and
>>> open a new one.  (Or rather, open a new one while the old one is
>>> closing - closing a failing TCP/IP connection can be slow.)
>>
>> I'm quite sure that MQTT retransmission mechanism is *not* based on a
>> new TCP connection. In MQTT, the TCP connection is persistent. It can
>> stay open for days without exchanging any real data. In this case, the
>> keepalive facility is there to detect a broken link.
>
> If the TCP/IP connection is working correctly, messages will be
> transmitted correctly to the broker.  If a QoS message fails to be
> transmitted - the MQTT client or server does not receive an acknowledge
> in time - then there are two possible issues.  One is that the
> server/broker application is in trouble.  The other is that there is an
> issue with the network.  In most cases, I would suspect the network
> first.  TCP/IP already has acknowledges and timeouts, so if it is a
> temporary problem then it is likely to be handled there.  By the time it
> reaches the attention of the application protocol's QoS handling, you
> are definitely at the point where a new TCP/IP connection is the right
> way to go - perhaps targeting a different IP address or via a different
> route.

Yes, this is the only solution for me too. Anyway, I don't know if this
behaviour (closing and reopening TCP connection) is described in the
MQTT specifications.

> The MQTT application already has to handle dropping and making new
> TCP/IP connections - even if the norm is for the connection to last for
> weeks at a time or more.  So creating a new TCP/IP link has a lot to
> gain, and very little to lose, and it is the standard way to handle such
> issues.

Here[1] the MQTT client implementation of lwip, a popular TCP/IP stack
for embedded systems.
When the timeout for the ACK is expired, this client only calls an
application callback with ERR_TIMEOUT. Maybe the decision to close and
reopen a new TCP connection is passed to the application.
I don't know if other MQTT clients implement an embedded mechanism that
automatically tries to solve the issue of lost ACKs by reopening a TCP
connection.

>>> I would actually have thought that UDP was a more natural choice for
>>> MQTT, rather than TCP - although older versions of MQTT did not have
>>> QoS and were therefore reliant on TCP's acknowledges and retries.
>>>
>>> (I always think its a shame that SCTP never caught on - among its
>>> many benefits, you don't have this head-of-line blocking issue.)
>>
>

[1] https://github.com/lwip-tcpip/lwip/blob/master/src/apps/mqtt/mqtt.c

Re: MQTT QoS vs TCP

<u4hrhe$2g1uj$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1608&group=comp.arch.embedded#1608

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Tue, 23 May 2023 09:55:26 +0200
Organization: A noiseless patient Spider
Lines: 154
Message-ID: <u4hrhe$2g1uj$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me> <u4f7v3$243vd$1@dont-email.me>
<u4fbef$24f2n$1@dont-email.me> <u4fhl7$259bh$1@dont-email.me>
<u4hnsq$2fj32$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 23 May 2023 07:55:26 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="c60f2a0a369b4ae10189540518b3d481";
logging-data="2623443"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18GbK77DE3FGUbbk5N/vkVSUvCxQmpkb5g="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:tBcaTYZ4Dbs/Xxb08HE/Z9kT4JI=
In-Reply-To: <u4hnsq$2fj32$1@dont-email.me>
Content-Language: en-GB
 by: David Brown - Tue, 23 May 2023 07:55 UTC

On 23/05/2023 08:53, pozz wrote:
> Il 22/05/2023 12:54, David Brown ha scritto:
>> On 22/05/2023 11:08, pozz wrote:
>>> Il 22/05/2023 10:09, David Brown ha scritto:
>>>> On 22/05/2023 09:11, pozz wrote:
>>>>> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>>>>>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>>>>>> I know TCP is able to guarantee the delivery of messages from
>>>>>>> sender to
>>>>>>> receiver, without corruption (thanks to checksums), in order and
>>>>>>> without
>>>>>>> duplication (thanks to sequence numbers).
>>>>>>>
>>>>>>> So I'm confused when I read something about MQTT QoS. For
>>>>>>> example, QoS 1
>>>>>>> (at least once) uses ack (PUBACK) and retransmissions (with
>>>>>>> packet ids)
>>>>>>> to guarantee that the message is delivered at least one time.
>>>>>>>
>>>>>>> I'm wondering how this is possible over a TCP connection that
>>>>>>> uses the
>>>>>>> same technique of ack and sequence numbers.
>>>>>>>
>>>>>>>  From what I know about TCP, if some data isn't well acknowledged
>>>>>>> by the
>>>>>>> receiver, the sender automatically resends the packets not acked.
>>>>>>> This
>>>>>>> is performed by TCP/IP stack or kernel, without the application
>>>>>>> knows
>>>>>>> anything about this.
>>>>>>>
>>>>>>> It seems to me it's impossible that a MQTT client needs to resend
>>>>>>> a MQTT
>>>>>>> message because it wasn't received by the broker. If this
>>>>>>> happens, TCP
>>>>>>> should signal the error to the application that should close and
>>>>>>> try to
>>>>>>> reopen the connection.
>>>>>>
>>>>>> TCP is only one of the lower levels of the protocol stack.
>>>>>> Data can sometimes be lost in the higher levels.
>>>>>
>>>>> In this case, there's only one higher level, that is MQTT
>>>>> application. How an application running on a machine could lost
>>>>> something? Network links aren't reliable, but applicaions running
>>>>> on a processor are reliable.
>>>>> Do you think about application crash or an entire machine crash
>>>>> that needs a reboot? In this case, after the reboot, the MQTT
>>>>> application usually doesn't know anything about the previous
>>>>> connection, timeout and lost messages... except it saved something
>>>>> on a non volatile memory.
>>>>>
>>>>>
>>>>>> Secondly, there is the issue of resend timeouts. If TCP fails to
>>>>>> deliver
>>>>>> the message past the MQTT retry time limit, then MQTT will resend the
>>>>>> message.
>>>>>
>>>>> What happens in this case? Suppose one TCP fragment with a single
>>>>> MQTT message (just for simplicity) sent by a client to the server
>>>>> (the broker) was lost. After a TCP timeout, the network stack
>>>>> autonomously resend the fragment until an ACK is received. Even if
>>>>> the MQTT application resend the MQTT message *before* TCP timeout,
>>>>> it will not be sent by TCP layer until the previous fragment is acked.
>>>>> Maybe, more exactly, on the receiver machine, the TCP layer will
>>>>> not pass the resent message to the application (the MQTT broker)
>>>>> before the lost TCP segment is received as well. When the lost TCP
>>>>> fragment is received, the broker will receive two MQTT messages:
>>>>> the "original" and the resent ones. I think it's impossible for the
>>>>> broker to receive the second transmission without receiving the first.
>>>>>
>>>>> So it seems to me the retransmission made at the MQTT level is
>>>>> completely useless... but I think I didn't get the real point here.
>>>>
>>>> I haven't used MQTT much, but generally if an application gets a
>>>> timeout and wants to retry, it will close the TCP/IP connection and
>>>> open a new one.  (Or rather, open a new one while the old one is
>>>> closing - closing a failing TCP/IP connection can be slow.)
>>>
>>> I'm quite sure that MQTT retransmission mechanism is *not* based on a
>>> new TCP connection. In MQTT, the TCP connection is persistent. It can
>>> stay open for days without exchanging any real data. In this case,
>>> the keepalive facility is there to detect a broken link.
>>
>> If the TCP/IP connection is working correctly, messages will be
>> transmitted correctly to the broker.  If a QoS message fails to be
>> transmitted - the MQTT client or server does not receive an
>> acknowledge in time - then there are two possible issues.  One is that
>> the server/broker application is in trouble.  The other is that there
>> is an issue with the network.  In most cases, I would suspect the
>> network first.  TCP/IP already has acknowledges and timeouts, so if it
>> is a temporary problem then it is likely to be handled there.  By the
>> time it reaches the attention of the application protocol's QoS
>> handling, you are definitely at the point where a new TCP/IP
>> connection is the right way to go - perhaps targeting a different IP
>> address or via a different route.
>
> Yes, this is the only solution for me too. Anyway, I don't know if this
> behaviour (closing and reopening TCP connection) is described in the
> MQTT specifications.
>

I haven't read the MQTT specifications - I don't even know what
documentation exists for the protocol. But implementation details like
this are not always covered in such documents, as it is really at a
level below the protocol itself. (The specifications for HTTP, for
example, don't say how many simultaneous connections a browser should
have to a web server, or when it should give up and retry.) So don't be
surprised if this is /not/ in the specs - that does not mean a client
cannot or should not make new TCP/IP connections.

>
>> The MQTT application already has to handle dropping and making new
>> TCP/IP connections - even if the norm is for the connection to last
>> for weeks at a time or more.  So creating a new TCP/IP link has a lot
>> to gain, and very little to lose, and it is the standard way to handle
>> such issues.
>
> Here[1] the MQTT client implementation of lwip, a popular TCP/IP stack
> for embedded systems.

This is a bit muddled. I am familiar with LWIP, but I don't know
whether you are talking about an MQTT client that you wrote yourself, or
which comes as part of newer LWIP, or which someone else contributed as
a sample.

> When the timeout for the ACK is expired, this client only calls an
> application callback with ERR_TIMEOUT. Maybe the decision to close and
> reopen a new TCP connection is passed to the application.

Yes, that would be the normal behaviour.

> I don't know if other MQTT clients implement an embedded mechanism that
> automatically tries to solve the issue of lost ACKs by reopening a TCP
> connection.
>

I don't know either. I can only tell you that if you are failing to
communicate on a TCP/IP connection, then making a new one (possibly
after a delay) is the normal way to handle things if you want automatic
recovery.

>
>>>> I would actually have thought that UDP was a more natural choice for
>>>> MQTT, rather than TCP - although older versions of MQTT did not have
>>>> QoS and were therefore reliant on TCP's acknowledges and retries.
>>>>
>>>> (I always think its a shame that SCTP never caught on - among its
>>>> many benefits, you don't have this head-of-line blocking issue.)
>>>
>>
>
> [1] https://github.com/lwip-tcpip/lwip/blob/master/src/apps/mqtt/mqtt.c

Re: MQTT QoS vs TCP

<u4j9hq$2mdu6$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1609&group=comp.arch.embedded#1609

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: pozzu...@gmail.com (pozz)
Newsgroups: comp.arch.embedded
Subject: Re: MQTT QoS vs TCP
Date: Tue, 23 May 2023 23:00:42 +0200
Organization: A noiseless patient Spider
Lines: 163
Message-ID: <u4j9hq$2mdu6$1@dont-email.me>
References: <u47g53$jgfj$1@dont-email.me>
<2991d458-41ff-4b92-80ad-9d8a93328437n@googlegroups.com>
<u4f4if$23mii$1@dont-email.me> <u4f7v3$243vd$1@dont-email.me>
<u4fbef$24f2n$1@dont-email.me> <u4fhl7$259bh$1@dont-email.me>
<u4hnsq$2fj32$1@dont-email.me> <u4hrhe$2g1uj$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 23 May 2023 21:00:43 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="3193bf130cce34bf139585509c2da88d";
logging-data="2832326"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/710kCMdXMgwf/vVhhzjzkkhu+9k4nm7U="
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.11.0
Cancel-Lock: sha1:dUj8paKJO+ft4fWCMvfh5cxMnU8=
In-Reply-To: <u4hrhe$2g1uj$1@dont-email.me>
 by: pozz - Tue, 23 May 2023 21:00 UTC

Il 23/05/2023 09:55, David Brown ha scritto:
> On 23/05/2023 08:53, pozz wrote:
>> Il 22/05/2023 12:54, David Brown ha scritto:
>>> On 22/05/2023 11:08, pozz wrote:
>>>> Il 22/05/2023 10:09, David Brown ha scritto:
>>>>> On 22/05/2023 09:11, pozz wrote:
>>>>>> Il 19/05/2023 17:14, Ed Prochak ha scritto:
>>>>>>> On Friday, May 19, 2023 at 5:42:14 AM UTC-4, pozz wrote:
>>>>>>>> I know TCP is able to guarantee the delivery of messages from
>>>>>>>> sender to
>>>>>>>> receiver, without corruption (thanks to checksums), in order and
>>>>>>>> without
>>>>>>>> duplication (thanks to sequence numbers).
>>>>>>>>
>>>>>>>> So I'm confused when I read something about MQTT QoS. For
>>>>>>>> example, QoS 1
>>>>>>>> (at least once) uses ack (PUBACK) and retransmissions (with
>>>>>>>> packet ids)
>>>>>>>> to guarantee that the message is delivered at least one time.
>>>>>>>>
>>>>>>>> I'm wondering how this is possible over a TCP connection that
>>>>>>>> uses the
>>>>>>>> same technique of ack and sequence numbers.
>>>>>>>>
>>>>>>>>  From what I know about TCP, if some data isn't well
>>>>>>>> acknowledged by the
>>>>>>>> receiver, the sender automatically resends the packets not
>>>>>>>> acked. This
>>>>>>>> is performed by TCP/IP stack or kernel, without the application
>>>>>>>> knows
>>>>>>>> anything about this.
>>>>>>>>
>>>>>>>> It seems to me it's impossible that a MQTT client needs to
>>>>>>>> resend a MQTT
>>>>>>>> message because it wasn't received by the broker. If this
>>>>>>>> happens, TCP
>>>>>>>> should signal the error to the application that should close and
>>>>>>>> try to
>>>>>>>> reopen the connection.
>>>>>>>
>>>>>>> TCP is only one of the lower levels of the protocol stack.
>>>>>>> Data can sometimes be lost in the higher levels.
>>>>>>
>>>>>> In this case, there's only one higher level, that is MQTT
>>>>>> application. How an application running on a machine could lost
>>>>>> something? Network links aren't reliable, but applicaions running
>>>>>> on a processor are reliable.
>>>>>> Do you think about application crash or an entire machine crash
>>>>>> that needs a reboot? In this case, after the reboot, the MQTT
>>>>>> application usually doesn't know anything about the previous
>>>>>> connection, timeout and lost messages... except it saved something
>>>>>> on a non volatile memory.
>>>>>>
>>>>>>
>>>>>>> Secondly, there is the issue of resend timeouts. If TCP fails to
>>>>>>> deliver
>>>>>>> the message past the MQTT retry time limit, then MQTT will resend
>>>>>>> the
>>>>>>> message.
>>>>>>
>>>>>> What happens in this case? Suppose one TCP fragment with a single
>>>>>> MQTT message (just for simplicity) sent by a client to the server
>>>>>> (the broker) was lost. After a TCP timeout, the network stack
>>>>>> autonomously resend the fragment until an ACK is received. Even if
>>>>>> the MQTT application resend the MQTT message *before* TCP timeout,
>>>>>> it will not be sent by TCP layer until the previous fragment is
>>>>>> acked.
>>>>>> Maybe, more exactly, on the receiver machine, the TCP layer will
>>>>>> not pass the resent message to the application (the MQTT broker)
>>>>>> before the lost TCP segment is received as well. When the lost TCP
>>>>>> fragment is received, the broker will receive two MQTT messages:
>>>>>> the "original" and the resent ones. I think it's impossible for
>>>>>> the broker to receive the second transmission without receiving
>>>>>> the first.
>>>>>>
>>>>>> So it seems to me the retransmission made at the MQTT level is
>>>>>> completely useless... but I think I didn't get the real point here.
>>>>>
>>>>> I haven't used MQTT much, but generally if an application gets a
>>>>> timeout and wants to retry, it will close the TCP/IP connection and
>>>>> open a new one.  (Or rather, open a new one while the old one is
>>>>> closing - closing a failing TCP/IP connection can be slow.)
>>>>
>>>> I'm quite sure that MQTT retransmission mechanism is *not* based on
>>>> a new TCP connection. In MQTT, the TCP connection is persistent. It
>>>> can stay open for days without exchanging any real data. In this
>>>> case, the keepalive facility is there to detect a broken link.
>>>
>>> If the TCP/IP connection is working correctly, messages will be
>>> transmitted correctly to the broker.  If a QoS message fails to be
>>> transmitted - the MQTT client or server does not receive an
>>> acknowledge in time - then there are two possible issues.  One is
>>> that the server/broker application is in trouble.  The other is that
>>> there is an issue with the network.  In most cases, I would suspect
>>> the network first.  TCP/IP already has acknowledges and timeouts, so
>>> if it is a temporary problem then it is likely to be handled there.
>>> By the time it reaches the attention of the application protocol's
>>> QoS handling, you are definitely at the point where a new TCP/IP
>>> connection is the right way to go - perhaps targeting a different IP
>>> address or via a different route.
>>
>> Yes, this is the only solution for me too. Anyway, I don't know if
>> this behaviour (closing and reopening TCP connection) is described in
>> the MQTT specifications.
>>
>
> I haven't read the MQTT specifications - I don't even know what
> documentation exists for the protocol.  But implementation details like
> this are not always covered in such documents, as it is really at a
> level below the protocol itself.  (The specifications for HTTP, for
> example, don't say how many simultaneous connections a browser should
> have to a web server, or when it should give up and retry.)  So don't be
> surprised if this is /not/ in the specs - that does not mean a client
> cannot or should not make new TCP/IP connections.
>
>>
>>> The MQTT application already has to handle dropping and making new
>>> TCP/IP connections - even if the norm is for the connection to last
>>> for weeks at a time or more.  So creating a new TCP/IP link has a lot
>>> to gain, and very little to lose, and it is the standard way to
>>> handle such issues.
>>
>> Here[1] the MQTT client implementation of lwip, a popular TCP/IP stack
>> for embedded systems.
>
> This is a bit muddled.  I am familiar with LWIP, but I don't know
> whether you are talking about an MQTT client that you wrote yourself, or
> which comes as part of newer LWIP, or which someone else contributed as
> a sample.

In the link, there's the official MQTT client implementation of lwip
project.

>> When the timeout for the ACK is expired, this client only calls an
>> application callback with ERR_TIMEOUT. Maybe the decision to close and
>> reopen a new TCP connection is passed to the application.
>
> Yes, that would be the normal behaviour.
>
>> I don't know if other MQTT clients implement an embedded mechanism
>> that automatically tries to solve the issue of lost ACKs by reopening
>> a TCP connection.
>>
>
> I don't know either.  I can only tell you that if you are failing to
> communicate on a TCP/IP connection, then making a new one (possibly
> after a delay) is the normal way to handle things if you want automatic
> recovery.
>
>>
>>>>> I would actually have thought that UDP was a more natural choice
>>>>> for MQTT, rather than TCP - although older versions of MQTT did not
>>>>> have QoS and were therefore reliant on TCP's acknowledges and retries.
>>>>>
>>>>> (I always think its a shame that SCTP never caught on - among its
>>>>> many benefits, you don't have this head-of-line blocking issue.)
>>>>
>>>
>>
>> [1] https://github.com/lwip-tcpip/lwip/blob/master/src/apps/mqtt/mqtt.c
>


Click here to read the complete article
1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor