Subject: [core] comments and support for draft-carey-core-std-msg-vs-trans-adapt-00

From: Benjamin Black <[email-masked]>
Date: Sun, 12 Jul 2015 16:16:28 -0700

i have been experimenting with coap for several months for use in
production iot applications. i find its design much cleaner and mature than
things like mqtt. the main weakness i have encountered is lack of support
for a connection-oriented transport. many deployments will involve needing
to connect through 3rd party nat or proxy systems that will not support the
required udp communication or http-coap translation.

reading the tcp and websockets drafts i was disappointed that they involved
significant changes to the protocol semantics. i wholeheartedly agree with
everything in draft-carey-core-std-msg-vs-trans-adapt-00. there is an
additional concern not mentioned in that draft that i believe critical. as
mentioned in the tcp draft, rfc7252 does not specify a message length
field, instead assuming the transport is able to delineate records. tcp
lacks this, requiring additional modifications to the message structure.

i support the following:

1) adoption of the (standard ietf) recommendation in
draft-carey-core-std-msg-vs-trans-adapt-00 to cleanly delineate the
standard message layer from the transport.
2) adoption of the recommendation in
draft-carey-core-std-msg-vs-trans-adapt-00 to maintain strict adherence to
rfc7252 for that standard message layer.
3) adoption of the recommendation in
draft-carey-core-std-msg-vs-trans-adapt-00 to require definitions of new
transports specify how they implement the standard set of messaging
primitives.
4) further to #3, require transports either support record delineation (as
websockets does) or define a record envelope to encapsulate standard
messages (as tcp would).

failing to adopt these 4 points will, i believe, significantly increase
implementation complexity for coap frameworks and for applications using
them. worse, this increase in complexity will deliver no benefits. even
something as otherwise flawed as mqtt gets this right. without coap
adopting these, i and others will have to choose between using an inferior
protocol like mqtt or implementing tcp and websockets coap in non-standard
but reasonable ways..

b

---

From: Klaus Hartke <[email-masked]>
Date: Tue, 14 Jul 2015 22:39:55 +0200

> i support the following:

Can you expand on the "why" of this? What are you missing that would
be provided by CON/ACK/NON/RST over TCP/WebSockets?

Klaus

---

From: Benjamin Black <[email-masked]>
Date: Wed, 15 Jul 2015 11:08:38 -0700

tim's draft covers much of this in great detail. changing the protocol
semantics and expecting every different _transport_ implementation to have
its own message parsing implementation, since udp, tcp and websockets all
use different message formats, adds complexity with no benefit and is not
something i have seen in any other ietf protocol.

a couple of specific scenarios of interest to me:

1) i'd like to be able to indicate via CON/NON whether or not a message
should be persisted to a kafka broker before acknowledgement. many sensor
scenarios are tolerant of loss of sensor readings but want reliability for
alert or control messages. without CON i can either persist everything
before ack, with significant performance penalty, or ack before persisting
anything, losing the durability required. while i could build _another_
protocol on top to do this, i shouldn't have to. coap has the needed
functionality when using udp transport. it should have the exact same
functionality over tcp or websockets.

2) i'd like to have a tcp/udp coap gateway that uses udp for communication
for nearby devices and tcp for communication with central collection and
control services. a command sent over tcp, which needs confirmation, is
sent over tcp which does not permit CON messages. how does the proxy know
to use CON when sending to the udp recipient? how does the proxy know not
to use CON when sending less important messages? again, i could build
another protocol on top of coap to make up for the loss of parts of coap
over tcp, but why should i have to?

if the semantics of the coap protocol are not maintained when using non-udp
transports, then they are not transports and not coap. they are new,
coap-like protocols on specific transports. i don't understand the logic
behind making changes like this. they require much more code, much more
thought about the numerous edge cases created, and are confusing to anyone
who has to use more than one. going down the path of changing coap
transport by transport will result in non-standard incompatible
implementations that maintain coap semantics. that would be a serious
failure of this working group.

b

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jul 2015 20:42:34 +0200

Benjamin,

> semantics 

I am not aware of any protocol semantics that are changing.

> and expecting every different _transport_ implementation to
> have its own message parsing implementation, 

Nope.  Only the 4-byte header is slightly rearranged.  The bulk of the
parsing code is in the option handling.  The header simply has to be
different between UDP/DTLS (datagram) and TCP/TLS/Websockets (reliable
byte stream) because there needs to be some message delimiting.  That
message delimiting code can still spit out UDP-like 4-byte headers if it
helps your implementation.

> since udp, tcp and
> websockets all use different message formats, adds complexity with no
> benefit and is not something i have seen in any other ietf protocol.

Sending parts of a message that have no function and will be
mis-interpreted differently by different implementations is prone to add
a lot of real-world complexity.

OK, the following is much more useful:

> should be persisted to a kafka broker before acknowledgement. many
> sensor scenarios are tolerant of loss of sensor readings but want
> reliability for alert or control messages. without CON i can either
> persist everything before ack, with significant performance penalty, or
> ack before persisting anything, losing the durability required. while i
> could build _another_ protocol on top to do this, i shouldn't have to.
> coap has the needed functionality when using udp transport. it should
> have the exact same functionality over tcp or websockets.

This is a bit hard to parse for me because I don't know what these
messages do.  Are you talking about PUT or POST requests?  These have
responses, which are also the application layer acknowledgements.
I would have expected the *resources* to have the necessary semantics
about persistence, as opposed to the client indicating its preference in
every exchange.  But maybe I need to understand the example some more.
In any case, CON and NON are about message layer semantics, not about
application semantics -- you gave them a meaning they don't have.
(Of course you are free to do this in a proprietary implementation, but
it is not something that the standard is specifying, so you won't
interoperate very well.)

> 2) i'd like to have a tcp/udp coap gateway that uses udp for
> communication for nearby devices and tcp for communication with central
> collection and control services. a command sent over tcp, which needs
> confirmation, is sent over tcp which does not permit CON messages. how
> does the proxy know to use CON when sending to the udp recipient? how
> does the proxy know not to use CON when sending less important messages?
> again, i could build another protocol on top of coap to make up for the
> loss of parts of coap over tcp, but why should i have to?

Since a reliable protocol (TCP) was used to talk to the proxy, I would
expect the proxy to use the reliable mode on the UDP side as well; it is
a bit hard to handle the request/response processing if you don't have
reliability.  (That is also true UDP-to-UDP -- I don't think a proxy is
in any way obligated to use unreliable mode if the request to it was
unreliable.)  But there probably should be a way to instruct a proxy to
use unreliable mode on the UDP side -- this is probably related to the
problems we have in proxying multicast.  Abhijan's no-response option
may come in useful here as well.

> if the semantics of the coap protocol are not maintained when using
> non-udp transports, then they are not transports and not coap. they are
> new, coap-like protocols on specific transports. i don't understand the
> logic behind making changes like this. they require much more code, much
> more thought about the numerous edge cases created, and are confusing to
> anyone who has to use more than one. going down the path of changing
> coap transport by transport will result in non-standard incompatible

I would understand all the rage if there was a proposal to actually
change anything, but I don't understand how those semantics you seem to
be relying on were part of the standard in the first place.

Still, I would like to make sure that CoAP provides the functions you
need for your application, so we should try to continue the discussion
on the detail level, maybe with some examples next.

Grüße, Carsten

---

From: Benjamin Black <[email-masked]>
Date: Wed, 15 Jul 2015 14:18:46 -0700

> I am not aware of any protocol semantics that are changing.
the tcp and websockets transport drafts both eliminate CON and ACK. that's
a semantic change.

> helps your implementation.
other than the changes, nothing is changed? this is an unnecessary change
and as you are clearly saying it does require additional implementation
around message parsing specific to the transport.

> a lot of real-world complexity.
they will only be mis-interpreted if the specification is poor. that is one
of the central points in tim's draft: by not clearly delineating the
different layers you push _more_ complexity into the transport specs. for
example, rather than removing parts of the protocol, you could specify that
a tcp ack for the bytes in a coap message should be interpreted as a coap
ack. i can then put a completely unmodified coap layer on top of this tcp
transport. if you are intentionally mixing application and transport
concerns, that should be made explicit and clear and compelling arguments
provided.

> responses, which are also the application layer acknowledgements.

as i said, i want to be able to signal _both_ intents. relying on a
response doesn't help because the intent is to influence what happens
_before_ the response is sent.

> it is not something that the standard is specifying, so you won't
as my second example addresses, it isn't quite that simple to separate the
two concerns.

> expect the proxy to use the reliable mode on the UDP side as well; it is

why would you assume that? and if that is a reasonable assumption, why
bother with NON in udp transport? just assume everything is CON.

> may come in useful here as well.
there is a way to instruct a proxy to use reliable or unreliable mode: CON
and NON. rather than rebuild it in another way, we should use the existing
mechanism.

> be relying on were part of the standard in the first place.
there is no rage, i am supporting a clearly written draft that explains in
great detail how proper layering enables simple and consistent
implementation and use of coap, while inconsistent mixing of application
and transport concerns leads to complexity and confusion.

> on the detail level, maybe with some examples next.
great, please adhere to best common practice and decouple the coap
messaging layer from the transport. if you prefer not to, then the onus is
on you to provide clear and compelling arguments for deviating.

b

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jul 2015 23:42:15 +0200

Benjamin,

I really need those examples in order to understand what you are trying to achieve.

Example:

 please adhere to best common practice and decouple the coap 
messaging layer from the transport. if you prefer not to, then the onus is 
on you to provide clear and compelling arguments for deviating. 

This statement does not make sense to me, because the messaging layer *is* providing the transport layer functions for UDP/DTLS, so how do I decouple them?

I think this exchange would be more efficient if we can stop communicating in generalities and get down to something more concrete.

It seems to me (but I’m still trying to understand) you would prefer to ship around UDP CoAP messages in TCP.
How do you do the delimiting?  How does that not change the processing?  What does your code do with the message types and the message IDs?  Timers?  What do you do when retransmissions start to queue up because TCP is throttling down? Etc.
This may all be very clear to you, but I haven’t seen it and have trouble imagining it.

Grüße, Carsten

On 15 Jul 2015 at 23:19:35, Benjamin Black ([email-masked]) wrote:

> on the detail level, maybe with some examples next. 
great,

---

From: "Carey, Timothy (Timothy)" <[email-masked]>
Date: Thu, 16 Jul 2015 13:33:09 +0000

Carsten,

I certain do agree with Ben on the his comments about semantic changes and the need for a clear specification. I won’t address that further in the thread.

Maybe I can help with his persistent message scenario (1) – so inline <TAC>

From: Benjamin Black [mailto:[email-masked]]
Sent: Wednesday, July 15, 2015 4:19 PM
To: Carsten Bormann
Cc: [email-masked] WG; Klaus Hartke
Subject: Re: [core] comments and support for draft-carey-core-std-msg-vs-trans-adapt-00

On Wed, Jul 15, 2015 at 11:42 AM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
Benjamin,

the tcp and websockets transport drafts both eliminate CON and ACK. that's a semantic change.

other than the changes, nothing is changed? this is an unnecessary change and as you are clearly saying it does require additional implementation around message parsing specific to the transport.

they will only be mis-interpreted if the specification is poor. that is one of the central points in tim's draft: by not clearly delineating the different layers you push _more_ complexity into the transport specs. for example, rather than removing parts of the protocol, you could specify that a tcp ack for the bytes in a coap message should be interpreted as a coap ack. i can then put a completely unmodified coap layer on top of this tcp transport. if you are intentionally mixing application and transport concerns, that should be made explicit and clear and compelling arguments provided.

as i said, i want to be able to signal _both_ intents. relying on a response doesn't help because the intent is to influence what happens _before_ the response is sent.

<TAC> The question is when does the receiver acknowledge receipt of the message (not process the request). Does the receiver do this when the message/adaptation layer reads the message into its memory buffer or does the receiver acknowledge the receipt of the message after the message is stored in a persistent environment. The sender would indicate if the message should be ACKed after persistent storage or while still in the memory buffer. The reason is simply a confirmation that message was received and is durable verse just received.

as my second example addresses, it isn't quite that simple to separate the two concerns.

why would you assume that? and if that is a reasonable assumption, why bother with NON in udp transport? just assume everything is CON.

there is a way to instruct a proxy to use reliable or unreliable mode: CON and NON. rather than rebuild it in another way, we should use the existing mechanism.

there is no rage, i am supporting a clearly written draft that explains in great detail how proper layering enables simple and consistent implementation and use of coap, while inconsistent mixing of application and transport concerns leads to complexity and confusion.

great, please adhere to best common practice and decouple the coap messaging layer from the transport. if you prefer not to, then the onus is on you to provide clear and compelling arguments for deviating.

b

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Jul 2015 17:51:08 +0200

Hi Tim,

thanks for chiming in.

> <TAC> The question is when does the receiver acknowledge receipt of the
> message (*not process the request*). Does the receiver do this when the
> message/adaptation layer reads the message into its memory buffer or
> does the receiver acknowledge the receipt of the message after the
> message is stored in a persistent environment. The sender would indicate
> if the message should be ACKed after persistent storage or while still
> in the memory buffer. The reason is simply a confirmation that message
> was received and is durable verse just received.

That is an interesting point.

It seems there is a need for some application layer QoS here.
This is an interesting topic.
(Actually, we will have a talk about IoT QoS at T2TRG.)

However, the NON/CON bit controls message layer reliability, not
application layer durability.  I might want to have message layer
retransmissions (CON) and still not need application layer durability
(actually, that combination is 90 % of the applications that I'm
interested in: I get all application layer reliability I need with the
response).  In the TCP case, we always get message layer reliability, so
there is no need for NON/CON.  In any case, application layer durability
is better done using a checkpoint mechanism; this allows the receiver to
amortize expensive stable storage operations over a set of messages.
This is particularly easy to do for TCP, because that connection already
links the state.  For UDP, maybe a simple option could do this.

Grüße, Carsten

---

From: "Carey, Timothy (Timothy)" <[email-masked]>
Date: Thu, 16 Jul 2015 16:10:00 +0000

Carsten,

Ok - I was just explaining the need. I don't really have an opinion either way on this issue.

However, I do disagree with your statement.
"In the TCP case, we always get message layer reliability, so there is no need for NON/CON"  - This isn't true - you do not have 100% reliability until you know the message had made it to the receiver's application layer - this includes movement/coping from the TCP buffers to the Message layer buffers which TCP cannot account for. 

BR,
Tim

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Thursday, July 16, 2015 10:51 AM
To: Carey, Timothy (Timothy)
Cc: Benjamin Black; [email-masked] WG; Klaus Hartke
Subject: Re: [core] comments and support for draft-carey-core-std-msg-vs-trans-adapt-00

Hi Tim,

thanks for chiming in.

> <TAC> The question is when does the receiver acknowledge receipt of 
> the message (*not process the request*). Does the receiver do this 
> when the message/adaptation layer reads the message into its memory 
> buffer or does the receiver acknowledge the receipt of the message 
> after the message is stored in a persistent environment. The sender 
> would indicate if the message should be ACKed after persistent storage 
> or while still in the memory buffer. The reason is simply a 
> confirmation that message was received and is durable verse just received.

However, the NON/CON bit controls message layer reliability, not application layer durability.  I might want to have message layer retransmissions (CON) and still not need application layer durability (actually, that combination is 90 % of the applications that I'm interested in: I get all application layer reliability I need with the response).  In the TCP case, we always get message layer reliability, so there is no need for NON/CON.  In any case, application layer durability is better done using a checkpoint mechanism; this allows the receiver to amortize expensive stable storage operations over a set of messages.
This is particularly easy to do for TCP, because that connection already links the state.  For UDP, maybe a simple option could do this.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Jul 2015 21:19:00 +0200

Carey, Timothy (Timothy) wrote:
> this includes movement/coping from the TCP buffers to the Message layer buffers which TCP cannot account for. 

How much is that information worth?
If you did a message layer with persistence*) and somehow can make your
peer know that information, that peer could make use of it.
But none of this is interoperable.

Grüße, Carsten

*) I think that is a suboptimal design, but we don't have to agree on that.

---

From: Benjamin Black <[email-masked]>
Date: Sun, 19 Jul 2015 22:28:20 -0700

carsten,

the fundamental problem (almost said core problem!) is the way rfc7252
mixes transport and application together without a well-defined
delineation. you are acknowledging that this is so and that it is a problem
when you say CON/ACK are only intended for unreliable transports. it is
confirmed again in both the tcp and websocket transport drafts when they
describe the changes to the messages when using reliable transport.

had things been arranged such that transport and application remained
independent there would be no discussion about using CON/ACK with reliable
transports because the concept would not exist outside of the udp
transport. the tcp and websocket drafts would also not have to say anything
about differences from rfc7252 because the application messages would be
totally unchanged. exactly the things removed are the things that are in
fact udp transport, not coap application protocol.

as rfc7252 went out with this situation there is no undoing it. however, it
can at least be addressed for the reliable transports, and any future
unreliable transports, by defining the actual coap application protocol in
its own draft and then updating the tcp and websocket drafts to refer to
that draft while only specifying the transport-specific concerns.

there is a secondary problem that tim and i have both been describing which
is that the mere reception of bytes into a buffer on the receiving side is
not sufficient to declare a message delivered. if CON/ACK from rfc7252 are
going to be excluded from use in reliable transports, then another
mechanism needs to be defined. again, this must be done in a
transport-independent manner with transport drafts defining how the
application layer semantics are provided.

b

---

From: Simon Lemay <[email-masked]>
Date: Mon, 20 Jul 2015 15:20:33 +0200

Hi all,

First of all, thank you Timothy and Benjamin for taking the time and reviewing the spec.

I have been evaluating this problem in great detail as well, especially since IETF 92, as it became very apparent to me that it was not as clear as it could be (and i agree that this is something we need to add it the draft).

Since UDP is stateless, I have always seen the CON/NON feature as a way to add reliability to the transport mechanism.  The feature was for me always the way to express that the message was received.  The Nature of a Request/response protocol was the answer to determine if the Application layer was successful in accomplishing the task;  every Action warrants an answer.  The draft clearly state  that the semantic to deal with a response that was not return was completely up to the implementation.

This is due to the fact that every action is different. Hence having a one solution "fit all” for this would be very dangerous and would lead to more problem then it is trying to solve.  Either the protocol would be to verbose, or miss cases.  I personally would not like to have a protocol that as 2 level of acknowledgment plus a response.

In UDP, the way i see it, when you send a message, the ACK from the CON type need to go out as soon as possible.  Of Course if it can supply the response with the ACK, then it should do so, they will then be combined.  But in any case a response will be sent out as or with the result for the request.

In TCP, this is always the case. Once the complete request is received and reassembled, the application layer can process the content of the TCP payload.  Then the application will process the message and send back the appropriate response.

If we evaluate the different failure scenario we see that there is no a lot of difference between the UDP and TCP.  They both require to send back a response.  In the UDP case, there is nothing preventing a server to send all response separate from the ACK.

If the transfer from transport to application fails, no response will be sent, hence the client will need to take action.  Depending on the idempotentness (no sure that’s a word) of the request, different action will need to take place, and that also depends on the nature and intent of the action.  If the transfer succeeds, then there is 4 outcome.

	1. the content is complete garbage —> message is then dropped
	2. the content is coap, but erroneous —> (UDP: send a RST, TCP: sever the connection)
	3. the content is coap, but fails —> send response with error code
	4. the content is coap and succeeds —> send response according to the request

Having played a lot with CoAP over TCP.  early on I found that the CON was not useful, it was only adding more transaction with no meaning and using bandwidth.  I quickly switch to NON for all messages and was getting the same result.

Now in term of implantation, i do understand that keeping all the features intact would save a lot of time and make it very easy to have a clear cut implementation. But i do think that this light tweak in feature is worth the extra time as it still offers clearcut implementation possibility and the benefit for a high traffic of short messages would add up quite rapidly.

Simon Lemay
> On Jul 20, 2015, at 7:28 AM, Benjamin Black <[email-masked]> wrote:
> 
> carsten,
> 
> the fundamental problem (almost said core problem!) is the way rfc7252 mixes transport and application together without a well-defined delineation. you are acknowledging that this is so and that it is a problem when you say CON/ACK are only intended for unreliable transports. it is confirmed again in both the tcp and websocket transport drafts when they describe the changes to the messages when using reliable transport.
> 
> had things been arranged such that transport and application remained independent there would be no discussion about using CON/ACK with reliable transports because the concept would not exist outside of the udp transport. the tcp and websocket drafts would also not have to say anything about differences from rfc7252 because the application messages would be totally unchanged. exactly the things removed are the things that are in fact udp transport, not coap application protocol.
> 
> as rfc7252 went out with this situation there is no undoing it. however, it can at least be addressed for the reliable transports, and any future unreliable transports, by defining the actual coap application protocol in its own draft and then updating the tcp and websocket drafts to refer to that draft while only specifying the transport-specific concerns.
> 
> there is a secondary problem that tim and i have both been describing which is that the mere reception of bytes into a buffer on the receiving side is not sufficient to declare a message delivered. if CON/ACK from rfc7252 are going to be excluded from use in reliable transports, then another mechanism needs to be defined. again, this must be done in a transport-independent manner with transport drafts defining how the application layer semantics are provided.
> 
> 
> b
> 
> 
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: "Carey, Timothy (Timothy)" <[email-masked]>
Date: Mon, 20 Jul 2015 16:33:57 +0000

Simon,

Thanks for your detailed explanation.

I do indeed agree with you that implementations, in general, would only need to send NON; only the applications that MUST ensure that the message made it to the application layer would use CON and incur the extra expense; but we should allow for that. In your response you said that receipt acknowledgement should be the response itself and I can see that its just a different semantic (meaning) than receipt acknowledgement of UDP.

I didn't quite understand your last comment:
"Now in term of implantation, i do understand that keeping all the features intact would save a lot of time and make it very easy to have a clear cut implementation. But i do think that this light tweak in feature is worth the extra time as it still offers clearcut implementation possibility and the benefit for a high traffic of short messages would add up quite rapidly."

Which light tweak feature are you referring?

BR,
Tim

-----Original Message-----
From: Simon Lemay [mailto:[email-masked]] 
Sent: Monday, July 20, 2015 3:21 PM
To: Benjamin Black
Cc: Carsten Bormann; Carey, Timothy (Timothy); Klaus Hartke; [email-masked] WG
Subject: Re: [core] comments and support for draft-carey-core-std-msg-vs-trans-adapt-00

Hi all,

Simon Lemay
> If you did a message layer with persistence*) and somehow can make 
> your peer know that information, that peer could make use of it.

---

From: Simon Lemay <[email-masked]>
Date: Tue, 21 Jul 2015 13:17:00 +0200

Hi Tim,

Yes sorry, that was poorly phrased,

by light feature tweak, i meant that the transport reliability is done in the transport layer rather than in the application layer.

Cheers,

Simon

> On Jul 20, 2015, at 6:33 PM, Carey, Timothy (Timothy) <[email-masked]> wrote:
> 
> Simon,
>