Subject: [core] asynchronous responses

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 15 Oct 2010 23:44:14 +0200

Hi,

I recently read the CoAP document and I have a question that was
already discussed here in July. The question is:

  How can I reliably match the responses to asynchronous requests?

It seems that there is a missing identifier for the request/response
transactions on the upper CoAP layer. The text says that the URI is
echoed back, which however may not be sufficient and it is relatively
expensive in terms of the number of octets needed.

Did you consider other approaches, e.g. using option in the first ACK
that carries a upper transaction identifier (utid) and that will also
be shipped with the following CON message?

   Client                 Server
      |                     |
      |     CON tid=48      |
      | code=GET http://n.. |
      +-------------------->|
      |                     |
      |     ACK tid=48      |
      | code=000 [etid=42]  |
      |<--------------------+
      |                     |
        ... Time Passes ...
      |                     |
      |     CON tid=783     |
      | code=200 [etid=42]  |
      |<--------------------+
      |                     |
      |     ACK tid=783     |
      | code=000 [etid=42]  |
      +-------------------->|
      |                     |

I just saw that something similar has been suggested in coap-misc-06.
However, the text in coap-misc-06 seems to suggest that the client
invoking the method needs to send the token; since the client does not
control whether a request is processed in synchronous or asynchronous
mode, it seems better to let the server generate a token when needed.
(This also saves space in the request that has to carry the URI.)

/js

PS: I changed the style of the figure, trying to make it clearer that
    both the method and the response code are carried in the same
    field, also putting options in brackets.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Robert Quattlebaum <[email-masked]>
Date: Sun, 17 Oct 2010 11:48:02 -0700

Hello Juergen!

>  How can I reliably match the responses to asynchronous requests?

I was just about to start a new thread to discuss this (and a possible solution), but since you started this one I'll go ahead and talk about that here. :)

In my opinion the current answer to your question is simple: You can't---at least not reliably, or easily, or efficiently.

Obviously this is a problem. 

> be shipped with the following CON message?

Using the same TID as the first transaction isn't a good enough solution though: Think about the case where a device is sending a CoAP request to itself and some of the packets are dropped. Confusion ensues. There are likely other subtle issues that are not immediately obvious. As Carsten once mentioned to me off-list, these types of things are unfortunately non-trivial. 

Another alternative, which you mentioned, involves clients adding a Token option to every request transaction they send (because you can't predict ahead of time which responses will be asynchronous), but that is obviously wasteful---if effectively adds another three bytes to the CoAP minimal overhead.

But...

I have come up with what I believe to be an optimal solution to this problem, requiring minimal changes to the CoAP spec:

Instead of using an arbitrary TID for the async response transaction, we should define the TID of the second transaction to be the one's complement (bitwise invert) of the original request TID. This would make it easy to correlate async responses to the original requests without using the same numeric TID for both transactions. This fixes the talking-to-yourself problem and would probably rectify other issues as well. 

For all I know something like this may have been suggested before and dismissed for a good (but not obvious) reason. However, it seems to, at least superficially, address everyone's concerns.

I would love to hear some thoughts about this suggestion.

__________________
Robert Quattlebaum
Jabber: [email-masked]
eMail:  [email-masked]
www:    http://www.deepdarc.com/

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 17 Oct 2010 22:50:52 +0200

> Another alternative, which you mentioned, involves clients adding a Token option to every request transaction they send (because you can't predict ahead of time which responses will be asynchronous…

I did not propose to use the same TID nor did I propose that the
client sends a token in every CON. I thought my proposal was clear,
but apparently not. (I even included a figure in my original post.)

> Instead of using an arbitrary TID for the async response transaction, we should define the TID of the second transaction to be the one's complement (bitwise invert) of the original request TID. Thi…

I doubt this works since the TIDs other clients may use in the future
are unpredictable and so I run into rare but possible conflicts with
TIDs I have been using to talk with clients and TIDs clients want me
to use in asynchronous responses.

/js

---

From: Robert Quattlebaum <[email-masked]>
Date: Sun, 17 Oct 2010 15:08:45 -0700

Hello Juergen!

> but apparently not. (I even included a figure in my original post.)

Sorry, for some reason when I first looked at the diagram I thought I saw TID's that were the same for the original request and the async response. Looking at it again, it is quite clear that they are not. I must have gotten my eyes crossed. I apologize for my misunderstanding, I'll make an attempt to read more carefully in the future.

> to use in asynchronous responses.

Good point. This can be fixed by also defining that the MSB of the TID of a synchronous request must always be set to 0. Then there would never be such a conflict, because the TID of the async response would always have the MSB set (and, thus, it is always a value that would never be chosen by that client otherwise). 

In any case, I agree that some sort of solution (whether yours, mine, or something else) is needed. Async responses are incredibly awkward to use otherwise. 

__________________
Robert Quattlebaum

---

From: Robert Quattlebaum <[email-masked]>
Date: Sun, 17 Oct 2010 15:18:54 -0700

I just wanted to address some individual observations about your original message. 

> be shipped with the following CON message?

I'm not sure if this was considered or not, but this sounds like a much more reasonable approach than relying solely on the URI-path parameter. Including the URI-path option is probably still useful for certain applications, but alone it seems problematic.

I would like to make a small suggestion. Might it be easier to simply use the original TID as the value of your etid/utid option, instead of an entirely new identifier? The resulting timeline would then look like this:

  Client                 Server
     |                     |
     |     CON tid=48      |
     | code=GET http://n.. |
     +-------------------->|
     |                     |
     |     ACK tid=48      |
     |      code=000       |
     |<--------------------+
     |                     |
       ... Time Passes ...
     |                     |
     |     CON tid=783     |
     | code=200 [etid=48]  |
     |<--------------------+
     |                     |
     |     ACK tid=783     |
     |      code=000       |
     +-------------------->|
     |                     |

> (This also saves space in the request that has to carry the URI.)

I agree entirely with this sentiment.

>    field, also putting options in brackets.

Again, I apologize for my misunderstanding earlier.

__________________
Robert Quattlebaum

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 18 Oct 2010 01:27:17 +0200

>  How can I reliably match the responses to asynchronous requests?

I don't have time right now to do a detailed response, but my question would be:
Why do you want to?

(If you indeed need to, that's what the token was meant for.  And yes, the idea of a server-supplied token/ticket/promise/... is interesting, so we should discuss this.  But for now I'd like to understand what specific motivation you have for wanting the binding.)

One of the interesting properties of asynchronous responses is that they are self-describing.  They could be sent by multicast and would still work.

So what do we gain by binding them to a specific request?
(Yes, I'm focusing on GET requests here.)

One of the nice things about CoAP is that it is not trying to be a generic transaction protocol with a REST protocol layered on top, but a REST protocol with a transaction protocol purpose-built to serve just the REST protocol.

Gruesse, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 18 Oct 2010 17:44:19 +0300

I'd like to add to this discussion that we actually have two types of asynchronous responses:

1. Delayed response to a request like in Section 2.1.2 of coap-02.

2. Repeated responses to a subscription as in hartke-coap-observe-02. This already discuses the possible use of Tokens.

In cases of a delayed GET response or a notification to a subscription, the URI is enough for most use cases. For example when a proxy is using subscriptions to update a cache, which then serves HTTP clients via polling, identification using the URI makes the most sense. I do see how optionally including a Token saves space and makes more sense for certain applications. So leave it up to the client to include a Token in a request. If a Token is present in a request, the server simply includes that instead of a URI. 

For delayed responses to POST, PUT, DELETE, it might make more sense to always use a Token. Here we can either require a client to always include a Token (might now make sense), or for the server to generate a Token in the ACK to a request that can't be answered. The same Token is then expected by the client in the delayed response. The same would obviously apply to delayed GET. 

So I agree with Carsten, for GET cases we don't need to require a Token, but might want to use it in some cases. For delayed responses in general it might make sense to let the server generate the Token in the ACK as the client doesn't know if the response will be delayed or not.

Zach  

On Oct 18, 2010, at 2:27 AM, Carsten Bormann wrote:

> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Peter Bigot <[email-masked]>
Date: Mon, 18 Oct 2010 09:59:17 -0500

The problem is that an asynchronous response is a new CoAP transaction which
provides the ReST response in a ReST transaction where the request was a
previous, now completed, CoAP transaction.  The transaction ID is irrelevant
to the ReST level, so should not be relied upon to do the correlation, but
something does need to indicate that an incoming message is the response
component of a specific ReST transaction.

The Token option seems to satisfy this requirement adequately.  I would
argue that, if a server cannot provide a response in the transaction
acknowledgement, and was not provided a Token to use to couple its eventual
response to the original request, the request must be rejected because there
is no facility to correlate the response to the request.  Another error code
is required.

Robert's alternative of an XOR'd transaction ID would eliminate this
requirement, though we'd have to reserve a bit as always zero in origin TIDs
to ensure an origin server doesn't end up using the same TID "too soon" for
two transactions (one originating with it, and one mandated by a response to
a client).  Probably also have to relax the requirement on increasing TIDs
from an endpoint.  Messy, but doable.  I'd use the Token: some clients may
not be able to support asynchronous responses (can't hold the pending state
while other transactions occur), and leaving it up to the client to control
their use simplifies their implementation.

Carsten, it sounds like you're proposing use of a split ReST transaction
mechanism.  This is an interesting approach that might provide a solution
for reconciling multicast at the transport level with ReST.  If all CoAP
transactions destined for a multicast address were treated as asynchronous,
and the responses were new ReST transactions, consistency could be
maintained.  But I don't think it works without a lot more detail.

Unless you want CON to become a ReST operation like GET, which I think has a
lot of consequences.  It may be too late to work those out.

Or if you accept the premise that a ReST GET can legitimately elicit
multiple state representations in response.  I still don't, but won't bother
arguing about it any more.

Peter

On Sun, Oct 17, 2010 at 6:27 PM, Carsten Bormann <[email-masked]> wrote:

> (If you indeed need to, that's what the token was meant for.  And yes, the
> idea of a server-supplied token/ticket/promise/... is interesting, so we
> should discuss this.  But for now I'd like to understand what specific
> motivation you have for wanting the binding.)
>
> One of the interesting properties of asynchronous responses is that they
> are self-describing.  They could be sent by multicast and would still work.
>
> One of the nice things about CoAP is that it is not trying to be a generic
> transaction protocol with a REST protocol layered on top, but a REST
> protocol with a transaction protocol purpose-built to serve just the REST
> protocol.
>
> Gruesse, Carsten
>

---

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 18 Oct 2010 10:52:54 -0700

Hello Carsten!

> One of the interesting properties of asynchronous responses is that they are self-describing.  They could be sent by multicast and would still work.

I did some more thinking about this case. Originally I did not like that idea at all, but you mentioning this has made me consider something I hadn't thought of before. I'll elaborate here for the sake of others.

Only using the URI options for correlating allows you to implement a device that only has data at discrete moments in time. In such a case, a GET request becomes a "one-time subscription" request. The response could even be multicasted, allowing the server to not even keep state about pending requests. It would work as follows:

A client sends a GET request for something to a server device. The server device ALWAYS returns an ACK response with a code of zero. Many devices could do this all at once, and the server would respond to all of them with simple ACK and a response code of zero. When the device actually has data available, it then multicasts a response which all of the devices which requested the resource earlier will receive. You can't do this if all of the devices are expecting the response to be tailored to their specific client-defined token.

But Juergen's suggestion would still work great in this case, because the server generates the identifier which is used to relate the requests. In this case, the server would simply always return the same identifier (etid, in Jeurgen's suggestion) in the ACK for the original request. It would make client implementations easier without preventing the above mentioned scenario. (Although there may be security implications)

I now like Juergen's concept better than my own suggestions now.

__________________
Robert Quattlebaum