Subject: [core] Token length (was Re: Comments on draft-ietf-core-coap-03)

From: Klaus Hartke <[email-masked]>
Date: Tue, 2 Nov 2010 18:22:58 +0100

Zach Shelby wrote:
>> 3.2.
>> Table 2 lists the Token Option with a length of 1-2 B. That prevents a
>> lot of interesting usage cases wrt statelessness. What's the reasoning
>> to limit a token to 1-2 B?
>
> I take responsibility for this one. Let's ask the question the other way around. Why do you need more than 64k token possibilities?

When an application receives a response, it typically wants to
continue executing from where it left off when it sent the request. So
when it sends the request, it saves its state, and when it receives
the response, it restores the state.

There are two options to save the state for this purpose:

(a) save it locally, assign it a number and include that number as token, or

(b) save it in the token.

In the first case, the client needs to take the token from the
response, retrieve the locally saved state, and restore it. In the
second case, the client can restore the state directly from the token
in the response.

The second option is very useful, as you've noted yourself: If the
application needs to know the URI of the resource to continue
execution when it receives a response, it is useful to have the URI
included in the response. This allows, for example, a proxy to observe
more than 64k resources at the same time, without having to keep any
local state at all. It would be very useful to enable this for
arbitrary data.

Klaus

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Wed, 3 Nov 2010 08:58:16 +0100

> (b) save it in the token.

When the server that has to handle and store the Token is a constrained
node, this is an undesiderable behaviour.

Constrained devices have strong limitations in RAM, so requesting the node
to store an arbitrary amount of data as "Token" of the served request is not
efficient; also handling variable length Tokens can lead to a more complex
implementation, because static memory allocation is tipically used on that
kind of systems (e.g.: TinyOS).

Angelo

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 13:54:11 +0100

> kind of systems (e.g.: TinyOS).

It's not an _arbitrary_ amount of data; it still needs to fit in a
single option (270 bytes at most).

A URI can be up to 810 bytes in length (authority+path+query), and it
doesn't seem to a concern to include that in every notification.

Klaus

---

From: Zach Shelby <[email-masked]>
Date: Wed, 3 Nov 2010 15:04:55 +0200

> single option (270 bytes at most).

I was just going to ask if that is what you had in mind. So Token Option would be 1-270 B. 

> doesn't seem to a concern to include that in every notification.

That does not require state on the server though, just over the air. The server already knows the URI of the resource it is sending a notification about. What Angelo (and myself) are concerned about is forcing a server to hold even more state about each observation. The current state is not too bad, the IP address + port of the observer and the lifetime (4B). But adding up to a 270 B token starts to get extreme. Many of these nodes have 4-8 kB of RAM in total. What that means in practice is that observations with a large token might be rejected because of lack of RAM, or a node will be able to handle very few observations.  

Are you suggesting forcing the use of a Token with coap-observe? I was assuming that if no Token was provided then the notification would simply include the Uri-Path of the resource. 

Zach 

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 14:23:45 +0100

Zach Shelby wrote:
> That does not require state on the server though, just over the air. The server already knows the URI of the resource it is sending a notification about.
> [...]

Yes, it knows the authority and the path. But not the query - which
may potentially be different for each observer and which needs to be
included in every notification (a notification about the resource
/sensors/temperature?unit=celsius does not provide a new state for the
resource /sensors/temperature?unit=fahrenheit if both resources are
observed at the same time).

Klaus

---

From: Zach Shelby <[email-masked]>
Date: Wed, 3 Nov 2010 15:30:07 +0200

> observed at the same time).

Agreed.

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Nov 2010 10:01:58 -0500

On 11/3/10 8:04 AM, Zach Shelby wrote:
> Are you suggesting forcing the use of a Token with coap-observe? I was 
> assuming that if no Token was provided then the notification would 
> simply include the Uri-Path of the resource.

I've been trying to figure out how subscription correlation (for the 
purpose of refreshing subscriptions) would work without tokens. In fact, 
coap-observe is a bit under-specified regarding matching subscription 
refreshes to the original subscription.

Clearly, when a client provides a token, the server should be able to 
recognize refreshes as being (a) to the same resource, (b) from the same 
client (IP and port), and (c) with the same token.

Do we want to make it so that a subscription without a token would 
simply be correlated by being (a) to the same resource and (b) from the 
same client?

On a slightly related note, there should probably be a sentence or two 
in coap-observe that talks about how clients unsubscribe from a 
resource. Once we have an explicit subscription correlation model, all 
the mechanisms are actually in place: refreshing a subscription with a 
duration of "0" would cause the subscription to immediately expire (this 
is a natural property of the current design). It would be nice if the 
draft pointed this out.

/a

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 17:56:47 +0100

Adam Roach wrote:
> I've been trying to figure out how subscription correlation (for the purpose
> of refreshing subscriptions) would work without tokens. In fact,
> Do we want to make it so that a subscription without a token would simply be
> correlated by being (a) to the same resource and (b) from the same client?

It is the intention that a subscription is identified by (a) the URI
of the resource (including Uri-Query) and (b) the identity of the
client (IP and port). A client may or may not reuse the same token
when refreshing a subscription. I will clarify this in coap-observe.

> On a slightly related note, there should probably be a sentence or two in
> coap-observe that talks about how clients unsubscribe from a resource. Once
> we have an explicit subscription correlation model, all the mechanisms are
> actually in place: refreshing a subscription with a duration of "0" would
> cause the subscription to immediately expire (this is a natural property of
> the current design). It would be nice if the draft pointed this out.

Refreshing a subscription with a duration of 0 seconds actually
doesn't unsubscribe the client immediately. It sets the remaining
lifetime of the subscription to 0 seconds, but also retrieves the
current state of the resource (it's indistinguishable from a basic GET
request). I agree it would be nice to have this mention in the draft,
including the "surprise" that a non-subscription-related GET request
can affect a subscription. I will add this to coap-observe.

Klaus

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Nov 2010 12:42:17 -0500

On 11/3/10 11:56 AM, Klaus Hartke wrote:
> request).

Right. We do the same thing in SIP (unless the subscriber uses an etag 
mechanism similar to HTTP's, which could be easily retrofitted onto CoAP 
at a later date if people really want it).

> I agree it would be nice to have this mention in the draft, including 
> the "surprise" that a non-subscription-related GET request can affect 
> a subscription. I will add this to coap-observe.

Are you sure you want to treat a GET with no Subscription-Lifetime the 
same as a GET with a Subscription-Lifetime present but set to zero? As 
you mention, this can lead to the surprise you describe above.

You can avoid this situation by treating a GET with no 
Subscription-Lifetime as incapable of affecting subscriptions. That's 
certainly what I would have expected.

/a

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 20:10:52 +0100

Adam Roach wrote:
>> Refreshing a subscription with a duration of 0 seconds actually doesn't
>> unsubscribe the client immediately. It sets the remaining lifetime of the
>> subscription to 0 seconds, but also retrieves the current state of the
>> resource (it's indistinguishable from a basic GET request).
>
> mechanism similar to HTTP's, which could be easily retrofitted onto CoAP at
> a later date if people really want it).

I imagine caching to work as follows with coap-observe:

C: GET coap://server/resource for 3600s
S: 200 (Etag: 0x0dc2) "<?xml..."
S: 200 (Etag: 0x6fdb) "<?xml..."
S: 200 (Etag: 0x29bc) "<?xml..."
S: 200 (Etag: 0x7b39) "<?xml..."
S: 200 (Etag: 0x2152) "<?xml..."
C: GET coap://server/resource for 3600s (I have 0x2152, 0x7b39 and 0x29bc)
S: 200 (Etag: 0x37ba) "<?xml..."
S: 304 (Etag: 0x7b39)
S: 200 (Etag: 0x5cb0) "<?xml..."
S: 304 (Etag: 0x2152)
S: 304 (Etag: 0x7b39)
C: GET coap://server/resource for 3600s (I have 0x7b39, 0x2152 and 0x5cb0)
S: 304 (Etag: 0x5cb0)
S: 304 (Etag: 0x2152)
etc.

That is, a client can cache one or more responses, indicate to the
server which responses it has cached, and gets notified to which
cached state the resource changed to.

>> I agree it would be nice to have this mention in the draft, including the
>> "surprise" that a non-subscription-related GET request can affect a
>> subscription. I will add this to coap-observe.
>
> Are you sure you want to treat a GET with no Subscription-Lifetime the same
> as a GET with a Subscription-Lifetime present but set to zero?

Why would a client want to GET the current state of a resource during
the time it has a subscription to that resource? It already knows the
current state.

Klaus

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Nov 2010 15:16:05 -0500

> cached state the resource changed to.

That seems somewhat ill defined. In your example, there is an 
implication that the client promises to keep the cached versions around 
until the next GET.

For example: Consider a case in which I cache the last three resource 
representations I've received (as your client above seems to do), and 
send the etags for all three of them in a GET. The server then sends 
dozens of differing resource representations, followed by a 304 
indicating one of the etags I sent with my GET. The client has long 
since discarded this old version in favor of the more recent documents. 
Is that valid?

I guess the client can recover by sending a new GET without the 
offending etag, which should (?) result in an immediate 200 containing 
the actual resource state. Is that what you have in mind?

An alternate approach would be including the etag in the ACKs (assuming 
the server responses are sent confirmable, which isn't always the case).

> the time it has a subscription to that resource? It already knows the

It might, or it might not; section 2.2:

   "It is not necessary that... the server sends a notification
    response for every single state change."

If the client has a good reason to make sure it has the current state 
(instead of something approximating the current state), it needs some 
way of being able to do that.

/a

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Nov 2010 15:28:25 -0500

> cached state the resource changed to.

As an aside, I don't see anywhere in core-coap or core-observe that 
indicates that clients can include more than one Etag in a request. I 
agree it can be useful if specified the way you imply, but think you 
need to have some more explicit language in a document somewhere if you 
expect people to actually use Etags that way.

/a

---

From: Zach Shelby <[email-masked]>
Date: Wed, 3 Nov 2010 22:34:51 +0200

On Nov 3, 2010, at 10:28 PM, Adam Roach wrote:

>> That is, a client can cache one or more responses, indicate to the server which responses it has cached, and gets notified to which cached state the resource changed to.
> 
> As an aside, I don't see anywhere in core-coap or core-observe that indicates that clients can include more than one Etag in a request. I agree it can be useful if specified the way you imply, but think you need to have some more explicit language in a document somewhere if you expect people to actually use Etags that way.

We discussed this earlier, but decided only to allow a single Etag in a header for now. Although Klaus and Carsten are playing around with ideas for multiple Etags, I would prefer that we take the approach of completing coap-observe with a single Etag and then later explore more advanced functionality. OK?

Zach

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 22:07:58 +0100

> We discussed this earlier, but decided only to allow a single Etag in a header for now. Although Klaus and Carsten are playing around with ideas for multiple Etags, I would prefer that we take the…

coap-03 section 3.2.7 doesn't say that "This option MUST NOT occur
more than once in a header." :-)

But, yes, we can explore multiple Etags in a request later. (HTTP's
If-None-Match header allows multiple entity tags.)

Klaus

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 3 Nov 2010 22:09:48 +0100

Here's an updated example for caching in coap-observe with a single Etag:

C: GET coap://server/resource for 3600s (I have 0x2152)
C: GET coap://server/resource for 3600s (I have 0x7b39)

Adam Roach wrote:
> That seems somewhat ill defined. In your example, there is an implication
> that the client promises to keep the cached versions around until the next
> GET.

Yes, that's what I have in mind. The client needs to keep the response
it indicated to the server cached until the next subscription refresh.
It can cache one or more additional responses. When it resubscribes,
it discards all but one response, according to some caching strategy.
It indicates to the server which response it kept, and keeps that
response until the next refresh.

> I guess the client can recover by sending a new GET without the offending
> etag, which should (?) result in an immediate 200 containing the actual
> resource state. Is that what you have in mind?

No, I think it doesn't make sense to tell the server that a client has
cached a specific state, and immediately retrieve that state when the
resource changed to it. If a client doesn't want to keep around some
old response, it can always resubscribe with another Etag or without
any Etag.

> An alternate approach would be including the etag in the ACKs (assuming the
> server responses are sent confirmable, which isn't always the case).

A server can't conclude from an ACK that the client has added a
response to its cache. We could invent some protocol to keep the
client's cache and what the server thinks is in the client's cache in
sync, but I think we shouldn't go down this road.

> (instead of something approximating the current state), it needs some way of
> being able to do that.

If a client has a subscription and wants to make sure it has the
current state, it can send a GET request. If it wants to continue the
subscription, it includes a Subscription-Lifetime Option. If it
doesn't want to continue the subscription, it omits it. I think there
is no problem here.

Klaus

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Nov 2010 16:37:00 -0500

> doesn't want to continue the subscription, it omits it. I think there 

You used the term "surprise" to describe this situation, and I've found 
"surprise" to generally be a bad property in a protocol. My experience 
is that surprising aspects of protocols are where implementors tend to 
make invalid assumptions and write code that doesn't quite interoperate.

So I agree with you: what you propose is surprising. I also agree with 
you that it works. There's another option that works without any 
surprises. I have a natural bias towards this other approach because of 
the value I place on interoperability. You don't seem to like this 
second approach, but have not yet explained why.

/a

---

From: Zach Shelby <[email-masked]>
Date: Wed, 3 Nov 2010 23:49:47 +0200

On Nov 3, 2010, at 11:37 PM, Adam Roach wrote:

>> If a client has a subscription and wants to make sure it has the current state, it can send a GET request. If it wants to continue the subscription, it includes a Subscription-Lifetime Option. If it doesn't want to continue the subscription, it omits it. I think there is no problem here.
> 
> You used the term "surprise" to describe this situation, and I've found "surprise" to generally be a bad property in a protocol. My experience is that surprising aspects of protocols are where implementors tend to make invalid assumptions and write code that doesn't quite interoperate.
> 
> So I agree with you: what you propose is surprising. I also agree with you that it works. There's another option that works without any surprises. I have a natural bias towards this other approach because of the value I place on interoperability. You don't seem to like this second approach, but have not yet explained why

Adam has a valid point here, I also think we should not remove the subscription due to a normal GET. This is the most logical behavior and doesn't hurt anything. I propose we make a ticket on this.

Zach

---

From: Guido Moritz <[email-masked]>
Date: Thu, 4 Nov 2010 13:56:08 +0100

Sorry for being maybe a bit too late to this discussion, but I am a bit
confused about correlation of subscriptions, notifications and subscription
refresh.

First of all I would like to express that I would say using the Token Option
is a bit misleading because Tokens are to identify the transaction and each
single subscription, notification and subscription refresh is a single
transaction for itself which might be CON or NON. Let's say a subscription
causes a notification each 25 days. So as a client I would appreciate if the
server can ACK my subscription to be sure I will get the notification.

Using the URI of the subscription would probably result in long URIs
(especially due to the long queries for filtering).

So I would propose to use a single EventID for the subscription. The EventID
can be very compact because it identifies the subscription together with
IP+Port of client (which must be stored anyway) and there might be not many
subscriptions to distinguish from. In case of a CON subscription the server
includes the EventID together with the Token (and without the URI) in the
subscription ACK. If the subscription is NON, the server may include the
EventID in the first notification and only in this first notification also
the URI. For each subsequent notification or refresh of the subscription the
EventID is included (without any URI). The URI is known by both endpoints
and not needs to be transmitted unless a change is required by the client.
The URI also must not be stored on both sides, only the "application logic"
behind the URI and the query. Thus the transaction layer and the application
logic behind the subscription are decoupled more logical.

The nice thing about this subscription identification is that even no
URI+query must be included for subscription refreshing, only the new
subscription time and the EventID.

BTW: Including such an EventID is the way how WS-Eventing avoids including
complex filtering/query information which both endpoints already know in
each message and it works nice even in only slightly complex queries. 

Guido

> -----Ursprüngliche Nachricht-----
> Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
> Klaus Hartke
> Gesendet: Mittwoch, 3. November 2010 17:57
> An: core
> Betreff: Re: [core] Subscription Refresh Correlation (was Re: Token length
> (was Re: Comments on draft-ietf-core-coap-03))
> 
> Adam Roach wrote:
purpose
> > Do we want to make it so that a subscription without a token would
simply
> be
in
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 4 Nov 2010 21:47:09 +0800

As a general observation, the DNS protocol (RFC 1035), which is trying to work in the constrained space of 512-byte UDP packets, went very well with repeating the request parameters in every response.  It might be worthwhile to have a look why that was done and what advantages it has up to today.

Gruesse, Carsten