Subject: [core] Proposal for a new subscription mechanism in CoAP

From: Gilman Tolle <[email-masked]>
Date: Fri, 18 Jun 2010 13:58:30 -0400

As mentioned in Ticket #1 and #2, the subscribe/notify system in the  
current CoAP draft is underspecified. For example:

* Unless the client's URL structure exactly mirrors the server's, a  
client creating a subscription needs to specify a destination URL for  
delivery of the updated resource, and there's no mechanism for that in  
the draft.
* There's no description of what CoAP method is used to deliver the  
resource.
* There's no mechanism for unsubscribing, querying, or managing  
existing subscriptions.

I'd like to suggest an alternate approach to subscribe/notify that  
treats subscriptions as RESTful resources, and does not introduce any  
new HTTP methods.

Start by defining two "well-known" URI namespaces: "/.sub" and "/.subs".

When a client subscribes to a URL (e.g. "/info"), they prepend "/.sub"  
to that URL, and then POST a message to the new URL (e.g. "/.sub/info").

The MIME type of the client's POST body is application/x-www-form- 
urlencoded, and the client specifies the subscription parameters as  
key-value pairs.

The most important key-value pair is "cb", short for "callback". So,  
the POST body would contain "cb=coap://client_addr/my/info/receiver".  
Because application/x-www-form-urlencoded is extensible, we can define  
additional parameters now or in the future. For example, we can define  
parameters for "suggested subscription lifetime", "maximum send  
interval", "authentication to use on delivery", etc.

If the "cb" parameter is omitted, we can fall back to the method in  
the draft -- the server assumes that the delivery URL is the source  
address, the default CoAP port (because the source port is ephemeral),  
and a path that mirrors the path being subscribed to.

The response to a successful subscription POST should be "201  
Created", with a Location header that includes a new URL on the server  
within the "/.subs" well-known URI namespace. For example, the server  
could send a Location of "/.subs/2", if the server is assigning  
integer IDs to new subscriptions. If the server does not support  
subscription on that resource, it can return one of the HTTP error  
codes instead.

 From then on, the server begins to POST new entity bodies to the  
callback URL as appropriate.

The returned subscription URI ("/.subs/2"), can now be stored by the  
client and used to:

* query its subscription status with a GET on that URI, with the body  
defined as application/x-www-form-urlencoded and containing  
information about the subscription (callback URL, remaining lifetime,  
etc)
* refresh its subscription or change its subscription parameters with  
a PUT to that URI using the application/x-www-form-urlencoded  
parameter set described above
* cancel its subscription with a DELETE on that URI

In addition, if the server chooses to support it (modulo security and  
discoverability choices), the server can provide browsing of  
subscriptions by supporting a GET on "/.subs".

What do you think?

Gil

---

From: Paul Duffy <[email-masked]>
Date: Fri, 18 Jun 2010 18:38:15 -0400

> treats subscriptions as RESTful resources, and does not introduce any 

+++1

> The MIME type of the client's POST body is 
> application/x-www-form-urlencoded, and the client specifies the 
> subscription parameters as key-value pairs.
>
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

---

From: Henning Schulzrinne <[email-masked]>
Date: Sat, 19 Jun 2010 04:42:49 -0400

In my opinion, a good example how religious adherence to REST leads to obscure and naming-based mechanisms that hide what's really happening behind arbitrary namespace conventions. 

Henning

On Jun 18, 2010, at 1:58 PM, Gilman Tolle wrote:

> As mentioned in Ticket #1 and #2, the subscribe/notify system in the current CoAP draft is underspecified. For example:
> 
> * Unless the client's URL structure exactly mirrors the server's, a client creating a subscription needs to specify a destination URL for delivery of the updated resource, and there's no mechanism for that in the draft.
> * There's no description of what CoAP method is used to deliver the resource.
> * There's no mechanism for unsubscribing, querying, or managing existing subscriptions.
> 
> I'd like to suggest an alternate approach to subscribe/notify that treats subscriptions as RESTful resources, and does not introduce any new HTTP methods.
> 
> When a client subscribes to a URL (e.g. "/info"), they prepend "/.sub" to that URL, and then POST a message to the new URL (e.g. "/.sub/info").
> 
> The MIME type of the client's POST body is application/x-www-form-urlencoded, and the client specifies the subscription parameters as key-value pairs.
> 
> The most important key-value pair is "cb", short for "callback". So, the POST body would contain "cb=coap://client_addr/my/info/receiver". Because application/x-www-form-urlencoded is extensible, we can define additional parameters now or in the future. For example, we can define parameters for "suggested subscription lifetime", "maximum send interval", "authentication to use on delivery", etc.
> 
> If the "cb" parameter is omitted, we can fall back to the method in the draft -- the server assumes that the delivery URL is the source address, the default CoAP port (because the source port is ephemeral), and a path that mirrors the path being subscribed to.
> 
> The response to a successful subscription POST should be "201 Created", with a Location header that includes a new URL on the server within the "/.subs" well-known URI namespace. For example, the server could send a Location of "/.subs/2", if the server is assigning integer IDs to new subscriptions. If the server does not support subscription on that resource, it can return one of the HTTP error codes instead.
> 
> From then on, the server begins to POST new entity bodies to the callback URL as appropriate.
> 
> The returned subscription URI ("/.subs/2"), can now be stored by the client and used to:
> 
> * query its subscription status with a GET on that URI, with the body defined as application/x-www-form-urlencoded and containing information about the subscription (callback URL, remaining lifetime, etc)
> * refresh its subscription or change its subscription parameters with a PUT to that URI using the application/x-www-form-urlencoded parameter set described above
> In addition, if the server chooses to support it (modulo security and discoverability choices), the server can provide browsing of subscriptions by supporting a GET on "/.subs".
> 
> What do you think?
> 
> Gil
>

---

From: Zach Shelby <[email-masked]>
Date: Sun, 20 Jun 2010 11:36:04 +0300

Hey, Ticket #2 was off limits before Ticket #1 is closed ;-) 

No really - it is super hard to talk about methods and solutions before we know what the architecture and requirements are. I don't think finding a solution will be very hard once we nail that down. But right now everyone with a solution has a different set of requirements ;-( 

Cullen, I think the input to your Ticket #1 thread on what is SUB/NOT was pretty good already. And then Carsten added some more. Could you guys please write a 2-page I-D on this so that we can converge and move on to Ticket #2. We would need that next week in order to get a new SUB/NOT mechanism into -01 of the draft before the cutoff. 

How does that sound as a way forward?

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 20 Jun 2010 10:56:50 +0200

On Jun 20, 2010, at 10:36, Zach Shelby wrote:

> 2-page I-D 

Working on it.  Will be more than two pages, though.

(There will also be another I-D on misc stuff that is missing for CoAP completion.)

Gruesse, Carsten

---

From: Paul Duffy <[email-masked]>
Date: Sun, 20 Jun 2010 09:49:28 -0400

I am not understanding why subscription resources need to be special 
cased with dedicated COAP methods.  Subscriptions are resources. they 
are created, have a lifetime, are deleted, can be modified, can be 
enumerated.  It makes no sense to me to add a special case method for 
creation (SUBSCRIBE), yet still be using the other methods to manage the 
subscriptions (GET, DELETE, etc.)

Yes, URI structure and payload definitions will be required for 
subscriptions.  Why is this a problem?

My main concern is that each incremental step COAP takes away from HTTP 
messaging semantics will result in difficult interoperability with the 
rest of the Web.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 20 Jun 2010 16:15:29 +0200

On Jun 20, 2010, at 15:49, Paul Duffy wrote:

> Subscriptions are resources. they are created, have a lifetime, are deleted, can be modified, can be enumerated.  

Why do you think that is the right design?

The main job of what you are calling a subscription is to manage conversation state, things like transport addresses, as opposed to application state.
Why is the resource level the right way to manage that state?

I'm sure that this design can be made to work; I'm just wondering what its advantages and disadvantages are.

Let me illustrate this point with a couple of slight modifications of your statement:

> Connections are resources. they are created, have a lifetime, are deleted, can be modified, can be enumerated.  
> Cached representations are resources. they are created, have a lifetime, are deleted, can be modified, can be enumerated.  

I'm pretty sure that in both cases this approach would lead to a contorted design; there is a reason why this state stays below the level of resources and their representation.

So what is it that makes the desire of one connector to remain informed about a resource, itself a resource?
(I'm not saying it is not, I just want to hear good arguments why that makes sense.  Or why it does not make sense.)

Gruesse, Carsten

---

From: Paul Duffy <[email-masked]>
Date: Sun, 20 Jun 2010 11:16:43 -0400

> Why do you think that is the right design?

Two choices...

- a subscription URI as child to the subscribed parent URI.   
protocol://{address}/somepath/subscription.  You subscribe directly to 
the resource.  Very OO

- a single subscription URI for the endpoint  
protocol:.//{address}/subscription.  All subscriptions managed @ a 
single URI.

Payloads defined appropriately

> The main job of what you are calling a subscription is to manage conversation state, things like transport addresses, as opposed to application state.

Disagree.  The main job of a subscription is to monitor changes to 
application state, and alert interested parties to those changes in 
application state.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 20 Jun 2010 17:51:34 +0200

On Jun 20, 2010, at 17:16, Paul Duffy wrote:

> Disagree.  The main job of a subscription is to monitor changes to application state, and alert interested parties to those changes in application state.

If you really believe that, then it's the job of the application to fill in the subscription.
Problem solved, nothing to do at the CoAP level.

I think this is a bit more subtle.  What is it that we expect the protocol to do?
Maybe nothing, and we *are* indeed done.

I'd love to see your payloads, BTW.

Gruesse, Carsten

PS.: Trying to support a position with "Very OO" is not going to win over any REST proponents :-)

---

From: Shidan <[email-masked]>
Date: Sun, 20 Jun 2010 12:37:45 -0400

I think Paul is absolutely right that the important subscriptions are
those of application state. I fully agree with him that gatewaying
with standard HTTP will be simpler if we consider subscriptions as
resources.

But the question is will it make sensor networks more complicated than
they need be?  CoAp is not the web, its for real-time P2P sensor
networks, where at the application level, there its realistic to
consider cases with more or as many sources as sinks and nodes as both
(at the application level). Some sort of intrinsic event bus with
event filtering in the network makes sense.

I think at the end of the day, do we want something that interfaces
with HTTP as nice as possible or something that's a nicer fit for
sensor networks.

Sent from my phone

---

From: Zach Shelby <[email-masked]>
Date: Sun, 20 Jun 2010 20:19:49 +0300

> with HTTP as nice as possible or something that's a nicer fit for

This is a critical point to solve for Ticket #1 - SUB/NOT architecture and requirements. 

I would say our first priority is to make a RESTful application protocol for embedded M2M application requirements (energy, building automation etc.). RESTful != HTTP, thus that does not mean we must do everything as well (or as badly) as HTTP. The secondary priority is to define an HTTP mapping, which obviously should be as simple as possible. 

Regarding proxying SUB/NOT - let's keep in mind the kinds of cases in which CoAP-HTTP proxying will actually be used, then that helps quite a bit in understanding how to optimize the HTTP mapping. For example, everyone seems to be assuming that subscriptions will be end-to-end across an CoAP-HTTP proxy. In practice what I have been seeing so far is a split approach as you want to cache and aggregate subscriptions in a proxy anyways. There are also security relationships that lean towards a split approach. Therefore I would say our main priority is a split SUB/NOT architecture WRT HTTP mapping.

As caching is an important feature of such a proxy, that brings us one more requirement. Notifications MUST be very easily identified with the resource they represent  so that the same cache used for responses to GETs can be leveraged. This keeps the caching feature of a CoAP-HTTP proxy simple. In practice this means the notification must include the URL of the  resource. 

Zach

---

From: "Stuber, Michael" <[email-masked]>
Date: Sun, 20 Jun 2010 12:34:38 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Zach Shelby
> 
> I would say our first priority is to make a RESTful application
protocol
> for embedded M2M application requirements (energy, building automation
> etc.). RESTful != HTTP, thus that does not mean we must do everything
as
> well (or as badly) as HTTP. The secondary priority is to define an
HTTP
> mapping, which obviously should be as simple as possible.

+1
 
> Regarding proxying SUB/NOT - let's keep in mind the kinds of cases in
> which CoAP-HTTP proxying will actually be used, then that helps quite
a
> bit in understanding how to optimize the HTTP mapping. For example,
> everyone seems to be assuming that subscriptions will be end-to-end
> across an CoAP-HTTP proxy. In practice what I have been seeing so far
is
> a split approach as you want to cache and aggregate subscriptions in a
> proxy anyways. There are also security relationships that lean towards
a
> split approach. Therefore I would say our main priority is a split
> SUB/NOT architecture WRT HTTP mapping.

I think Zach raises a good point w/r/t security.  While this depends on
the as yet to be determined security design, assuming that the security
design follows TLS/DTLS or some similar security session oriented
design, the security conversation will end at the proxy.  If this holds
true, implementations will likely follow one of two major design
paradigms:  Smart proxy that offers a variety of translation
capabilities --or -- CoAp end-to-end.  

I don't think that HTTP mapping (or not) should form the basis of this
decision.  I think the basis of this decision should be what the impact
is to the embedded devices.  In this regard, I have a request (and feel
free to direct me to a previous email if I missed it):

Zach, and proponents of native sub/not, can you explain the advantages
of having a native sub/notify command?  

Paul, and proponents of sub/not being managed as a resource, can you
explain the advantages of doing so?

(I can come up with arguments for both but I'd like to hear your
reasoning, specifically with respect to M2M communications in the
embedded space.  I'd really like to set the mapping question aside for
the moment, and just look at the merits of it in M2M context if
possible.)

---

From: Thomas Herbst <[email-masked]>
Date: Mon, 21 Jun 2010 09:50:04 -0700

RESTful != HTTP  is  true, but CoAP < HTTP should be true - we should be defining the minimal 
Subset of HTTP to make binary on a reliable UDP base.  

Sensor networks need to be able to interface with the rest of the world to be useful.  Defining elements that do not cleanly map to HTTP makes these strange little devices.  Why bother doing IP at that point? 

Pub/Sub should be defined as a BCP that can use CoAP.

Zach - could you elaborate on the proxying paragraph? I'd think a gateway would be a dumb device, not an intelligent proxy certainly not breaking end-to-end security. 

tom

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zach Shelby
Sent: Sunday, June 20, 2010 10:20 AM
To: Shidan
Cc: core
Subject: Re: [core] Proposal for a new subscription mechanism in CoAP

Zach

---

From: [email-masked]
Date: Mon, 21 Jun 2010 16:58:45 +0000

I am writing the http proxy for the coap draft assuming it is simple and not application gateway.

Don
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: Thomas Herbst <[email-masked]>
Sender: [email-masked]
Date: Mon, 21 Jun 2010 09:50:04 
To: Zach Shelby<[email-masked]>; Shidan<[email-masked]>
Cc: core<[email-masked]>
Subject: Re: [core] Proposal for a new subscription mechanism in CoAP

tom

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zach Shelby
Sent: Sunday, June 20, 2010 10:20 AM
To: Shidan
Cc: core
Subject: Re: [core] Proposal for a new subscription mechanism in CoAP

Zach

---

From: Shidan <[email-masked]>
Date: Mon, 21 Jun 2010 13:25:18 -0400

> Subset of HTTP to make binary on a reliable UDP base.

Not sure about this I think at this point all we can say is that we
should get CoAP ≈ HTTP, this might amount to CoAP < HTTP but I think
Zach is right that this needs much more thought.

> Sensor networks need to be able to interface with the rest of the world to be useful.  Defining elements that do not cleanly map to HTTP makes these strange little devices.  Why bother doing IP at…

I'm not sure I agree with your statement in this context. The internet
is filled with much much more dirtier mappings than CoAP with pubsub
to standard HTTP. XMPP, IAX, H323, etc, SIP server, Email to HTTP, and
I can spend years writing a list on systems that are far more
complicated than our current discussion permeating the net.

I haven't personally decided either way, but I believe we have to
study this as Michael mentioned, what both arguments offer and to do
that, I would personally like to understand the use cases of the proxy
more myself. Will systems need an intelligent proxy that can do things
like cache, aggregate  and broker events or will most be dumb proxies?

---

From: Shidan <[email-masked]>
Date: Mon, 21 Jun 2010 13:27:08 -0400

Again, as my general sentiment, I think we should be considering this
for the near future  use cases as well as the present.

---

From: Zach Shelby <[email-masked]>
Date: Wed, 23 Jun 2010 18:49:35 +0300

On Jun 20, 2010, at 10:34 PM, Stuber, Michael wrote:

>  In this regard, I have a request (and feel

I am not arguing that SUB/NOT must be realized using only native methods/messages. As an engineer, my first priority is that we all understand the goal and requirements, and then design for that. The design currently in coap-00 was a straw-man we put together just to see what such a think would look like based on the known requirements. I am not stuck on that specific solution really. 

Firstly, I do see that our main goal according to the charter is to optimize this for M2M application needs. We are not chartered to blindly do HTTP compression here.

Secondly, this working group is defining a RESTful transfer protocol, *not* an application design pattern. Therefore CoAP should only define a low-level conversational mechanism for making it possible to asynchronously observe a resource. The designers of HTTP didn't have subscribe/notify as a design requirement, but we do. 

Regarding how to make a subscription, I can see ways of doing it both as meta-data about a subscription, or subscriptions as well-known resources. Pretty neutral on which way to go, although meta-data probably fits a transfer protocol better.

Regarding how to make notifications, here it is so far clear to me that overloading POST or PUT is wrong. Firstly, the target URI referenced in the request is handled by the application protocol, not the transfer protocol. Secondly, the target URI does not make it easy for a proxy to match what resource this representation is about in a cache. 

> explain the advantages of doing so?

Resources are application constructions, if we want to deal with subscription/notify purely at the application then we are done. No need to specify it in CoAP at all, just leave it to any of the 10 different design patterns as with HTTP today.

Zach

---

From: Zach Shelby <[email-masked]>
Date: Wed, 23 Jun 2010 23:03:45 +0300

> I am writing the http proxy for the coap draft assuming it is simple and not application gateway.

Yep, this is exactly what we should describe in the draft. However we of course need to mention that caching may after be performed along with mapping, pointing to the caching section of the draft.

Implementation details and advanced features of proxies are out of scope of the document, I agree. We also don't want to start describing how to map Sub/Notify to the many ways of doing that in HTTP. So probably we end up not specifying how to map Sub/Notify at all, but rather leave that to proxy implementors to support the techniques they want (HTTP polling, HTTP webhooks, SIP, XMPP etc.)?

Zach

---

From: Zach Shelby <[email-masked]>
Date: Wed, 23 Jun 2010 23:44:54 +0300

> Subset of HTTP to make binary on a reliable UDP base.  

We are defining a RESTful protocol for constrained M2M applications. Yes, that should be simpler than HTTP, but I don't see a reason to be limited to a strict subset of HTTP as our requirements are pretty different. 

> Sensor networks need to be able to interface with the rest of the world to be useful.  Defining elements that do not cleanly map to HTTP makes these strange little devices.  Why bother doing IP at…

In many M2M applications you will see CoAP be used end-to-end, or between peer devices. Sure in some cases you want to access CoAP resources via HTTP using a proxy but that is not the main use case. Mappings to other protocols will also be common. More commonly there will be middleware, databases and web GUIs in between mortal HTTP browsers and low-level CoAP resources. IP doesn't mean end-to-end at the applications layer. 

> Pub/Sub should be defined as a BCP that can use CoAP.

You mean as a separate draft? 

> Zach - could you elaborate on the proxying paragraph? I'd think a gateway would be a dumb device, not an intelligent proxy certainly not breaking end-to-end security. 

Proxies are by no means very dumb when you start really implementing a production one useful for something.  Sure, you could make a simple mapping function which doesn't perform caching, aggregation or anything but I find that of limited use. But that is out of scope for CoRE. 

My understanding: Intelligence is not what splits security in a proxy. IPsec obviously stops at L3. At L4 we are dealing with UDP (DTLS) on one side, TCP (TLS) on the other and likely different trust models. I don't see how your security could be end-to-end unless it is object security? In a CoAP-CoAP proxy we could tunnel DTLS through I guess?

Zach