Subject: [core] http observe unification

From: Peter Bigot <[email-masked]>
Date: Thu, 28 Oct 2010 07:43:09 -0500

Cullen:

In the telecon yesterday, you objected to my proposal that the solution to
the observation problem be left to designers of specific systems.  It
sounded like you felt a reference document describing alternative approaches
that applications might take would not be adequate.  Could you elaborate a
little more here on what you believe CoAP must provide, and in what
timeframe?

I understood you to say that some IETF working group was nearly complete
with a proposed standard way to implement observation in HTTP.  Is that the
Httpbis group?  To which of their documents were you referring?
draft-loreto-http-bidirectional is somewhat relevant (wrt streaming
responses).

The clearly relevant one is draft-roach-sip-http-subscribe
s
, but that's based on SIP event notification.  Were we to take that
approach, it might be best to define a CoAP-like protocol "csip", rather
than integrate that functionality directly into CoAP.  On first glance, it
appears this would be at least as much work as defining CoAP has been.  It
would, though, give us a solution to observation, notification, and group
communication that is compatible with the HTTP family of protocols, so is
certainly worth considering.

Thanks.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Oct 2010 15:56:00 +0200

Hi Peter,

I can't speak for Cullen (who is fast asleep right now, I hope), but you pressed what I think is a similar button with me, so I'll give you my interpretation.

> In the telecon yesterday, you objected to my proposal that the solution to the observation problem be left to designers of specific systems.  It sounded like you felt a reference document describing alternative approaches that applications might take would not be adequate.  Could you elaborate a little more here on what you believe CoAP must provide, and in what timeframe?

Most of us are used to take existing protocols and kludge together a solution for a problem using these protocols.  (One could argue that much of the Internet works that way.)  However, all these kludges have limitations and complexities caused by their kludginess.  When designing something new, it is important to separate in one's mind the desire to reuse a neat kludge that once worked for us, from a *good* way to solve the problem.  (As usual, this is balanced by the need to avoid NIH -- component reuse is a good idea, and a fit rarely has to be perfect.)

So, back to the specific problem, I think what would be interesting to see is a spec on resource observation that is developed well enough that these issues can be evaluated.  HTTP has shown us that there is no end of kludges that can be invented to solve related problems -- we have to find out which approaches are good for CoAP (where goodness includes, but is *not* limited to, ease of building boxes that map to HTTP).

In the end we want to have *the* way to handle asynchronous communication in CoAP, or -- if we need to have multiple ways -- one each for a well-defined area of application.  "Let the market decide" often just means people play games whose kludge goes into which widely used product first -- in the end we all lose.

(To avoid misunderstandings -- I'm not saying that, say, every solution that treats a subscription as a REST resource in their own right is a kludge, I'm just feeling that way when the main motivation appears to be that one would need to do this in HTTP.  CoAP is not HTTP.  On the other hand there may be a need to do both the lightweight and the heavyweight solution if each of them does not solve all the problems the other does and the unsolved problems are important enough to warrant a second solution.)

> I understood you to say that some IETF working group was nearly complete with a proposed standard way to implement observation in HTTP.  Is that the Httpbis group?  To which of their documents were you referring?  draft-loreto-http-bidirectional is somewhat relevant (wrt streaming responses).

I think that comment was with respect to websockets.  Websockets is interesting as it is not an application protocol in the strict sense, but a way to securely (for some definition of security) derive a (minimally structured) transport to a browser from a URI.  The other half, how notifications should look like and how they interact with resource state, is outside the scope of websockets.  CoAP does not need websockets, because we already know how to send packets in our constrained networks, and there is no browser security to take care of.

Of course, there is half a dozen other hybi proposals that are more semantically rich.
http://trac.tools.ietf.org/bof/trac/wiki/HyBi#Proposals lists about half of them.

> The clearly relevant one is draft-roach-sip-http-subscribe, but that's based on SIP event notification.  Were we to take that approach, it might be best to define a CoAP-like protocol "csip", rather than integrate that functionality directly into CoAP.  On first glance, it appears this would be at least as much work as defining CoAP has been.  It would, though, give us a solution to observation, notification, and group communication that is compatible with the HTTP family of protocols, so is certainly worth considering.

I think this draft is a good source of ideas on how to combine HTTP resources with other protocols.  I don't see a need to draw in yet another protocol, though.

Gruesse, Carsten

---

From: Peter Bigot <[email-masked]>
Date: Thu, 28 Oct 2010 09:28:42 -0500

I think the IETF activity I was looking for was "HyBi", a term I didn't
recognize; thanks for the pointer.  From its Wiki page:

The HyBi <http://trac.tools.ietf.org/bof/trac/wiki/HyBi> activity of the
IETF coordinates work in the IETF related to several proposals for
bidirectional, "long poll", and "reverse" HTTP: mechanisms where the
connection is still initiated by the client but communication is initiated
by the server.

This clearly states the problem for HTTP, which involves concepts of
connection and client/server communication.  CoAP does not have a concept of
"connection", and the client/server distinction is, in my opinion, a residue
of the assumption in REST that a resource is owned by a single entity (the
server) and interaction with it is always initiated by a different entity
(the client).  It is not necessarily appropriate for all M2M communication,
where an explicit peer-to-peer model has significant value.

There are a variety of approaches.  I'm still collecting base information,
then will drop below the working group reflector to coordinate with Kerry,
Klaus, Angelo, and others who have expressed a specific interest in the
problem, with the intent of coming up with a set of use cases and common
terminology that can be used to evaluate these approaches.  I have other
responsibilities which may delay visible progress until next week, but we'll
still aim to have something for discussion in Beijing.

Peter

On Thu, Oct 28, 2010 at 8:56 AM, Carsten Bormann <[email-masked]> wrote:

> Hi Peter,
>
> I can't speak for Cullen (who is fast asleep right now, I hope), but you
> pressed what I think is a similar button with me, so I'll give you my
> interpretation.
>
> > In the telecon yesterday, you objected to my proposal that the solution
> to the observation problem be left to designers of specific systems.  It
> Most of us are used to take existing protocols and kludge together a
> solution for a problem using these protocols.  (One could argue that much of
> the Internet works that way.)  However, all these kludges have limitations
> and complexities caused by their kludginess.  When designing something new,
> it is important to separate in one's mind the desire to reuse a neat kludge
> that once worked for us, from a *good* way to solve the problem.  (As usual,
> this is balanced by the need to avoid NIH -- component reuse is a good idea,
> and a fit rarely has to be perfect.)
>
> So, back to the specific problem, I think what would be interesting to see
> is a spec on resource observation that is developed well enough that these
> issues can be evaluated.  HTTP has shown us that there is no end of kludges
> that can be invented to solve related problems -- we have to find out which
> approaches are good for CoAP (where goodness includes, but is *not* limited
> to, ease of building boxes that map to HTTP).
>
> In the end we want to have *the* way to handle asynchronous communication
> in CoAP, or -- if we need to have multiple ways -- one each for a
> well-defined area of application.  "Let the market decide" often just means
> people play games whose kludge goes into which widely used product first --
> in the end we all lose.
>
> (To avoid misunderstandings -- I'm not saying that, say, every solution
> that treats a subscription as a REST resource in their own right is a
> kludge, I'm just feeling that way when the main motivation appears to be
> that one would need to do this in HTTP.  CoAP is not HTTP.  On the other
> hand there may be a need to do both the lightweight and the heavyweight
> solution if each of them does not solve all the problems the other does and
> the unsolved problems are important enough to warrant a second solution.)
>
> I think that comment was with respect to websockets.  Websockets is
> interesting as it is not an application protocol in the strict sense, but a
> way to securely (for some definition of security) derive a (minimally
> structured) transport to a browser from a URI.  The other half, how
> notifications should look like and how they interact with resource state, is
> outside the scope of websockets.  CoAP does not need websockets, because we
> already know how to send packets in our constrained networks, and there is
> no browser security to take care of.
>
> > The clearly relevant one is draft-roach-sip-http-subscribe, but that's
> based on SIP event notification.  Were we to take that approach, it might be
> best to define a CoAP-like protocol "csip", rather than integrate that
> functionality directly into CoAP.  On first glance, it appears this would be
> at least as much work as defining CoAP has been.  It would, though, give us
> a solution to observation, notification, and group communication that is
> compatible with the HTTP family of protocols, so is certainly worth
> considering.
>
> I think this draft is a good source of ideas on how to combine HTTP
> resources with other protocols.  I don't see a need to draw in yet another
> protocol, though.
>
> Gruesse, Carsten
>
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Oct 2010 17:20:06 +0200

On Oct 28, 2010, at 16:28, Peter Bigot wrote:

> drop below the working group reflector to coordinate with Kerry, Klaus, Angelo, and others who have expressed a specific interest in the problem,

I look forward to the results!

(with WG chair hat:)

Let me take the opportunity to ask everyone that we keep technical discussions on the mailing list as much as reasonably possible.  

Of course, sometimes it is better to form a small design team to explore new directions, and there may be subjects (such as considerations influenced by future products) that simply can't be discussed on the list.
But, if you feel comfortable with your material out on the list, please don't refrain from making use of it.
There are often important nuggets of rationale that are lost when a close group discussion finally moves into the open.

(I'll revise this statement slightly when we exceed 25 messages per day.  We have had ~ 1000 messages in the ~ eight months this list has existed, so we currently average at about four messages per day.  Somehow activity seems to peak around I-D deadlines and physical meetings, of course...)

Gruesse, Carsten

---

From: <[email-masked]>
Date: Thu, 28 Oct 2010 16:29:39 +0100

the hybi workgroup has been in something of a mess - there's been a dustup over whether it's a W3C (WHATWG) or IETF workgroup, drafts have been stalled, editors have been changed.

I would not recommend basing anything here on hybi's output.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Bigot
Sent: 28 October 2010 15:29
To: Carsten Bormann
Cc: core
Subject: Re: [core] http observe unification

I think the IETF activity I was looking for was "HyBi", a term I didn't recognize; thanks for the pointer.  From its Wiki page:

The HyBi<http://trac.tools.ietf.org/bof/trac/wiki/HyBi> activity of the IETF coordinates work in the IETF related to several proposals for bidirectional, "long poll", and "reverse" HTTP: mechanisms where the connection is still initiated by the client but communication is initiated by the server.

This clearly states the problem for HTTP, which involves concepts of connection and client/server communication.  CoAP does not have a concept of "connection", and the client/server distinction is, in my opinion, a residue of the assumption in REST that a resource is owned by a single entity (the server) and interaction with it is always initiated by a different entity (the client).  It is not necessarily appropriate for all M2M communication, where an explicit peer-to-peer model has significant value.

There are a variety of approaches.  I'm still collecting base information, then will drop below the working group reflector to coordinate with Kerry, Klaus, Angelo, and others who have expressed a specific interest in the problem, with the intent of coming up with a set of use cases and common terminology that can be used to evaluate these approaches.  I have other responsibilities which may delay visible progress until next week, but we'll still aim to have something for discussion in Beijing.

Peter

On Thu, Oct 28, 2010 at 8:56 AM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
Hi Peter,

Gruesse, Carsten

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 28 Oct 2010 10:22:06 -0600

On 10/28/10 9:29 AM, [email-masked] wrote:
> the hybi workgroup has been in something of a mess - there's been a
> dustup over whether it's a W3C (WHATWG) or IETF workgroup, drafts have
> been stalled, editors have been changed.
>  

That characterization sounds like FUD to me.

1. HyBi is an IETF working group.

2. Drafts get stalled in all sorts of working groups, but that doesn't
prevent other groups from using the final output.

3. The editor did change, but a change in editors does not mean that the
output is suspect. In this case I think the change has led to smoother
discussions within the WG.

If folks here have concerns about re-using the output of the HyBi WG, I
suggest that they contact the WG chairs or the responsible AD (cc'd).

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 29 Oct 2010 14:48:36 -0600

(inline and all of this in my individual contributor roll not my co-chair roll )

On Oct 28, 2010, at 6:43 AM, Peter Bigot wrote:

> Cullen:
> 
> In the telecon yesterday, you objected to my proposal that the solution to the observation problem be left to designers of specific systems.

Yes - I think there should be at document that describes at least one way to do this. I don't care if there could be many ways to do it. I can see that there are some designs where you need to change or extend the underlying COAP protocol and there are also designs where it is layered on top with no change to COAP. I don't care too much which of the two it is as in the end that is really just moving where the bits are in the packet which I could care less about. 

>  It sounded like you felt a reference document describing alternative approaches that applications might take would not be adequate.  

sorry if I gave that impression - I'd love to see a document like that - it would help understand the design space. I am more concerned about having less than 1 solution than the problem of having more than 1 solution. I do not think the document that describes a solution has to be the same RFC as the base spec COAP spec and devices could choose if they were just doing the base spec or if they implemented base spec + an observation solution. 

> Could you elaborate a little more here on what you believe CoAP must provide, and in what timeframe?

I don't care if it is in COAP, an extortion to COAP, or just a way of using COAP but I would like to see the WG produce a way of having a way for Application A to to say it wants to know about changes to resource B and when B changes, a message gets sent to A saying it changed. It's not really very complicated what I want. As my friend Dean says, there's many ways to skin this cat, I rather not see us use a potato peeler. Oh, and let me be clear on one more thing about it - I do not care in the slightest if anyone thinks it is REST or not, I just want my message when the resource changes and something that is reasonable to implement. I've got a pretty deep understanding of REST and I don't see how the concepts of REST actually help much with the conversation here. I do think discussion of what devices need to store state, for how long, and how much do help with the design as well as discussion of what is idempotent. The security and flexibility implications of various solutions are also interesting to consider along with the obvious issues of size, complexity, and efficiency. 

> I understood you to say that some IETF working group was nearly complete with a proposed standard way to implement observation in HTTP.  Is that the Httpbis group?  To which of their documents were…

HyBy

http://tools.ietf.org/wg/hybi/draft-ietf-hybi-thewebsocketprotocol/

> The clearly relevant one is draft-roach-sip-http-subscribe, but that's based on SIP event notification.  Were we to take that approach, it might be best to define a CoAP-like protocol "csip", rathe…

I helped design the SIP Sub/Notify system and it's design goals were very different from what we are trying to do here. At a high level it is isomorphic to what we want but as you did into the details, it was optimized for something fairly different than the CoAP use cases. I'm certainly not pushing the approach that was used in SIP thought I tend to fall into using the terminology from SIP just because I know it well.

---

From: Adam Roach <[email-masked]>
Date: Fri, 29 Oct 2010 16:56:53 -0500

> I helped design the SIP Sub/Notify system and it's design goals were very different from what we are trying to do here. At a high level it is isomorphic to what we want but as you did into the deta…

Oh, yeah, wow. You don't want to use SIP for this, unless you change 
what the "Co" stands for in "CoAP." You don't really even want to try to 
mirror its semantics into a CoAP-related protocol. One of the things I'm 
finding refreshing about CoAP at this stage of its life is that no one 
has started adding in giant arrays of nuclear-powered kitchen sinks. At 
least, not yet.

What we've done with SIP events is arbitrarily flexible, which is why 
it's used for everything from presence to dialog state monitoring, user 
agent configuration, and even certificate management. We've even seen 
proposals to use it to monitor automotive data in real-time. Really.

You don't need this.

Now, what's kind of nice about RFC 5989 (draft-roach-sip-http-subscribe 
was published as an RFC last night) is that it's a very simple use of 
SIP events. Just about any explicit observation model you come up with 
for CoAP should be able to interwork with RFC 5989 (at least, as long as 
you're using CoAP servers and HTTP clients -- I wouldn't want to try to 
make it go in the other direction, but I don't think that's on the table).

I really wouldn't focus on the semantics of the SIP events mechanism 
when you're designing CoAP observations, anyway. There are other ways to 
monitor for changes to a resource, such as PubSubHubbub, that are 
designed for certain niches.

In terms of a model that is simple, deployed, and works over UDP 
(well... and proprietary), you might look here for design inspiration: 
http://msdn.microsoft.com/en-us/library/aa143117%28EXCHG.65%29.aspx

/a

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 30 Oct 2010 08:20:25 +0200

On Oct 29, 2010, at 22:48, Cullen Jennings wrote:

> I do not care in the slightest if anyone thinks it is REST or not, I just want my message when the resource changes and something that is reasonable to implement. I've got a pretty deep understanding of REST and I don't see how the concepts of REST actually help much with the conversation here.

What could be RESTful about observation?  Indeed, we won't get that much direct guidance from the REST scriptures here..

I think the obvious (and interesting) property we want to achieve is that a proxy should be able to fan out observation notifications, i.e., if three clients want to observe the same resource through the same proxy, the proxy can make do with observing it once.  This property should be part of the architecture, not an application-specific hack.

(Why is that property interesting?  Because it allows constrained nodes to minimize their direct communication.)

Gruesse, Carsten

---

From: Peter Bigot <[email-masked]>
Date: Sat, 30 Oct 2010 07:44:27 -0500

On Sat, Oct 30, 2010 at 1:20 AM, Carsten Bormann <[email-masked]> wrote:

> > I do not care in the slightest if anyone thinks it is REST or not, I just
> want my message when the resource changes and something that is reasonable
> to implement. I've got a pretty deep understanding of REST and I don't see
> how the concepts of REST actually help much with the conversation here.
>
> What could be RESTful about observation?  Indeed, we won't get that much
> direct guidance from the REST scriptures here..
>

Much could be RESTful about observation.  It sounds like there's little
interest in pursuing the issue from that perspective, though.

The meta-concept behind REST is that having a well-defined set of
constraints to guide the architectural decisions helps ensure a scalable and
successful system.  Up to now, we've been using REST as the constraints to
help assess alternative approaches in CoAP.  What constraints are we using
for this problem?

I think the obvious (and interesting) property we want to achieve is that a
> proxy should be able to fan out observation notifications, i.e., if three
> clients want to observe the same resource through the same proxy, the proxy
> can make do with observing it once.  This property should be part of the
> architecture, not an application-specific hack.
>

Proposal: A proxy, as you are using the term, is an application.

Consequently, its behavior should be defined in a separate document from the
one that specifies the CoAP protocol: data encoding, supported metadata
(options), and transaction state machines.

(Why is that property interesting?  Because it allows constrained nodes to
> minimize their direct communication.)
>

Yes.  Observation in a constrained environment really involves four distinct
roles, not two.  Whether some of those roles are implemented on the same
end-point is necessarily going to be a system-specific decision.

And I'm really not looking forward to trying to explain over email why that
is so.

Peter