Subject: [core] CoAP and HTTP - Philosophy and Direction

From: Brian Frank <[email-masked]>
Date: Sat, 4 Dec 2010 15:03:40 -0500

Sometimes I think it is useful to just take a step back and reevaluate
exactly what we are trying to accomplish.

Personally, I got involved in the CoAP effort for one primary reason: I
believe that every piece of information associated with the Internet of
Things should have a URI.  Not just any URI, but an "http" URI - so that
smart devices could integrate with the web at large.  So for me the whole
idea of CoAP is not really that desirable, rather it is a necessary evil.
 It is necessitated by the fact that HTTP doesn't run
over constrained networks and in constrained devices:

  - number of bytes on the wire matters greatly (http text headers too
verbose)

  - network bandwidth, network characteristics, and device processing/memory
make TCP hard

  - sleeping nodes are big pain

I realize that not everyone necessarily shares my philosophy, but to
me personally the whole point of CoAP is to seamless integrate to HTTP so
that the Internet of Things can integrate into the web at large.

If you share this philosophy, then probably one of the most critical aspects
of CoAP is the CoaP/HTTP mapping Section 7.  Up to this point we (me
included) have sort of been treating this section as the thing to do last
after all the important stuff has been done.  But after reviewing a lot of
the tickets and outstanding issues, I think we have been putting the cart
before the horse so to speak.  HTTP is the end-goal, CoAP is just the means
to the end.  I personally would like to see every feature of CoAP evaluated
in the context of its HTTP mapping.

Just to review a couple of the biggies:

Proxy Terminology (#72)
==================
This is what started me on this whole thought process.  The more I started
thinking about terminology, the more I started thinking that the most
important proxy/gateway is the HTTP-to-CoAP functionality.  And I really
didn't see this discussed on the mailing list (perhaps I missed it though).
The key idea is that "proxy" could mean different things:

  - there is the actual IP address where you direct requests to (explicit
proxies, interception proxies, reverse proxies etc)

  - there is caching semantics (which is both a proxy issue *and* a
end-client issue)

  - there is actual protocol translations HTTP->CoAP and CoAP->HTTP (and
potentially others like XMPP, etc)

  - there are potentially MIME content translations (for example XML <->
compress XML/binary as it goes thru HTTP <-> CoAP)

I am not sure we can really tackle proxy terminology without tackling some
of the latter issues which haven't really received much discussion.

Caching Semantics (#78)
===================
It may just be a process/document structure issue.  But I suspect that the
technique we use to tackle section 7 and how we cross reference RFC 2616
will actually dictate how we should fix caching to do more cross reference
of the HTTP spec itself.

Deferred Requests
==============
I understand how we got here.  But it doesn't really fit the simple HTTP
req/res model.  So I think this discussion has to include the HTTP side of
things.

Pub/Sub Observations
=================
I know there are application tricks to do this with HTTP, but how will we
map it?  I personally am finding this requirement is creating a lot of
complexity.  Furthermore based on my experience, the state management of the
current design is too complicated and has too much overhead to implement in
constrained devices.  I just want to raise the flag that I worry about this
one.  I'd love to see it, but I am more than happy to live with a solid,
simple req/res model that works seamlessly with HTTP.

A couple key take away points:

  - are we actually all on the same page that seamless HTTP integration is a
critical aspect of this effort?

  - assuming yes to the above, then should we make HTTP mapping the top
priority and see how everything else fits into that?

Just my random thoughts...
Brian

---

From: Peter Bigot <[email-masked]>
Date: Sun, 5 Dec 2010 11:49:28 -0600

I share your desire to have all resources associated with a URI, but not the
further step that assumes all those URIs should be accessible by HTTP.  I
think the scheme portion of the URI serves a critical role in allowing
domain-specific protocols to be used where appropriate.  A specific example
will be for resources involved in group communications, where the
client/server point-to-point model of HTTP is simply a non-starter.

Beyond that, I agree with your assessment that CoAP's integration with HTTP
is far too complex to bolt on as an afterthought.  I had thought to use
REST, rather than simply HTTP, as the primary architectural model for CoAP,
but found there to be insufficient agreement on what that meant or the
degree to which it should drive the solution.  (With that approach, it would
be a matter of expressing REST operations in HTTP and vice-versa, a
technique that's reasonably well understood.)

I think the group had already determined that CoAP was not going to be a
transliteration of HTTP.  Consequently, I believe HTTP/CoAP integration can
be made seamless but that the effort involved is really a separate
initiative will include things like how to translate group operations into
something meaningfully expressible by HTTP operations.  This will
necessarily involve transformation, aggregation, and separation of the
resource representations, an issue that is currently completely unaddressed.

Peter

On Sat, Dec 4, 2010 at 2:03 PM, Brian Frank <[email-masked]> wrote:

>   - network bandwidth, network characteristics, and device
> processing/memory make TCP hard
>
> If you share this philosophy, then probably one of the most critical
> aspects of CoAP is the CoaP/HTTP mapping Section 7.  Up to this point we (me
>   - are we actually all on the same page that seamless HTTP integration is
> a critical aspect of this effort?
>
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>
>

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 6 Dec 2010 09:18:08 +0100

Thanks Brian for your message, I share completely your point(s).

To me also seems that CoAP design is too much oriented on the design
of a stand-alone application protocol, instead of focusing on REST and
HTTP mapping as a top priority.

Observe in my opinion has been designed as an efficient stand-alone
mechanism, leaving in the to do list the mapping to/from HTTP; this
should not be the design procedure and could lead to (big?) problems
when integrating with the "legacy" web.

CoAP priority should be the design of efficient *and* HTTP mappable
solutions; if the proposed solution maps hardly to HTTP, introduces
state (requiring a stateful entity to be correctly mapped) or can't
work in both directions CoAP->HTTP and HTTP->CoAP, then it should be
reconsidered.

Angelo

---

From: Yoav Nir <[email-masked]>
Date: Mon, 6 Dec 2010 12:36:23 +0200

I don't think the problem is in the design of Observe. The problem is with the requirement. In some cases, observers require immediate notification. Picture a light switch that has been flipped, with the light fixture observing. 

The best that HTTP can provide is either polling ("are you on now?  No. (time passes) Are you on now? No. (time passes) Are you on now? Yes.") or else long connections waiting for a response ("Are you being turned on? (time passes) Timeout. Are you being turned on? (less time passes) Yes."). Both are crude and inefficient bastardizations of HTTP.

I don't see a way to map that observer required behavior to HTTP.

Yoav

On Dec 6, 2010, at 10:18 AM, Angelo P. Castellani wrote:

> Scanned by Check Point Total Security Gateway.

---

From: "Clark, Gilbert" <[email-masked]>
Date: Mon, 6 Dec 2010 06:24:46 -0500

For what it's worth:

I agree that supporting some subset of HTTP is important, but I'd be wary of
designing CoAP around its HTTP support.  CoAP networks are likely to have
very different characteristics from networks running HTTP; CoAP should
therefore be primarily designed to operate best in the environment into
which it's being deployed.

Also, I see the state required by OBSERVE as being part of an HTTP server's
cache.  In this case, though, instead of caching content, the server is
caching a GET and passing back data to tell the client what the GET is
called so it knows when it gets a response; the server has the option of
telling the client it won't do so, so no state is actually *forced* onto the
server; instead, the server is choosing to cache the request with the
understanding that supporting 10K of cached GET requests could go a long way
to minimizing power / CPU utilization in the long run.

Additionally, HTTP assumes that the server involved in a connection will be
up the majority of the time.  I'd imagine that, for many constrained
devices, the opposite would be true: they'd be sleeping for the vast
majority of their lifetime, which means that timing a poll to hit one of the
brief periods the device was awake would be difficult at best.  An
alternative to this approach might be forcing the device to wake every time
it was polled, which seems like it might be pretty bad for networks where
power and / or network utilization was a constraint.

I agree that there's probably not a good way to make OBSERVE visible to an
HTTP client; at the same time, I also feel like a 1:1 mapping should not be
required as long as the CoAP <--> HTTP translator is able to support a
viable subset of HTTP.  For example, in the case of OBSERVE, the CoAP <-->
HTTP translator could actually fulfill HTTP GET requests by OBSERVE-ing the
resource specified in the GET, and caching content between updates.  This
would minimize load on the CoAP network while maintaining HTTP
compatibility, and HTTP wouldn't necessarily have to know the translator was
cheating and using OBSERVE.

/$0.02

--Gilbert Clark

---

From: Yoav Nir <[email-masked]>
Date: Mon, 6 Dec 2010 14:59:15 +0200

I think that would work well. I don't know if the caching proxy defined in section 5 of CoAP does this, but the proxy can do the OBSERVE with the actual server, while the HTTP client can poll the proxy. This is not a 1:1 mapping, but should be close enough. The lamp needs immediate access to the light switch, so it should use OBSERVE, but a control station that polls the light switches in the building at 10 PM to see which one is still on can rely on HTTP.

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 6 Dec 2010 18:07:54 +0100

Yoav,
in my opinion there at least two ways to cope better with HTTP, the
first is more simple but has minor pros and some cons, the second more
ambitious but can lead to some bigger advantages.

a aka simple alternative) support HTTP streaming introducing CoAP
chunked transfer encoding

HTTP supports chunked Transfer-Encoding, through this feature is
possible to push data to clients.

This mechanism can be ported to CoAP and can be mapped directly to
HTTP by an in-the-middle "proxy".

Status associated with every notification, which is useful and present
in the current design, has to be dropped in HTTP and/or can be present
only in CoAP<->CoAP communication.

This alternative however doesn't introduce significant advantages over
the observe one. Unconvenient state is still present: persistent tcp
connections and soft-state during streaming. Moreover could be harder
to define how to cache data (single chunks can be cached?).

b aka hard/ambitious alternative) only servers can support observations

In M2M environments, interactions are expected to be using a
peer-to-peer communication fashion.

For this reason to realize the subscription functionality, the
subscription can be defined as an interaction between resources (so
between servers).

Every message should be defined in terms of source and destination
URI, so that resource A requests resource B to notify its value (on
change, every X secs, etc.).

Moreover notifications can also carry the status code associated with
that particular notification.

This kind of mechanism requires the definition of a header in HTTP,
however CoAP observe itself to be mapped to HTTP probably requires a
Subscription-lifetime header as well.

Main advantage is that with this kind of solution, proxy implementors
can choose whether to support natively CoAP observe or not, and they
can even map it to a long polling or HTTP streaming session.

But also is possible to pass-through the proxy without requiring the
establishment of state inside it. Moreover proxies can implement
policies to redirect incoming subscriptions to a single local
proxy-owned subscription.

However, as said before, this technique doesn't specify an obliged
solution but supports flexibly more message exchanges and so system
architectures.

CoAP header compression: using Token option
Caching: still possible through request caching
Security: to be evaluated (some effort may be needed)

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 6 Dec 2010 18:17:09 +0100

On Mon, Dec 6, 2010 at 12:24, Clark, Gilbert <[email-masked]> wrote:
> cache.

Caching the state can be hardly implementable, because that state is
required to match responses so has to be promptly available to the
proxy process whether usually caching mechanisms rely on a more looser
storage support that could require higher access time..

Moreover the bigger state to be mantained could be *one TCP
connection* open for *every subscriber*, this is required to push the
notification to the client.

> power and / or network utilization was a constraint.

This can be obtained also with an alternative, subscription itself is
an high-level solution to this.. I don't think we should not support
subscription.

> viable subset of HTTP.

In my opinion the problem we face is not really about finding a way to
map, but about finding a *simple* and *scalable* way to map CoAP to
HTTP.

> For example, in the case of OBSERVE, the CoAP <-->

Sure, I agree that the subscription technique proposed should
obviously support caching in some way.

Regards,
Angelo

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 6 Dec 2010 12:37:45 -0500

> The best that HTTP can provide is either polling ("are you on now?  No. (time passes) Are you on now? No. (time passes) Are you on now? Yes.") or else long connections waiting for a response ("Are…

---------------------------------------------
I agree that HTTP is the wrong protocol for group operations and
status information.

What about using XMPP (IM chat) instead? The XMPP protocol already
supports concepts like presence, status, group and individual
conversations, etc. A fixed function chat client (the light switch)
does not take a lot of resources.  The chat server could run on a wifi
router (multiple free implementations exist) and handle proxies for
sleeping devices.

XMPP also has a serverless mode which should be adaptable to directly
use mutlticast LANs packets.
http://xmpp.org/extensions/xep-0174.html

For direct browser access to the devices, use http or run a Javascript
chat client in the browser.

-- 
Jon Smirl
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 Dec 2010 18:41:39 +0100

One very useful rule that this WG has been following so far for the design of CoAP is: 

	When in doubt, do it like HTTP.

There is absolutely no doubt here, though.  There is nothing we can follow.

On Dec 6, 2010, at 18:07, Angelo P. Castellani wrote:

> HTTP supports chunked Transfer-Encoding,

Yes.

> through this feature is

(If you mean "push" in the sense of deferred response:)

No.

Yes, I'm aware of "forever-frames" and all those wonderful asynchronous AJAX hacks.
(I built a little toy based on long-polls in 2005; http://tzi.org:2000/s6.html if you are interested).
These hacks are not a part of HTTP, they are abusing HTTP, and they work only because the idiosyncrasies of certain specific implementations happen to line up just right.

I'll leave the last word to Brian Carpenter, who prefixed his review of draft-loreto-http-bidirectional-05.txt with:

> This is real example of protocol abuse. HTTP wasn't designed for this and doesn't do this properly. 

Gruesse, CArsten

---

From: "Clark, Gilbert" <[email-masked]>
Date: Mon, 6 Dec 2010 15:14:06 -0500

Hi:

Responses inline.

> storage support that could require higher access time..

Couldn't you generate the value for the token option based on a hashed
combination of the resource URI and the subscriber, eliminating the need for
additional state (at the cost of additional CPU)?  I'm behind on list
traffic, so I'm not sure how those are generated at the moment, but in
general I believe it might be possible for a token to be implicitly defined.

Wouldn't the state be required only to match requests, and not responses?

Also, I'm not sure I understand why state would need to be promptly
available; you could sacrifice update granularity to allow for deferred
processing, at the cost of storing a token option / queued update.

> notification to the client.

Correction: I was intending to argue that the state required by OBSERVE
could arguably be considered to be part of a server's cache (and therefore
could be included as part of a REST architecture).  I didn't mean to say
that this would be a good thing to try to implement within HTTP.

--

> map, but about finding a *simple* and *scalable* way to map CoAP to

One straightforward approach to scaling a HTTP <--> CoAP proxy as described
in my original e-mail would be to make each CoAP node a subdomain (e.g.
node1.proxy.coapiscool.net), and implement the DNS lookup service such that
it will return the appropriate HTTP <--> CoAP proxy for a given node.

In any case, my issue with stating the problem as "finding a simple and
scalable way to map CoAP to HTTP" is that it's ambiguous.  What parts of
HTTP and CoAP should be interoperable?  Why?  What is "simple"?  What is
"scalable"?   I vote we come up with a concrete problem statement and then
try to argue about its implementation; it could be that we're all in violent
agreement and we just don't know it yet.

> obviously support caching in some way.

s/should/must/

--Gilbert Clark

---

From: Paul Duffy <[email-masked]>
Date: Mon, 06 Dec 2010 19:57:27 -0500

Strong agreement with Brian's and Angelo's points.

COAP <--> HTTP mapping needs to be as simple and stateless as possible.  
Else you run the risk of islanding COAP from the rest of the Web.

I cringe, however,  when I see existing Web referred to as "legacy".  
How about "ubiquitously deployed and in no danger of going away anytime 
soon"?

Cheers

On 12/6/2010 3:18 AM, Angelo P. Castellani wrote:
> On Sat, Dec 4, 2010 at 21:03, Brian Frank<[email-masked]>  wrote:
>>    - there are potentially MIME content translations (for example XML<->
>> compress XML/binary as it goes thru HTTP<->  CoAP)

---

From: Zach Shelby <[email-masked]>
Date: Tue, 7 Dec 2010 11:03:11 +0200

Gilbert,

This was exactly how I envision observe being used. The philosophy is that observe is a mechanism a CoAP-HTTP caching proxy can use to update its cache. HTTP clients can request resources using polling, long polling or with some other mechanism. To an HTTP client observe is not visible, in other words, it does not require a mapping. It is up to the proxy how it wants to decide when observation should be used. We have a ticket already to add examples of this to the next version of core-observe. 

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Zach Shelby <[email-masked]>
Date: Tue, 7 Dec 2010 11:34:53 +0200

On Dec 4, 2010, at 10:03 PM, Brian Frank wrote:

> Sometimes I think it is useful to just take a step back and reevaluate exactly what we are trying to accomplish.

Good idea.

> 
> Personally, I got involved in the CoAP effort for one primary reason: I believe that every piece of information associated with the Internet of Things should have a URI.  Not just any URI, but an "http" URI - so that smart devices could integrate with the web at large.  So for me the whole idea of CoAP is not really that desirable, rather it is a necessary evil.  It is necessitated by the fact that HTTP doesn't run over constrained networks and in constrained devices:
> 
>   - number of bytes on the wire matters greatly (http text headers too verbose)
> 
>   - network bandwidth, network characteristics, and device processing/memory make TCP hard
> 
> I realize that not everyone necessarily shares my philosophy, but to me personally the whole point of CoAP is to seamless integrate to HTTP so that the Internet of Things can integrate into the web at large.  

That may be your main use case for HTTP, but it clearly is just one use case for the WG. I would say a simple HTTP mapping is an important aspect of CoAP, but fulfilling the other M2M application requirements are just as important. Many of the application people will be using this for in the M2M industry won't require HTTP mapping at all. For the long-term evolution of the Internet of Things, I agree that HTTP mapping is important. 

> If you share this philosophy, then probably one of the most critical aspects of CoAP is the CoaP/HTTP mapping Section 7.  Up to this point we (me included) have sort of been treating this section as the thing to do last after all the important stuff has been done.  But after reviewing a lot of the tickets and outstanding issues, I think we have been putting the cart before the horse so to speak.  HTTP is the end-goal, CoAP is just the means to the end.  I personally would like to see every feature of CoAP evaluated in the context of its HTTP mapping.

We haven't forgotten about HTTP mapping by any means, Section 7 is important to the document. On the list I think everyone has been assuming HTTP mapping as an underlying requirement, thus it is not brought up in every post. Not every feature of CoAP affects HTTP mapping, for example the transaction model of CoAP is invisible to HTTP. 

Although there is no ticket for it (I will make one now), Klaus has done some work on improvements for the HTTP mapping section and we plan on having a better table there e.g. for response code mappings in both directions. Is there something else in particular missing from the HTTP mapping section?    

> This is what started me on this whole thought process.  The more I started thinking about terminology, the more I started thinking that the most important proxy/gateway is the HTTP-to-CoAP functionality.  And I really didn't see this discussed on the mailing list (perhaps I missed it though). The key idea is that "proxy" could mean different things:
> 
>   - there is the actual IP address where you direct requests to (explicit proxies, interception proxies, reverse proxies etc)
> 
>   - there is caching semantics (which is both a proxy issue *and* a end-client issue)
> 
>   - there is actual protocol translations HTTP->CoAP and CoAP->HTTP (and potentially others like XMPP, etc)
> 
>   - there are potentially MIME content translations (for example XML <-> compress XML/binary as it goes thru HTTP <-> CoAP)
> 
> I am not sure we can really tackle proxy terminology without tackling some of the latter issues which haven't really received much discussion.

I think we can tackle these in the text. Already we made the decision to separate the caching semantics from the proxy function. Reverse proxies are really out of scope, although we do plan on mentioning them. HTTP translation is already explained in section 7. Although MIME translations are out of scope for CoAP, I don't think it would hurt to mention that this is something a proxy may want to do.  

> 
> Caching Semantics (#78)
> ===================
> It may just be a process/document structure issue.  But I suspect that the technique we use to tackle section 7 and how we cross reference RFC 2616 will actually dictate how we should fix caching to do more cross reference of the HTTP spec itself.

Exactly.

> 
> Deferred Requests
> ==============
> I understand how we got here.  But it doesn't really fit the simple HTTP req/res model.  So I think this discussion has to include the HTTP side of things.

This is a transaction layer issue, and is invisible to request/response and HTTP. 

> 
> Pub/Sub Observations
> =================
> I know there are application tricks to do this with HTTP, but how will we map it?  I personally am finding this requirement is creating a lot of complexity.  Furthermore based on my experience, the state management of the current design is too complicated and has too much overhead to implement in constrained devices.  I just want to raise the flag that I worry about this one.  I'd love to see it, but I am more than happy to live with a solid, simple req/res model that works seamlessly with HTTP.

There were already some good comments on this, but just to summarize. The main point is that core-observe does not require mapping. Instead it is just a cache refresh mechanism to an HTTP proxy. I envision that more explicit REST pub/sub frameworks would be used end-to-end if you want more sophisticated features. core-observe is not meant to compete with or replace those frameworks, they have different goals. We have APs to both add good examples and discuss how observe differs from other frameworks in the next version of that document.

>   - are we actually all on the same page that seamless HTTP integration is a critical aspect of this effort?

Everyone agrees that HTTP mapping is an important requirement.

>   - assuming yes to the above, then should we make HTTP mapping the top priority and see how everything else fits into that?

I disagree that it is the top priority. But I don't think that any of our current design issues are at odds here. There really isn't anything in CoAP that makes HTTP mapping particularly difficult, we did a lot of work to make things that don't work in the HTTP world invisible to it. Mostly I think some more editorial work is needed in the next versions of core-coap and core-observe to portrait that clearly.

Zach