Subject: [core] Next/Range Options (re: draft-ietf-core-block-00)

From: Robert Quattlebaum <[email-masked]>
Date: Thu, 21 Oct 2010 10:41:15 -0700

Below is an informal document I prepared as I was fleshing out the details of how exactly a Next option would work. My goal was to develop a chunking mechanism that had all of the benefits of the Block option without the serious drawbacks when serving dynamically generated content.

I am providing it here to add to the discussion about draft-ietf-core-block-00. Ultimately I would like to see draft-ietf-core-block adopt some or all of what I describe below because I believe that it is a more flexible mechanism and makes the minimum implementation easier to implement.

Feedback is welcome.

--------------------------------------

# Content Chunking and Random Access Header Options #

This informal document outlines a mechanism of chunking large CoAP content sizes into smaller more convenient sizes using an alternative method than described in draft-ietf-core-block-00.

## Header Options ##

This chunking mechanism introduces two new header options: Range and Next.

The "Next" option specifies an opaque server-defined token which is used for retrieving the next chunk of content from the server when the content was too large to fit in a previous request. It is a critical option.

Range specifies two integers: "max-requested-payload-size" and "requested-content-offset". It is an elective header, and not required to be handled or implemented by the server. The offset may be omitted, and if so is assumed to be zero. "max-requested-payload-size" is largely a suggestion that the server may or may not obey, depending on implementation constraints. (In the examples below, the value of the range option is represented by two numbers: the "max-requested-payload-size" and "requested-content-offset", respectively)

### Range Option Format ###

The exact wire format of the Range option is TBD, but the following description may be used as a starting point.

The contents of the Range header may contain one or two values. The first value is "max-requested-payload-size", and is bound to the range of [0,16383]. The second (optional) value, "requested-content-offset", is unbounded. Each value is encoded using the method described in <http://www.w3.org/TR/exi/#encodingUnsignedInteger>.

### Why not use the Block option? ###

While the Block option is functionally similar to the Range option, the Block option has significant differences: 1) It is critical instead of elective, 2) it is much more coarse in terms of block sizes and offsets, and 3) it includes a mechanism for indicating the end of content.

It is important that the Range option be elective because the Range option can be used to indicate a "max-requested-payload-size" for chunked transfers. Since the "max-requested-payload-size" is simply a recommendation for the server and the default "requested-content-offset" is zero even without the Range header, the response will always be valid (but perhaps not polite, in the sense that the server may ignore the "max-requested-payload-size"). In cases where the "requested-content-offset" is non-zero, the client can easily determine from the response if the server supported the option or not from the response code: a server which supports the header would ALWAYS return 206 (Partial Content) to such a request, and a server which doesn't would return 200 (OK).

The coarse nature of the Block option appears to make it somewhat incompatible with varying payload sizes. This is most clearly evident in the PUT and POST cases, described in a section below.

The mechanism for indicating the end of content is not necessary in the GET request case, as this condition is indicated by the absence of the Next header. For PUT and POST requests (as described in a section below), the content is terminated by the first empty payload. 

## GET Requests ##

If at any time the server determines that the payload of a response is a subset of the entire content represented by the requested resource, it MUST reply with a 206 (Partial Content) response code. (A corollary of this is that if the Range header is present with a non-zero "requested-content-offset" that the correct positive response code will always be 206)

If there is additional content after what is included in the payload of the response, the server MUST include a Next option in the response containing an opaque token which the client would use to fetch the rest of the content in a subsequent request. If the payload in the response represents the end of the content, then a Next option MUST NOT be included in the response. The absence of a Next header in the response indicates the end of the content.

While the client MAY include a Range option for an initial request, subsequent requests to retrieve the rest of the content (which include a Next option) SHOULD NOT include a Range option. If the Range option is included, it MUST contain the exact same value as the Range option from the original request. The results of doing otherwise are undefined.

The client can influence the payload chunking threshold of the server by including a Range option in the initial request. In the presence of the Range option in the initial request, the server SHOULD make every attempt to keep payload sizes below the "max-requested-payload-size" specified by the option.

The presence of Range header in a 206 response from a server is optional, except in the case of an asynchronous 206 response---where it is required for correlation purposes.

Using the Range header with a non-zero "requested-content-offset" requires random-access support, which is OPTIONAL for the server to implement. In cases where clients make a request for a non-zero "requested-content-offset" to a server which does not support random access, the server MUST NOT respond with a 206 (Partial Content) response and SHOULD respond with a 501 (Not Implemented). A server response of 200 (OK) to such a request would indicate to the client that the Range option is not supported by the server.

While it would be possible for a client to implement a chunked download ignoring the Next header and using only the Range header, such an implementation is strongly NOT RECOMMENDED because it would only be successful with servers which support random access.

### Example 1 ###

Client is requesting example.txt, which is 192 bytes long. The internal server chunking threshold is 64 bytes. 

Client -> Server, TT=CONF, TID=0001
	GET /example.txt

Server -> Client, TT=ACK, TID=0001
	206 Partial Content
	Next: 0x01
	<bytes 0-63>

Client -> Server, TT=CONF, TID=0002
	GET /example.txt
	Next: 0x01

Server -> Client, TT=ACK, TID=0002
	206 Partial Content
	Next: 0x02
	<bytes 64-127>

Client -> Server, TT=CONF, TID=0003
	GET /example.txt
	Next: 0x02

Server -> Client, TT=ACK, TID=0003
	206 Partial Content
	<bytes 128-191>

### Example 2 ###

Client is requesting example.txt, which is 192 bytes long. The client requests a chunk size of 32 bytes. Note that while this example includes the value of "max-requested-payload-size" in the Next option, the value of the Next option is implementation-specific to the server and may contain any value or no value.

Client -> Server, TT=CONF, TID=0001
	GET /example.txt
	Range: 32 0

Server -> Client, TT=ACK, TID=0001
	206 Partial Content
	Next: 0x01 0x20
	<bytes 0-31>

Client -> Server, TT=CONF, TID=0002
	GET /example.txt
	Next: 0x01 0x20

... snip ...

Server -> Client, TT=ACK, TID=0005
	206 Partial Content
	Next: 0x05 0x20
	<bytes 128-159>

Client -> Server, TT=CONF, TID=0006
	GET /example.txt
	Next: 0x05 0x20

Server -> Client, TT=ACK, TID=0006
	206 Partial Content
	<bytes 160-191>

### Example 3 ###

Client is requesting example.txt, which is 128 bytes long. The server responds with an asynchronous response.

Client -> Server, TT=CONF, TID=0001
	GET /example.txt

Server -> Client, TT=ACK, TID=0001
	Code: 0

Server -> Client, TT=CONF, TID=3001
	206 Partial Content
	Next: 0x01
	Range: 64 0
	<bytes 0-63>

Client -> Server, TT=ACK, TID=3001
	Code: 0

Client -> Server, TT=CONF, TID=0002
	GET /example.txt
	Next: 0x01

Server -> Client, TT=ACK, TID=0002
	Code: 0

Server -> Client, TT=CONF, TID=3002
	206 Partial Content
	Range: 64 64
	<bytes 64-127>

Client -> Server, TT=ACK, TID=3002
	Code: 0

### Example 5 ###

Client is requesting example.csv, which is dynamically populated by the server with rows from a database in a CSV format. The full generated content looks like this:

id, name, min, max, value
1, sensor01, 0.0, 1.0, 0.28
2, temp, -40, 80, 0
3, nil1, 0, 0, 0
4, nil2, 0, 0, 0
5, pressure05, 0.0, 20.0, 5.82
6, sensor02, 0.0, 1.0, 0.95

The server sends responses with a max chunk threshold of 64. The payload size of the response varies from request-to-request as the rows are retrieved. 

Client -> Server, TT=CONF, TID=0001
	GET /example.csv

Server -> Client, TT=ACK, TID=0001
	206 Partial Content
	Content-Type: text/csv
	Etag: 1
	Next: 0x02
	"id, name, min, max, value"
	"1, sensor01, 0.0, 1.0, 0.28"

Client -> Server, TT=CONF, TID=0002
	GET /example.csv
	Next: 0x02

Server -> Client, TT=ACK, TID=0002
	206 Partial Content
	Content-Type: text/csv
	Etag: 1
	Next: 0x05
	"2, temp, -40, 80, 0"
	"3, nil1, 0, 0, 0"
	"4, nil2, 0, 0, 0"

Client -> Server, TT=CONF, TID=0003
	GET /example.csv
	Next: 0x05

Server -> Client, TT=ACK, TID=0003
	206 Partial Content
	Etag: 1
	Content-Type: text/csv
	"5, pressure05, 0.0, 20.0, 5.82"
	"6, sensor02, 0.0, 1.0, 0.95"

## POST and PUT requests ##

POST and PUT require a slightly different process.

In the case where the client wants to send a request with a larger amount of data than it is comfortable fitting into a single CoAP payload, the client would send only a subset of the data to the server, along with both a Range option, describing what slice of the data in question is being sent. [TBD: perhaps we should also require that the initial request include an empty Next option to force a failure if chunked transfers are not supported by the server]

The payload size of the requests which make up the transfer is client-defined, and may vary from request-to-request.

The use of the Next header for POST and PUT requests is ultimately determined by the server implementation and is entirely optional.

If successful, the server would then respond with a response code of 100 (Continue), indicating that the client should send the next slice. The response would include a Range option describing the range of the data that was just received, and possibly a Next option. If the server response includes a "Next" option, that option and value should be included in the next request by the client.

The end of the content is signified by a request with a zero-length payload size, at which point the server MUST NOT respond with a 100 (Continue) response code. The final response from the server MUST NOT include a Range or Next option.

### Example 5 ###

Client is sending example.txt, which is 128 bytes long. Chunk size is 64 bytes.

Client -> Server, TT=CONF, TID=0001
	POST /example.txt
	Range: 64 0
	<bytes 0-63>
	
Server -> Client, TT=ACK, TID=0001
	100 Continue
	Range: 64 0
	Next: 0x40

Client -> Server, TT=CONF, TID=0002
	POST /example.txt
	Range: 64 64
	Next: 0x40
	<bytes 64-127>

Server -> Client, TT=CONF, TID=0002
	100 Continue
	Range: 64 64
	Next: 0x80

Client -> Server, TT=CONF, TID=0003
	POST /example.txt
	Range: 0 128
	Next: 0x80

Server -> Client, TT=CONF, TID=0003
	201 Created
	Etag: 1

## Proxies and Caches ##

In general, a chunked transaction should be treated as a single REST transaction by caches and proxies. An exception is a CoAP-CoAP proxy, in which case no special considerations are required.

Caches which implement this mechanism MUST cache the entire content of a chunked request. Caches do not need to respect the chunk boundaries used for fetching the original content, and may choose chunk boundaries based on whatever criteria is most convenient.

__________________
Robert Quattlebaum
Jabber: [email-masked]
eMail:  [email-masked]
www:    http://www.deepdarc.com/

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 22 Oct 2010 00:21:58 +0200

Hi Robert,

thanks for making these thoughts available to the WG.

I don't want to do a detailed comment just yet, but I want to explain some of the thinking behind the block option and show how it is almost equivalent to yours.

The block option is a design that strives to be minimal while addressing the following requirements:
1) REST transfers can be chopped up into pieces without requiring a transport/transaction protocol that does segmentation/reassembly;
2) both sides get some say in how large these pieces are;
3) pieces are entirely self-describing, so there is never confusion about how to put them together again;
4) servers that have access to the entire binary representation can implement GET in a stateless fashion,
5) the encoding is extremely compact.

block does not try to address:
a) arbitrarily sized pieces
b) semantic fragmentation

It seemed easiest to achieve (5) and (4) in combination with (3) by using fixed, power-of-two-sized blocks (hence "block" option) together with an integer number pacing through these blocks.
The desire for self-describing packets led me to do the protocol progress by having the client perform arithmetic on the identifiers, which maybe is not the best way to do this.

An alternative would be to make slice identifiers opaque to the client.
A server could still use the compact representation used by current block if it wants to, or use some semantic mechanism.

Thought experiment: So how could "block" be changed to do this protocol?

-- the M bit would be deleted; the absence of a block option in a response (while a block option is present in the request) would indicate the last slice.
(This goes a bit against the self-describing aspect.)

-- in all but the first slice, the client would echo the block option it got from the server in the previous slice (i.e., instead of "I want block 1 now" it would say "I have block 0 now, give me the next").  The server would then increment the block number (or do whatever it needs to do).  This gives the server more freedom in the way it encloses the block size in the option value (if it wants to be stateless but still offer a slice size choice); the current encoding would simply become one of many implementation strategies.
(Again, the message is now less self-describing than block was.)

-- the slice size preferred by the client would be indicated outside the block option mechanism; we just had an exchange on that aspect.  (I'm not sure we want to conflate this with a range option, though.)

After this transformation, the block option is pretty much identical to your next option; the option value is now entirely opaque to the client.
Your proposal enhances this by modifying the response codes (which would probably need some more detail, as in contrast to HTTP we are not just using this for GET).

What do we gain?  A lot more flexibility.

What do we lose?  The simplicity we had because we didn't have the flexibility (only 1/2 :-) here).

Maybe, to facilitate debugging tools, the document could still describe an encoding of the opaque slice identifier that looks like the current block option (slice number + slice size, minus M bit) as a suggested format for those cases where it happens to fit.

Is this change worth it?  I'm not yet decided, but I like the fact that we have an existence proof now.

Gruesse, Carsten

PS.: Do we really want to have a range option?  I don't think so.  I think we'll need a really convincing usage scenario to make that case.

PPS.: An empty block option might be another way to indicate the last slice, so we don't feel a need to change response codes.

---

From: Robert Quattlebaum <[email-masked]>
Date: Thu, 21 Oct 2010 17:40:07 -0700

Hello Carsten!

> Thought experiment: So how could "block" be changed to do this protocol?

Ultimately that was the goal with my previous email, but for clarity sake I was referring to my proposal as the "Next" option. In reality the name of the options is immaterial. It could just as easily be named the block option, but I felt referring to it as such might be confusing.

> (This goes a bit against the self-describing aspect.)

As long as the response code of the last response is 206 (Partial Content) instead of 200 (OK), the absence of the M bit doesn't make the protocol any less self-describing. If the response code were always 200, then there would be no way for the client to differentiate between a fully contained response and the last response of a chunked request. (e.g., 206 responses are always chunked, and the last chunk is denoted by the absence of a block option)

> -- in all but the first slice, the client would echo the block option it got from the server in the previous slice (i.e., instead of "I want block 1 now" it would say "I have block 0 now, give me t…

Sounds great to me!

> (Again, the message is now less self-describing than block was.)

With the next option, all requests were self-describing as far as the server was concerned---but replies were not necessarily self-describing to the client. I anticipated that this wasn't a huge problem because if the client is keeping state then there is no longer a need for the response to be self-describing, as the TID would be adequate. (The TID is obviously not adequate in the case of an async response, which is why I required the range option to always be included for async responses)

However, if a stateless client really needs to the responses to be self-describing, there is already a facility to do so: the Token option. The client could simply use the Token option to store the number of bytes it has received so far in the transfer. This would make all responses that the client would receive self-describing, at least to the client.

> -- the slice size preferred by the client would be indicated outside the block option mechanism; we just had an exchange on that aspect.  (I'm not sure we want to conflate this with a range option,…

I just picked the name "Range" for the option because it seemed to be so similar. We could easily use a different name to describe a suitably similar mechanism. Originally I was just going to re-use the block option because it seems to be encoding similar information, but I decided against this for the reasons described in my proposal.

> After this transformation, the block option is pretty much identical to your next option; the option value is now entirely opaque to the client.

Again, sounds great! While I think the other aspects of my proposal are still important, having this alone would be a huge win, IMHO. 

> Your proposal enhances this by modifying the response codes (which would probably need some more detail, as in contrast to HTTP we are not just using this for GET).

I added the use of the 206 (Partial Response) and 100 (Continue) response codes to avoid some ambiguities, as well as to encourage more "RESTful" thinking of these sorts of transactions. A 200 code implies (to me anyway) that the response of the entire REST transaction is self-contained, whereas the other response codes indicate an ongoing REST transaction. But that's just me.

> What do we lose?  The simplicity we had because we didn't have the flexibility (only 1/2 :-) here).

While the new mechanism would certainly be less strict (and more flexible), did we really loose any simplicity? It seems to me that doing this actually makes implementations *more* simple.

As you mentioned earlier, a server which implements the current mechanism could be very easily updated to support this new mechanism, likely by changing a single line. Zero additional complexity. The client would no longer be mutating the block option value before sending it back to the server, so this would make client implementations even more simple.

> Maybe, to facilitate debugging tools, the document could still describe an encoding of the opaque slice identifier that looks like the current block option (slice number + slice size, minus M bit)…

This is effectively what the Range option contained, but with higher resolution. 

> PS.: Do we really want to have a range option?  I don't think so.  I think we'll need a really convincing usage scenario to make that case.

As I said earlier, I named it "Range" because it seemed to be descriptive of what I was using it for, but the name isn't really important: what's important is what it does. Here was the goals I was trying to accomplish with the Range option:

	* Provide a way for the client to influence the chunking threshold that the server uses.
	* Provide a way for the client to associate an asynchronous response with the GET request.
	* Enable random access for GET requests, when supported by the server. (Kind of a bonus, as least as far as this spec is concerned)

In my proposal, the server would optionally include a range option in GET responses (except async responses, where they were required) for debugging purposes.

I used the option heavily in the PUT and POST cases, but this could be easily avoided by only using the "opaque" block option---since that alone would allow the server to keep track of the transfer in a stateless fashion and the client is most likely stateful anyway. If the client really does want to (somehow) be stateless in such a case, it can preserve its state using the Token option.

> PPS.: An empty block option might be another way to indicate the last slice, so we don't feel a need to change response codes.

Why avoid using more descriptive response codes, especially when they are codes which are already used for the same thing in HTTP?

__________________
Robert Quattlebaum

---

From: Peter Bigot <[email-masked]>
Date: Fri, 22 Oct 2010 04:54:39 -0500

How about the following to reconcile all this and restore both simplicity
and orthogonality?  There are two independent proposals here (separated by a
key concept), and both can be extensions to core CoAP.

Size management:

- Add an elective option MessageSize, which when used in a request encodes a
one or two byte unsigned integer representing the maximum size of the CoAP
(non-error) response message that should be sent to the client.  Give it a
default value of 1152 or whatever.  Servers use this to determine whether
the resource response can be sent in one message or needs to be chunked.
This use of the option appears only in requests.

- Add an error code TooLarge sent by the server when the representation to
be sent to the client exceeds the size it has specified (or defaulted) and
the server declines (for whatever reason) to pre-emptively chunk it.  The
response message would use MessageSize to convey the total resource size to
the client.  In this use, the value of the option might be larger than two
bytes, if we feel it's reasonable for any CoAP resource to ever have a
representation larger than 65535 bytes.

This eliminates the question of how the server determines whether it should
use any multi-part response like the block option, and provides a facility
for influencing chunking on inherently multi-part responses like Robert's
use cases.

A Comment on Symmetry:

Unlike HTTP, CoAP can assume that each endpoint is capable of being both a
server and a client.  This means we have the option of asymmetry: in
particular if we have a mechanism for GET that allows multi-part
transactions for large resource representations, we do not need to specify a
related mechanism for PUT or POST if doing so is difficult.  What we do
instead is define a content type representing a URI, and do a PUT/POST where
the body is the URI that the server must in turn retrieve in order to get
the actual content of the PUT/POST.  It might do so by sending an ACK to
make the client's transaction asynchronous, retrieving the content from the
client, then executing the operation.

Chunked Retrievals:

A requested resource may be larger than the client has requested, or may
have a content-type that naturally supports chunking.  In either case, the
server may choose to send the resource representation in pieces.

It does so following essentially Robert's process.  A 206 Partial Content
response messsage is sent, including an opaque PartialContentId option.  The
client retrieves the next chunk by re-submitting the request but adding the
PartialContentId option provided in the last chunk.  The final chunk
includes a PartialContentId option of length zero.  All pieces use the same
response code (206).  The value of the resource is the byte-wise catenation
of the individual responses.

Comments:

- Functionally, PartialContentId is similar to HTTP's Content-Range, but
does not reveal information other than whether the piece is the last chunk
of the representation (is the option value length zero or not).  I don't
think CoAP needs a Range option or anything that involves random access
internal to a resource description.  If the resource has internal structure,
provide a RESTful interface to it, e.g. sub-resources identified by the
hierarchical path.

- By accepting the asymmetry of chunking only for responses to GET, we
always leave it up to the client whether it is willing to use a multi-part
transaction, and to determine how large the chunks are.  If we push chunks
at a server, we need a way for the server to say "Too big" or to try to get
the client to stop.   CoAP's client/server symmetry makes that unnecessary.

Peter

On Thu, Oct 21, 2010 at 7:40 PM, Robert Quattlebaum <[email-masked]>wrote:

> Ultimately that was the goal with my previous email, but for clarity sake I
> was referring to my proposal as the "Next" option. In reality the name of
> the options is immaterial. It could just as easily be named the block
> option, but I felt referring to it as such might be confusing.
>
> > -- the M bit would be deleted; the absence of a block option in a
> response (while a block option is present in the request) would indicate the
> last slice.
> As long as the response code of the last response is 206 (Partial Content)
> instead of 200 (OK), the absence of the M bit doesn't make the protocol any
> less self-describing. If the response code were always 200, then there would
> be no way for the client to differentiate between a fully contained response
> and the last response of a chunked request. (e.g., 206 responses are always
> chunked, and the last chunk is denoted by the absence of a block option)
>
> > -- in all but the first slice, the client would echo the block option it
> got from the server in the previous slice (i.e., instead of "I want block 1
> now" it would say "I have block 0 now, give me the next").  The server would
> then increment the block number (or do whatever it needs to do).  This gives
> the server more freedom in the way it encloses the block size in the option
> value (if it wants to be stateless but still offer a slice size choice); the
> current encoding would simply become one of many implementation strategies.
>
> Sounds great to me!
>
> With the next option, all requests were self-describing as far as the
> server was concerned---but replies were not necessarily self-describing to
> the client. I anticipated that this wasn't a huge problem because if the
> client is keeping state then there is no longer a need for the response to
> be self-describing, as the TID would be adequate. (The TID is obviously not
> adequate in the case of an async response, which is why I required the range
> option to always be included for async responses)
>
> However, if a stateless client really needs to the responses to be
> self-describing, there is already a facility to do so: the Token option. The
> client could simply use the Token option to store the number of bytes it has
> received so far in the transfer. This would make all responses that the
> client would receive self-describing, at least to the client.
>
> > -- the slice size preferred by the client would be indicated outside the
> block option mechanism; we just had an exchange on that aspect.  (I'm not
> sure we want to conflate this with a range option, though.)
>
> I just picked the name "Range" for the option because it seemed to be so
> similar. We could easily use a different name to describe a suitably similar
> mechanism. Originally I was just going to re-use the block option because it
> seems to be encoding similar information, but I decided against this for the
> reasons described in my proposal.
>
> > After this transformation, the block option is pretty much identical to
> your next option; the option value is now entirely opaque to the client.
>
> Again, sounds great! While I think the other aspects of my proposal are
> still important, having this alone would be a huge win, IMHO.
>
> > Your proposal enhances this by modifying the response codes (which would
> probably need some more detail, as in contrast to HTTP we are not just using
> this for GET).
>
> I added the use of the 206 (Partial Response) and 100 (Continue) response
> codes to avoid some ambiguities, as well as to encourage more "RESTful"
> thinking of these sorts of transactions. A 200 code implies (to me anyway)
> that the response of the entire REST transaction is self-contained, whereas
> the other response codes indicate an ongoing REST transaction. But that's
> just me.
>
> While the new mechanism would certainly be less strict (and more flexible),
> did we really loose any simplicity? It seems to me that doing this actually
> makes implementations *more* simple.
>
> As you mentioned earlier, a server which implements the current mechanism
> could be very easily updated to support this new mechanism, likely by
> changing a single line. Zero additional complexity. The client would no
> longer be mutating the block option value before sending it back to the
> server, so this would make client implementations even more simple.
>
> > Maybe, to facilitate debugging tools, the document could still describe
> an encoding of the opaque slice identifier that looks like the current block
> option (slice number + slice size, minus M bit) as a suggested format for
> those cases where it happens to fit.
>
> > PS.: Do we really want to have a range option?  I don't think so.  I
> think we'll need a really convincing usage scenario to make that case.
>
> As I said earlier, I named it "Range" because it seemed to be descriptive
> of what I was using it for, but the name isn't really important: what's
> important is what it does. Here was the goals I was trying to accomplish
> with the Range option:
>
>        * Enable random access for GET requests, when supported by the
> server. (Kind of a bonus, as least as far as this spec is concerned)
>
> In my proposal, the server would optionally include a range option in GET
> responses (except async responses, where they were required) for debugging
> purposes.
>
> I used the option heavily in the PUT and POST cases, but this could be
> easily avoided by only using the "opaque" block option---since that alone
> would allow the server to keep track of the transfer in a stateless fashion
> and the client is most likely stateful anyway. If the client really does
> want to (somehow) be stateless in such a case, it can preserve its state
> using the Token option.
>
> > PPS.: An empty block option might be another way to indicate the last
> slice, so we don't feel a need to change response codes.
>
> Why avoid using more descriptive response codes, especially when they are
> codes which are already used for the same thing in HTTP?
>
> __________________
> Robert Quattlebaum

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 22 Oct 2010 12:04:48 +0200

On Oct 22, 2010, at 11:54, Peter Bigot wrote:

> Unlike HTTP, CoAP can assume that each endpoint is capable of being both a server and a client.  

Is that so?
I don't know of any requirement of this sort so far.
(Maybe you mean something different with "client"/"server" here.)

Gruesse, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Fri, 22 Oct 2010 13:14:08 +0300

> (Maybe you mean something different with "client"/"server" here.)

It is often the case that a server will also include client functionality (if supporting asynchronous responses or subscription, they require that you can initiate transactions), but not the other way around. You very well may have client-only implementations. Considering that we are dealing with constrained devices, we can't assume or require that they will implement both.

Zach

> 
> Gruesse, Carsten
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Zach Shelby <[email-masked]>
Date: Fri, 22 Oct 2010 13:38:08 +0300

> 	* Provide a way for the client to associate an asynchronous response with the GET request.

This is what a Token is for already.

> 	* Enable random access for GET requests, when supported by the server. (Kind of a bonus, as least as far as this spec is concerned)

Now this is an interesting case. Do we want to enable random access to a resource? When does random access make sense? At first I was thinking that this could be used as a way to recover from the failure of GETs on some blocks if you are requesting multiple blocks in parallel. However, with the opaque model, you could still do this if the client would keep some state on the opaque tags for open requests. 

> In my proposal, the server would optionally include a range option in GET responses (except async responses, where they were required) for debugging purposes.

Token can be used for this?

> I used the option heavily in the PUT and POST cases, but this could be easily avoided by only using the "opaque" block option---since that alone would allow the server to keep track of the transfer…

Yep.

Zach

---

From: Peter Bigot <[email-masked]>
Date: Fri, 22 Oct 2010 05:52:01 -0500

I simply meant that a CoAP end-point at the network level uses the same
mechanism to send and receive messages that may be requests or responses.
Unlike HTTP, where clients do not natively provide a service entry point,
thus causing the millions (billions?) of dollars worth of pain associated
with trying to do pub/sub or callbacks over HTTP for the last dozen years.

Is it not a fair constraint to say that a client that doesn't support
minimal server functionality is unable to transmit large resource
descriptions?  I think that's all it would take.  Being a server need not
require supporting resource discovery; the code footprint should be little
more than is required to support asynchronous responses.

I see this as an opportunity to simplify CoAP by removing this rather
artificial asymmetry, but if y'all think it's important to keep it, so be
it.

BTW: Please don't lose the MaximumSize option proposal just because my
symmetry comment didn't fly.  I believe that has independent value, solving
several problems: how clients influence size, how servers determine the need
to chunk, and how a client can get an estimate of the resource
representation size without retrieving it (HEAD functionality).

Peter

On Fri, Oct 22, 2010 at 5:14 AM, Zach Shelby <[email-masked]> wrote:

> It is often the case that a server will also include client functionality
> (if supporting asynchronous responses or subscription, they require that you
> can initiate transactions), but not the other way around. You very well may
> have client-only implementations. Considering that we are dealing with
> constrained devices, we can't assume or require that they will implement
> both.
>
> Zach
>
> >
> > Gruesse, Carsten
> >

---

From: Zach Shelby <[email-masked]>
Date: Fri, 22 Oct 2010 13:58:49 +0300

Peter, 

Thanks for trying to reconcile these, comments in-line. You and Robert do realize you'll have to help us edit this document  :-) Starts to look like a re-write soon if we go in this direction...

On Oct 22, 2010, at 12:54 PM, Peter Bigot wrote:

> How about the following to reconcile all this and restore both simplicity and orthogonality?  There are two independent proposals here (separated by a key concept), and both can be extensions to core CoAP.
> 
> Size management:
> 
> - Add an elective option MessageSize, which when used in a request encodes a one or two byte unsigned integer representing the maximum size of the CoAP (non-error) response message that should be sent to the client.  Give it a default value of 1152 or whatever.  Servers use this to determine whether the resource response can be sent in one message or needs to be chunked.  This use of the option appears only in requests.

Would that eliminate the current text describing what the maximum size of a CoAP message is? In reality the default of MessageSize would serve that purpose.

> - Add an error code TooLarge sent by the server when the representation to be sent to the client exceeds the size it has specified (or defaulted) and the server declines (for whatever reason) to pre-emptively chunk it.  The response message would use MessageSize to convey the total resource size to the client.  In this use, the value of the option might be larger than two bytes, if we feel it's reasonable for any CoAP resource to ever have a representation larger than 65535 bytes.

Considering that many client's have limited memory, it would be useful for a client to know how large a resource is before it starts to retrieve it. MessageSize could be a variable length uint. 

> 
> This eliminates the question of how the server determines whether it should use any multi-part response like the block option, and provides a facility for influencing chunking on inherently multi-part responses like Robert's use cases.
> 
> A Comment on Symmetry:
> 
> Unlike HTTP, CoAP can assume that each endpoint is capable of being both a server and a client.  This means we have the option of asymmetry: in particular if we have a mechanism for GET that allows multi-part transactions for large resource representations, we do not need to specify a related mechanism for PUT or POST if doing so is difficult.  What we do instead is define a content type representing a URI, and do a PUT/POST where the body is the URI that the server must in turn retrieve in order to get the actual content of the PUT/POST.  It might do so by sending an ACK to make the client's transaction asynchronous, retrieving the content from the client, then executing the operation.

Although I don't agree that we can assume all endpoints can act as a server and client, I do find some logic here though. It might make sense to limit the use of chunked requests for GET (although we still need to look at how hard PUT/POST actually are). Just because we would do that though, doesn't mean we need to build in a callback URI feature. This kind of interface belongs to the REST interface design of the application, not for the transfer protocol. If chunked transfers only are supported for GET, then REST interfaces will naturally be designed in that way. We have to be careful about designing too much into the transfer protocol.

> Chunked Retrievals:
> 
> A requested resource may be larger than the client has requested, or may have a content-type that naturally supports chunking.  In either case, the server may choose to send the resource representation in pieces.

Considering that the chunked mode is optional, I would say the server MUST return the resource in one message if it is smaller than MessageSize. The WG was pretty clear in Maastricht that chunking should be an optional feature.

> It does so following essentially Robert's process.  A 206 Partial Content response messsage is sent, including an opaque PartialContentId option.  The client retrieves the next chunk by re-submitting the request but adding the PartialContentId option provided in the last chunk.  The final chunk includes a PartialContentId option of length zero.  All pieces use the same response code (206).  The value of the resource is the byte-wise catenation of the individual responses.

WFM

> 
> Comments:
> 
> - Functionally, PartialContentId is similar to HTTP's Content-Range, but does not reveal information other than whether the piece is the last chunk of the representation (is the option value length zero or not).  I don't think CoAP needs a Range option or anything that involves random access internal to a resource description.  If the resource has internal structure, provide a RESTful interface to it, e.g. sub-resources identified by the hierarchical path.

My only requirement for random access is to recover failed chunks. I think PartialContentId would be sufficient for doing that if the client keeps a little state on open requests (it has to anyways).

> - By accepting the asymmetry of chunking only for responses to GET, we always leave it up to the client whether it is willing to use a multi-part transaction, and to determine how large the chunks are.  If we push chunks at a server, we need a way for the server to say "Too big" or to try to get the client to stop.   CoAP's client/server symmetry makes that unnecessary.

We still need to think through the use of this with POST/PUT, although I do like the idea of keeping this as simple as possible.

Zach

> 
> Peter
> 
> On Thu, Oct 21, 2010 at 7:40 PM, Robert Quattlebaum <[email-masked]> wrote:
> Hello Carsten!
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 22 Oct 2010 14:01:14 +0200

On Oct 22, 2010, at 12:58, Zach Shelby wrote:

> I would say the server MUST return the resource in one message if it is smaller than MessageSize

If MessageSize is a client option, that would force the server to use a size that doesn't make sense server-side.  How does the client get to choose a good messagesize without forcing the server to do something uncomfortable?  I still think this should be some kind of negotiation.

Gruesse, Carsten

---

From: Robert Quattlebaum <[email-masked]>
Date: Fri, 22 Oct 2010 05:05:26 -0700

Hello Zach!

On Oct 22, 2010, at 3:38 AM, Zach Shelby wrote:

> Now this is an interesting case. Do we want to enable random access to a resource? When does random access make sense?

I'm just taking a stab in the dark with this one, but lets see where it goes.

Let's say we have an HTTP-CoAP proxy. An HTTP client makes a GET request to the proxy for a CoAP resource with a Range header. Without random access, the proxy would have to start retrieving the CoAP resource from the start until it gets to the requested offset---which may be inefficient. In such a case the HTTP-CoAP proxy would likely simply always return a full 200 response, ignoring the range header in the HTTP request.

Another possibility is where a constrained device is running some sort of compiled script which, due to resource limitations, is served by another CoAP node and never stored entirely locally. Random access would allow jumps to earlier parts of the compiled script without needing to restart the transfer from scratch. Thus without random access this scenario becomes unfeasible. (I say *compiled* script here because random access doesn't really help when jumping in a non-compiled script)

Do either of these cases sound compelling? The first case is a bit of a stretch, but the second case sounds like it may be interesting. 

> At first I was thinking that this could be used as a way to recover from the failure of GETs on some blocks if you are requesting multiple blocks in parallel. However, with the opaque model, you could still do this if the client would keep some state on the opaque tags for open requests. 

Not sure how you could request multiple blocks in parallel without implying random access, especially with an opaque model.

> Token can be used for this?

Yes, but since you can't predict which responses will be asynchronous you will need to add a Token to every request you send out (which, IMHO, would be unfortunate). Mandating the range option in the async responses was a way to make them self-describing without any context.

__________________
Robert Quattlebaum

---

From: Robert Quattlebaum <[email-masked]>
Date: Fri, 22 Oct 2010 05:35:15 -0700

Hello Peter!

> - Add an error code TooLarge sent by the server when the representation to be sent to the client exceeds the size it has specified (or defaulted) and the server declines (for whatever reason) to pr…

Two quick observations:

A server may not be immediately able to reliably determine the entire size of the retrieved content in an efficient manner, so for the TooLarge case the "MessageSize" option should be RECOMMENDED.

Second, In the first case you are using MessageSize to convey the largest single CoAP message (header+options+payload) that the client is willing to accept. Sounds reasonable. However, in the second case you are using the same option to contain an entirely different value: the size of the fully retrieved content of the requested resource. Perhaps we could define two options with the same numerical option identifier: For requests, it would be a "Message-Size" (or, as I would prefer, "Max-Message-Size"), and for responses it would be a "Resource-Content-Size" or something similar. This way it is clear that the values of this option in these two contexts are not related in function.

__________________
Robert Quattlebaum

---

From: Peter Bigot <[email-masked]>
Date: Fri, 22 Oct 2010 08:56:15 -0500

On Fri, Oct 22, 2010 at 5:58 AM, Zach Shelby <[email-masked]> wrote:

> Peter,
>
> Thanks for trying to reconcile these, comments in-line. You and Robert do
> realize you'll have to help us edit this document  :-) Starts to look like a
> re-write soon if we go in this direction...
>

Rewrite of which document?  core-coap or core-block?  I'd assumed core-coap
was pretty nearly frozen, but if not, let's do this.

 On Oct 22, 2010, at 12:54 PM, Peter Bigot wrote:
> > - Add an elective option MessageSize, which when used in a request
> encodes a one or two byte unsigned integer representing the maximum size of
> the CoAP (non-error) response message that should be sent to the client.
>  Give it a default value of 1152 or whatever.  Servers use this to determine
> whether the resource response can be sent in one message or needs to be
> chunked.  This use of the option appears only in requests.
>
> Would that eliminate the current text describing what the maximum size of a
> CoAP message is? In reality the default of MessageSize would serve that
> purpose.
>

Only if this makes it into the CoAP base document, in which case it would
change that text (I believe the restriction from section 4 to fit in an IP
datagram should remain).  I propose that it should be a critical option if
it's in the base document, as failure to understand and respect it could
result in lost data.

> > - Add an error code TooLarge sent by the server when the representation
> to be sent to the client exceeds the size it has specified (or defaulted)
> and the server declines (for whatever reason) to pre-emptively chunk it.
>  The response message would use MessageSize to convey the total resource
> size to the client.  In this use, the value of the option might be larger
> than two bytes, if we feel it's reasonable for any CoAP resource to ever
> have a representation larger than 65535 bytes.
>
> Considering that many client's have limited memory, it would be useful for
> a client to know how large a resource is before it starts to retrieve it.
> MessageSize could be a variable length uint.
>

That was my assumption.  In requests, it need be no larger than 16 bits
(unless an IPv6 datagram can be larger than 65536 octets?).  In the response
it might require more bits.  We already (or used to) support the concept of
variable-length integers as header option values, so nothing special needed
here.

A client that doesn't know how big the resource is and just wants to find
out can send a request for it with a MessageSize of zero.  Unless the
resource has no representation (fits in zero bytes), the server would be
obliged to return an error message.  As I hinted in my wording, MessageSize
restrictions should not apply to error responses.  I don't want to get
started defining support for reassembling error messages.

>
> >
> > This eliminates the question of how the server determines whether it
> should use any multi-part response like the block option, and provides a
> facility for influencing chunking on inherently multi-part responses like
> Robert's use cases.
> >
> > A Comment on Symmetry:
> >
> > Unlike HTTP, CoAP can assume that each endpoint is capable of being both
> a server and a client.  This means we have the option of asymmetry: in
> Although I don't agree that we can assume all endpoints can act as a server
> and client, I do find some logic here though. It might make sense to limit
> the use of chunked requests for GET (although we still need to look at how
> hard PUT/POST actually are). Just because we would do that though, doesn't
> mean we need to build in a callback URI feature. This kind of interface
> belongs to the REST interface design of the application, not for the
> transfer protocol. If chunked transfers only are supported for GET, then
> REST interfaces will naturally be designed in that way. We have to be
> careful about designing too much into the transfer protocol.
>

Yes; sorry I said "can assume" instead of "could assume" ;-)

Yes, it's exactly that all this can be done at the REST level.  Callback URI
support does not require anything special from CoAP.

> Considering that the chunked mode is optional, I would say the server MUST
> return the resource in one message if it is smaller than MessageSize. The WG
> was pretty clear in Maastricht that chunking should be an optional feature.
>

I agree with Carsten that the server should have the right to decline the
size if it knows something the client doesn't.  Carsten may or may not agree
with me that, if it does so, it should deny the request with TooLarge rather
than automatically switch to some chunking protocol.

Small mess here: if the server declines the proposed fragment size, but the
representation is larger than the proposed fragment size, there's currently
no mechanism for the server to suggest an acceptable fragment size.  I
propose that, in that situation, the server reject with TooLarge, and the
client re-submit with the options that enable the chunking protocol.  The
chunking protocol in turn should explicitly allow the server to use a
smaller fragment than the client suggests.

> My only requirement for random access is to recover failed chunks. I think
> PartialContentId would be sufficient for doing that if the client keeps a
> little state on open requests (it has to anyways).
>

This works fine if it's necessary for the client to re-transmit a request
because the server didn't receive it.

It doesn't work so fine if the server received it, updated things, and its
response (with a new PartialContentId) was lost.  If it has to be robust in
that situation, this rules out the use of a constant PartialContentId that
the server treats as a "Next" indicator rather than encoding segment
identification information.

Either PartialContentId must contain the complete (opaque) state required
for the server to reconstruct the current position (the REST-ful, but
possibly inefficient, way), or the server must encode segment identification
information in the PartialContentId and further be able to reconstruct
earlier messages.  This might not be possible in an on-demand generation
situation, in which case we need a CantRecreate error.

Also need an error for garbled PartialContentId content.

> We still need to think through the use of this with POST/PUT, although I do
> like the idea of keeping this as simple as possible.
>

I look forward to seeing somebody's proposal.

I suspect that just defining chunk support for GET would make the code
footprint of a full implementation noticeably smaller than one that also had
to support a distinct PUT/PUSH chunking solution.

---

From: Peter Bigot <[email-masked]>
Date: Fri, 22 Oct 2010 08:57:38 -0500

On Fri, Oct 22, 2010 at 7:35 AM, Robert Quattlebaum <[email-masked]>wrote:

> A server may not be immediately able to reliably determine the entire size
> of the retrieved content in an efficient manner, so for the TooLarge case
> the "MessageSize" option should be RECOMMENDED.
>

Actually, I'd intended it as the server's best estimate of the total size of
the resource representation, which may not even be a valid message size when
coming from the client.  So recommended/mandatory is irrelevant; it's
informational.  I'm not sure whether the server has to guarantee that the
resource will fit within the limit.  If so, the option would be omitted if
the server can't determine an upper bound on the final representation size.

> Second, In the first case you are using MessageSize to convey the largest
> single CoAP message (header+options+payload) that the client is willing to
> accept. Sounds reasonable. However, in the second case you are using the
> same option to contain an entirely different value: the size of the fully
> retrieved content of the requested resource. Perhaps we could define two
> options with the same numerical option identifier: For requests, it would be
> a "Message-Size" (or, as I would prefer, "Max-Message-Size"), and for
> responses it would be a "Resource-Content-Size" or something similar. This
> way it is clear that the values of this option in these two contexts are not
> related in function.
>

I agree; we just have the legacy issue of trying to reduce the number of
header options that have to be recognized.  Are people are OK with saying
that, say, header option type 7 means "MaxMessageSize" on requests and
"ResourceRepresentationSize" on responses (including error responses)?

---

From: Zach Shelby <[email-masked]>
Date: Sat, 23 Oct 2010 22:46:27 +0300

On Oct 22, 2010, at 4:56 PM, Peter Bigot wrote:

> Rewrite of which document?  core-coap or core-block?  I'd assumed core-coap was pretty nearly frozen, but if not, let's do this.

;-) coap-block of course. What we are discussing there doesn't affect core-coap much. Of course we need to keep them aligned. 

Zach

---

From: Zach Shelby <[email-masked]>
Date: Sat, 23 Oct 2010 22:50:26 +0300

On Oct 22, 2010, at 4:57 PM, Peter Bigot wrote:

> I agree; we just have the legacy issue of trying to reduce the number of header options that have to be recognized.  Are people are OK with saying that, say, header option type 7 means "MaxMessageSize" on requests and "ResourceRepresentationSize" on responses (including error responses)?

This is what I had in mind.

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Oct 2010 18:36:15 +0200

OK, I wrote up this slicing stuff.

Since I can't submit a new draft at this time, I put it in as draft-bormann-core-coap-block-01.
Shoot me.  I'm not trying to compete with the WG draft, I just want to have something written up that is not entirely handwaving.

I think the design I'm presenting is complete, and this will enable us to compare generalized slicing with block (and/or possibly simplify slicing by dropping some of the objectives I had in mind).

Enjoy at:
	http://tools.ietf.org/html/draft-bormann-core-coap-block

(Once the tools server has picked it up -- right now you get to see the -00 there; so go to http://www.ietf.org/internet-drafts/draft-bormann-core-coap-block-01.txt for black-and-white version.)

Gruesse, Carsten

---

From: Peter Bigot <[email-masked]>
Date: Mon, 25 Oct 2010 07:47:44 -0500

The proposed size option, as well as the asynchronous delay estimate option,
would both IMO be appropriate for core-coap and would significantly decrease
the number of cases where compliant behavior is unclear.

Peter

On Sat, Oct 23, 2010 at 2:46 PM, Zach Shelby <[email-masked]> wrote:

> > Rewrite of which document?  core-coap or core-block?  I'd assumed
> core-coap was pretty nearly frozen, but if not, let's do this.
>
>
> ;-) coap-block of course. What we are discussing there doesn't affect
> core-coap much. Of course we need to keep them aligned.
>
> Zach
>
> --