Subject: Re: [core] draft-bormann-core-coap-block-01.txt now all sliced up (Re: Next/Range Options (re: draft-ietf-core-block-00))

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 1 Nov 2010 14:10:29 -0700

Hello Carsten!

Overall, draft-bormann-core-coap-block-01 <http://tools.ietf.org/html/draft-bormann-core-coap-block-01> appears to address most, if not all, of my original concerns. In particular, I like how this proposal allows for both sliced transfers in both directions for POST/PUT requests. My largest concern is that this proposal adds five new header options.

I'll skip over the largely aesthetic concerns, ie: "slice" vs "chunk", etc.

>    When a representation is larger than can be comfortably transferred
>    in a single UDP datagram, the Continuation options can be used to
>    indicate a sliced transfer.  The value of both Continuation-request
>    and Continuation-response options is an opaque sequence of bytes that
>    is used to link up one slice transaction with the next in a sequence
>    of transactions.

I assume that the distinction between the request and response is necessary for asynchronous response correlation. I'd love to consolidate these together, but I'm not sure how yet. 

>    (TBD: Does the continuation request have to supply the URI and other
>    parameters again along with the Continuation-request option, or does
>    the server make sure the Continuation-response provides all
>    information required, e.g., by sending back a descriptor or by
>    encoding the request parameters in the Continuation-response?)

I would say yes, because doing otherwise seems like it would require more complicated implementations. I don't think there is a great deal to be gained by not including those headers. 

>    Each continuation response in a sliced transfer carries the response
>    code that, based on information currently available, the server
>    expects the transfer to receive at the end of the entire sequence of
>    slices, e.g. a 200 code (encoded as 80 decimal) for a successful
>    resource representation retrieval.  However, only the response code
>    of the last slice transferred (i.e., without a Continuation-response
>    option) is binding.

I don't see the benefit of having this capability, other than returning obnoxiously long custom 404 error messages. It also makes it impossible for a HTTP-CoAP proxy to incrementally populate an HTTP response without receiving (and caching) the entire content first---because the response code goes first in the HTTP response and you don't know the "real" response code until the end of the transfer. 

I am inclined to either always use 206(Partial Content), or always using 200(OK) for "sliced" transfers. I prefer 206 over 200 because 200 implies to me that the response is self-contained. 

>    For PUT and POST requests, a client has to indicate that it requests
>    a Continuation-response option from the server whenever it is unable
>    or unwilling to provide the entire (rest of the) representation in
>    the current transaction, i.e. that the slice(s) transferred so far
>    are only a prefix of the entire representation to be transferred.
>    This is done by supplying a Continuation-required option in the
>    request.  The request that carries the last slice of the
>    representation does not contain a Continuation-required option.

What is the response code of the intermediate responses from the server? 100 (Continue)?

>    However, the response to that last forward-transferring request may
>    carry a Continuation-response option, in case the representation
>    returned from the PUT or POST request is larger than can be
>    comfortably transferred in one message.  The exchange then continues
>    as it would for a sliced GET request.

Interesting. I like this.

If we specify the end of the request content as a message with no content, then we might be able to do away with the "continuation-required" option. In such a case we would use an empty "continuation-request" to signify the start of the transfer. Just a thought.

> 3.  The Size-Estimate option
> 
>    (This is new functionality not provided by the Block option.)
> 
>    The party slicing up a resource representation for sliced transfer
>    may have an idea of the size of the entire resource representation in
>    bytes.  Providing this size as an estimate may be beneficial for the
>    other party.  If provided, it should be sent with the first slice,
>    and SHOULD provide a close upper bound of the total size that will be
>    transferred.  In a PUT or POST transfer, both sides MAY provide a
>    Size-Estimate for their respective resource representation to be
>    transferred.

I'm somewhat neutral on this new option. I wouldn't loose any sleep if we dropped it, but if there is a compelling case for it I don't mind.

>    o  An attacker might attempt to forge Continuation-response values to
>       obtain increased access.  Where the structure of the Continuation-
>       response might make this possible, the server should validate it
>       e.g. by including an HMAC computed from a secret only known to the
>       server.  If replay attacks might be problem, the server should
>       mitigate them, e.g. by adding a timestamp or a sequence number
>       into the protected data.

I can imagine someone using a pointer to the next item in their linked list as the value of the continuation-request, which would be a bad, bad idea. Perhaps we should call out cases like this specifically.

__________________
Robert Quattlebaum
Jabber: [email-masked]
eMail:  [email-masked]
www:    http://www.deepdarc.com/

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Nov 2010 22:56:25 +0100

On Nov 1, 2010, at 22:10, Robert Quattlebaum wrote:

> five new header options

Yes, that's what gives me pause, too.
However, one option (the Size-Estimate option) is completely unrelated to the rest of the draft -- if we want the capability, we'll have to have an option like this.
This leaves four options for the continuation mechanism: one for message size negotiation and three for the actual state machine.
Every attempt I have made to consolidate them made things more complicated instead of simpler.

The beauty of the block option is that things just happen to fall together nicely.
(However, block does not solve the extended PUT/POST response problem; this would probably need a second option.)

As a general rule, I accept generalizing approaches only when things become simpler or stay as simple through generalizing.
But here, the generalization also addresses a somewhat unfortunate limitation.  So it's not as clear-cut.

We probably should make a decision within the course of November.

Gruesse, Carsten

PS.: I tried to use terminology that is different from everything people are used to, hence no chunks etc.  
E.g., HTTP chunking is solving a completely different problem, so reusing the term will confuse people.

---

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 1 Nov 2010 17:24:15 -0700

Hello Carsten!

> Every attempt I have made to consolidate them made things more complicated instead of simpler.

After doing some more thinking, I think that we can take the option count down from five to three if we remove the extended PUT/POST response support (which is something previous proposals didn't support anyway):

 * Replace the Message-Size and Size-Estimate with a single option value that is interpreted as Message-size in requests and Size-Estimate in responses. (We would however loose the ability to send a size-estimate to the sever for sliced PUT/POST request content)
 * Continuation-Requried would be removed. The start of a sliced PUT/POST request would be denoted by the presence of an empty Continuation-response header. The end of a sliced PUT/POST request would be denoted by a message with no content.

My Range header was basically my attempt to reduce the header count further by using the byte offset field as a way to correlate async responses, as well as specify the maximum message size. This allows us to merge the Continuation-requested and Continuation-required options. We still loose the extended PUT/POST though. The name 'Range' however implies the possibility of random access, which (based on previous messages in this mailing list) seems to be a method of access that the WG doesn't want to promote. I made a few passes at trying to make a compelling case for random access in <http://www.ietf.org/mail-archive/web/core/current/msg00849.html>, but it didn't seem to catch on. 

But if we dissect the Range header, we get two values: Max-message-size and Continuation-offset. If we don't allow random access, the Max-message-size only makes sense in requests, and the continuation-offset would only make sense in async responses. This means that we can overload a single header option value to mean "Max-message-size" for requests and "Continuation-offset" for responses, at the expense of Size-estimate header option. Since we can then use Continuation-offset to correlate the async response, we can then merge the Continuation-request and Continuation-response header options as well. That gives us four header options using two header option indexes:

RequestHeaders/ResponseHeaders
Continuation-response/Continuation-request
Max-message-size/Continuation-offset

Zach seemed to imply in <http://www.ietf.org/mail-archive/web/core/current/msg00864.html> that this sort of header option index combo is acceptable. This would reduce the header index count to two, sacrificing the Size-estimate header option and the extended PUT/POST response support.

It might even still be possible to do extended PUT/POST response support using this mechanism, but I'll have to do some more thinking about that.

__________________
Robert Quattlebaum