Subject: [core] Review of draft-bosh-core-new-block-00.txt

From: Christian Amsüss <[email-masked]>
Date: Mon, 11 May 2020 14:55:41 +0200

Hello BOSH authors,
hello CoRE,

I'd like to offer a rough review of the new-block draft. I got
interested in it both because I was curious how it would address the
original DOTS use case, how it'd be useful to other cases that have been
brought up (a-realist comes to mind), and how it'd address other pending
issues around block-wise (eg.  the need for an extra Request-Tag when
access can not guaranteed to be independent per block).

As I understand this proposal, it addresses several issues
simultaneously:

* In Block1, there is no way for the server to indicate missing blocks.

* For Block2, a client can not request more than one block in a single
  request -- especially for observations, and when it would like
  multiple non-contiguous blocks.

My main concern here is the way token values are (re)used, and how that
interacts with proxies (intermediaries): The way multiple responses
arrive on the same token makes the option inherently proxy-unsafe, which
is not something that can entirely be avoided in the context of
block-wise transfer where atomic processing is desired (which I
personally consider more of a corner case). I am also concerned that the
option's implication covering all sub-layers of CoAP (talking about the
request body in the same paragraph as about CON/NON details) makes
things more complicated than they need to be. (CoAP deliberately has all
these sub-layers in a single protocol to allow several optimizations,
but solutions that stick to one layer are preferable in my opinion).

I don't quite understand how the Block Identifier is interacting with
resources that are not updated atomically, and how (in Block4 use) it
differs from the ETag.

There are several other concerns (down to the code 2.05 of the Figure 4
example which there is already 4.08 for), but they're probably best
addressed when the rest is out of the way.

My impression is that all of this can be achieved more generally by
combining tools we already have, or are in development:

* For the server to indicate missing blocks in processing, a response
  payload for 4.08 Request Entity Incomple would be sufficient; that
  could be a simple CBOR array indicating the byte offsets of missing
  slices or a more generic format like [1].

  Then, figures 3 and 4 would look like this:

           CoAP        CoAP
          Client      Server
            |          |
            +--------->| PUT /path B1:0/1/1024 No-Response:2.xx
            +----X     | PUT /path B1:1/1/1024 No-Response:2.xx
            +----X     | PUT /path B1:2/1/1024 No-Response:2.xx
            +--------->| PUT /path B1:3/1/1024 No-Response:2.xx
            |          |
            |<---------+ 4.18, payload `missing 1024-3072`
            +--------->| PUT /path B2:1/1/1024 No-Response:2.xx
            +----X     | PUT /path B2:2/1/1024 No-Response:2.xx
            |          |
            +--------->| PUT /path B1:4/1/1024 No-Response:2.xx
            |<---------+ 4.18, payload `missnig 2048-3072`
                ...

  Note that this is mentioning no message IDs, tokens or message types;
  the client would probably pick NON for the No-Response requests as
  long as it's within its probing rate. It may skip the No-Response and
  send a CON whenever it feels uncomfortable with the silence.

  This *does* expose one difference to the B3 behavior described, and I
  can't tell yet whether that's good or bad: Unlike with B3, whether a
  "these blocks are missing" message comes depends on the server's
  implementation. If it requires blocks to arrive sequentially, it'd
  send a 4.18 immediately at the next block (as is depicted above). If
  it spools them all up, it'd send a cumulative error to the final
  request.

  If it does not do atomic processing, it would just process the request
  with the resource's last state -- which is a bug in this use case but
  a feature in others. For cases where this could bite, I trust (hope?)
  that protocol designers have the wisdom to ensure that this does not
  happen at application level.

* For returning multiple responses to a single request, there is the
  core-responses draft[2] that would need more work but should
  eventually be where the specification for sending data on a token
  that already had its response could be done.

  Assuming that a later version of it offers a proxy-unsafe
  Lesiure-For-Responses:uint option that grants the serve a number of
  additional responses short-term responses on the same token (which may
  contain a Response-For option, but may also not if other options
  already clearly make the response incompatible to the original request
  but suitable for processing anyway), figures 5 and 6 could look like
  this:

           CoAP        CoAP
          Client      Server
            |          |
            +--------->| GET /path T:0xf0 O:0 B2:0/0/1024
            |<---------+ 2.05 T:0xf0 O:1234 B2:0/1/1024 ETag: 21
            |<---------+ 2.05 T:0xf0 O:1234 B2:1/1/1024 ETag: 21
            |<---------+ 2.05 T:0xf0 O:1234 B2:2/1/1024 ETag: 21
            |<---------+ 2.05 T:0xf0 O:1234 B2:3/0/1024 ETag: 21
                ...
               [[Observe triggered]]
            |<---------+ 2.05 T:0xf0 O:1235 B2:0/1/1024 ETag: 22
            |<---------+ 2.05 T:0xf0 O:1235 B2:1/1/1024 ETag: 22
            |<---------+ 2.05 T:0xf0 O:1235 B2:2/1/1024 ETag: 22
            |<---------+ 2.05 T:0xf0 O:1235 B2:3/0/1024 ETag: 22

  Note that tokens *are* shown here now, as that very option would
  interfere with token handling, and thus be proxy unsafe.

  If any of the triggered messages does *not* make it through, the
  client is at leisure to send a `GET /path T:0xf1 B2:2/0/1024` and
  reassemble it with the rest of the burst provided the ETag matches, as
  it always could.

  That outlined mechanism would not allow specifically requesting what
  should be in the follow-up blocks. Like in HTTP/2 push, the server
  would send what it thinks best, which would in the blockwise case be
  the next n blocks -- so this is where a new option *would* be helpful,
  but that option could be very much noncommittal, and I'd expect some
  such options to be specified along along with such a
  Leisure-For-Responses option (eg. "Send these and those blocks", "if
  the response is a link just assume I'll send a GET there next", "and
  send me a current representation after the PATCH I've sent", "and let
  me know if anything else changed as a result").

Looking forward to more direct discussion on Wednesday.

Kind regards,
Christian

[1]: https://tools.ietf.org/html/draft-bosh-core-new-block-00
[2]: https://tools.ietf.org/html/draft-bormann-core-responses-00

-- 
The detailed semantics of CoAP methods are "almost, but not entirely
unlike" [HHGTTG] those of HTTP methods.
[HHGTTG]: Adams, D., "The Hitchhiker's Guide to the Galaxy", October 1979.
  -- Shelby, et al., Internet-Draft Constrained Application Protocol (CoAP)

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 14 May 2020 00:46:09 +0200

Hello,

following up on the interim discussion, I'm trying to make the examples
a bit more plastic and tangible.

An example of Block1 flow (the previous one was slightly flawed relative
to its description) -- assuming the server is spooling up the complete
request, and illustrating it by the client asking for ackowledgements
every few blocks to err out early (though in the DOTS use case,
everything would probably just be pumped out in NONs with at most a CON
or a NON without No-Response at the end):

            CoAP        CoAP
           Client      Server
             |          |
             +----X     | NON PUT /path B1:0/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:1/1/1024 No-Response:2.xx
             +----X     | NON PUT /path B1:2/1/1024 No-Response:2.xx
             +--------->| CON PUT /path B1:3/1/1024
             |          |
             |<---------+ ACK 2.31
             +--------->| NON PUT /path B1:4/1/1024 No-Response:2.xx
             +--------->| CON PUT /path B1:5/0/1024
             |          |
             |<---------+ 4.18, payload `missing 0-1024, 2048-3072`
             +--------->| NON PUT /path B1:0/1/1024 No-Response:2.xx
             +--------->| CON PUT /path B1:2/1/1024
             |<---------+ ACK 2.04 Changed

Those who are baffled by the 2.04 response to the non-terminal final
block: The server is just making ample use of the 7959's license around
"whether to handle the request for the entire body atomically, or any
mix of the two". It could certainly be discussed whether the block that
triggered the initial 4.18 would be acceptable, though, or whether the
client would need to retransmit that.

The server in this example is going out of its way to ensure all
received data is processed. More typical servers would probably start
4.18'ing to the second request already (as their atomic updates need to
go from the start), and may require blocks to arrive not too far off, up
to the strictest servers that need messages in the right sequence.

Except maybe for the corner cases if the initial and the last block,
this should all already be within the specified block-wise behavior (the
new media type for the 4.18 being the only addition).

{Followers of a "updates should never be atomic but always per-block"
school of thought might take more liking to an alternative model in
which all blocks are accepted, and the client can verify that a full
representation has been uploaded by getting a representation of the
resoruce in a 4.18-payload equivalent "content sparseness
representation". This works best if resources are created by a PUT
instead of updated by a PUT, and can be made into an efficient exchange
using the Leisure-For-Responses option as in footnote [2]).

For the Block2 case, in addition to the generic non-traditional
extension proposal[1], let's have one option More-Blocks-Hint, which
indicates ranges of content the client would like to receive on the
Leisure-For-Responses tickets. Unlike Leisure-For-Responses, while
technically still hop-by-hop, this option can be elective and proxy-safe
-- sure, proxies work better if they treat it hop by hop, but a proxy's
ignorance of it won't break proxying. Then, a similar Block2 exchange
could look like this:

            CoAP        CoAP
           Client      Server
             |          |
             +----------> GET /path B2:0/0/1024 Leisure-For-Responses:3
             |          | More-Blocks-Hint:1,2,3 (or `1024-end`)
             |    X-----+ 2.05 B1:0/1/1024
             <----------+ 2.05 B1:1/1/1024
             |    X-----+ 2.05 B1:2/1/1024
             <----------+ 2.05 B1:3/1/1024
             +----------> GET /path B2:0/0/1024 Leisure-For-Responses:3
             |          | More-Blocks-Hint:2,4,5 (or `2048-3072, 4096-end`)
             <----------+ 2.05 B1:0/1/1024
             <----------+ 2.05 B1:2/1/1024
             <----------+ 2.05 B1:3/1/1024
             <----------+ 2.05 B1:4/0/1024

I hope this makes the proposal more tangible.

Kind regards
Christian

[1]: https://mailarchive.ietf.org/arch/msg/core/KPDCO9mQNzYjAuGwjV6FWWEkAH8/

[2]:

            CoAP        CoAP
           Client      Server
             |          |
             +----X     | NON PUT /path B1:0/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:1/1/1024 No-Response:2.xx
             +----X     | NON PUT /path B1:2/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:3/1/1024 No-Response:2.xx
             |          | Leisure-For-Response:1
             |<---------+ NON 2.05 For-Request:(Uri:/path,
             |          | Accept:application/sparseness+cbor), payload
             |          | `missing 0-1024, 2048-3072`
             +--------->| NON PUT /path B1:0/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:2/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:4/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:5/0/1024 Leisure-For-Response:1
             |<---------+ NON 2.05 For-Request:(Uri:/path,
             |          | Accept:application/sparseness+cbor), payload
             |          | `all here`

-- 
There's always a bigger fish.
  -- Qui-Gon Jinn

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 14 May 2020 08:58:38 +0200

Hi Christian,

Picking one detail here:

> On 2020-05-14, at 00:46, Christian Amsüss <[email-masked]> wrote:
> 
>             |<---------+ ACK 2.31

That is the provisional response that is already defined in RFC 7959.

>             |<---------+ 4.18, payload `missing 0-1024, 2048-3072`

That is not a provisional response; receiving it would stop the client logic for a block-wise transfer.  I think this message needs to be clearly identified as a provisional response as well. 

Similarly, but on the level of the individual request, each non-final response would need to indicate that further responses will be sent on this request (i.e., with the current token).
(And that behavior would need to be solicited by the client; I think that is your more-blocks-hint.)

A completely separate question is how to encode all this information; we could add options to go along with the basic block1/2 machinery (bonus: some of these could be elective, which might help in a non-DOTS incremental deployment situation), or we could simply define a new set of options like Block3/4.
The latter can be made very compact, but it also starts to require some non-trivial encoding work (with optional/variable length parts and all that), which would better be covered by the existing Option decoding logic.

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 14 May 2020 12:12:54 +0200

Hello Carsten, CoRE,

> That is the provisional response that is already defined in RFC 7959.

Yes, this was just to illustrate how No-Response and Blockwise interact.

> >             |<---------+ 4.08, payload `missing 0-1024, 2048-3072`
> 
> That is not a provisional response; receiving it would stop the client
> logic for a block-wise transfer.  I think this message needs to be
> clearly identified as a provisional response as well. 

Mh, I *was* missing out the M bit value in the response, which indicates
whether the server treated as part of an atomic operation. There's ample
leeway for the server to treat parts of the transfer as atomic or
non-atomic, so if it does respond 4.08 Request Entity Incomplete that
doesn't necessarily relate to the *whole* message, but at least the last
block needs to acknowledge all the previous ones.

Maybe some redesign of block-wise *is* necessary for the Block1 case.

> Similarly, but on the level of the individual request, each non-final
> response would need to indicate that further responses will be sent on
> this request (i.e., with the current token).
>

(Assuming this is about the Block2 example),

The indication for the additional responses on the same token is given
with Leisure-for-Responses; More-Blocks-Hint is just the indication
*what* the client would like to have with those (and I've kept it
separate from Leisure-For-Responses so a proxy doesn't need to know it).

> A completely separate question is how to encode all this information;
> we could add options to go along with the basic block1/2 machinery

I'd slightly lean toward making More-Blocks-Hint independent of the
block options, as this would ease handling them across a
non-caching-but-possibly-fragmenting proxy. On the downside, encoding
that in CBOR takes 6 byte per hole (2x uin16_t).

For best encoding efficiency, the option could sure be merged into a
more powerful Block2 option that would be proxy-unsafe. That could then
also imply `Leisure-For-Responses` in a single go as well.

(In figuring out how to encode this all, it may well be that
Leisure-For-Responses never hits the wire, and is always implied by some
other options. Then still, a Block-with-Extras option and
Response-Forwarding[1], as well as any other longlived-token options,
could refer to a single description of how those are done rather than
touching the token process each on there own -- and implementers would
just flag those options as "having longlived token properties").

KR
c

[1]: https://mailarchive.ietf.org/arch/msg/core/AwYqnQu703V5RGR43JQxRslkYsw

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: "Jon Shallow" <[email-masked]>
Date: Fri, 15 May 2020 19:47:54 +0100

Hi Christian,

Thank you and all the others on the on the Interim Conference Call on
Wednesday - I found it very useful.

I have been trying to get my mind around everything, including the new
potential options and how I can practically implement things in the
potentially aggressive DOTS environment.

In summary, I am concerned with your proposal as it stands as uses new
additional options (one specific to Block2 usage) that modify the semantics
of how Block1 and Block2 are used as defined in RFC7959 along with some
potential design / implementation flaws.

That said, there are some good ideas that may need to be wrapped into the
Block3/BLock4 alternative.

Please see inline comments prefixed with Jon>

Regards

Jon

> -----Message d'origine-----
> To : Christian Amsüss [mailto:[email-masked]] Sent : Thu 14 May
2020 00:46
>  To : BOUCADAIR Mohamed TGI/OLN Cc : [email-masked] 
>  Re: [core] Review of draft-bosh-core-new-block-00.txt
> 
> Hello,
> 
> following up on the interim discussion, I'm trying to make the 
> examples a bit more plastic and tangible.
> 
> An example of Block1 flow (the previous one was slightly flawed 
> relative to its description) -- assuming the server is spooling up the 
> complete request, and illustrating it by the client asking for 
> ackowledgements every few blocks to err out early (though in the DOTS 
> use case, everything would probably just be pumped out in NONs with at 
> most a CON or a NON without No-Response at the end):

Jon> For DOTS, this would be NON without "No-Response".  However the Client
may not see the response (DOTS handles this already for the "all fits in one
packet" PUT case).

> block: The server is just making ample use of the 7959's license 
> around "whether to handle the request for the entire body atomically, 
> or any mix of the two". It could certainly be discussed whether the 
> block that triggered the initial 4.18 would be acceptable, though, or 
> whether the client would need to retransmit that.

Jon> There is nothing stopping the 4.18 response not including the
triggering block as "missing" if the server otherwise is happy with the
block.

> 4.18'ing to the second request already (as their atomic updates need 
> to go from the start), and may require blocks to arrive not too far 
> off, up to the strictest servers that need messages in the right sequence.

Jon> DOTS only cares that all of the blocks have been received before
atomically processing the entire "body".  My suggestion would be that the
potentially multiple 4.18 responses come under Congestion Control.

Jon> There is a reasonable chance that the blocks are sent out in sequence,
but may arrive at the server out of sequence
as we have  moved away from the "Lock Stepped" model of rfc7959 with the
2.31 Continue and need to update the sending of a 4.08 if Block1s are out of
order as per

" A server MAY
   also return a 4.08 error code for any (final or non-final) Block1
   transfer that is not in sequence; therefore, clients that do not have
   specific mechanisms to handle this case SHOULD always start with
   block zero and send the following blocks in order.
"

Jon> There is as issue with data corruption with using the above  mechanism.
Say there is the following sequence in
a DOTS environment

            CoAP        CoAP
           Client      Server
             |          |
             +----X     | NON PUT /path B1:0/1/1024 No-Response:2.xx
             +--------->| NON PUT /path B1:3/0/1024
             |    X<----+ 4.18, payload `missing 0-1024`
sometime later, new data
             +--------->| NON PUT /path B1:0/1/1024 No-Response:2.xx
             +----X     | NON PUT /path B1:3/0/1024
             |<---------+ ACK 2.04 Changed

Jon> The server then ends up with Block2-0 from the second data set and
Block2-1 from the first.  This is why we
 included the Block-ID in the Block3 option.  We could consider using the
Token, keeping it the same Token for
 each set of blocks that comprise a body, but that breaks the usage of
Tokens definition as there may be a non
 2.xx response in the case of the first block2-0 transmission in my example.

" A Token is
   used to match responses to requests independently from the underlying
   messages (Section 5.3)
"

Jon> I still believe that we need a Block-ID - whether embedded in a Block3
option or as a separate option.  With Block3, the M (more) bit set is
equivalent to No-Response:2.xx.

Jon> I am also concerned that we are changing how Block1 usage is defined in
RFC 7959 (change in behaviour
 with No-Response Option and removal of 2.31 lock-stepping) so the RFC
containing the No-Response option
 will be updating RFC 7959.

> this should all already be within the specified block-wise behavior 
> (the new media type for the 4.18 being the only addition).
> 

Jon> Not quite - see above

> resoruce in a 4.18- payload equivalent "content sparseness 
> instead of updated by a PUT, and can be made into an efficient 
> exchange using the Leisure-For-Responses option as in footnote [2]).

Jon> For DOTS, after the initial PUT (of telemetry information), there will
be PUT updates to the same resource as telemetry information gets updated.

Jon> With [2], RFC7252 refers to a 2.01 or 2.04 response for a PUT.  2.05 is
only used in response to a GET.
  [1] Section for Leisure-For-Responses looks like it is defined for GET (or
FETCH) stating

" Leisure-For-Responses
---------------------

This new option (elective, proxy-unsafe, uint valued) allows the server to
send up as many responses as indicated in the option value in addition to
the requested response.
"

Jon> And not PUT.

> -- sure, proxies work better if they treat it hop by hop, but a 
> proxy's ignorance of it won't break proxying. Then, a similar Block2 
> exchange could look like this:
> 
>             CoAP        CoAP
>            Client      Server
>              |          |

Jon> As discussed on the Conf call, we need to have a concept of Block-ID
for each "body" of data, which could be the Etag so that the secondary GET
requests are pulling the data for the correct "body".  However, I believe
ETag cannot be used, as when used with a GET to get missing blocks, the
server (or cache) could respond with a 2.03 as per

"
   A server can issue a 2.03 Valid response (Section 5.9.1.3) in place
   of a 2.05 Content response if one of the ETags given is the entity-
   tag for the current representation, i.e., is valid; the 2.03 Valid
   response then echoes this specific ETag in a response option.
"

Jon> so believe that we still need a Block-ID here.

Jon> The Token for the secondary GET will be different and so the secondary
set of responses will use the secondary
 token (as does the first set of responses use the initial token).

Jon> If Observe:0 is used for the initial GET, then all Observe responses
will need to track and use the initial
 More-Blocks-Hint and Leisure-For-Responses in the Observe subscription for
the resource on the server.

Jon> Again, I am concerned that we are modifying RFC 7959 Block2 Usage in
that the client does not have to
 request the next block as per

"   When sending a 2.05 (Content) notification, the server only sends the
   first block of the representation.  The client retrieves the rest of
   the representation as if it had caused this first response by a GET
   request, i.e., by using additional GET requests with Block2 Options
   containing NUM values greater than zero.
"

Jon> More-Blocks-Hint would be a new Option that is only relevant to Block2.
If we went down the Block4 route, this could either be embedded in the
Block4 option, or the Block4 option can be repeated in the GET request
defining in the "missing" blocks.  In effect, Block4 request for block0 is
'give me all of the blocks hint' and a block4 request for block!=0 is 'send
me this missing block'.

Jon: The benefit of Leisure-For-Responses is that the number of blocks that
can sequentially sent without waiting can be controlled.  On the conf call
10 packets was mentioned as potentially being a good number before delays
should be inserted between packets.

Jon> In the DOTS case if the response fits into a single packet and the
response is dropped then the client does not care if in an attack scenario.
If there are multiple blocks, the client would elect to either try to
recover the missing blocks or just ignore the partial response.

~Jon

> 
> Kind regards
> Christian
> 
> [1]:
> https://mailarchive.ietf.org/arch/msg/core/KPDCO9mQNzYjAuGwjV6FWWE
> kAH8/
> 
> [2]:
> 
>             CoAP        CoAP
>            Client      Server
>              |          |

> -----Message d'origine-----
> De : Christian Amsüss [mailto:[email-masked]]
> Envoyé : jeudi 14 mai 2020 00:46
> À : BOUCADAIR Mohamed TGI/OLN
> Cc : [email-masked]
> Objet : Re: [core] Review of draft-bosh-core-new-block-00.txt
> 
> Hello,
> 
> following up on the interim discussion, I'm trying to make the examples a
> bit more plastic and tangible.
> 
to
> its description) -- assuming the server is spooling up the complete
request,
> and illustrating it by the client asking for ackowledgements every few
> blocks to err out early (though in the DOTS use case, everything would
> probably just be pumped out in NONs with at most a CON or a NON without
> No-Response at the end):
> 
>             CoAP        CoAP
>            Client      Server
>              |          |
> "whether to handle the request for the entire body atomically, or any mix
of
> the two". It could certainly be discussed whether the block that triggered
> the initial 4.18 would be acceptable, though, or whether the client would
> need to retransmit that.
> 
> The server in this example is going out of its way to ensure all received
data
> is processed. More typical servers would probably start 4.18'ing to the
> second request already (as their atomic updates need to go from the
start),
> and may require blocks to arrive not too far off, up to the strictest
servers
> that need messages in the right sequence.
> 
> Except maybe for the corner cases if the initial and the last block, this
> should all already be within the specified block-wise behavior (the new
> media type for the 4.18 being the only addition).
> 
> 
> school of thought might take more liking to an alternative model in which
> all blocks are accepted, and the client can verify that a full
representation
> has been uploaded by getting a representation of the resoruce in a 4.18-
> payload equivalent "content sparseness representation". This works best if
> resources are created by a PUT instead of updated by a PUT, and can be
> made into an efficient exchange using the Leisure-For-Responses option as
> in footnote [2]).
> 
> 
> For the Block2 case, in addition to the generic non-traditional extension
> proposal[1], let's have one option More-Blocks-Hint, which indicates
ranges
> of content the client would like to receive on the Leisure-For-Responses
> tickets. Unlike Leisure-For-Responses, while technically still hop-by-hop,
> this option can be elective and proxy-safe
could
> look like this:
> [1]:
> https://mailarchive.ietf.org/arch/msg/core/KPDCO9mQNzYjAuGwjV6FWWE
> kAH8/
> 
> [2]:
> 
>             CoAP        CoAP
>            Client      Server
>              |          |