Subject: Re: [core] #15: Consolidate maximum sizes (encodings, options table, maximum datagram)

From: Zach Shelby <[email-masked]>
Date: Fri, 2 Jul 2010 11:08:07 +0300

Any other comments on this one? Carsten didn't like the "physical inteface", I can remove that. Otherwise I think discussing in terms of MTU is the best we can do.

Zach

On Jun 30, 2010, at 1:39 PM, core issue tracker wrote:

> #15: Consolidate maximum sizes (encodings, options table, maximum datagram)
> --------------------------------+-------------------------------------------
> Reporter:  hartke@…            |       Owner:     
>     Type:  defect              |      Status:  new
> Priority:  minor               |   Milestone:     
> Component:  coap                |     Version:  1.0
> Severity:  Active WG Document  |    Keywords:     
> --------------------------------+-------------------------------------------
> 
> Comment(by zach@…):
> 
> - The URI and option length discrepency is being fixed already.
> 
> - Regarding the maximum message size, I propose text like this which is
> similar to that used by other UDP based protocols such as m-DNS:
> 
> "The resulting datagram carrying a CoAP message MUST fit within a single
> MTU of that phsical interface, which is typically 1500 bytes for an
> Ethernet interface and 1280 bytes for a 6LoWPAN interface."
> 
> -- 
> Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/15#comment:2>
> core <http://tools.ietf.org/core/>
> 

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Guido Moritz <[email-masked]>
Date: Fri, 2 Jul 2010 12:37:46 +0200

I don't like the differentiation depending on the 'link layer'. 1280 seems
to be reasonable as the magic number of 6LoWPAN networks in general ;)

Guido

> -----Ursprüngliche Nachricht-----
> Von: [email-masked] [mailto:[email-masked]] Im Auftrag
> von Zach Shelby
> Gesendet: Freitag, 2. Juli 2010 10:08
> An: Carsten Bormann; core
> Cc: core issue tracker
> Betreff: Re: [core] #15: Consolidate maximum sizes (encodings, options
> table, maximum datagram)
> 
> Any other comments on this one? Carsten didn't like the "physical
> inteface", I can remove that. Otherwise I think discussing in terms of
> MTU is the best we can do.
> 
> Zach
> 
> > Ticket URL:
> <http://trac.tools.ietf.org/wg/core/trac/ticket/15#comment:2>
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: "Oflynn, Colin" <[email-masked]>
Date: Fri, 2 Jul 2010 15:37:02 -0600

Hello,

I would also support just forcing a size of 1280. This 'artificial' limit serves to stop people from trying to use CoAP for stuff it wasn't designed for, even if their link-layer really does support bigger packets.

Regards,

  -Colin

-----Original Message-----
From: [email-masked] on behalf of Guido Moritz
Sent: Fri 02/07/2010 11:37
To: 'Zach Shelby'; 'Carsten Bormann'; 'core'
Cc: 'core issue tracker'
Subject: Re: [core] #15: Consolidate maximum sizes (encodings, options table,maximum datagram)

Guido

> > Reporter:  hartke@.            |       Owner:
> > Comment(by zach@.):

---

From: "core issue tracker" <[email-masked]>
Date: Mon, 05 Jul 2010 08:44:26 -0000

#15: Consolidate maximum sizes (encodings, options table, maximum datagram)
--------------------------------+-------------------------------------------
 Reporter:  hartke@…            |        Owner:        
     Type:  defect              |       Status:  closed
 Priority:  minor               |    Milestone:        
Component:  coap                |      Version:  1.0   
 Severity:  Active WG Document  |   Resolution:  fixed 
 Keywords:                      |  
--------------------------------+-------------------------------------------
Changes (by zach@…):

  * status:  new => closed
  * resolution:  => fixed

Comment:

 Based on the list discussion it seems that we should keep the MUST fit in
 a datagram and recommend a size when the MTU is unknown of 1280 byte. The
 text about 6LoWPAN was left out as this really is link-specific, and
 should be obvious to the implementor on such a constrained node/network.
 The text for coap-01 is thus planned as:

 "The length of the Payload in a CoAP message is calculated from the
 datagram length. A CoAP message MUST fit within a single datagram. When
 the MTU is not known, then the minimum IPv6 MTU of 1280 bytes SHOULD be
 assumed."

 This text would change if the Block Option from coap-misc would be added
 later.

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/15#comment:3>
core <http://tools.ietf.org/core/>

---

From: Zach Shelby <[email-masked]>
Date: Mon, 5 Jul 2010 11:47:42 +0300

Yep, I removed the 6LoWPAN text as that should be obvious to people using it. For the vast majority of LAN link-layers that MTU is 1500 bytes, so letting them use that doesn't stretch the use of CoAP that much. In the case of e.g. 6LoWPAN the MTU would be 1280 bytes. The current text assumes 1280 bytes unless the MTU is otherwise known.

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 5 Jul 2010 11:29:30 +0200

On Jul 2, 2010, at 12:37, Guido Moritz wrote:

> 1280 seems

Well, 1280 is the magic number of IPv6 MTUs.
6LoWPAN does support 1280 byte packets, but splitting a packet into 15 or 20 fragments is expensive in terms of reduced probability of delivery.

Worse, Protocols like RPL can steal some of these bytes (and the IP and UDP headers will, too).
CoAP implementations should be able to operate on any UDP datagram that fits into the IPv6 MTU.
 
But having a good idea of what is actually efficient to use also helps.
So I would rarely send a CoAP payload of more than 1024 bytes; adding in CoAP options (the total size of which depends a lot on the length of the URI), CoAP fixed header (4 bytes), UDP header and IP header then is likely to leave some room for RPL and other tunnel stunts.
For applications such as firmware transfers on 6LoWPAN, I would stick with a payload size of 64 (1 fragment) or 128 (two fragments); that's why coap://ns.tzi.org:61616/ has 128 as the default block size.

Gruesse, Carsten

---

From: "Oflynn, Colin" <[email-masked]>
Date: Mon, 5 Jul 2010 04:14:45 -0600

Hello,

> When
> assumed.

I think this text is somewhat dangerous, and would prefer just a fixed limit still. It's not entirely clear what MTU you are talking about.

e.g.: Someone could be sending a CoAP packet on an Ethernet network, and not know that the other end of it will actually be a 6LoWPAN network. Thus they use the MTU of the Ethernet network which seems in line with the specification, but will cause problems on 6LoWPAN.

Or am I missing something here?

Regards,

  -Colin

-----Original Message-----
From: [email-masked] on behalf of core issue tracker
Sent: Mon 05/07/2010 09:44
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [core] #15: Consolidate maximum sizes (encodings, options table, maximum datagram)
 
Changes (by zach@.):

  * status:  new => closed
  * resolution:  => fixed

Comment:

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 5 Jul 2010 12:29:30 +0200

On Jul 5, 2010, at 12:14, Oflynn, Colin wrote:

> fixed limit

1152?

Divides the space between 1280 (IPv6 MTU) and 1024 (useful largest payload size) evenly between CoAP headers/options and IPv6/UDP/extension(RH, ESP, ...) headers.

Gruesse, Carsten

---

From: "Oflynn, Colin" <[email-masked]>
Date: Mon, 5 Jul 2010 04:49:58 -0600

Hello,

I think unfortunately the question of "optimum packet size" will always be "it depends". I'm assuming in the following you *need* to send a larger packet for some reason. 

Performing the fragmentation at the 6LoWPAN layer should be the most efficient, as you don't waste having IP/UDP/CoAP headers on each packet. This would suggest sending as large an IPv6 packet as possible. Under reasonable network conditions a 6LoWPAN network can do this pretty reliably (1280 byte delivery).

But in a bad RF environment though the lower layers may not deliver this message as reliably. Normally 802.15.4 will retry sending each 6LoWPAN fragment three times, after which it gives up. You then need to resend the entire IPv6 packet until all the fragments get through. As each time you are sending many (say 10) fragments, this could take a few tries! This would suggest keeping the IPv6 packet as small as possible, so each individual IPv6 packet is likely to get through, and when they fail you aren't resending as much.

Again I think it comes down to CoAP being a simple protocol. It will work great for small packets basically no matter what. It should work pretty well for larger packets too even. But if you want to transfer lots of data (firmware, etc) it's better to use something designed for the purpose. 

Another consideration for CoAP data is not just the transfer of it, but actually buffering the data in memory. I would guess most constrained (ie: 8-bit processor) devices are designed around having a 1280 byte IP buffer. The actual resources need to be designed with this in mind too!

Regards,

   -Colin

-----Original Message-----
From: [email-masked] on behalf of Carsten Bormann
Sent: Mon 05/07/2010 10:29
To: Guido Moritz
Cc: core
Subject: Re: [core] #15: Consolidate maximum sizes (encodings, options table,maximum datagram)

 

Gruesse, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 5 Jul 2010 13:59:03 +0300

Colin,

You captured the situation perfectly. So now just to get that down in text for coap-01 - the hard part. Seems I closed that ticket prematurely.

Carsten pointed that we have two options:

1. Define that this must fit in an IPv6 packet with an MTU of 1280 bytes. 
or
2. Define a fixed octet limit of e.g. 1152. At that point why not just say 1024 bytes.

Regarding 6LoWPAN how about we do the following. We include some text about this in the introduction, which is then also referenced from the text about maximum packet sizes. We can't force applications to avoid adaptation layer fragmentation, but we can point out that it was a design requirement.  

The Block Option discussion we can have separately to this. But as Carsten pointed out it shouldn't affect this text as you would use a block size < 1024 bytes anyways. 

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 5 Jul 2010 13:15:56 +0200

> 2. Define a fixed octet limit of e.g. 1152. At that point why not just say 1024 bytes.

Because that would mean the biggest CoAP payload that is a power of two in size would be 512.

(Historical anecdote: Why was the IPv6 MTU set at 1280?  Because 1500 is the Ethernet MTU, and 1024 bytes is a useful payload size.
Allowing for reasonable headers both internal and external, 1280 = (1024 + (1536-1024)/2) sounded round enough and about right.
(1500 is a rather odd number in binary, hence 1536 in the above formula.))

> block size < 1024 bytes anyways. 

I'd make that ≤ 1024 bytes, which makes 1152 as the message an attractive number: (1024 + (1280-1024)/2).

1024 bytes of payload sounds right for power-line applications.
For 6LoWPAN, 64 or 128 may be closer to optimal.

BTW, CoAP *is* designed to do large transfers (if you include the block option).
It may not be *fast* (as in saturating the link), because it is lock-step (until we add TCP-like congestion control as another option).
But it is efficient in every other way.
(And I wouldn't want to saturate a 6LoWPAN anyway.)

Gruesse, Carsten

---

From: "Oflynn, Colin" <[email-masked]>
Date: Mon, 5 Jul 2010 05:21:32 -0600

Hi Zach,

>Carsten pointed that we have two options:

Both work for me.

Using #2 w/ the 1024 option you suggest would leave room for extra IP header options to be slotted in later. Not sure if that is actually something we care about to be honest?

>We can't force applications to avoid adaptation layer fragmentation, but we can point out that it was a design requirement. 

To me sounds like a good idea, other thoughts? I think it's easy for someone to pick up any spec without having a full appreciation of the design space. A quick summary of fragmentation problems might help one understand why the emphasis on small packet sizes is present.

Regards,

  -Colin

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]]
Sent: Mon 05/07/2010 11:59
To: Oflynn, Colin
Cc: Carsten Bormann; Guido Moritz; core
Subject: Re: [core] #15: Consolidate maximum sizes (encodings, options table, maximum datagram)
 
Colin,

Zach

---

From: Peter Bigot <[email-masked]>
Date: Mon, 5 Jul 2010 08:17:03 -0500

>
> Seems I closed that ticket prematurely.
>

Too bad; I was all set to +1 the proposed wording.

I understand the desire to define a small fixed limit so people don't forget
that "constrained" is significant and try to use CoAP to stream HD video.
But I can't see a rationale for any numeric limit that doesn't end up
unnecessarily coupling CoAP to a specific technology.  If I have a whole
infrastructure set up with CoAP, I might really like to be able to re-use it
on a network where the MTU is, say, 2048 bytes.  I'd be kinda upset if I had
to replace the entire system because the specification included arbitrary
assumptions about maximum packet size.

Fragmentation is hard, so I think CoAP should make it somebody else's
problem.  What's wrong with referring to the MTU of the underlying transport
protocol (as long as "MTU" is defined)?  I might even have a system where,
for technical reasons (e.g., the devices have only 512B of RAM) the MTU
isn't as large as IPv6 requires.  I ought to be able to use CoAP in that
system.

In any closed system designers will be able to determine the appropriate
MTU.  For open systems with unmanaged clients, well, degradation due to loss
of network or link-layer fragments is a risk in any communication path.  At
least if clients are using IPv6 there's infrastructure to determine the
end-to-end MTU limits, and adjust the application messages accordingly.

I agree with Colin that an introductory section on packet sizes,
fragmentation, and their effects on CoAP architecture and use would help
avoid problems with folks not familiar with the constrained domain.

Peter

On Mon, Jul 5, 2010 at 5:59 AM, Zach Shelby <[email-masked]> wrote:

> Colin,
>
> You captured the situation perfectly. So now just to get that down in text
> for coap-01 - the hard part. Seems I closed that ticket prematurely.
>
> Regarding 6LoWPAN how about we do the following. We include some text about
> this in the introduction, which is then also referenced from the text about
> maximum packet sizes. We can't force applications to avoid adaptation layer
> fragmentation, but we can point out that it was a design requirement.
>
> The Block Option discussion we can have separately to this. But as Carsten
> pointed out it shouldn't affect this text as you would use a block size <
> 1024 bytes anyways.
>
> Zach
>
> > I think unfortunately the question of "optimum packet size" will always
> be "it depends". I'm assuming in the following you *need* to send a larger
> packet for some reason.
> >
> > Performing the fragmentation at the 6LoWPAN layer should be the most
> efficient, as you don't waste having IP/UDP/CoAP headers on each packet.
> This would suggest sending as large an IPv6 packet as possible. Under
> reasonable network conditions a 6LoWPAN network can do this pretty reliably
> (1280 byte delivery).
> >
> > But in a bad RF environment though the lower layers may not deliver this
> message as reliably. Normally 802.15.4 will retry sending each 6LoWPAN
> fragment three times, after which it gives up. You then need to resend the
> entire IPv6 packet until all the fragments get through. As each time you are
> sending many (say 10) fragments, this could take a few tries! This would
> suggest keeping the IPv6 packet as small as possible, so each individual
> IPv6 packet is likely to get through, and when they fail you aren't
> resending as much.
> >
> > Again I think it comes down to CoAP being a simple protocol. It will work
> great for small packets basically no matter what. It should work pretty well
> for larger packets too even. But if you want to transfer lots of data
> (firmware, etc) it's better to use something designed for the purpose.
> >
> > Another consideration for CoAP data is not just the transfer of it, but
> actually buffering the data in memory. I would guess most constrained (ie:
> 8-bit processor) devices are designed around having a 1280 byte IP buffer.
> The actual resources need to be designed with this in mind too!
> >
> > Regards,
> >
> >    -Colin
> >
> >
> >
> >
> > 6LoWPAN does support 1280 byte packets, but splitting a packet into 15 or
> 20 fragments is expensive in terms of reduced probability of delivery.
> >
> > So I would rarely send a CoAP payload of more than 1024 bytes; adding in
> CoAP options (the total size of which depends a lot on the length of the
> URI), CoAP fixed header (4 bytes), UDP header and IP header then is likely
> to leave some room for RPL and other tunnel stunts.
> > For applications such as firmware transfers on 6LoWPAN, I would stick
> with a payload size of 64 (1 fragment) or 128 (two fragments); that's why
> coap://ns.tzi.org:61616/ has 128 as the default block size.
> >
> > Gruesse, Carsten
> >

---

From: Zach Shelby <[email-masked]>
Date: Tue, 6 Jul 2010 15:45:51 +0300

I understand both arguments there. I added introduction text about adaptation layer fragmentation, and referenced that. Now this gives you the obvious hard upper bound as this is UDP, plus lets you use a Path MTU if you know it. I propose this text:

   The length of the Payload in a CoAP message is calculated from the
   datagram length.  While specific link layers make it beneficial to
   keep CoAP messages small enough to fit into their link layer packets
   (see Section 1), this is a matter of implementation quality.  The
   CoAP specification itself provides only an upper bound to the message
   size.  A CoAP message SHOULD fit within a single IP packet and MUST
   fit within a single IP datagram.  If the Path MTU is not known for a
   destination, an MTU of 1280 octets SHOULD be assumed.

On Jul 5, 2010, at 4:17 PM, Peter Bigot wrote:

> I understand the desire to define a small fixed limit so people don't forget that "constrained" is significant and try to use CoAP to stream HD video.  But I can't see a rationale for any numeric limit that doesn't end up unnecessarily coupling CoAP to a specific technology.  If I have a whole infrastructure set up with CoAP, I might really like to be able to re-use it on a network where the MTU is, say, 2048 bytes.  I'd be kinda upset if I had to replace the entire system because the specification included arbitrary assumptions about maximum packet size.
> 
> Fragmentation is hard, so I think CoAP should make it somebody else's problem.  What's wrong with referring to the MTU of the underlying transport protocol (as long as "MTU" is defined)?  I might even have a system where, for technical reasons (e.g., the devices have only 512B of RAM) the MTU isn't as large as IPv6 requires.  I ought to be able to use CoAP in that system.
> 
> In any closed system designers will be able to determine the appropriate MTU.  For open systems with unmanaged clients, well, degradation due to loss of network or link-layer fragments is a risk in any communication path.  At least if clients are using IPv6 there's infrastructure to determine the end-to-end MTU limits, and adjust the application messages accordingly.
> 
> I agree with Colin that an introductory section on packet sizes, fragmentation, and their effects on CoAP architecture and use would help avoid problems with folks not familiar with the constrained domain.
> 
> Peter
>

---

From: "Oflynn, Colin" <[email-masked]>
Date: Tue, 6 Jul 2010 06:46:53 -0600

Hi Zach,

This looks really good to me. Specifying path MTU closes my concern about 'which MTU will people use', and the paragraph is strongly worded enough to get the idea across of e.g.: don't send big packets unless you know what you are doing!

Thanks,

  -Colin

---

From: Peter Bigot <[email-masked]>
Date: Tue, 6 Jul 2010 07:52:58 -0500

Yes, thank you.

Peter

On Tue, Jul 6, 2010 at 7:46 AM, Oflynn, Colin <[email-masked]>wrote:

>  Hi Zach,
>
> This looks really good to me. Specifying path MTU closes my concern about
> 'which MTU will people use', and the paragraph is strongly worded enough to
> get the idea across of e.g.: don't send big packets unless you know what you
> are doing!
>
> Thanks,
>
>
>   -Colin
>
>
> From: Zach Shelby [mailto:[email-masked] <[email-masked]>]
> I understand both arguments there. I added introduction text about
> adaptation layer fragmentation, and referenced that. Now this gives you the
> obvious hard upper bound as this is UDP, plus lets you use a Path MTU if you
> know it. I propose this text:
>
> > I understand the desire to define a small fixed limit so people don't
> forget that "constrained" is significant and try to use CoAP to stream HD
> video.  But I can't see a rationale for any numeric limit that doesn't end
> up unnecessarily coupling CoAP to a specific technology.  If I have a whole
> > You captured the situation perfectly. So now just to get that down in
> text for coap-01 - the hard part. Seems I closed that ticket prematurely.
> >
> > Regarding 6LoWPAN how about we do the following. We include some text
> about this in the introduction, which is then also referenced from the text
> about maximum packet sizes. We can't force applications to avoid adaptation
> layer fragmentation, but we can point out that it was a design requirement.
> >
> > The Block Option discussion we can have separately to this. But as
> Carsten pointed out it shouldn't affect this text as you would use a block
> size < 1024 bytes anyways.
> >
> > Zach
> >
> > > But in a bad RF environment though the lower layers may not deliver
> this message as reliably. Normally 802.15.4 will retry sending each 6LoWPAN
> > > Again I think it comes down to CoAP being a simple protocol. It will
> work great for small packets basically no matter what. It should work pretty
> well for larger packets too even. But if you want to transfer lots of data
> > > 6LoWPAN does support 1280 byte packets, but splitting a packet into 15
> or 20 fragments is expensive in terms of reduced probability of delivery.
> > >
> > > So I would rarely send a CoAP payload of more than 1024 bytes; adding
> in CoAP options (the total size of which depends a lot on the length of the