Subject: Re: [Cbor] Predicting the size of a CBOR byte string

From: Francesca Palombini <[email-masked]>
Date: Thu, 5 Oct 2017 10:59:32 +0000

Hi,

(chair hat off) as a user of CBOR, I also have come up with this question, mostly because from CBOR's own nature, it is interesting to know what's the upper bound overhead that CBOR encoding can add in particular cases. I could definitely see something about that in the document, maybe even just as an appendix.

What does everybody else think?

Francesca

> -----Original Message-----
> From: CBOR [mailto:[email-masked]] On Behalf Of Carsten Bormann
> Sent: den 12 augusti 2017 09:02
> To: Brian E Carpenter <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Cbor] Predicting the size of a CBOR byte string
> 
> Hi Brian,
> 
> > On Aug 12, 2017, at 06:40, Brian E Carpenter
> <[email-masked]> wrote:
> >
> > Hi,
> >
> > I have a practical question. Does anyone have any tips for predicting
> > the size of a CBOR byte string in advance?
> 
> Are you talking about a single CBOR byte string and its serialization (=
> encoded data item)?
> 
> The (definite length) serialization is the size of the byte string, which becomes
> the body of the data item, plus the size of the head, which is
> 
> 1 for 0..23 bytes
> 2 for 24..255 bytes
> 3 for 256..65535 bytes
> 5 for 2^16 to 2^32-1 bytes
> 9 for 2^32 to 2^64-1 bytes
> 
> > Yes, I realise that looks like a silly question, so I'd better
> > explain. In some contexts there may be a practical limit on the size
> > of CBOR byte string that can be sent. (The GRASP protocol, for
> > example, has a maximum allowed message size.)
> >
> > So, it would be very useful before starting to encode a series of
> > objects into CBOR to be able to predict the resulting number of bytes
> > (or more realistically, an upper bound).
> 
> Hmm, maybe you aren’t talking about a single byte string.
> 
> The size of a encoded CBOR data item is the size of the heads + the size of
> the bodies.
> The heads are one byte (initial byte) plus any additional bytes for exceeding
> 23 bytes/data items (see above).
> The bodies are the data themselves.
> 
> So, essentially, add all string sizes (text string, byte string), add one byte for
> each node in the tree, and then add additional bytes for any nodes that are
> "larger than 23” (integer size, number of bytes, number of data items, tag
> number, floating point value).
> 
> >
> > (I'm quite aware that doing the actual encoding is the only reasonable
> > way to get an exact answer.)
> 
> An encoder could provide a less expensive counting mechanism that
> exercises the serialization mechanism without actually creating an output,
> just producing a byte count.
> 
> > Of course it isn't hard to invent a rule of thumb for a particular
> > environment, but I've noticed that (in Python 3) the ratio between
> > Python object size and CBOR size is different between my Windows and
> > my Linux machines.
> 
> The CBOR objects are the same in both cases, no?
> How do you measure what you call the “Python object size”?
> (Obviously, there are also 32-bit and 64-bit implementations of Python and
> many other languages, which will differ in the size of internal
> representations.)
> 
> Grüße, Carsten
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Jim Schaad <[email-masked]>
Date: Thu, 5 Oct 2017 09:17:41 -0700

Personally I don't need this, but I am use to computing this type of thing for other types of encodings so this is nothing new to me.  Other than that I don't really have an opinion.

Jim

> From: CBOR [mailto:[email-masked]] On Behalf Of Francesca
> Palombini
> Sent: Thursday, October 5, 2017 4:00 AM
> To: Carsten Bormann <[email-masked]>; Brian E Carpenter
> <[email-masked]>
> Cc: [email-masked]
> (chair hat off) as a user of CBOR, I also have come up with this question,
> mostly because from CBOR's own nature, it is interesting to know what's the
> upper bound overhead that CBOR encoding can add in particular cases. I
> could definitely see something about that in the document, maybe even just
> as an appendix.
> 
> > > I have a practical question. Does anyone have any tips for
> > > predicting the size of a CBOR byte string in advance?
> >
> > Are you talking about a single CBOR byte string and its serialization
> > (= encoded data item)?
> >
> > The (definite length) serialization is the size of the byte string,
> > which becomes the body of the data item, plus the size of the head,
> > which is
> >
> > 1 for 0..23 bytes
> > 2 for 24..255 bytes
> > 3 for 256..65535 bytes
> > > objects into CBOR to be able to predict the resulting number of
> > > bytes (or more realistically, an upper bound).
> >
> > The size of a encoded CBOR data item is the size of the heads + the
> > size of the bodies.
> > So, essentially, add all string sizes (text string, byte string), add
> > one byte for each node in the tree, and then add additional bytes for
> > any nodes that are "larger than 23” (integer size, number of bytes,
> > number of data items, tag number, floating point value).
> >
> > >
> > > (I'm quite aware that doing the actual encoding is the only
> > > reasonable way to get an exact answer.)
> >
> > exercises the serialization mechanism without actually creating an
> > output, just producing a byte count.
> >
> > (Obviously, there are also 32-bit and 64-bit implementations of Python
> > and many other languages, which will differ in the size of internal
> > representations.)
> >
> > Grüße, Carsten
> >

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 6 Oct 2017 09:38:07 +1300

In my own code where this issue came up, I tried a pragmatic
approach, namely application layer fragmentation of the data object
when it got to be a bit too big in Python terms. But that doesn't
work reliably because Python object "size" is a slippery concept.
So I ended up using len(cbor.dumps(object)) anyway, which makes for
redundant calls to dumps().

Regards
   Brian

---

From: Jim Schaad <[email-masked]>
Date: Thu, 5 Oct 2017 14:25:46 -0700

That is the same basic approach that I have always used, but I generally have a flag which does just the sizing and does not fill the buffer as I go.

Jim

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Thursday, October 5, 2017 1:38 PM
> To: Jim Schaad <[email-masked]>; 'Francesca Palombini'
> <[email-masked]>; 'Carsten Bormann' <[email-masked]>
> Cc: [email-masked]
> In my own code where this issue came up, I tried a pragmatic approach,
> namely application layer fragmentation of the data object when it got to be a
> bit too big in Python terms. But that doesn't work reliably because Python
> object "size" is a slippery concept.
> > Personally I don't need this, but I am use to computing this type of thing for
> other types of encodings so this is nothing new to me.  Other than that I
> don't really have an opinion.
> >
> > Jim
> >
> >
> >> (chair hat off) as a user of CBOR, I also have come up with this
> >> question, mostly because from CBOR's own nature, it is interesting to
> >> know what's the upper bound overhead that CBOR encoding can add in
> >> particular cases. I could definitely see something about that in the
> >> document, maybe even just as an appendix.
> >>
> >>> Are you talking about a single CBOR byte string and its
> >>> serialization (= encoded data item)?
> >>>
> >>>> explain. In some contexts there may be a practical limit on the
> >>>> size of CBOR byte string that can be sent. (The GRASP protocol, for
> >>> So, essentially, add all string sizes (text string, byte string),
> >>> add one byte for each node in the tree, and then add additional
> >>> bytes for any nodes that are "larger than 23” (integer size, number
> >>> of bytes, number of data items, tag number, floating point value).
> >>>
> >>>>
> >>>> Python object size and CBOR size is different between my Windows
> >>>> and my Linux machines.
> >>>
> >>> (Obviously, there are also 32-bit and 64-bit implementations of
> >>> Python and many other languages, which will differ in the size of
> >>> internal
> >>> representations.)
> >>>
> >>> Grüße, Carsten
> >>>