Subject: [Cbor] Predicting the size of a CBOR byte string

From: Brian E Carpenter <[email-masked]>
Date: Sat, 12 Aug 2017 16:40:40 +1200

Hi,

I have a practical question. Does anyone have any tips for
predicting the size of a CBOR byte string in advance?

Yes, I realise that looks like a silly question, so I'd better
explain. In some contexts there may be a practical limit on the
size of CBOR byte string that can be sent. (The GRASP protocol,
for example, has a maximum allowed message size.)

So, it would be very useful before starting to encode a series
of objects into CBOR to be able to predict the resulting number
of bytes (or more realistically, an upper bound).

(I'm quite aware that doing the actual encoding is the only
reasonable way to get an exact answer.)

Of course it isn't hard to invent a rule of thumb for a particular
environment, but I've noticed that (in Python 3) the ratio between
Python object size and CBOR size is different between my Windows and my
Linux machines. So my rule of thumb isn't portable.

Any ideas?
 
Regards
   Brian

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 12 Aug 2017 09:02:15 +0200

Hi Brian,

> On Aug 12, 2017, at 06:40, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi,
> 

Are you talking about a single CBOR byte string and its serialization (= encoded data item)?

The (definite length) serialization is the size of the byte string, which becomes the body of the data item, plus the size of the head, which is

1 for 0..23 bytes
2 for 24..255 bytes
3 for 256..65535 bytes
5 for 2^16 to 2^32-1 bytes
9 for 2^32 to 2^64-1 bytes

> of bytes (or more realistically, an upper bound).

Hmm, maybe you aren’t talking about a single byte string.

The size of a encoded CBOR data item is the size of the heads + the size of the bodies.
The heads are one byte (initial byte) plus any additional bytes for exceeding 23 bytes/data items (see above).
The bodies are the data themselves.

So, essentially, add all string sizes (text string, byte string), add one byte for each node in the tree, and then add additional bytes for any nodes that are "larger than 23” (integer size, number of bytes, number of data items, tag number, floating point value).

> reasonable way to get an exact answer.)

An encoder could provide a less expensive counting mechanism that exercises the serialization mechanism without actually creating an output, just producing a byte count.

> Linux machines.

The CBOR objects are the same in both cases, no?
How do you measure what you call the “Python object size”?
(Obviously, there are also 32-bit and 64-bit implementations of Python and many other languages, which will differ in the size of internal representations.)

Grüße, Carsten