Subject: Re: [apps-discuss] High-volume benchmark (was Re: cbor-05)

From: Carsten Bormann <[email-masked]>
Date: Mon, 9 Sep 2013 17:53:51 +0200

On Sep 9, 2013, at 17:44, "Pierre Thierry" <[email-masked]> wrote:

> Carsten,
> 
> in draft-bormann-cbor-05, you mention that "The format must be
> applicable to (…) high-volume applications." in section 1.1.
> 
> Do you already have a benchmark in place to measure the volume that a
> CBOR implementation is able to manage? Do you plan to compare CBOR
> implementations to implementations of other similar formats?

Hi Pierre,

of course, I do have some informal benchmarks for my implementations; their purpose is to avoid performance regressions when evolving the code.

However, it has been my experience that, unless you make mistakes in defining the format (e.g., requiring extensive data conversions, two-pass implementations, or back-patching) the actual performance of a data format encoder/decoder is largely dependent on system issues, such as the data model you are using and the memory allocation principles.  So I'm not sure a formal benchmark would help a lot.

What kind of application do you have in mind? Maybe I can come up with a better, more tailored benchmark for that.

Grüße, Carsten

---

From: "Pierre Thierry" <[email-masked]>
Date: Mon, 9 Sep 2013 17:44:33 +0200

Carsten,

Curiously,
Pierre Thierry
-- 
[email-masked]
OpenPGP OxD9D50D8A

---

From: Pierre Thierry <[email-masked]>
Date: Mon, 23 Sep 2013 23:49:38 +0200

Scribit Carsten Bormann dies 09/09/2013 hora 17:53:
> What kind of application do you have in mind? Maybe I can come up
> with a better, more tailored benchmark for that.

I actually don't, I'm wondering if formats like CBOR, BULK or BSON
would fare differently and how. I suspect that a high-volume benckmark
should include trivial calculations to ensure correctness but that it
also should avoid storing any data to avoid storage to interfere with
measurements. Although to be fair, multiple benchmarks would probably
be necessary: at least pure reading speed (calculate only) and
throughput (where the program both reads and writes serialized data).

Curiously,
Pierre Thierry
-- 
[email-masked]
OpenPGP 0xD9D50D8A