Subject: [core] SenML and delta encoding for samples

From: Julien Vermillard <[email-masked]>
Date: Mon, 02 Feb 2015 19:45:39 +0000

Hi,
I'm taking a look at SenML with a CBOR encoding for replacing a proprietary
representation for telemetry values.

For now it's fitting my needs but I'm just surprised about the lack of
delta encoding for values (it's present for timestamp only).

Here my use case:

We send large number of samples of vehicle sensors:
time, latitude, longitude, altitude, accelerometer (x, y, z) and some extra
sensors. Everything is sampled at 10 Hz, so the difference between each
sample can be minimal. Smaller the delta value is, smaller the json or cbor
encoding would be.

The file can be really large (megabytes), so delta encoding save a lot of
space (30% for a protocol buffer based encoding).

A crude example, of delta encoding support for samples could look likes:

{"e":[
        { "n": "sensorA", "dv":20 },
        { "n": "sensorB", "t": -5, "dv": 0.2 },
        { "n": "sensorB", "t": -4, "dv": 0.30 },
        { "n": "sensorA", "t": -3, "dv": -10.14 },
        { "n": "sensorB", "t": -2, "dv": 0.5 },
        { "n": "sensorA", "t": -1, "dv": 20.6 },
        { "n": "sensorB", "t": 0,   "dv": 0.7 }],
    "bn": "urn:dev:mac:0024befffe804f/",
    "bt": 1276020076,
    "ver": 1,
    "bv" : { "sensorA" : 1000,
              "sensorB": 250} }

vs

{"e":[
        { "n": "sensorA", "dv":1020 },
        { "n": "sensorB", "t": -5, "dv": 250.2 },
        { "n": "sensorB", "t": -4, "dv": 250.30 },
        { "n": "sensorA", "t": -3, "dv": 990.14 },
        { "n": "sensorB", "t": -2, "dv": 250.5 },
        { "n": "sensorA", "t": -1, "dv": 1020.6 },
        { "n": "sensorB", "t": 0,   "dv": 250.7 }],
    "bn": "urn:dev:mac:0024befffe804f/",
    "bt": 1276020076,
    "ver": 1
}

I can provide an example based on real world samples if it helps.

By the way putting "e" at the end of the document would help streaming and
reduce memory consumption, I wonder why all the samples are this way in the
RFC, it could be misleading.

Julien

---

From: Christian Amsüss <[email-masked]>
Date: Tue, 24 Feb 2015 18:45:57 +0100

Hello Julien,

I am working with similar data and have not come up with a good solution
yet.

>     "bv" : { "sensorA" : 1000,

Such a scheme would reduce the size of my transmissions as well; my
example 271k uncompressed file (no timestamps) goes down to 248k with
the established bt mechanism, to 217k with dt (see below), to 210k with
dt and dv, to 188k when absence of dv implies dv=0 (samples arrive
irregularly, so the presence of a record does matter in my application),
and down to 130k when I compress names to digits.

For CBOR, I'm getting 207k / 207k / 212k / 217k / 178k / 124k, but the
numbers show that the implementation I used for testing (the Python
version from [1]) does not apply number compression.

The number compression part is quite relevant, actually, because
numerics matter here; a delta compression that produces small numbers in
JSON might produce even bigger representations in CBOR ("0.1" vs a 64bit
float with tag), so much care would have to be taken when doing those
calculations. Unless one wants to deal with lossy compression at
transmission time (which would be neigh impossible to disambiguate for
duplicates or similar afterwards), that means that the numeric values
have to be designed for compressibility beforehand; once a number like
0.1d shows up there, it will always compress badly with those schemes.

If the dv scheme does work out to good savings using appropriate number
representations, it should be considered for t as well, as the savings
of the time compression decrease when a SenML file spans several weeks.
Storing the time in a "dt" field (which would in your example always
have the value of 1) would be trivial then.

> RFC, it could be misleading.

The CBOR RFC does allow "CBOR-based protocol [to] prescribe a specific
order of serialization", I'm not sure about JSON.

I don't see the benefit of a "bv" here; the delta compression could
always start from zero and have its initial value in the first entry of
e. Same goes for bt if the proposed dt is used. What's left in common
use is bn -- somehow dealing with names would give the best results with
your and my data sets. A solution might look like that:

{"n": {0: "sensorA", 1: "sensorB"},
 "e": [{"n":0, dv:20}, {"n":1, "t":-5, "dv": 0.2}]}

Would that still be in the spirit of SenML?

Best regards
Christian

[1] https://code.google.com/p/cbor/

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: "Peintner, Daniel (ext)" <[email-masked]>
Date: Wed, 25 Feb 2015 09:19:37 +0000

Hi Christian, Julien, all,

If message size matters I think EXI is a good alternative.
The original data Julien provided can be represented with the according XML schema in about 100 Bytes.

You can give it a try [1],

-- Daniel

[1] exificient.sourceforge.net/exificient.jnlp

-----Ursprüngliche Nachricht-----
Von: core [mailto:[email-masked]] Im Auftrag von Christian Amsüss
Gesendet: Dienstag, 24. Februar 2015 18:46
An: Julien Vermillard
Cc: [email-masked]
Betreff: Re: [core] SenML and delta encoding for samples

Hello Julien,

I am working with similar data and have not come up with a good solution yet.

>     "bv" : { "sensorA" : 1000,

Such a scheme would reduce the size of my transmissions as well; my example 271k uncompressed file (no timestamps) goes down to 248k with the established bt mechanism, to 217k with dt (see below), to 210k with dt and dv, to 188k when absence of dv implies dv=0 (samples arrive irregularly, so the presence of a record does matter in my application), and down to 130k when I compress names to digits.

For CBOR, I'm getting 207k / 207k / 212k / 217k / 178k / 124k, but the numbers show that the implementation I used for testing (the Python version from [1]) does not apply number compression.

The number compression part is quite relevant, actually, because numerics matter here; a delta compression that produces small numbers in JSON might produce even bigger representations in CBOR ("0.1" vs a 64bit float with tag), so much care would have to be taken when doing those calculations. Unless one wants to deal with lossy compression at transmission time (which would be neigh impossible to disambiguate for duplicates or similar afterwards), that means that the numeric values have to be designed for compressibility beforehand; once a number like 0.1d shows up there, it will always compress badly with those schemes.

If the dv scheme does work out to good savings using appropriate number representations, it should be considered for t as well, as the savings of the time compression decrease when a SenML file spans several weeks.
Storing the time in a "dt" field (which would in your example always have the value of 1) would be trivial then.

> By the way putting "e" at the end of the document would help streaming 
> and reduce memory consumption, I wonder why all the samples are this 
> way in the RFC, it could be misleading.

The CBOR RFC does allow "CBOR-based protocol [to] prescribe a specific order of serialization", I'm not sure about JSON.

I don't see the benefit of a "bv" here; the delta compression could always start from zero and have its initial value in the first entry of e. Same goes for bt if the proposed dt is used. What's left in common use is bn -- somehow dealing with names would give the best results with your and my data sets. A solution might look like that:

Best regards
Christian

---

From: Julien Vermillard <[email-masked]>
Date: Wed, 25 Feb 2015 13:11:17 +0000

> version from [1]) does not apply number compression.

Number compression is critical, I use integers for a better result.
Maybe a solution is a base value and a multiplication factor in the header
and only integer for the sample delta values.

> JSON might produce even bigger representations in CBOR ("0.1" vs a 64bit

0.1 as 64bit float is probably not a good idea,
CBOR support 16/32/64bit floating number precisions.

> and
> e.

Same goes for bt if the proposed dt is used. What's left in common
> use is bn --

Good idea, it would be simpler than what I proposed.

> somehow dealing with names would give the best results with

I don't know.. of course it would save a lot of bytes, but maybe a second
pass with gzip/huffman-coding would achieve the same result and keep the
format in the spirit of SenML?
Anyway I think you are right, does compression feature should be part of
SenML or no ? if no, we should also ditch the delta time feature.

Julien