Subject: [core] Fwd: New Version Notification for draft-jennings-core-senml-01.txt

From: Ari Keränen <[email-masked]>
Date: Mon, 6 Jul 2015 23:11:19 +0300

Hi all,

We have now submitted an updated version of the SenML draft (see the 
announcement below).

The main changes from previous revision:
- Added CBOR representation format (thanks Carsten!)
- Added recommendation for the order of the JSON attributes
- Added text how the JSON format can be used in a stream-like fashion
- Removed 'minOccurs="0"' from XML schema (is not allowed in attributes)
- Lots of smaller editorial fixes and changes

Comments and reviews are welcome.

Cheers,
Ari

-------- Forwarded Message --------
Subject: New Version Notification for draft-jennings-core-senml-01.txt
Date: Mon, 6 Jul 2015 12:57:00 -0700
From: [email-masked]

A new version of I-D, draft-jennings-core-senml-01.txt
has been successfully submitted by Ari Keranen and posted to the
IETF repository.

Name:		draft-jennings-core-senml
Revision:	01
Title:		Media Types for Sensor Markup Language (SENML)
Document date:	2015-07-06
Group:		Individual Submission
Pages:		26
URL: 
https://www.ietf.org/internet-drafts/draft-jennings-core-senml-01.txt
Status:         https://datatracker.ietf.org/doc/draft-jennings-core-senml/
Htmlized:       https://tools.ietf.org/html/draft-jennings-core-senml-01
Diff: 
https://www.ietf.org/rfcdiff?url2=draft-jennings-core-senml-01

Abstract:
    This specification defines media types for representing simple sensor
    measurements and device parameters in the Sensor Markup Language
    (SenML).  Representations are defined in JavaScript Object Notation
    (JSON), Concise Binary Object Representation (CBOR), eXtensible
    Markup Language (XML), and Efficient XML Interchange (EXI), which
    share the common SenML data model.  A simple sensor, such as a
    temperature sensor, could use this media type in protocols such as
    HTTP or CoAP to transport the measurements of the sensor or to be
    configured.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Ari Keränen <[email-masked]>
Date: Tue, 7 Jul 2015 00:14:00 +0300

Hi all,

One of the main open issues with the SenML draft is whether we should 
add support for nested elements and inheriting values.

Currently, if one wants to send e.g. multiple current and voltage 
values, something like this is done:

    {"bn": "urn:dev:mac:0024befffe804ff1/",
     "bt": 1276020076,
     "bu": "A",
     "e":[
         { "n": "voltage", "t": -5, "u": "V", "v": 120.1 },
         { "n": "voltage", "t": -3, "u": "V", "v": 120.4 },
         { "n": "voltage", "t": -1, "u": "V", "v": 120.5 },
         { "n": "voltage", "t": 0,  "u": "V", "v": 120.1 },
         { "n": "current", "t": -4, "v": 1.30 },
         { "n": "current", "t": -3, "v": 0.14e1 },
         { "n": "current", "t": -2, "v": 1.5 },
         { "n": "current", "t": -1, "v": 1.6 },
         { "n": "current", "t": 0, "v": 1.7 }]
    }

With a large batch of measurements repeating the name creates quite a 
bit of overhead and also it might be valuable to be able to define 
multiple base/default units.

One possible solution would be to allow nested elements. Something like:

    {"bn": "urn:dev:mac:0024befffe804ff1/",
     "bt": 1276020076,
     "nested":[{
  	"bu": "V",
         "bn": "voltage",
         "e":[
           { "t": -5, "v": 120.1 },
           { "t": -3, "v": 120.4 },
           { "t": -1, "v": 120.5 },
           { "t": 0,  "v": 120.1 }]
         },
         {
         "bu": "A",
         "bn": "current",
         "e":[
           { "t": -4, "v": 1.30 },
           { "t": -3, "v": 0.14e1 },
           { "t": -2, "v": 1.5 },
           { "t": -1, "v": 1.6 },
           { "t": 0, "v": 1.7 }]
         }
       ]
     }

For long batches of measurements this could save considerable amount of 
bandwidth. But as a downside this complicates SenML generation and 
parsing quite a bit.

Would this added complexity be worth it? And/or is there a better way to 
achieve the same result?

Cheers,
Ari

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 07 Jul 2015 00:06:39 +0200

> - Added recommendation for the order of the JSON attributes

Hmm.  This means

-- If I want to heed that recommendation, I can't use standard JSON
implementations, because map order typically can't be influenced.

-- As a consumer, I still can't rely on this order (it is just a
recommendation), so nothing is gained.

Lots of downside, no upside.
Maybe I don't understand this recommendation.

Grüße, Carsten

---

From: Ari Keränen <[email-masked]>
Date: Tue, 7 Jul 2015 01:21:18 +0300

> implementations, because map order typically can't be influenced.

Yes, that's why it's just a recommendation.

However, how do the implementations folks use for constrained devices 
currently work? I know the one I did keeps the order since I used simple 
structures (i.e., nothing fancy like hash maps) to keep the 
implementation simple.

> Maybe I don't understand this recommendation.

You don't necessarily need to rely on the order to get benefits. The key 
benefit is that with a simple parser you can calculate full name and 
time based on base name and time while parsing the data. If the bn and 
bt are in the end, you don't know the result before the end of the 
structure. You can live with that, but it's just more extra cycles you 
need to use.

Cheers,
Ari

---

From: Andrew Mcgregor <[email-masked]>
Date: Tue, 7 Jul 2015 10:02:21 +1000

How about:
   {"bn": "urn:dev:mac:0024befffe804ff1/",
    "bt": 1276020076,
    "bu": "A",
    "e":[
        { "n": "voltage", "t": [-5, -3, -1, 0], "u": "V", "v": [120.1,
120.4, 120.5, 120.1] },
        { "n": "current", "t": [-4, -3, -2, -1, 0], "v": [1.30, 0.14e1,
1.5, 1.6, 1.7] },
   }

Even more compact, and probably easier to parse. If you require the value
and time series to have a different name when they are lists, what about:
   {"bn": "urn:dev:mac:0024befffe804ff1/",
    "bt": 1276020076,
    "bu": "A",
    "e":[
        { "n": "voltage", "ts": [-5, -3, -1, 0], "u": "V", "vs": [120.1,
120.4, 120.5, 120.1] },
        { "n": "current", "ts": [-4, -3, -2, -1, 0], "vs": [1.30, 0.14e1,
1.5, 1.6, 1.7] },
   }

On 7 July 2015 at 07:14, Ari Keränen <[email-masked]> wrote:

> Hi all,
>
> One of the main open issues with the SenML draft is whether we should add
> support for nested elements and inheriting values.
>
> Currently, if one wants to send e.g. multiple current and voltage values,
> something like this is done:
>
> With a large batch of measurements repeating the name creates quite a bit
> of overhead and also it might be valuable to be able to define multiple
> base/default units.
>
> bandwidth. But as a downside this complicates SenML generation and parsing
> quite a bit.
>
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 07 Jul 2015 14:47:40 +0200

> structures (i.e., nothing fancy like hash maps) to keep the

Right.  Implementers on the constrained side will have few problems with
this.  If the assumption is that SenML will mainly be originated by
embedded implementations, it may be possible to do this.

> structure. You can live with that, but it's just more extra cycles you

That assumes that the actual CPU used here matters (or what do you mean
by "cycles"?).  On a server platform, I'll get the whole map handed to
me anyway.  On an embedded platform, I may not even have space for two
code paths, one that handles the efficient case and one for the general
case.

So I still think requesting this is a bit questionable, because it asks
for something JSON generally can't do.

If we really want to make this into a feature, we need to go for arrays
(in "record" style, see
https://tools.ietf.org/html/draft-greevenbosch-appsawg-cbor-cddl-06#section-2).

I.e., turn

{"head1":"value1", "head2":"value2", "e" : [...]}

into

[{"head1":"value1", "head2":"value2"}, [...]]

(or even

[{"head1":"value1", "head2":"value2"}, ...]

if we want to superoptimize this).

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 8 Jul 2015 12:09:06 +0200

Hello,

On Tue, Jul 07, 2015 at 01:21:18AM +0300, Ari Keränen wrote:
> structures (i.e., nothing fancy like hash maps) to keep the implementation
> simple.

my embedded version currently throws an error when trailing "bn" values
are encountered. It could be easily made to parse them in a single-block
request, but for updates received in blockwise mode that's out of the
question.

On Tue, Jul 07, 2015 at 02:47:40PM +0200, Carsten Bormann wrote:
> [...]

This would make implementing a correct SenML parser on a constrained
system *much* easier, and might make other extensions easier as well
(see "Nested elements for SenML" thread, switching there to stay on
topic).

I'd prefer this form over the form without a second list ([{...}, e1,
e2, ...]), as it is clearer and only costs 1-2 bytes.

Best regards
Christian

-- 
Christian Amsüss                      | Energy Harvesting Solutions GmbH
founder, system architect             | headquarter:
mailto:[email-masked]  | Arbeitergasse 15, A-4400 Steyr
tel:+43-664-97-90-6-39                | http://www.energyharvesting.at/
                                      | ATU68476614

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 8 Jul 2015 12:15:01 +0200

Hello Ari,

if we follow the headers-first syntax suggested by Carsten[1], there
might be a way that is flexible in allowing all header attributes to be
batched, but avoids the complexity of nesting:

On Tue, Jul 07, 2015 at 12:14:00AM +0300, Ari Keränen wrote:
>         "e":[...1...]
>         "e":[...2...]

could be represented like this:

[
  {"bn": "urn:dev:mac:0024befffe804ff1/voltage",
   "bt": 1276020076,
   "bu": "V"
   },
  [...1...],
  {"bn": "current",
   "bu": "A"
   },
  [...2...]
]

The semantics of the later headers would be updating the former. That
should be comparatively easy to parse without the need for memory
management or nesting depth limits.

Best regards
Christian

[1] https://www.ietf.org/mail-archive/web/core/current/msg06254.html

---

From: Ari Keränen <[email-masked]>
Date: Wed, 8 Jul 2015 18:07:23 +0300

> embedded implementations, it may be possible to do this.

Yes, or if you are in control of both endpoints. I'm thinking more along 
the lines of "if it's easy to do so, do this".

> code paths, one that handles the efficient case and one for the general

CPU cycles and also memory. If you can have the full values while 
parsing the structure you can (potentially) act accordingly and then 
forget all the parts you have already processed. If you get the base 
values only in the end, you can't do that; or you need to parse the 
structure twice to do that.

But good point regarding the two code paths. I'm thinking it would be 
only a few extra lines of c-code to take advantage of the optimized way, 
but would probably need to implement it to know for sure. And anyway 
non-optimized code would work for each approach equally well -- no harm 
done.

> if we want to superoptimize this).

That might be an option too. What are the downsides of this approach?

Cheers,
Ari

---

From: Ari Keränen <[email-masked]>
Date: Wed, 8 Jul 2015 18:17:31 +0300

Hi,

> request, but for updates received in blockwise mode that's out of the

Do you mean that you currently support *only* base values in the beginning?

And block-wise transfer indeed complicates things even further since it 
may take a while to receive the base values if they are in the very end 
of a large SenML.

> (see "Nested elements for SenML" thread, switching there to stay on

Could you please elaborate a bit how this makes parser easier? Do you 
mean because of the fixed ordering?

Thanks,
Ari

---

From: Ari Keränen <[email-masked]>
Date: Wed, 8 Jul 2015 18:25:13 +0300

Hi Christian,

>    {"bn": "urn:dev:mac:0024befffe804ff1/voltage",

That looks like a reasonable approach to me. Although, in the example 
above, I guess we would need full "bn" also before the second 
measurement array.

Cheers,
Ari

---

From: Ari Keränen <[email-masked]>
Date: Wed, 8 Jul 2015 18:33:41 +0300

Thanks Andrew. This is pretty interesting too. I wonder what's the 
implementation complexity of this compared to the Christian's proposal?

And of course it also depends a bit whether we do Carsten's proposal for 
the headers-first issue.

Cheers,
Ari

On 07/07/15 03:02, Andrew Mcgregor wrote:
> Even more compact, and probably easier to parse. If you require the
> value and time series to have a different name when they are lists, what
> about:
>          { "n": "voltage", "ts": [-5, -3, -1, 0], "u": "V", "vs":
> [120.1, 120.4, 120.5, 120.1] },
>          { "n": "current", "ts": [-4, -3, -2, -1, 0], "vs": [1.30,
> 0.14e1, 1.5, 1.6, 1.7] },
>     }
>
> On 7 July 2015 at 07:14, Ari Keränen <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>     One of the main open issues with the SenML draft is whether we
>     should add support for nested elements and inheriting values.
>
>     With a large batch of measurements repeating the name creates quite
>     a bit of overhead and also it might be valuable to be able to define
>     For long batches of measurements this could save considerable amount
>     of bandwidth. But as a downside this complicates SenML generation
>     and parsing quite a bit.
>
>     Would this added complexity be worth it? And/or is there a better
>     way to achieve the same result?
>
>
>     Cheers,
>     Ari
>
>     [email-masked] <mailto:[email-masked]>
> Andrew McGregor | SRE [email-masked]
> <mailto:[email-masked]> | +61 4 1071 2221

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 08 Jul 2015 17:39:38 +0200

Maybe the easiest way to phrase the semantics is that each map in that
top-level array is applied to the previous state of the variables as a
merge-patch (RFC 7386)?  (We could still allow nesting, where making a
copy of that state is helpful.)

On a meta level, one thing we'll need to understand is how much change
we want to absorb -- SenML as defined today is in use in various places.
 I'm not saying major change is wrong, all I'm saying is that this
should be done as a conscious decision.  Of course, we can also "pull a
COSE" and decide to use a new format for CBOR and stay with the old
format for JSON, but that also would need to be a conscious decision*).

Grüße, Carsten

*) In COSE it was easy to make this decision because the crypto aspect
made sure that there is no easy conversion between COSE and JOSE.  Here,
there probably is. Obviously, the conversion could be required to do
more than just JSON <-> COSE and some map key transcoding.

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 9 Jul 2015 08:52:59 +0200

Hello Ari,

> Do you mean that you currently support *only* base values in the beginning?

Yes, it only supports that ordering. That is not conformant to the
draft, but for the header values i'm actually using ("bt", not "v") is
easy to achieve even when using a full-featured JSON library (eg. using
Python's sort_keys=True, which sorts "bn", "bt", "e", "v" and is thus
happens to work for the time being).

> Could you please elaborate a bit how this makes parser easier? Do you mean
> because of the fixed ordering?

With arbitrary sequences, I have to parse twice through a structure I
can't hold in memory. With fixed sequence (no matter how achieved), I
can store the relevant "b" values in one data structure, update another
to always contain the latest "e" element, and call back interested
parties (eg. resources in the batch)  all in one pass.

Best regards
Christian

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 9 Jul 2015 09:28:44 +0200

Hi Ari, hallo Carsten,

On Wed, Jul 08, 2015 at 06:25:13PM +0300, Ari Keränen wrote:
> [ data about `urn:dev:mac:0024befffe804ff1/voltage` and `/current` ]
>
> That looks like a reasonable approach to me. Although, in the example above,
> I guess we would need full "bn" also before the second measurement array.

That, depends on the semantics of "bn". I've always seen "bn" like
HTML's `base href=` or turtle's `@base`, updating the current base URI
(initialized with the resource address) as relative URIs -- but that's
being discussed in [1].

On Wed, Jul 08, 2015 at 05:39:38PM +0200, Carsten Bormann wrote:
> merge-patch (RFC 7386)? 

That would rule out URI joining from one header group to the next, but
if the resulting latest "bn" is always applied relative to the
document's URI.

In the above example, that would mean duplicating that base URI, but I'd
be OK with it as it for sake of reusing both RFCs and code components.

> (We could still allow nesting, where making a copy of that state is
> helpful.)

That would enable deduplication of the base URI again. We'd stil have to
deal with nesting limitation and figure out a suitable syntax -- if
there are use cases for it, why not.

> we want to absorb -- SenML as defined today is in use in various places.

If this were only about nesting, I'd be more cautious about this, but
with the issue of header sequence, existing implementations are either
incomplete (like mine, requiring a sequence that is not specified or can
not be provided by all implementations easily) or much more complex than
they need to be (for parsing multi-block SenML with headers at the end).
I don't see how this can be fixed in a backwards compatible way.

At least, with the [header, events] format, it would be immediately
obvious that a file is senml-01, and if parsers need to be compatible in
a transition period, they can switch on the first character.

Best regards
Christian

[1] https://mailarchive.ietf.org/arch/msg/core/I-dxnrJg2tvcY4LMv4sgSm8pWRU