Subject: [core] SenML JSON syntax (with multiple base objects)

From: Ari Keränen <[email-masked]>
Date: Tue, 20 Oct 2015 17:06:33 +0000

Hi CoRE WG folks,

Final SenML open issue for today is whether we should have simpler format for the JSON. Currently the draft uses nested array for the measurement entries as discussed in the Prague meeting (see for example [1]). However, it could be simpler to use just a single array and have entry and base objects at the same level. 

For example:

[
 {"bn": "urn:dev:mac:0024befffe804ff1/current",
 "bt": 1276020076,
 "bu": "A"},
  {  "t": -3, "v": 0.14e1 },
  {  "t": -2, "v": 1.5 },
  {  "t": -1, "v": 1.6 },
  {  "t": 0,  "v": 1.7 },
 {"bn": "urn:dev:mac:0024befffe804ff1/voltage", "bu":"V" },
  {  "t": -2, "v": 120.05 },
  {  "t": -1, "v": 120.1 },
]

The base objects would use the same JSON merge patch format and logic as before, but all entry objects would be simply inside the root array, potentially intermixed with base objects.

One challenge with this format is that when parsing, one does not immediately know whether it is a base object or entry object, but this should be possible to discover from the member names ("b*"). Of course to accommodate for new base and entry members, we would need to have logic that makes the difference clear even if there are unknown members in the objects (for example, "only b* member names allowed in base object and no b* in entries").

Seems sensible to me. What do you think?

Cheers,
Ari

[1] https://tools.ietf.org/html/draft-jennings-core-senml-01#section-6.1.2

---

From: Michael Koster <[email-masked]>
Date: Tue, 20 Oct 2015 11:06:24 -0700

Hi Ari,

Is the proposed format simpler because it removes the need to have an “e” tag and instead associates a data object with a base object by position in the array?

I think having base parameters and data array wrapped together in an object has some advantages, particularly when the object represents a set of resources as opposed to a sequence of measurements from a base resource. In this case the base name is the object or collection name and the array represents the resources in the collection.

[
  { “bn”: ”/collection1/“,
    “e”: [
      {“n”: “item1”, “sv”, “value1”},
      {“n”: “item2”, “sv”, “value2”},
      {“n”: “item3”, “sv”, “value3”},
      {“n”: “item4”, “sv”, “value4”}
    ]
  }
]

This structure correctly represents the array elements as items within the base name of a collection.

Additionally, this format can be extended using the same mechanism by adding another tag for another class of element. For example, let’s say I wanted to extend senml to add descriptive metadata in the form of RFC6690 hyperlinks encoded in link-format+json. I could easily add a “l” element extension for links and create a new content-format which can be processed and understood as a simple extension of SenML:

[
  { “bn”: ”/collection1/“,
    “e”: [
      {“n”: “item1”, “sv”, “value1”},
      {“n”: “item2”, “sv”, “value2”},
      {“n”: “item3”, “sv”, “value3”},
      {“n”: “item4”, “sv”, “value4”}
    ]
    “l”: [
      {“href”: “item1”, “if”, “core.s”, “rt”: “type1”, “ct”: “50"},
      {“href”: “item2”, “if”, “core.s”, “rt”: “type2”, “ct”: “50"},
      {“href”: “item3”, “if”, “core.s”, “rt”: “type3”, “ct”: “50"},
      {“href”: “item4”, “if”, “core.s”, “rt”: “type4”, “ct”: “50"}
  }
]

For the use case of representing items in a collection or data object (e.g. OMA LWM2M, CoRE Inerfaces, OIC) I would prefer the current serialization.

Best regards,

Michael

On Oct 20, 2015, at 10:06 AM, Ari Keränen <[email-masked]> wrote:

> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Oct 2015 20:46:02 +0200

Michael Koster wrote:
> Is the proposed format simpler because it removes the need to have an
> “e” tag and instead associates a data object with a base object by
> position in the array?

Old SenML has the problem that there is no way in JSON (or CBOR, which
is modeled on JSON here) to ensure that the base values come before the
measurements.  This has to be an array unless it's OK to force an
implementation to keep the measurements until finally the base values
come through.

The "simplification" that has been proposed today is that the clear
delineation between base values and measurements would be lost and an
implementation has to examine each element of the array whether it is a
base object or a measurement.  I have no idea how that simplifies
things, except for the superficial property of having fewer brackets.
To the contrary, making the structure very explicit is a recipe that
allows very simple implementations.

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Tue, 20 Oct 2015 20:46:30 +0200

Hello Michael,

> “e” tag and instead associates a data object with a base object by

the main advantage of the [{base}, [elements]] serialization is that
constrained devices have a chance of streaming decoding. With the base
and element data contained in an object, it is impractical to prescribe
a sequence of serialization (as it precludes the use of common JSON
serializers whose data structures don't know order inside an object).

There has been some discussion at [1] and [2], which has led to the the
currently proposed syntax.

> I think having base parameters and data array wrapped together in an
> object has some advantages, particularly when the object represents a
> set of resources as opposed to a sequence of measurements from a base
> resource. In this case the base name is the object or collection name
> and the array represents the resources in the collection.

Granted, the grouping between the base name and the elements is not that
(visually) tight any more, but can still be expressed with the same base
name mechanisms. How is the multiple-resources case different from the
multiple-timestamps case here?

> Additionally, this format can be extended using the same mechanism by
> adding another tag for another class of element. For example, let’s
> say I wanted to extend senml to add descriptive metadata in the form
> of RFC6690 hyperlinks encoded in link-format+json. I could easily add
> a “l” element extension for links and create a new content-format
> which can be processed and understood as a simple extension of SenML:

That indeed has not come up in the previous discussion, and would not be
easily feasible with the new syntax.

In my opinion, it raises the question of how generic SenML should
attempt to be. My personal view of it is that SenML is a way of
encapsulating several resource representations (be they of different
points in time or different resource) in a single message. With that in
mind, maybe the following would work for you (rephrasing your example
into senml-02 syntax, with comments):

  [
    { “bn”: ”/collection1/“ }
    [ 
      /* the sv values are identical to what the client would receive
         when GETting the respective /collection1/item as text/plain */
      {“n”: “item1”, “sv”, “value1”},
      {“n”: “item2”, “sv”, “value2”},
      {“n”: “item3”, “sv”, “value3”},
      {“n”: “item4”, “sv”, “value4”},

      /* the "object value" is what the client would receive when
         GETting /collection1/ with mime type
         application/link-format+json */
      {"n": "", "ov": [
        {“href”: “item1”, “if”, “core.s”, “rt”: “type1”, “ct”: “50"},
        {“href”: “item2”, “if”, “core.s”, “rt”: “type2”, “ct”: “50"},
        {“href”: “item3”, “if”, “core.s”, “rt”: “type3”, “ct”: “50"},
        {“href”: “item4”, “if”, “core.s”, “rt”: “type4”, “ct”: “50"}
      ]}
    ]
  ]

That would need another data type in addition to string-, (numeric)- and
boolean value ("sv", "v", "bv") and thus allow encapsulating any other
JSON based response. Parsers in applications that don't expect "ov"
wouldn't even need to get more complicated, as they are already required
to ignore key-value pairs they don't understand.

This still doesn't give a way to add arrays in parallel to the previous
"e" element, but for this application, it would automatically give
streamability also to your "l" list.

What do you think of the above arrangement?

Kind regards
Christian

[1] https://www.ietf.org/mail-archive/web/core/current/msg06254.html
[2] http://recordings.conf.meetecho.com/Playout/watch.jsp?recording=IETF93_CORE_II&chapter=chapter_1 starting at 1:18 

-- 
Christian Amsüss                      | Energy Harvesting Solutions GmbH
founder, system architect             | headquarter:
mailto:[email-masked]  | Arbeitergasse 15, A-4400 Steyr
tel:+43-664-97-90-6-39                | http://www.energyharvesting.at/
                                      | ATU68476614

---

From: Michael Koster <[email-masked]>
Date: Tue, 20 Oct 2015 12:52:19 -0700

Thanks for the reply! My comments below.

Michael

> multiple-timestamps case here?

It’s more than a simple visual relationship. I’m used to JSON tools that create an in-memory data structure that conforms to the JSON serialization. With the “old” SenML model, the elements of the object identified by “bn” are rendered as an array within the element identified by “bn” and tagged by “e”. 

The new construct more than just enables streaming, it forces serial interpretation, i.e. it *requires* streaming. I need to transform the in-memory representation if I want it to look the same as the familiar one. Not that I can’t do that, but to illustrate that it’s more than just a visual preference thing. It’s a matter of “impedance match” between the software machine and the in-memory representation I get from a JSON destringifier.
> into senml-02 syntax, with comments):
SenML is already being used to represent simple collections in CoRE Interfaces, OMA LWM2M, and OIC. Whether to have it be extensible and evolvable or not is certainly a tradeoff against complexity and stream processing ability. I would lean toward evolvability. 

Would it make sense to create a new content-format that optimizes for streaming processing?

>        {“href”: “item4”, “if”, “core.s”, “rt”: “type4”, “ct”: “50"}

I was going to ask for “ov” for the ability to represent items that are themselves objects, so that’s a very useful feature for collections. However, representation of the links in a collection as a collection mapped to one of the items does not lead to a natural in-memory representation either. 

> What do you think of the above arrangement?
I think it’s a substantial compromise in the ability to represent data structure to get streaming processing ability. But I do like the idea of a “ov” element for object values.

Best regards,

Michael

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 27 Oct 2015 15:08:18 +0900

I’ve been toying with the idea that there is only one type of object and there is an array of theses. If a base name is found in the the object, it applies to all the values in that object plus any later objects in the array until something else overrides that base name. 

So the JSON might look like 

[
  { “bn”: ”/collection1/“ },

  { “bn”: ”/collection2/“ },

  {“n”: “thing1”, “sv”, “value1”},
  {“n”: “thing2”, “sv”, “value2”}
]

Which could be the same thing as say 

[
  { “bn”: ”/collection1/“ , “n”: “item1”, “sv”, “value1”},
  {“n”: “item2”, “sv”, “value2”},

  { “bn”: ”/collection2/“, “n”: “thing1”, “sv”, “value1”},
  { “bn”: ”/collection2/“, “n”: “thing2”, “sv”, “value2”}
]

I’m thinking implementation would 
> On Oct 21, 2015, at 4:52 AM, Michael Koster <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Oct 2015 09:12:55 +0100

Maps (JSON objects) are not fully compatible with streaming processing — you don’t know in which order the map elements will arrive.
So this is OK if the assumption is that each element of the top level array will be fully ingested before acting on it.
(Yes, we already have this problem with the other metadata in the measurements objects.)

I still like the two-level structure*) that we have been discussing, as it allows processing the elements of the measurements arrays knowing there is no serial dependency between their entries.  Might *look* more complicated, but actually is simpler, because the code can focus positionally on either processing base information or on processing measurements.

Grüße, Carsten

*) as in [+(base, [+measurement])]
On 27 October 2015 at 07:06:56, Cullen Jennings ([email-masked]) wrote:

If a base name is found in the the object, it applies to all the values in that object

---

From: Alexander Pelov <[email-masked]>
Date: Tue, 27 Oct 2015 21:02:29 +0100

Dear all,

I’m not sure to whether this has been discussed, so please forgive my ignorance on the topic. At one point, we considered the use of Deterministic Multimaps in CoOL.

A deterministic multimap, is a map, which can have duplicated key entries, and where the order of the elements is preserved. It seems to me that this structure fits perfectly the need of SenML.

Unfortunately, there is no native JSON/CBOR support for deterministic multimaps (DMMs). However, there is an extremely efficient way of representing them with arrays. In fact, it is as efficient, as the native map representation.

The representation is the following: 
[element1, element2, …, elementN]

Where, for every element:
- if it is NOT an array, it is the key, and the following element is the corresponding value.
- if it IS an array, its first element provides the key, and the rest of the array is the value, which is a DMM (processed recursively).

This allows for an efficient encoding of DMM, where the keys can be of any type EXCEPT arrays. In addition, it is fully compatible with JSON and CBOR. In CBOR we could have the luxury of allocating a tag for DMMs.

-------------------------------------------------------------------------------------------------------------
Two simple examples:

1)
[key1, value1, key2, value2, …, keyN, valueN]
Is interpreted as a DMM with the following structure:
{
  key1 : value1,
  key2 : value2, 
  …
  keyN : valueN
} // note that key1 can be the same as key2, …, keyN, and the order is preserved

2)
[key1, value1, key2, value2, [key3, keyA1, valueA1, keyA2, valueA2, [keyA3, keyB1, valueB1, keyB2, valueB2]], …, keyN, valueN]
Is interpreted as a DMM with the following structure:
{
  key1 : value1,
  key2 : value2, 
  key3 : {
    keyA1 : valueA1,
    keyA2 : valueA2,
    keyA3 : {
      keyB1, valueB1,
      keyB2, valueB2
    }
  },
  …
  keyN : valueN
}
-------------------------------------------------------------------------------------------------------------

Best,
Alexander

> Le 27 oct. 2015 à 09:12, Carsten Bormann <[email-masked]> a écrit :
> 
> On 27 October 2015 at 07:06:56, Cullen Jennings ([email-masked] <mailto:[email-masked]>) wrote:
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/core <https://www.ietf.org/mailman/listinfo/core>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Oct 2015 00:22:48 +0300

Where, for every element:
- if it is NOT an array, it is the key, and the following element is the corresponding value.
- if it IS an array, its first element provides the key, and the rest of the array is the value, which is a DMM (processed recursively).
(Why is the array always a DMM?  Should be just any sequence of values that are in the multimap under the key given.)

Sounds worthy of a CBOR tag.  Do you have this written up?

(I’d get rid of “deterministic”, I think the thing is just ordered.)

This allows for an efficient encoding of DMM, where the keys can be of any type EXCEPT arrays. 
I don’t understand the last comment.  If the key is an array, say for the multimap equivalent of {“a”: “b”, [1]: [2]}, you could represent it with OMM([“a”, “b”, [[1], [2]]]), where OMM is the tag for ordered multimaps.  Yes, this requires a little care when encoding the keys.  (There are a few other details about the data model semantics that probably need to be written down.)

Grüße, Carsten

---

From: Alexander Pelov <[email-masked]>
Date: Tue, 27 Oct 2015 22:41:08 +0100

> Le 27 oct. 2015 à 22:22, Carsten Bormann <[email-masked]> a écrit :
> 

Yes, this could be any sequence. I chose to make it specifically an array, as to facilitate the encoding/decoding procedure. I suppose this could be also a map, or other sequence.. As long as there is interest in it.
> Sounds worthy of a CBOR tag.  Do you have this written up?

Yep, I have a draft version (not submitted), called - as you proposed - CBOR Ordered Collection of Pairs (COCOP). 
> I don’t understand the last comment.  If the key is an array, say for the multimap equivalent of {“a”: “b”, [1]: [2]}, you could represent it with OMM([“a”, “b”, [[1], [2]]]), where OMM is the tag…

Actually, you are right. With some care, this case is also handled.

Best,
Alexander