Subject: [netmod] JSON encoding of anydata: approximating an operational leaf-list

From: Jan Kundrát <[email-masked]>
Date: Mon, 21 Feb 2022 11:44:07 +0100

Hi,
last year we published some work [1] about using IETF YANG-push for 
telemetry streaming in the context of optical networks. One of the use 
cases was continualy sending spectral scans, and for us that meant updating 
a list of roughly 15-20k items at least once per second.

Think of this as a list of (frequency, power) pairs where the "frequency" 
can be implied, so the data was really a "a long vector of numbers". We 
were not able to use either `list` or a `leaf-list` directly for 
performance reasons. The root cause is that `leaf-list` traditionally used 
to be "an ordered set". It was changed to allow duplicate values in YANG 
1.1, and that for ops data only. Still, various implementations try to look 
at the operational leaf-lists and helpfully compute diffs, they look for 
item moves, etc. This is not cheap for 15-20k items.

In the end we used a model like this:

 leaf lowest-frequency {
   type frequency-ghz;
   config false;
   mandatory true;
   description "Lowest frequency slice in the `p` list.";
 }

 leaf step {
   type frequency-ghz;
   config false;
   mandatory true;
   description "Each subsequent `p` result is this far from the previous 
one.";
 }

 anydata p {
   config false;
   description "Measured power. Sorted by frequency, starting at 
`lowest-frequency`, step size `step`. Individual items are decimal64 
numbers.";
 }

The JSON-encoded data looked like this:

 {
   "something: {
     "lowest-frequency": 123456,
     "step": 1,
     "p": ["0.003", "0.001", "0.001", "0.24", "0.37"]
   }
 }

Our internal implementation uses libyang, and we're now updating form 
libyang v1 to libyang v2, which assumes that our `p` has to be encoded as 
JSON object. This of course produces an invalid JSON:

    "p": {["0.003", "0.001", "0.001", "0.24", "0.37"]}

I have a few questions:

- Can `anydata` represent a JSON array directly? On one hand the standard 
says that "An anydata instance is encoded in the same way as a container, 
i.e., as a name/object pair", but then it also allows the `empty` thing via 
`[null]`. Or is it perhaps allowed only for *members* of this faux 
container?

- In case `anydata` is not a good match, I suppose `anyxml` will work fine 
because the example in the RFC is already using an array. Perhaps we could 
also use JSON numbers directly without encoding as a string as well.

- Is there a better approach than this one? Can I reasonably expect "all 
the YANG implementations out there" to stop looking at operational 
leaf-lists as essentially ordered sets? I think the answer is "no", but I 
am looking for suggestions.

With kind regards,
Jan

[1] https://ieeexplore.ieee.org/document/9457112

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 21 Feb 2022 12:38:41 +0100

On 21. Feb 2022, at 11:44, Jan Kundrát <[email-masked]> wrote:
> Sorted

➔ ordered

Interesting discussion!

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 21 Feb 2022 12:41:36 +0100

>   "p": {["0.003", "0.001", "0.001", "0.24", "0.37”]}

I read 7951 as saying that the value part of the anydata pair is a JSON object (map) with YANG names as keys.

"p": {“ietf-blub:blub”: ["0.003", "0.001", "0.001", "0.24", "0.37"]}

I have no idea how the YANG model hints at what keys you should be using there.

Grüße, Carsten

---

From: Martin Björklund <[email-masked]>
Date: Mon, 21 Feb 2022 12:44:15 +0100 (CET)

Jan Kundrát <[email-masked]> wrote:
> cases was continualy sending spectral scans, and for us that meant
> updating a list of roughly 15-20k items at least once per second.
> 
> Think of this as a list of (frequency, power) pairs where the
> "frequency" can be implied, so the data was really a "a long vector of
> numbers". We were not able to use either `list` or a `leaf-list`
> directly for performance reasons. The root cause is that `leaf-list`
> traditionally used to be "an ordered set". It was changed to allow
> duplicate values in YANG 1.1, and that for ops data only. Still,
> various implementations try to look at the operational leaf-lists and
> helpfully compute diffs, they look for item moves, etc. This is not
> cheap for 15-20k items.
> 
> libyang v1 to libyang v2, which assumes that our `p` has to be encoded
> as JSON object. This of course produces an invalid JSON:
> 
> - Can `anydata` represent a JSON array directly?

No.

> On one hand the
>   standard says that "An anydata instance is encoded in the same way as
>   a container, i.e., as a name/object pair", but then it also allows the
>   `empty` thing via `[null]`. Or is it perhaps allowed only for
>   *members* of this faux container?

RFC 7950 says:

   The "anydata" statement is used to represent an unknown set of nodes
   that can be modeled with YANG, [...]

And 7951 says:

   An anydata instance is encoded in the same way as a container, i.e.,
   as a name/object pair.

So with "anydata p", a correct JSON example could be:

  "p": {"x": ["0.003", "0.001", "0.001", "0.24", "0.37"]}

The "could be modeled with YANG" part is satisfied with for example:

  leaf-list x {
    config false;
    type decimal-64 {
      fraction-digits 3;
    }
  }

> 
> - In case `anydata` is not a good match, I suppose `anyxml` will work
>   fine because the example in the RFC is already using an array. Perhaps
>   we could also use JSON numbers directly without encoding as a string
>   as well.

I guess this could work.  I don't know if implementations handle
anyxml consistently.

>   leaf-lists as essentially ordered sets? I think the answer is "no",
>   but I am looking for suggestions.

It is clear from RFC 7950 that duplicate values are allowed in
operational state.

/martin

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod