Subject: [netmod] json empty leaf encoding strangeness [Re: WG Last Call for draft-ietf-netmod-yang-json-04 (until 2015-06-29)]

From: Christian Hopps <[email-masked]>
Date: Sat, 18 Dec 2021 01:47:21 -0500

Hi,

I was just going over json encoding of YANG, and got to the empty leaf encoding and it's odd use of "[null]" rather than just "null". I read the explanation, but had a hard time believing what it said. So, running code time. The first thing I tried was Python to see if something weird happened:

In [1]: import json
In [2]: json.loads('{"foo": null}')
Out[2]: {'foo': None}

Nope, seems fine. Then checked the "definitive source" for json, javascript:

js> JSON.parse("{\"foo\": 1, \"bar\": null}")
({foo:1, bar:null})

Seems fine here too.. 

So I went searching the mailing list and found this:

> On Jun 29, 2015, at 5:49 AM, Ladislav Lhotka <[email-masked]> wrote:

> Juergen Schoenwaelder <[email-masked]> writes:

...
>> - I am not sure I understand the argument why [null] is better than

>>  null for the empty type. Perhaps it is but the text does not really

>>  tell me.

> "foo": null

> may sometimes appear as if the "foo" instance is not present at all - in

> some languages this is probably more serious than in others. This is how

> it works in Python:

>>>> import json

>>>> obj = json.loads('{"foo": null, "bar": [null]}')

>>>> "YES" if obj["foo"] else "NO"

> 'NO'

>>>> "YES" if obj["bar"] else "NO"

> 'YES'

> So the special value "[null]" seems safer and it cannot appear in any

> other context.

The justifying example here is, unfortunately, wrong. One does not (normally) check for the presence of a dictionary item in python by trying to access the value of that item (which is what the code above is doing). The standard python code to check for the presence of an item is as follows, and it works fine with a "null" json value:

In [1]: import json
In [2]: o = json.loads('{"foo": null}')
In [3]: o
Out[3]: {'foo': None}
In [4]: "YES" if "foo" in o else "NO"
Out[4]: 'YES'

Indeed in python if you try and check for a missing item the way the email above describes, and the item is missing, an exception is raised:

In [5]: o["bar"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)

One *could* catch the "KeyError" to check for missing items which will also work using "null" son values for empty leafs, but the more common form is "if key in dict" in my experience.

Furthermore, using "access the value" for presence check will also "fail" for zero values and for empty strings, but we don't have odd encodings for integers or strings...

In [1]: import json
In [2]: o = json.loads('{"foo": 0, "bar": ""}')
In [3]: o
Out[3]: {'foo': 0, 'bar': ''}
In [4]: "YES" if o["foo"] else "NO"
Out[4]: 'NO'
In [5]: "YES" if o["bar"] else "NO"
Out[5]: 'NO'

Point being, evaluating an items value is not the right way to check for "presence", it's the way to check the "value" of the item. An empty leaf has no value so it's basically never correct to try to access that value in code.

Javascript is basically the same as python here:

js> o = JSON.parse('{"foo": null}')
({foo:null})
js> if ("foo" in o) { "YES" } else { "NO" }
"YES"

So, do we think it would it ever be possible to change (fix) this encoding oddity?

Thanks,
Chris.

---

From: Eliot Lear <[email-masked]>
Date: Sat, 18 Dec 2021 09:20:19 +0100

Possibly.  The issue with fixing it is that it would create an ambiguity 
between null and [  null ].  I am not sure how often [ null ] would not 
have that semantic equivalence in earlier work. This is further 
complicated by a lack of self-descriptive versioning in YANG serializations.

Still it seems like the right thing to do, in order to fall into line 
with the target language spec.

Eliot

On 18.12.21 07:47, Christian Hopps wrote:
> Hi,
>
> I was just going over json encoding of YANG, and got to the empty leaf 
> encoding and it's odd use of "[null]" rather than just "null". I read 
> the explanation, but had a hard time believing what it said. So, 
> running code time. The first thing I tried was Python to see if 
> something weird happened:
>
>     In [1]: import json
> The justifying example here is, unfortunately, wrong. One does not 
> (normally) check for the presence of a dictionary item in python by 
> trying to access the value of that item (which is what the code above 
> is doing). The standard python code to check for the presence of an 
> item is as follows, and it works fine with a "null" json value:
>
>     In [1]: import json
> Indeed in python if you try and check for a missing item the way the 
> email above describes, and the item is missing, an exception is raised:
>
>     In [5]: o["bar"]
> One *could* catch the "KeyError" to check for missing items which will 
> also work using "null" son values for empty leafs, but the more common 
> form is "if key in dict" in my experience.
>
> Furthermore, using "access the value" for presence check will also 
> "fail" for zero values and for empty strings, but we don't have odd 
> encodings for integers or strings...
>
>     In [1]: import json
> Point being, evaluating an items value is not the right way to check 
> for "presence", it's the way to check the "value" of the item. An 
> empty leaf has no value so it's basically never correct to try to 
> access that value in code.
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Eliot Lear <[email-masked]>
Date: Sat, 18 Dec 2021 09:54:11 +0100

Just following this up, I see in my own work it occurs a little bit, so 
it is out there.  It doesn't create ambiguities, tho, because it is used 
in null-valued leafs.  It just creates syntax breakage.  Some parsers 
will have a problem with recognizing a naked null as valid.  On the 
other hand, in the case of MUD, we separately added versioning into the 
model, so we could bump this if the serialization updates.

On 18.12.21 09:20, Eliot Lear wrote:
>
> Possibly.  The issue with fixing it is that it would create an 
> ambiguity between null and [  null ].  I am not sure how often [ null 
> ] would not have that semantic equivalence in earlier work.  This is 
> further complicated by a lack of self-descriptive versioning in YANG 
> serializations.
>
>> I was just going over json encoding of YANG, and got to the empty 
>> leaf encoding and it's odd use of "[null]" rather than just "null". I 
>> read the explanation, but had a hard time believing what it said. So, 
>> One *could* catch the "KeyError" to check for missing items which 
>> will also work using "null" son values for empty leafs, but the more 
>> common form is "if key in dict" in my experience.
>>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 18 Dec 2021 13:59:13 +0100

On 2021-12-18, at 09:20, Eliot Lear <[email-masked]> wrote:
> 
> Still it seems like the right thing to do, in order to fall into line with the target language spec.

But that’s not what YANG was designed for.
YANG is a data modeling language, which incidentally has a couple of representation formats (YANG-XML, YANG-JSON, YANG-CBOR).
There is no intention for the YANG generic data model (the set of all data models YANG can express) to cover the generic data model of the representation format.

While RFC 8791 (and before it RFC 8040) extended YANG to be able to describe data in flight, there is no intention that YANG can be used to describe the entire expressibility of the representation format.  We have Relax-NG, CDDL, and a few other modeling approaches if we need that.

Grüße, Carsten

---

From: Christian Hopps <[email-masked]>
Date: Sat, 18 Dec 2021 08:35:41 -0500

> On Dec 18, 2021, at 7:59 AM, Carsten Bormann <[email-masked]> wrote:
> 

To be clear, the question I raised is about an encoding of YANG modeled data. In particular the JSON encoding of the YANG empty leaf element.

  https://datatracker.ietf.org/doc/html/rfc7951#section-6.9

the justification for having the very odd encoding was that there was some belief that some programming languages might have trouble with the obvious JSON "null" encoding (which stands for "no value" in JSON -- exactly what we want here). An old email gave a python example to try and illustrate this, except the python example was an incorrect use of python for this case (presence check) and not a problem with the natural JSON encoding for no value items. In fact correctly written python code has no problem at all with the more natural "null" encoding, and results in the correct internal representation in python to boot.

This isn't about the YANG specification in general, or making YANG conform to any particular language. It's just about the a choice in the specification for the encoding of YANG in a particular format (JSON).

Thanks,
Chris.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 18 Dec 2021 15:13:42 +0100

> This isn't about the YANG specification in general, or making YANG conform to any particular language. It's just about the a choice in the specification for the encoding of YANG in a particular for…

I understand the objective to have a good representation for YANG data.

For instance, in YANG-CBOR, a YANG empty is represented as a CBOR null (0xf6, in case you need to visualize the bits); there is no expectation that this will be difficult to handle by CBOR implementations.

However, there is nothing wrong with representing empty with the array »[null]« in YANG-JSON.  It is just more complex, and apparently unnecessarily so.
Whether removing this complexity to save two characters is worth doing an incompatible upgrade to the YANG-JSON representation format can be debated.
I’d say: emphatically no.

The other motivation that came up in this thread is that it is natural to have null as a value in JSON and YANG cannot describe such formats via YANG-JSON.  This was what I was reacting to, but, as you say, that wasn’t your question.

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Sat, 18 Dec 2021 08:33:43 -0800

On Sat, Dec 18, 2021 at 6:14 AM Carsten Bormann <[email-masked]> wrote:

> > This isn't about the YANG specification in general, or making YANG
> conform to any particular language. It's just about the a choice in the
> specification for the encoding of YANG in a particular format (JSON).
>
> For instance, in YANG-CBOR, a YANG empty is represented as a CBOR null
> (0xf6, in case you need to visualize the bits); there is no expectation
> that this will be difficult to handle by CBOR implementations.
>
> However, there is nothing wrong with representing empty with the array
> »[null]« in YANG-JSON.  It is just more complex, and apparently
> unnecessarily so.
> Whether removing this complexity to save two characters is worth doing an
> incompatible upgrade to the YANG-JSON representation format can be debated.
Thank you for caring about backward compatibility and stability.

The other motivation that came up in this thread is that it is natural to
> have null as a value in JSON and YANG cannot describe such formats via
> YANG-JSON.  This was what I was reacting to, but, as you say, that wasn’t
> your question.
>
> Grüße, Carsten
>
>
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 18 Dec 2021 19:04:45 +0100

Christian Hopps <[email-masked]> writes:

> conform to any particular language. It's just about the a choice in
> the specification for the encoding of YANG in a particular format
> (JSON).

The original proposal (even before my individual draft appeared in the NETMOD WG) was actually just "null". We'd been discussing it back in March 2012 in a smaller group (Andy, Jürgen, Martin, Phil Shafer and myself), and Phil objected to this encoding of empty values. His arguments were, for example:

    'foo: null'?  Doesn't that make testing for empty values a major
    pain?  'if (foo)' would not work.

    JSON evolved from Javascript, so it must keep the javascript
    meanings for these keywords.

It is indeed true that tests in JavaScript cannot really distinguish between a non-existent member and member with the value of 'null'. I remember I wasn't happy about this change but I thought it wasn't a big deal.

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 18 Dec 2021 20:02:40 +0100

>    pain?  'if (foo)' would not work.

Same with »false«, which 7951 is not escaping into an array.
This argument simply doesn’t hold water.
As was mentioned, you would check »”foo” in o«, which works for null, false, and all other values that may happen to be falsy(*) in the platform.

>    meanings for these keywords.

That is a common misconception that must be stamped out, but we have [email-masked] for that discussion.

> It is indeed true that tests in JavaScript cannot really distinguish between a non-existent member and member with the value of 'null'. I remember I wasn't happy about this change but I thought it…

As mentioned, they can.
And, yes, this little unnecessary complexity is not a big deal even if it is based on mistaken beliefs; every non-trivial protocol has some unburied zombies in the basement.

Grüße, Carsten

(*) falsy: converted to false when implicitly coerced to a Boolean (Ant.: truthy)

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 19 Dec 2021 09:25:13 +0100

Carsten Bormann <[email-masked]> writes:

> Same with »false«, which 7951 is not escaping into an array.  This
> argument simply doesn’t hold water.  As was mentioned, you would
> check »”foo” in o«, which works for null, false, and all other
> values that may happen to be falsy(*) in the platform.

The whole story is that Phil wanted to have both empty and boolean values as JSON strings. The result was a compromise, and I thought that [null] was a much better choice because, unlike "null" string, it cannot otherwise appear as a valid value of a YANG leaf.

> [email-masked] for that discussion.

It's not that I am defending these views. I did argue with RFC 4627, but not too hard, as this seemed like a marginal issue at that time - much more fundamental objections against the JSON representation were raised back then.

> And, yes, this little unnecessary complexity is not a big deal even if it is based on mistaken beliefs; every non-trivial protocol has some unburied zombies in the basement.

Right, in hindsight it was a mistake. It is also worth noting that [null] is only an on-the-wire value, so e.g. None can be used internally in Python. I have written a lot of related running code and don't find it a serious problem.  

Lada

---

From: Christian Hopps <[email-masked]>
Date: Mon, 20 Dec 2021 16:53:14 -0500

> On Dec 18, 2021, at 11:33 AM, Andy Bierman <[email-masked]> wrote:
> 

I guess I left as implicit in my question of whether we could do something to fix it, that that "could" implied a fix wouldn't break things. I certainly care about stability and backward compatibility.

Thanks,
Chris.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 20 Dec 2021 14:22:58 -0800

On Mon, Dec 20, 2021 at 1:53 PM Christian Hopps <[email-masked]> wrote:

> I guess I left as implicit in my question of whether we could do something
> to fix it, that that "could" implied a fix wouldn't break things. I
> certainly care about stability and backward compatibility.
>
>

good point.

Sometimes we forget that changing the RFCs is the easy part.
People that don't even know this mailing list exists have to learn and
implement the changes.
Then the operator community has to deal with tool incompatibilities for
years to come.

The "industry" expects stability from YANG at this point.
They are focused on implementing and deploying YANG models.
The foundation is good enough. Not perfect. Nobody is expecting that.

Thanks,

Andy