Subject: [netmod] JSON to XML lossy conversion

From: Michal Vaško <[email-masked]>
Date: Thu, 16 Jul 2020 16:24:43 +0200

Hello,
it seems that there are cases when YANG data represented in JSON include more information than when converted to XML. Example:

YANG:

module a {
  leaf l {
    type union {
      type string;
      type uint8;
    }
  }
}

JSON:

{
  "a:l": 20
}

These data will be parsed as the leaf "l" of type "uint8" based on the JSON specs [1]. Then, when printed again

XML:

<a:l>20<a:l>

this XML would be parsed as leaf "l" of type "string".

Similar problem occurs also for boolean. So I am just wondering, is this a known bug/feature, is it intentional? It is true that applications usually choose one format and stick to it but it is not a requirement and I can see this having possibly quite nasty consequences.

Regards,
Michal

[1] https://tools.ietf.org/html/rfc7951#section-6.10

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 Jul 2020 17:30:57 +0200

Hi Michal,

> Similar problem occurs also for boolean. So I am just wondering, is this a known bug/feature, is it intentional? It is true that applications usually choose one format and stick to it but it is not…

This has already been discussed several times in the past, for example in this thread:

https://mailarchive.ietf.org/arch/browse/netmod/?gbt=1&index=dqNrvzOV2Ur6RoX6rTW-ZzsV65g

In my view, if it is a bug, it is in the design of the union type in YANG - there is no general way to signal the actual union member type for an instance. I believe it is a good thing that some encodings allow for at least partial signalling.

Note that similar issues may arise even more often in CBOR, see e.g. comments for section 5.12 in

https://mailarchive.ietf.org/arch/msg/core/Zrb2yhSSdlouS6PI9qfsA1bsDB4/

Lada

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka 
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 16 Jul 2020 17:55:22 +0200

> Similar problem occurs also for boolean. So I am just wondering, is this a known bug/feature, is it intentional? It is true that applications usually choose one format and stick to it but it is not…

This is known and has already be known when the JSON format was
standardized. Whether it is a feature or a bug (and if so where
the bug is) depends on whom you talk to.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Michal Vaško <[email-masked]>
Date: Fri, 17 Jul 2020 08:28:35 +0200

Thanks for the answers and sorry for not searching the mailing list archive first.

However, I interpret your answers as "we know about it, it is not ideal, but it will stay the way it is". I was hoping for something better.

Regards,
Michal

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Jul 2020 08:44:57 +0200

> In my view, if it is a bug, it is in the design of the union type in YANG - there is no general way to signal the actual union member type for an instance.

Right.  The ambiguity is normally not a problem for a type choice (which is just a union of the possible values of all types given), unless what specific alternative was chosen is intended to carry semantics.

> I believe it is a good thing that some encodings allow for at least partial signalling.
> 

In the original YANG (XML-only), everything was represented as a text string, so the ambiguity was the highest we see now.  YANG-JSON and YANG-CBOR provide progressively more disambiguating information, so the interpretation (which chosen alternative) may be different from the one after converting to YANG-XML.

It gets slightly worse if the non-text type has a conversion to text that is not fully nailed down; I don’t know if that is a problem with the current set of YANG types, but any new type could of course worsen the situation.

The onus is now on the author of the YANG module to make sure that the varying levels of ambiguity do not cause a problem.  I would be interested to know whether there is any tool support for this.

Grüße, Carsten

---

From: Michal Vaško <[email-masked]>
Date: Fri, 17 Jul 2020 08:57:50 +0200

Hi Carsten,
you had an interesting idea to have tools that could warn about these problems (although that is hardly a proper solution) but it is not really possible because the problem may occur whenever there is union with a 'string' and 'int8' - 'int32', 'uint8' - 'uint32', or 'boolean', in any order. Meaning in lots of, if not most, unions. And I have considered only XML and JSON, I have not looked into CBOR, which may make it even worse.

Regards,
Michal

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 17 Jul 2020 09:02:20 +0200

It is as it is for now and hence module authors have to be careful
with unions. Fixing this issue might require a new version of YANG
(depending on how you look at the issue), something that has been
talked about (for some years) but so far no serious action has been
started to seriously work on a revision of YANG.

https://github.com/netmod-wg/yang-next/issues/72

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Jul 2020 09:11:22 +0200

> you had an interesting idea to have tools that could warn about these problems (although that is hardly a proper solution) but it is not really possible because the problem may occur whenever there…

What you can do is to define a metadata annotation specifying the union
member for a particular instance, and implement it in your tools. This
would be a solution independent of instance representation.

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Jul 2020 09:03:51 +0200

On 2020-07-17, at 08:57, Michal Vaško <[email-masked]> wrote:
> 
> I have not looked into CBOR

Please do; this is in 2nd WGLC right now.

Grüße, Carsten

---

From: Vladimir Vassilev <[email-masked]>
Date: Fri, 17 Jul 2020 12:03:37 +0200

> would be a solution independent of instance representation.

IMO this does not solve the problem that XML can not encode all values 
of the value space in certain awkward unions (JSON and CBOR can't do 
that either only the constraints are alternative supersets).

If the XML representation rules in the YANG definition were provided as 
an inherent constraint to the value space of the union type there was 
not going to be ambiguity and there would have been a bug in any 
encoding that does not enforce this exact constraint. This is not 
spelled out in RFC7950 but is a good rule of thumb to avoid unions that 
have value space that can not be represented in XML.

The alternative is to rework the XML representation to not have such 
limitation in a next version of YANG (e.g. adding XML attributes with 
the active member type index and name).

That said I think there is one underestimated feature of the union type 
being entirely mapped to a string also for integer types as it is in the 
current XML encoding definition. Without requirement for resolving which 
member type the data actually is representing but just validating there 
is at least one match allows implementations to encode the data without 
resolving a possibly complex logical relationship. For example a union 
of interface index and leafref to interface name (for whatever reason 
someone would design such a union). You really do not need to know if 
there is an interface named "2" in the candidate configuration when you 
send <edit-config> in XML. With JSON you have to resolve this.

/Vladimir

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Jul 2020 14:12:06 +0200

On 17. 07. 20 12:03, Vladimir Vassilev wrote:
>>> you had an interesting idea to have tools that could warn about these
>>> problems (although that is hardly a proper solution) but it is not
>>> really possible because the problem may occur whenever there is union
>>> with a 'string' and 'int8' - 'int32', 'uint8' - 'uint32', or
>>> 'boolean', in any order. Meaning in lots of, if not most, unions. And
>>> I have considered only XML and JSON, I have not looked into CBOR,
>>> which may make it even worse.

Why not? In XML, everything is encoded as text, and the annotation tells
how to parse it and interpret. So either the text representation
conforms to that precise type or otherwise it is an error. There is no
ambiguity.

Lada

>>>>> In my view, if it is a bug, it is in the design of the union type
>>>>> in YANG - there is no general way to signal the actual union member
>>>>> type for an instance.
>>>> Right.  The ambiguity is normally not a problem for a type choice
>>>> (which is just a union of the possible values of all types given),
>>>> unless what specific alternative was chosen is intended to carry
>>>> semantics.
>>>>
>>>> In the original YANG (XML-only), everything was represented as a
>>>> text string, so the ambiguity was the highest we see now.  YANG-JSON
>>>> and YANG-CBOR provide progressively more disambiguating information,
>>>> so the interpretation (which chosen alternative) may be different
>>>> from the one after converting to YANG-XML.
>>>>
>>>> It gets slightly worse if the non-text type has a conversion to text
>>>> that is not fully nailed down; I don’t know if that is a problem
>>>> with the current set of YANG types, but any new type could of course
>>>> worsen the situation.
>>>>
>>>> The onus is now on the author of the YANG module to make sure that
>>>> the varying levels of ambiguity do not cause a problem.  I would be
>>>> interested to know whether there is any tool support for this.
>>>>
>>>> Grüße, Carsten
>>>>
>>>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 17 Jul 2020 14:42:28 +0200

> conforms to that precise type or otherwise it is an error. There is no

An annotation that is only understood by some tools and not by others
is creating a new problem since different tools now start to interpret
data in different ways, i.e., the annotation harms interoperability.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Jul 2020 15:50:05 +0200

> data in different ways, i.e., the annotation harms interoperability.

I agree, it should eventually be standardized, if it turns out to be a
good idea. But to find out, experience with running code always helps.

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Jul 2020 18:09:52 +0200

On 2020-07-17, at 12:03, Vladimir Vassilev <[email-masked]> wrote:
> 
> IMO this does not solve the problem that XML can not encode all values of the value space in certain awkward unions (JSON and CBOR can't do that either only the constraints are alternative supersets).

The reason YANG-CBOR cannot do that is that the union can evolve, and might gain or lose some branches in this evolution.  (Otherwise we would have done something crude like numbering the branches.)

The term “union” of a set of types means that all values in all these types can be used.
It does not mean that any actual value identifies which of the branches it is in.

Apparently, this is not what YANG means with “union”.  ASN.1 has “choice”, but leaves the onus of all the branches looking different (usually using ASN.1 “tags”) to the specification writer.

The text in 9.12 of 7950 seems to say there is some sequential matching that does identify the branch.  It nicely ties this to the specifics of an XML encoding, and 6.10 of 7951 tells us it’s different in JSON.  

YANG-CBOR goes the extra mile here and defines special encodings that are only used within unions (6.6, 6.7, 6.12), which are meant to be a bit closer to the text-based encodings, so the outcome of all the above is closer to XML/JSON than it would be with the more optimized encoding we use otherwise.  This is a rather ugly rucksack, but we didn’t find a better way to keep compatibility with random YANG specs.  If you redesign unions (e.g., turning them into explicit choices), please do this in a way that allows YANG-CBOR to return to its more optimized ways.

Grüße, Carsten