Subject: [netmod] WG Last Call for draft-ietf-netmod-yang-json-04 (until 2015-06-29)

From: Kent Watsen <[email-masked]>
Date: Mon, 15 Jun 2015 22:49:28 +0000

This is a notice to start a NETMOD WG last call for the document "JSON Encoding of Data Modeled with YANG":

https://tools.ietf.org/html/draft-ietf-netmod-yang-json-04

Please indicate your support by Monday June 29, 2015 at 9PM EST.
We are not only interested in receiving defect reports, we are equally
interested in statements of the form:

  "I have reviewed I-D XYZ and I found no issues"
  "I have implemented the data model in I-D XYZ"
  "I am implementing the data model in I-D XYZ"
  "I am considering to implement the data model in I-D XYZ"

This is the first Last Call for this document.

Kent, as NETMOD co-chair

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 16 Jun 2015 13:47:00 +0200

>   "I have implemented the data model in I-D XYZ"

FWIW, I have implemented automatic data-model-driven conversions between
XML- and JSON-encoded instance documents, and vice versa, as pyang
plugins "jsonxsl" and "jtox".

Lada

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 24 Jun 2015 16:53:25 +0200

> Please indicate your support by Monday June 29, 2015 at 9PM EST.

Hi,

I have reviewed draft-ietf-netmod-yang-json-04.

- I am not sure I agree with the wording in section 3. Why is section
  8.3.3 only applicable to XML encoded data? Validation applies to
  datastores. While constraints are defined using XML-based notations
  such as XPATH, how the validation is carried out is not defined in
  the YANG specifications. I guess I actually disagree with the
  wording in section 3 of the JSON encoding I-D.

- It is unclear whether the 'if and only if' on page 4 means that an
  implementation that generates namespace prefixes that are not
  strictly needed is violating this I-D. I see the need for a MUST to
  include the module name if the parent node belongs to a different
  module. I am not sure why it is necessary to mandate minimal
  encodings (if that is the idea here). Whatever the answer is, it
  would be good to use RFC 2119 language.

- The reason for the requirement that list keys are encoded first in
  RFC 6020 is to make it easier to process data in a stream-oriented
  fashion. If keys can appear anywhere, they might appear at the very
  end and thus buffering is required in order to process data
  properly. Is this concern not relevant for the JSON encoding?
  Perhaps this is not relevant, but then we might also state this
  explicitly:

      As a consequence, implementations must be cable to buffer JSON
      encoded instances in order to locate keys that may appear at the
      end of a JSON encoded instance.

- I think that section 5.5 should say:

      If the data model for the data in an anydata instance is known,
      then the data must be encoded following the rules defined in
      this I-D.

  In other words, it is not arbitrary JSON with a few constraints but
  something that matches the JSON encoding rules once the data model
  is known.

- Section 6, I suggest s/mapped/encoded/.

- I do not understand 'An "enumeration" value is mapped in the same
  way as a string except that permitted values are defined by enum
  statements in YANG.' Perhaps this is simpler:

     An "enumeration" value is encoded as a JSON string. The allowed
     string values are the enumeration names assigned by the enum
     statement in YANG.

     The representation is identifical to the XML encoding, see
     sec. 9.6 in [RFC6020bis].

  I specifically tried to avoid 'value' in order to avoid confusion with
  the value YANG statement.

- Perhaps remove the reference to XML in section 6.5 from the
  definition of the encoding rules so that the JSON encoding rules are
  not tied into the XML encoding rules. It is OK to mention that it is
  the same.

     A "bits" value is encoded as a JSON string. Multiple bit names
     assigned by the bit statement in YANG are encoded as a
     space-separated string of bit names representing the individual
     bits that are set.

     The representation is identifical to the XML encoding, see
     sec. 9.7 in [RFC6020bis].

- Same as before for section 6.6:

     A "binary" value is encoded by first encoding the binary value in
     base64 and encoding the resulting base64-encoded value as a JSON
     string.

     The representation is identifical to the XML encoding, see
     sec. 9.8 in [RFC6020bis].

- I am not sure I understand the argument why [null] is better than
  null for the empty type. Perhaps it is but the text does not really
  tell me.

- I suggest to remove the reference to 9.13.3 in the definition. In
  fact, the representation is pretty different since XML uses
  namespace prefixes while JSON uses module names. (I must admit that
  I find the JSON representation more readable since it does not
  require XML namespace context.)

- Section 7: So what happens in the rare case of a binary value
  appearing in a RESTCONF URI? Is the resulting BASE64 value than
  simply subject to URI escaping rules?

  I assume this 'rare event' would happen if a list is indexed by a
  leaf of type binary, no? Are there any other cases?

/js

PS: Should RFC 6020bis change the section titles "XML Mapping Rules" to
    "XML Encoding Rules"? I think we really talk about 'encoding', not
    about 'mapping' and if we agree on this, we should try to be
    consistent with the terms we use.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Jun 2015 11:49:11 +0200

Hi Juergen,

thank you for the review.

>   datastores. While constraints are defined using XML-based notations

You are right that this section shouldn't talk about XML-encoded data,
i.e. serialized form. On the other hand, XPath 1.0 spec says: "XPath
operates on the abstract, logical structure of an XML document, …".

So I think a datastore needs to be represented, at least conceptually,
as XML infoset.

>   the YANG specifications. I guess I actually disagree with the

I don't think this is true. YANG spec doesn't say how "must" and "when"
statements are evaluated, and relies on XPath.

>   wording in section 3 of the JSON encoding I-D.

What specifically? Do you have any suggestions for changes?

>   strictly needed is violating this I-D. I see the need for a MUST to

Yes, that's the intention. Why is it unclear?

>   would be good to use RFC 2119 language.

Revision -02 used 2119 terms but there were objections against it:

https://mailarchive.ietf.org/arch/msg/netmod/xXS0uSKKu83qBQVCJ_CYmdsavUc

In fact, YANG spec also states syntax rules without using 2119 keywords,
for example "Each identifier starts with an uppercase or lowercase ASCII
letter or an underscore character, …", it doesn't say that it MUST NOT
start with anything else.

>   properly. Is this concern not relevant for the JSON encoding?

This cannot be required as long as our aim is interoperability of
implementations based on off-the-shelf JSON parsers, hence I_JSON. RFC
7493 states it clearly: "The order of object members in an I-JSON
message does not change the meaning of an I-JSON message."

>       end of a JSON encoded instance.

I think it is clear that an implementation needs to be able to handle
keys appearing anywhere in a list-entry-object. How it is handled is IMO
implementation-specific - I guess it also depends on where and how the
stream-oriented data processing is used.

>       then the data must be encoded following the rules defined in

Something like this should then probably appear in 6020bis, yet it doesn't.

I am unclear how one can determine that the data model is known because
anydata nodes don't provide this information. Perhaps it can be
specified in the description but then is it known including the
revisions of YANG modules involved? Moreover, the data model needn't be
known at all.

Therefore, I tend to think about anydata as kind of ad hoc data
that's just piggybacked inside YANG-modelled data. There may be rules
that this data have to conform to (expressed in YANG or not) but they
are out of scope from the perspective of the data model that's specified
in hello or yang-library.

That's why sec. 5.5 only states syntactic constraints to make sure that
the data "can be modelled with YANG".

BTW, I have a growing concern that anydata creates more problems than it
solves.

> - Section 6, I suggest s/mapped/encoded/.

Agreed.

>   the value YANG statement.

OK, good point.

>      sec. 9.8 in [RFC6020bis].

OK.

>   null for the empty type. Perhaps it is but the text does not really

"foo": null

may sometimes appear as if the "foo" instance is not present at all - in
some languages this is probably more serious than in others. This is how
it works in Python:

>>> import json
>>> obj = json.loads('{"foo": null, "bar": [null]}')
>>> "YES" if obj["foo"] else "NO"
'NO'
>>> "YES" if obj["bar"] else "NO"
'YES'

So the special value "[null]" seems safer and it cannot appear in any
other context.

> - I suggest to remove the reference to 9.13.3 in the definition. In

… definition of instance-identifier encoding.

>   require XML namespace context.)

Yes, the document could simply state an ABNF production for JSON-endoded instance-identifier without
further ado.

>   simply subject to URI escaping rules?

Right, and I agree the document should say this.

>   leaf of type binary, no? Are there any other cases?

I am not aware of any other case.

>     consistent with the terms we use.

I agree, it would be better.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 30 Jun 2015 11:56:00 +0200

> statements are evaluated, and relies on XPath.

RFC 6020:

   When a datastore is validated, all "must" constraints are
   conceptually evaluated once for each data node in the data tree, and
   for all leafs with default values in use (see Section 7.6.1).  If a
   data node does not exist in the data tree, and it does not have a
   default value, its "must" statements are not evaluated.

   [...]

   Also note that the XPath expression is conceptually evaluated.  This
   means that an implementation does not have to use an XPath evaluator
   on the device.  How the evaluation is done in practice is an
   implementation decision.
 
> What specifically? Do you have any suggestions for changes?

The problem is that RFC 6020 talks about datastore validation, not
about validation of a specific serialization. Hence, it does not
matter whether the data was XML or JSON (or CBOR or whatnext) encoded
- once the data is in the datastore, datastore validation takes
place. One way to implement this is to serialize everything to XML and
then to use XML gear to do the validation. But this implementation
strategy is not required.

> start with anything else.

If the goal is to define a strict implementation requirement, then I
think using RFC 2119 language is the preferred choice in the IETF. I
think the debate back then was whether it is reasonable to declare an
implementation that fails to produce minimal encodings as violating
the spec. How does it break a receiver if I send a redundant module
name? Your change of the MUST to 'if and only if' did cosmetics but it
did not address the concern raised.

> stream-oriented data processing is used.

I still prefer to have this explicitly stated. If the keys happen to
appear at the end, how can you process the data without some form of
buffering? Perhaps this wording is better (since it is less specific
how data is buffered):

      As a consequence, implementations must be cable to buffer data
      in order to locate keys that may appear at the end of a JSON
      encoded instance.

> revisions of YANG modules involved? Moreover, the data model needn't be

Yes, this is why the sentence starts with "If the data model ... is
known,"

> BTW, I have a growing concern that anydata creates more problems than it

I fail to see that but this is not relevant here as long as the JSON
document provides a means to encode anydata in an interoperable way.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 30 Jun 2015 14:56:30 +0200

>    on the device.  How the evaluation is done in practice is an

Yes, but the result must be guaranteed to be the same as if an XPath 1.0
processor is used, otherwise it makes really no sense.

> strategy is not required.

We have no explicit concept of a metamodel for datastores but IMO the
fact that we evaluate XPath expressions on top of it implies that the
datastore must be (congruent to) restricted XML infoset. Section 5 (Data
Model) is a crucial part of XPath spec, and it is XML.

> think using RFC 2119 language is the preferred choice in the IETF. I

I guess identifier syntax is a strict implementation requirement,
too. What the I-D defines is an unambiguous encoding of data.

> did not address the concern raised.

Yes, but these are two different things:

1. In my view, in any syntax or encoding specification 2119 keywords are
   not needed, in fact they are inappropriate.

2. There are strong technical reasons for *not* relaxing the namespace
   rules - it would again render of-the-shelf JSON parsers pretty much
   useless, as Martin correctly pointed out:

   https://mailarchive.ietf.org/arch/msg/netmod/mCtvup_YnL3gvr95EIe4sSFcKfs

> buffering? Perhaps this wording is better (since it is less specific

Maybe the keys are just irrelevant for the given purpose, e.g. when
streaming state data.

As a matter of fact, with JSON encoding the receiving side will mostly
buffer the whole payload anyway.

I can add that sentence, if you insist, but I see it as superfluous.

> document provides a means to encode anydata in an interoperable way.

It depends on what you call "interoperability", but if there is no
(precise) data model available for anydata content, then the round-trip
JSON client -> server -> XML client (or the other way around) may not be
possible.

So I don't get how "anydata" fixes "anyxml" problems (I personally
only have problem with the name).

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 30 Jun 2015 15:20:59 +0200

> Model) is a crucial part of XPath spec, and it is XML.

The YANG language does not require that XPath is used. It says that
XPath is conceptually evaluated. Should the text in an encoding
document not be consistent with that?

>    https://mailarchive.ietf.org/arch/msg/netmod/mCtvup_YnL3gvr95EIe4sSFcKfs

OK

> I can add that sentence, if you insist, but I see it as superfluous.

Thanks.

> only have problem with the name).

We are not discussing this anymore.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 30 Jun 2015 16:07:10 +0200

> On 30 Jun 2015, at 15:39, Juergen Schoenwaelder <[email-masked]> wrote:
> 
> On Tue, Jun 30, 2015 at 03:32:13PM +0200, Ladislav Lhotka wrote:
>> 
>>> On 30 Jun 2015, at 15:20, Juergen Schoenwaelder <[email-masked]> wrote:
>>> 
>> YANG *does* use XPath 1.0. Whether it is evaluated conceptually or otherwise is irrelevant, the results must always be the same. And evaluation of XPath requires a specific context and data model as defined in its spec.
>> 
> 
> The YANG language definition is rather clear that (i) datastore
> content is validated and (ii) that XPath expressions are conceptually
> evaluated. I think the JSON encoding should not say anything
> different.

RFC 6020(bis) says in sec. 6.4:

   The data model used in the XPath expressions is the same as that used in XPath 1.0 [
   XPATH], …

JSON's data model is different.

Therefore, I believe the yang-json document has to specify how (YANG-specific) JSON data map on the XPath 1.0 data model. The way how it is currently done is that JSON-encoded instances are (conceptually) mapped to XML-encoded instances, and it is assumed that the mapping of the latter to the XPath data model and XPath context is fully defined in YANG spec.

Maybe there is a better way how to achieve the same but I am not aware of it, and the current way is IMO better than hand-waving.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 30 Jun 2015 16:39:59 +0200

> Maybe there is a better way how to achieve the same but I am not aware of it, and the current way is IMO better than hand-waving.

The data once decoded goes into a datastore. It is the datastore that
is validated. I quote the relevant text again:

   When a datastore is validated, all "must" constraints are
   conceptually evaluated once for each data node in the data tree, and
   for all leafs with default values in use (see Section 7.6.1).

   [...]

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 01 Jul 2015 08:54:07 +0200

>> >>>>> default value, its "must" statements are not evaluated.

The text you substituted here with an ellipsis is actually quite
important for this discussion because it defines the context for XPath
evaluation (together with section 6.4), in particular the data tree on
which every XPath expression is evaluated. It is clear that the data
tree can also comprise state data, notification content or RPC
input/output, i.e. not only datastore content as you keep saying.

Terms like "context node" refer to the XPath data model as described in
sec. 5 of the XPath spec:

http://www.w3.org/TR/1999/REC-xpath-19991116/#data-model

(and section 6.4 in RFC 6020 says it explicitly).

We need to know, at least conceptually, how to construct the XPath data
tree from JSON text. For example, it has to be clear that leaf-list
entries encoded as a JSON array appear as sibling nodes in the data
tree, otherwise a "must" constraint specified for the leaf-list won't
work correctly. I don't think this is anyhow evident and IMO it has to
be addressed. This is the purpose of section 3 in
draft-ietf-netmod-yang-json-04.

Would it help if "validation" is replaced with "XPath evaluation"
throughout this section?

Lada