Subject: Re: [netmod] WG Last Call for draft-ietf-netmod-yang-json-04 (until 2015-06-29)

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 1 Jul 2015 09:21:14 +0200

On Wed, Jul 01, 2015 at 08:54:07AM +0200, Ladislav Lhotka wrote:
> Juergen Schoenwaelder <[email-masked]> writes:
> 
> > On Tue, Jun 30, 2015 at 04:07:10PM +0200, Ladislav Lhotka wrote:
> >> 
> >> > On 30 Jun 2015, at 15:39, Juergen Schoenwaelder <[email-masked]> wrote:
> >> > 
> >> > On Tue, Jun 30, 2015 at 03:32:13PM +0200, Ladislav Lhotka wrote:
> >> >> 
> >> >>> On 30 Jun 2015, at 15:20, Juergen Schoenwaelder <[email-masked]> wrote:
> >> >>> 
> >> >>> On Tue, Jun 30, 2015 at 02:56:30PM +0200, Ladislav Lhotka wrote:
> >> >>>> Juergen Schoenwaelder <[email-masked]> writes:
> >> >>>> 
> >> >>>>> On Mon, Jun 29, 2015 at 11:49:11AM +0200, Ladislav Lhotka wrote:
> >> >>>>>> Hi Juergen,
> >> >>>>>> 
> >> >>>>>> thank you for the review.
> >> >>>>>> 
> >> >>>>>> Juergen Schoenwaelder <[email-masked]> writes:
> >> >>>>>> 
> >> >>>>>>> On Mon, Jun 15, 2015 at 10:49:28PM +0000, Kent Watsen wrote:
> >> >>>>>>>> 
> >> >>>>>>>> This is a notice to start a NETMOD WG last call for the document "JSON Encoding of Data Modeled with YANG":
> >> >>>>>>>> 
> >> >>>>>>>> https://tools.ietf.org/html/draft-ietf-netmod-yang-json-04
> >> >>>>>>>> 
> >> >>>>>>>> Please indicate your support by Monday June 29, 2015 at 9PM EST.
> >> >>>>>>> 
> >> >>>>>>> Hi,
> >> >>>>>>> 
> >> >>>>>>> I have reviewed draft-ietf-netmod-yang-json-04.
> >> >>>>>>> 
> >> >>>>>>> - I am not sure I agree with the wording in section 3. Why is section
> >> >>>>>>> 8.3.3 only applicable to XML encoded data? Validation applies to
> >> >>>>>>> datastores. While constraints are defined using XML-based notations
> >> >>>>>> 
> >> >>>>>> You are right that this section shouldn't talk about XML-encoded data,
> >> >>>>>> i.e. serialized form. On the other hand, XPath 1.0 spec says: "XPath
> >> >>>>>> operates on the abstract, logical structure of an XML document, …".
> >> >>>>>> 
> >> >>>>>> So I think a datastore needs to be represented, at least conceptually,
> >> >>>>>> as XML infoset.
> >> >>>>>> 
> >> >>>>>>> such as XPATH, how the validation is carried out is not defined in
> >> >>>>>>> the YANG specifications. I guess I actually disagree with the
> >> >>>>>> 
> >> >>>>>> I don't think this is true. YANG spec doesn't say how "must" and "when"
> >> >>>>>> statements are evaluated, and relies on XPath.
> >> >>>>> 
> >> >>>>> RFC 6020:
> >> >>>>> 
> >> >>>>> When a datastore is validated, all "must" constraints are
> >> >>>>> conceptually evaluated once for each data node in the data tree, and
> >> >>>>> for all leafs with default values in use (see Section 7.6.1).  If a
> >> >>>>> data node does not exist in the data tree, and it does not have a
> >> >>>>> default value, its "must" statements are not evaluated.
> >> >>>>> 
> >> >>>>> [...]
> 
> The text you substituted here with an ellipsis is actually quite
> important for this discussion because it defines the context for XPath
> evaluation (together with section 6.4), in particular the data tree on
> which every XPath expression is evaluated. It is clear that the data
> tree can also comprise state data, notification content or RPC
> input/output, i.e. not only datastore content as you keep saying.
> 
> Terms like "context node" refer to the XPath data model as described in
> sec. 5 of the XPath spec:
> 
> http://www.w3.org/TR/1999/REC-xpath-19991116/#data-model
> 
> (and section 6.4 in RFC 6020 says it explicitly).
> 
> We need to know, at least conceptually, how to construct the XPath data
> tree from JSON text. For example, it has to be clear that leaf-list
> entries encoded as a JSON array appear as sibling nodes in the data
> tree, otherwise a "must" constraint specified for the leaf-list won't
> work correctly. I don't think this is anyhow evident and IMO it has to
> be addressed. This is the purpose of section 3 in
> draft-ietf-netmod-yang-json-04.
> 
> Would it help if "validation" is replaced with "XPath evaluation"
> throughout this section?

No. I continue to believe (a) a datastore is validated and not an XML
infoset (or something like that) and (b) that the evaluation of "must"
constraints is conceptual.

JSON is an encoding. It remains unclear why you think that the JSON
encoding has any impact of what happens with a datastore.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 1 Jul 2015 14:03:15 +0200

> On 01 Jul 2015, at 09:21, Juergen Schoenwaelder <[email-masked]> wrote:
> 

Both NETCONF and YANG are absolutely silent about the data model of a datastore, so I assume it can be pretty much anything. Can you explain how a “must” constraint is conceptually evaluated on, say, key-value database?

Lada

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 1 Jul 2015 14:33:13 +0200

> Both NETCONF and YANG are absolutely silent about the data model of a datastore, so I assume it can be pretty much anything. Can you explain how a “must” constraint is conceptually evaluated on, sa…

This is a question to answer by someone who implements it on a
key-value database.

I feel we are wasting time and energy here.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 1 Jul 2015 15:01:46 +0200

> On 01 Jul 2015, at 14:33, Juergen Schoenwaelder <[email-masked]> wrote:
> 

Note that RFC 6020 just has XML mapping sections and otherwise tells nothing about how instances of data nodes are represented in a datastore. I have thus always assumed that everything works fine because the datastore model is essentially XML Infoset, except that “data tree” is used instead of “infoset”,  “data node” instead of “information item” etc.

In any case, with RFC 6020 in hand I can only explain how JSON is mapped to XML. I have no idea how to get JSON data into a datastore of unknown data model (which you take for granted), and then conceptually evaluate XPath expressions on it.

> I feel we are wasting time and energy here.

Indeed, and it is frustrating. However, I’d like to know what to do with section 3 of the yang-json document.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 1 Jul 2015 07:20:04 -0700

On Wed, Jul 1, 2015 at 6:01 AM, Ladislav Lhotka <[email-masked]> wrote:

>
> > On 01 Jul 2015, at 14:33, Juergen Schoenwaelder <
> [email-masked]> wrote:
> >
> >>> On 01 Jul 2015, at 09:21, Juergen Schoenwaelder <
> [email-masked]> wrote:
> >>>
> >>>>>>> On 30 Jun 2015, at 15:39, Juergen Schoenwaelder <
> [email-masked]> wrote:
> >>>>>>>
> >>>>>>>>> On 30 Jun 2015, at 15:20, Juergen Schoenwaelder <
> [email-masked]> wrote:
> >>>>>>>>>
> >>>>>>>>>>>>>> This is a notice to start a NETMOD WG last call for the
> document "JSON Encoding of Data Modeled with YANG":
> >>>>>>>>>>>>>>
> >> Both NETCONF and YANG are absolutely silent about the data model of a
> datastore, so I assume it can be pretty much anything. Can you explain how
> a “must” constraint is conceptually evaluated on, say, key-value database?
> >>
> >
> Note that RFC 6020 just has XML mapping sections and otherwise tells
> nothing about how instances of data nodes are represented in a datastore. I
> have thus always assumed that everything works fine because the datastore
> model is essentially XML Infoset, except that “data tree” is used instead
> of “infoset”,  “data node” instead of “information item” etc.
>
> In any case, with RFC 6020 in hand I can only explain how JSON is mapped
> to XML. I have no idea how to get JSON data into a datastore of unknown
> data model (which you take for granted), and then conceptually evaluate
> XPath expressions on it.
>
> >
I agree with Juergen that the implementation of YANG constraints
on  a datastore is not XML-specific.  The text refers to data nodes
not XML elements.  It is quite possible for a server to populate data
nodes in a datastore without NETCONF of XML being involved.

We keep getting stuck on the difference between resource representations
on the wire, and implementation details within a client or server.
There is no requirement that they be the same "XML or JSON".
inside.

Lada
> > /js

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 02 Jul 2015 08:33:04 +0200

> not XML elements.  It is quite possible for a server to populate data

The text says in sec. 6.4: "The data model used in the XPath expressions
is the same as that used in XPath 1.0 [XPATH], …", and the data model in
[XPATH] is about elements, attributes, namespaces etc. In order to
correctly interpret [XPATH], one needs to map (conceptually, if you
wish) the data tree onto the XPath data model.

> nodes in a datastore without NETCONF of XML being involved.

And what about RPCs and notifications? In this case the accessible data
tree is defined as the corresponding XML instance document.

Consider this definition:

  rpc foo {
    input {
      leaf x {
        type uint8;
        must ". = ../y";
      }
      leaf y {
        type string;
      }
    }
  }

Then what's the result of conceptual XPath evaluation for this RPC
request?

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
  <foo xmlns="http://example.com/xtest">
    <x>+1</x>
    <y>1</y>
  </foo>
</rpc>

And how about this RPC input in JSON?

{
  "foorpc:input": {
    "x": +1,
    "y": "1"
  }
}

> on the wire, and implementation details within a client or server.

This is not my problem here.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 2 Jul 2015 13:08:04 -0700

> tree is defined as the corresponding XML instance document.

There is no possibility for data to be converted between
XML and JSON if this is the case.

> Then what's the result of conceptual XPath evaluation for this RPC

Just because you can construct a pathological
case comparing strings and numbers doesn't
mean anything.

Use "number(foo) < number(bar)" to ensure that
numeric comparisons are done correctly.

> This is not my problem here.

There are lots of XPath expressions that can be constructed
that create problems because of the way XPath converts
types by default.  That is why there are functions number(), string(), etc.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 4 Jul 2015 08:32:04 +0200

> On 02 Jul 2015, at 22:08, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

I don’t know why, and it’s not about converting. Are you saying JSON encoding of RPC operations is ill-defined?
 
> case comparing strings and numbers doesn't

Pathological?? Don’t forget that uint64 numbers are encoded as strings in JSON. Moreover, comparisons of two nodesets are *always* performed on string values, even if both have numeric types in YANG.

> numeric comparisons are done correctly.

That’s not the point. The above XPath expression is perfectly valid so its result should be deterministic and identical to "non-conceptual” XPath evaluation. 

Saying that all gaps in the spec are pathological examples is an extremely slippery slope.

> types by default.  That is why there are functions number(), string(), etc.

Funny enough, 6087bis says:

   The 'local-name', 'namespace-uri', 'name', 'string', and 'number'
   functions SHOULD NOT be used if the argument is a node-set.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 4 Jul 2015 10:09:45 +0200

> > numeric comparisons are done correctly.

If this +1 != "1" needs fixing, I think it needs fixing in the YANG
specification. For datastores, things are kind of handled since RFC
6020 says:

   Note that since all leaf values in the data tree are conceptually
   stored in their canonical form (see Sections 7.6 and 7.7), any XPath
   comparisons are done on the canonical value.

For notifications, this is kind of handled as well:

   When a NETCONF server sends data, it MUST be in the canonical form.

And of course, we expect the server to only send notifcations that are
valid anyway.

For RPCs, the obvious things to do would be to require that XPATH
evaluation of RPC arguments also takes place on the canonical
representation of values.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Jul 2015 11:43:26 +0200

> specification. For datastores, things are kind of handled since RFC

Yes.

>    comparisons are done on the canonical value.

Yes.

> representation of values.

Agreed, perhaps the following change in sec. 6.4.1 could be sufficient:

OLD:

   In the accessible tree, all leafs and leaf-lists with default values
   in use exist (See Section 7.6.1 and Section 7.7.2).

NEW:

   In the accessible tree, all leafs and leaf-lists with default values
   in use exist (See Section 7.6.1 and Section 7.7.2), and all leaf and
   leaf-list values are conceptually in the canoonical form.

Lada