Subject: Re: [netmod] yang json and anyxml

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Sep 2014 23:25:49 +0200

On Fri, Aug 29, 2014 at 11:31:01AM +0200, Ladislav Lhotka wrote:
> 
> On 28 Aug 2014, at 17:32, Andy Bierman <[email-masked]> wrote:
> 
> > Agreed -- if mixed mode XML and entities were allowed, then encoding
> > the XML as a string would be best.
> > 
> 
> OK, so would a string be better than an object? Something like this:
> 
> An XML element that is an instance of a YANG anyxml data node is translated to a name/value pair where the value is a JSON string. The XML-JSON mapping of this string is not defined by this document. That is, the content of the XML instance may appear unchanged in this string value, or it may be processed in any way that’s appropriate for a given application. In the latter case, the definition of the anyxml data node SHOULD contain a description specifying the mapping of the XML content to JSON and vice versa. 
> 

Lada,

I think we are talking about an _encoding_ not about
_transformations_.  The purpose of an encoding is to transfer data
from A with B without making any modifications. Why do you insist that
implementations can make arbitrary _transformation_? What does
"appropriate for a given application" mean? How does the server know
what an application needs and is it not far better to let the
applications do whatever they want to be doing instead of outsourcing
some of the work to the NC/RC server? I think I prefer this to be
clearly defined and interoperable.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 02 Sep 2014 10:55:56 +0200

> clearly defined and interoperable.

I am just saying that the encoding of the anyxml data *may* be different
in JSON and XML. One reason could be that the XML content may not be
valid without the "outer" XML context (NS-prefix mapping).  I don't get
why it should be an interoperability problem. Maybe a better term than
"application-specific" is "anyxml-data-node-specific".

In fact, I treated "anyxml" more like "anydata", and I also assume that XML
needn't be the default and ever-present encoding. If a server uses anyxml for
configuring a banner page with markup, and supports only JSON, I really
see no reason why the content should be XML, which is what anyxml
requires.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 3 Sep 2014 08:23:49 +0200

> see no reason why the content should be XML, which is what anyxml

As technical contributor, I fundamentally disagree. If I have

   leaf banner {
     type anyxml;
     description
       "An XHTML banner to show to the user."
   }

then the client should get the banner in the same form (XHTML)
regardless how the protocol messages are encoded. It is not the job of
an _encoding_ to try to do _transformations_ in the hope the client
finds them useful.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 3 Sep 2014 08:47:46 +0200

> an _encoding_ to try to do _transformations_ in the hope the client

Sorry, anyxml is not a type and hence the correct example would be:

    anyxml banner {
      type anyxml;
      description
        "An XHTML banner to show to the user.";
    }

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 3 Sep 2014 10:26:13 +0200

> an _encoding_ to try to do _transformations_ in the hope the client

But it is the job of the *data model* to specify the form and meaning of the anyxml content. In your (corrected) example, the description of the “banner” anyxml node says the content has to be XHTML, and then XHTML should indeed appear in both XML and JSON. However, if the description prescribes XHTML for XML and Markdown for JSON, then this could/should be understood by both the server and client.

A client that uses JSON could configure the banner page using Markdown, and another client that uses XML will retrieve the banner as XHTML. I don’t know what’s wrong with it. The data modeller is free to decide what makes sense and whether it is feasible for server implementations to offer/accept encoding-specific forms of the same information.

To be able to move forward, could you propose your text for sec. 3.3.10?

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 3 Sep 2014 10:54:20 +0200

On Wed, Sep 03, 2014 at 10:26:13AM +0200, Ladislav Lhotka wrote:
> 
> But it is the job of the *data model* to specify the form and
> meaning of the anyxml content. In your (corrected) example, the
> description of the “banner” anyxml node says the content has to be
> XHTML, and then XHTML should indeed appear in both XML and
> JSON. However, if the description prescribes XHTML for XML and
> Markdown for JSON, then this could/should be understood by both the
> server and client.

A data model where the content depends on the protocol encoding is a
gross layer violation and IMHO broken. That said, if you want an
object that can be either XHTML or MD, then anyxml is the wrong
construction for this union.

> A client that uses JSON could configure the banner page using
> Markdown, and another client that uses XML will retrieve the banner
> as XHTML. I don’t know what’s wrong with it. The data modeller is
> free to decide what makes sense and whether it is feasible for
> server implementations to offer/accept encoding-specific forms of
> the same information.

The server is expected to translate MD to XHTML? Anyway, you are
making a layer violation here - there is the data model and there
is an encoding on the wire. The encoding needs to be transparent.

> To be able to move forward, could you propose your text for

I believe there are two issues here. First there needs to be an
agreement what exactly anyxml is and which constraints actually
appy. This is related to Y34 and should be resolved in YANG 1.1.

If the answer here is that anyxml is defacto a subset of XML that
matches NETCONF protocol constraints and we only have elements and no
mixed content, no PIs etc, then the solution to 3.3.10 is likely what
I understand Andy implemented, namely that anyxml elements translate
into JSON objects but without having information of the underlying
data model (e.g. no distinction between numbers and strings).

[ And I would be happy if such a data model ignorant translation would
  generally be allowed, but this is a different topic. ]

Before writing text, I think we should see whether we can reach
agreement since the debate is not about the details of the solution
but the general approach to take.

/js

PS: Even if we conclude that mixed content or PIs may appear in
    anyxml,

---

From: Andy Bierman <[email-masked]>
Date: Wed, 3 Sep 2014 04:35:55 -0700

On Tue, Sep 2, 2014 at 11:47 PM, Juergen Schoenwaelder <
[email-masked]> wrote:

> > > >> An XML element that is an instance of a YANG anyxml data node is
> translated to a name/value pair where the value is a JSON string. The
> XML-JSON mapping of this string is not defined by this document. That is,
> the content of the XML instance may appear unchanged in this string value,
> or it may be processed in any way that's appropriate for a given
> application. In the latter case, the definition of the anyxml data node
> SHOULD contain a description specifying the mapping of the XML content to
> JSON and vice versa.
> > > >>
> > > >
> > > > Lada,
> > > >

>         "An XHTML banner to show to the user.";

It seems odd to me to debate proper YANG typing for
something that, by definition, has no YANG type.
It isn't YANG!  It's a hole in the YANG schema where
data that has no syntax and no semantics can be stored.
(IMO, a really bad idea for data modeling).

2) No NETCONF implementation supports mixed mode XML.
There is only 1 mention of mixed mode XML in RFC 6241 in 6.2.5

  o  Filtering of mixed content is not supported.

The intent was to disallow mixed mode in all operations, not
just subtree filtering.

3) IMO the IETF spends too much time on broken text, worrying about
the letter of the law when there is little to no operational value to
be gained. Most implementors have never even tried to send mixed mode
XML to their server, and don't even know what will happen.  Obviously
if customers were using this, the implementors would know exactly
whether this worked or not.

> /js

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 3 Sep 2014 14:12:43 +0200

On 03 Sep 2014, at 13:35, Andy Bierman <[email-masked]> wrote:

> 
> 
> 
> On Tue, Sep 2, 2014 at 11:47 PM, Juergen Schoenwaelder <[email-masked]> wrote:

I fully agree. My original idea for section 7.3.3 in yang-json was similar:

An instance of anyxml data node is translated to a name/value pair in JSON, where “name” is the name of the anyxml data node. Any statement about the “value” part is outside the scope of this document.

I still think it is the easiest and best solution.

I also think anyxml is largely unnecessary and string or binary leafs could be easily used instead.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 3 Sep 2014 14:15:39 +0200

> whether this worked or not.

If we conclude that anyxml already today is what anydata is supposed
to be, then we should align the definition of anyxml in RFC602bis with
the reality and move forward (and update the resolution of Y34 - I
think you have an action on this one anyway).

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 03 Sep 2014 14:26:51 +0200 (CEST)

Juergen Schoenwaelder <[email-masked]> wrote:
> to be

If we do that, what does that mean for NETCONF?  RFC 6241 uses
'anyxml' to describe <edit-config> content etc.  If we make this
change, it seems to indicate that NETCONF cannot relly be used with
any other modelling languages than YANG.  Maybe this is ok.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 3 Sep 2014 14:55:12 +0200

> any other modelling languages than YANG.  Maybe this is ok.

Either Andy is correct that this is what all implementations do,
namely restrict content to things that can be defined in YANG (note
that my wording is different from your wording) or Andy's claim is not
correct. The 'all' quantifier makes this difficult to evaluate.
However, we can do a specific concensus call if necessary to settle
explicitly this one point.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 03 Sep 2014 14:59:08 +0200 (CEST)

Juergen Schoenwaelder <[email-masked]> wrote:
> that my wording is different from your wording)

Right; I started to write that, but then I was thinking that we have
to define 'anyxml' as data modelled with YANG, since otherwise the
JSON mapping won't work.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 3 Sep 2014 10:27:20 -0700

> to define 'anyxml' as data modelled with YANG, since otherwise the

Here is the extension Yuma has been using since 2008 to deal with the
most common use-case:

   extension root {
      description
        "Used within a container definition to indicate it is
       really a root container for a conceptual NETCONF database,
       instead of just an empty container.  This is needed
       for yuma to correctly process any RPC method
       that contains a 'config' parameter.";

It is used in several protocol operations like /edit-config/input/config or
/get-config/output/data.

    container data {
        ncx:root;
     }

A plain parser will not know that "ncx:root" means the child nodes are
expected to be to-level YANG data nodes., so it should have probably
been designed to use anyxml:

   anyxml data {
       ncx:root;
   }

I would like to add a new solution proposal for Y34:

Solution Y34-04:

* Consider a new statement called "root" that provides the ncx:root
semantics.
   There may be restrictions on its use (i.e., may not be full
data-def-stmt)

   root config {
      description "Container of top-level YANG data nodes representing the
configuration";
   }

* For YANG 1.0 backward compatibility, allow anyxml to be used,
   except implementations MAY restrict the XML so it is not considered
interoperable.

 * YANG Doctors will review each use of anyxml when YANG 1.1 is adopted,
    and avoid its use if possible (e.g., root should be used instead)
   (Does that mean anyxml SHOULD NOT be used -- deprecated?)

 * Do not add a replacement statement for anyxml (anydata) to YANG 1.1

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 04 Sep 2014 10:26:49 +0200

Hi,

I think there are actually two different cases:

1. Container for data that eventually have to conform to a YANG data
   model, but this data model is not known in advance.

2. Location in the schema tree where another (known) schema is to be
   grafted.

I understand your root extension is #1, and this is probably what's
needed when defining RPCs that may contain data snippets conforming to
any data model.

#2 is analogical to externalRef in RELAX NG. The use case I had was a
module describing network topology - nodes, links etc. I wanted to be
able to include with each node also its configuration described by our
standard modules. Currently the only option is to have the node
configurations in separate files and refer to them from the topology.

I think #2 is also closer to what Kent's configlets need.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Sep 2014 09:56:04 +0200

>  * Do not add a replacement statement for anyxml (anydata) to YANG 1.1

This sounds like Solution Y34-02 with the additional restriction that
the nodes inside the container need to be a top-level YANG data nodes
representing configuration. Why is this additional restriction needed
/ helpful?

/js

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Sep 2014 01:20:41 -0700

> representing configuration. Why is this additional restriction needed

This restriction matches the usage of the <config> and <data> containers
as the root of an edit or retrieval.  This use-case does not have any
holes in the schema tree.   There are no nodes of unknown syntax
and semantics allowed.

IMO we are not doing anything useful wrt/ anyxml so we should
just ignore it and move on.  It will continue to be used and abused
by implementations, with limited interoperability.  Some vendors
do not even support it at all.

Andy

---

From: Vladimir Vassilev <[email-masked]>
Date: Thu, 11 Sep 2014 14:10:40 +0200

Hello,

 > holes in the schema tree.   There are no nodes of unknown syntax

There is one more use-case of anyxml that is not caused by holes in the 
schema tree. Modelling of a recursive tree in YANG is currently not 
possible due to the following sentence is RFC6020 7.11. The grouping 
Statement -  "A grouping MUST NOT reference itself, neither directly nor 
indirectly through a chain of other groupings."

A simple model of an OID tree illustrates the problem (check the "case 
oid-container" part). In that case one has to use anyxml since the 
deterministic option of using recursive grouping is illegal YANG. There 
are certain advantages using such recursive models. Is there reason for 
not removing the "A grouping MUST NOT reference itself..." requirement 
and thus removing one more unnecessary use-case for anyxml in YANG models?

module oids {

     namespace "http://transpacket.com/ns/oids";

     prefix "oids";

     description
         "OID resolver module.";

     revision 2012-11-01 {
         description
             "Initial revision";
     }

     typedef type-enumeration {
       type enumeration {
         enum ASN_INTEGER;
         enum ASN_OCTET_STR;
         enum ASN_COUNTER;
         enum ASN_COUNTER64;
         enum ASN_GAUGE;
         enum ASN_OID;
       }
     }
     grouping oids-grouping {

         list oid {
             key num;
             leaf num {
                 type uint16;
                 description "oid num";
             }
             container contents {
                 choice leaf-container {
                     case oid-leaf {
                         leaf type {
                             type type-enumeration;
                         }
                         leaf value {
                             type string;
                             description "String value";
                         }
                     }
                     case oid-container {
                         //container oids {
                         //    uses oids-grouping;
                         //}
                         anyxml oids;
                     }
                 }
             }
         }
     }

     container oids {
         config false;

         description
           "Top-level container for all oid database objects.";

         uses oids-grouping;
     }
}

Vladimir

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Sep 2014 07:35:25 -0700

On Thu, Sep 11, 2014 at 5:10 AM, Vladimir Vassilev <[email-masked]
> wrote:

> deterministic option of using recursive grouping is illegal YANG. There are
> certain advantages using such recursive models. Is there reason for not
> removing the "A grouping MUST NOT reference itself..." requirement and thus
> removing one more unnecessary use-case for anyxml in YANG models?
>
>
Recursive groupings would not work in YANG like they do in real languages
because the "uses" is automatically expanded.  It needs to be manually
instantiated by the client or server at run-time, not automatically expanded
at compile-time. At this point, YANG might as well provide abstract and
concrete classes,
instead of a crude hack like anyxml.

But thanks for demonstrating my point, which is that you are not using
anyxml
to comply with its interoperable semantics. You are creating restrictions on
the content that are not conveyed to the client.

A "standard client" should be able to send any valid XML in an edit,
and that will be accepted as "anyxml".  How many people are using anyxml
as it is defined, not hacking it to provide a missing feature in YANG?

Andy

---

From: Vladimir Vassilev <[email-masked]>
Date: Fri, 12 Sep 2014 12:27:38 +0200

> the content that are not conveyed to the client.

I agree leaving groupings non-recursive makes groupings implementation 
work simpler and resolvable at compile-time. I agree introduction of 
concrete classes is proper solution for the recursive model problem. At 
the moment one can either write a really big yang file with e.g. 128!! 
recursive container definitions if one accepts limit to the depth of the 
tree or just live with an anyxml like in the example and leave a hole in 
the schema not conveying the restrictions to the client.

> as it is defined, not hacking it to provide a missing feature in YANG?
Except for the <data> and <config> cases there are very few places we 
use "anyxml" to represent configuration for tools which are not modelled 
in YANG. For those cases I do not see any advantage of the complexity of 
translation to json and keeping some relevance to the properties of the 
parent container e.g. namespace. That is just too much intelligence to 
ask from the protocol layer. anyxml is a string type with restrictions 
defined by the xml standard where properties as e.g. identation may or 
may not be preserved. Adding "anyjson" (wonder why this was never 
mentioned) and "root" (a private case of concrete class definition) is 
good step to resolve the confusion. As for the netconf protocol I think 
it should communicate anyxml as xml string and anyjson as json string. 
While "root" is encoded with the underlying choice of protocol encoding. 
For the case of recursive models we as users will just have to leave a 
hole in the schema with either "anyxml" or "anyjson" or write big 
recursive yang files until/if ever more flexible concrete class 
definitions are introduced.

Vladimir
>
>
>
>
>     module oids {
>
>          namespace "http://transpacket.com/ns/__oids
>     <http://transpacket.com/ns/oids>";
>
>          prefix "oids";
>
>          description
>              "OID resolver module.";
>
>          revision 2012-11-01 {
>              description
>                  "Initial revision";
>          }
>

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 15 Sep 2014 16:11:48 +0200

Hello Vladimir,
I have been arguing to have recursive containment for quite some time. 
Have a good number of use-cases to support.
regards Balazs

On 2014-09-11 14:10, Vladimir Vassilev wrote:
> There is one more use-case of anyxml that is not caused by holes in 
> the schema tree. Modelling of a recursive tree in YANG is currently 
> not possible due to the following sentence is RFC6020 7.11. The 
> grouping Statement -  "A grouping MUST NOT reference itself, neither 
> directly nor indirectly through a chain of other groupings."
>
> deterministic option of using recursive grouping is illegal YANG. 
> There are certain advantages using such recursive models. Is there 
> reason for not removing the "A grouping MUST NOT reference itself..." 
> requirement and thus removing one more unnecessary use-case for anyxml 
> in YANG models?
>
>
> module oids {
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>
>

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
Senior Specialist
ECN: 831 7320                        Tel: +36-1-437-7320
Mobile: +36-70-330-7909              email: [email-masked]