Subject: [Netconf] usage of groupings

From: Martin Bjorklund <[email-masked]>
Date: Wed, 12 Nov 2014 10:13:55 +0100 (CET)

Hi,

At the meeting yesterday the issue of RESTCONF's usage of groupings to
define data structures came up.

I'd like to point out that it is perfectly legal to define groupings
that are not being used.

What is unorthodox is that RESTCONF ties the contents of these
groupings to a YANG module name / namespace not by 'uses' but in
english text in the document (this text needs to be clarified if we
keep this construct).  This isn't worse than defining the complete
structure in plain text.

The problem we're trying to solve is to have a single, formal,
definition of these structures, than can be encoded in XML or JSON.
If we were to use XSD it wouldn't work for JSON, and if we were to use
JSON schema (or whatever) it wouldn't work for XML.

One alternative to this is to define an extension statement:

  extension structure {
    argument name;
    description
      "Introduces a named data structure without any semantics
       attached.  The data nodes in the structure are defined in the
       namespace of the module defining the structure."
  }

Then we would replace the groupings with:

  rc:structure errors {
    container errors {
      ...
    }
  }

  rc:structure collection {
    container collection {
      ....
    }
  }

and so on.

Or even better, add such a statement to YANG 1.1.

(incidentally, we have such an extension in our implementation that we
use for this and similar purposes...)

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 12 Nov 2014 10:25:43 +0100

Martin,

is what you propose generally useful or are you just trying to solve a
corner problem with the definition of RESTCONF? If the later, the
price of adding a new construct may be high, no? Is the idea here to
provide a mechanism to define arbitrary XML/JSON protocol messages in
YANG?

And why would this go into YANG 1.1? If this mechanism is needed at
all, would it not be possible to define it separately as a proper YANG
extension? (And you know that the YANG 1.1 issue collection ended
months ago and this does not seem to qualify as a bug fix.)

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 12 Nov 2014 09:08:55 -0800

> keep this construct).  This isn't worse than defining the complete

+1

I would like to keep using groupings for this purpose.
What if I want to use these definitions in my data models?
I cannot do that unless the definitions are in groupings.

For example, let's say my app has a structured log file format
and I want to log the <errors> in the response?
It is perfectly reasonable to use the grouping instead of
cut and pasting.

For this purpose, a YANG extension is worthless.
Only a grouping provides the reusable definitions without
hacking or copying.

Andy

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 12 Nov 2014 18:43:33 +0100 (CET)

> provide a mechanism to define arbitrary XML/JSON protocol messages in

I think this could be useful in other cases than RESTCONF.  The goal
is not to provide a mechanism to define arbitrary XML/JSON messages,
but of course this might happen if people see that it can be used in
this way.

> extension?

It is unclear if such an extension can be said to define a new type of
data definition statement and as such make it possible to use normal
augment - e.g., would this be ok with the error structure:

  augment /rc:errors { ... }

> (And you know that the YANG 1.1 issue collection ended

Sure...

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 12 Nov 2014 18:44:27 +0100 (CET)

> Only a grouping provides the reusable definitions without

Well, you can certainly have both:

  grouping errors { ... }

  rc:structure error {
    uses errors;
  }

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 12 Nov 2014 07:47:05 -1000

> I'd like to point out that it is perfectly legal to define groupings

This was not the point of my initial comment.

> keep this construct).  This isn't worse than defining the complete

I think we should think twice before doing that. The handling of namespaces in groupings is tricky - people get often confused about the fact that the namespace is assigned according to the module where the grouping is used, not where it is defined. So breaking this rule and saying that in this particular case the namespace is that of the module where the grouping is defined is IMO very dubious. 

> JSON schema (or whatever) it wouldn't work for XML.

I understand that. My proposal was to use augments instead with an artificial target representing the HTTP response.

Lada

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 12 Nov 2014 19:10:45 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> I think we should think twice before doing that. The handling of namespaces in
> groupings is tricky - people get often confused about the fact that the
> namespace is assigned according to the module where the grouping is used, not
> where it is defined. So breaking this rule and saying that in this particular
> case the namespace is that of the module where the grouping is defined is IMO
> very dubious.

The idea is not to break the rule.  The idea is to express in text
that we have the equivalent of "uses errors", i.e. the instantiation
of the grouping that binds to the namespace.  I agree that this is
special and that is why I prefer to use an extension statement.

> target representing the HTTP response.

IMO an artificial target is worse.  Either you augment something that
doesn't exist, or you have to define data nodes that aren't really
data nodes.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 12 Nov 2014 09:57:51 -1000

> The idea is not to break the rule.  The idea is to express in text

For instance, an example in the text shows that the “collection” element is in the namespace of the ietf-restconf module but nothing in that module provides any reason for this.

Phil was right when he said that if this was done in another WG, we would think they don’t understand how namespaces work in groupings.

> special and that is why I prefer to use an extension statement.

OK.

> doesn't exist, or you have to define data nodes that aren't really

Well, it is clear in any case that YANG is somewhat misused here as a general schema language. I am all for making this use of YANG more “official” in YANG 1.1.

Lada

---

From: Phil Shafer <[email-masked]>
Date: Sun, 16 Nov 2014 16:16:17 -0500

>Well, you can certainly have both:

And they are distinct uses; groupings is used to reusability, where
rc:structure would be used for, well, whatever the extension defines
it as being used for.

My objection is that an existing mechanism (groupings) is being
repurposed in a way that it wasn't used for.  My question was "would
we be upset if another working group had done this?".

Also I wonder if we'll hear "I though groupings were used to
define protocol messages" ;^)

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Sun, 16 Nov 2014 13:27:24 -0800

> we be upset if another working group had done this?".

I would hope we would listen to their reasons first and try
to find out why YANG is being extended to represent
reusable data structures, including CLI parameters,
log files, audit files, and protocol messages.

> define protocol messages" ;^)

too late for that -- already being done

(YANG-ification has started.  ;-)

You shouldn't of helped create a DML with the best balance
of readability and automation features ever created :-)

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sun, 16 Nov 2014 13:39:17 -0800

> we be upset if another working group had done this?".

We wanted to provide a normative syntax and description that
is encoding-independent, which also supports the specific
XML and JSON encoding requirements for NETCONF and RESTCONF.
(CoMI will define a YANG to CBOR encoding, so the list will grow...)

This is going to be a real popular thing to do....

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 16 Nov 2014 22:42:10 +0100 (CET)

> This is going to be a real popular thing to do....

If we beleive this is going to be used more and more, I think we
should define a proper statement for this.  At least as an extension.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sun, 16 Nov 2014 14:31:41 -0800

> should define a proper statement for this.  At least as an extension.

I prefer a real statement in YANG 1.1.  There is already an
"anyxml + extensions" based solution available in YANG 1.0.

I would like it cover the following use-cases:

   1) template for YANG document root (<config> or <data> element)
   2) template for a container for a specific node type
   3) template for a container that has an "instance wrapper" list for
        instances from 1 or more 'contexts' (e.g YANG mount server instance)
   4) template for an abstract reusable data structure (te.g. ietf-restconf)

IMO, solution Y34-04 should be extended to support (2 - 4)

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 17 Nov 2014 03:09:34 -0500

>This is going to be a real popular thing to do....

Then it's even more important that we do it right.  Is there
any objection to defining an extension suitable to this purpose?

I strongly oject to repurposing groupings for this.  It's
a bad precendent.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 17 Nov 2014 03:12:51 -0500

>"anyxml + extensions" based solution available in YANG 1.0.

This is also a bad precendent.  If everyone thinks their new module
requires a new version of YANG, we're DOA.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 17 Nov 2014 09:35:47 +0100

> "anyxml + extensions" based solution available in YANG 1.0.

YANG 1.1 is not supposed to become a kitchen sink. What can be done
using extensions should be done using extensions.

Note that such a 'structure' extension would have to answer what all
the bits and pieces that are bound to the notion of datastores mean,
config true/falls, must and when expressions, ... (it could be that
many of these things would not mean anything but even that would have
to be defined).

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 17 Nov 2014 09:40:50 +0100 (CET)

> requires a new version of YANG, we're DOA.

Well, we *are* doing YANG 1.1 now so we do have an opportunity to get
it right (if adding it is right...)

For this particular statement, having it as a builtin YANG statement
has one advantage - we can use 'augment' on the resulting structure:

If we do it as an extension:

  module my-module {
    ...
    rc:structure my-structure { ... }
  }

normal augment cannot be used:

  augment /my:my-structure {  // won't work!
    ...
  }

An alternative could be to mark the extension as introducing an
augmentable node:

  extension structure {
    augmentable true; // new statement!
    ...
  }

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 17 Nov 2014 01:05:39 -0800

> requires a new version of YANG, we're DOA.

It's hard to even respond to a strawman comment like this.
I suppose OpenDaylight will stop using YANG as an IDL know they
have been properly educated on the use of YANG...

Anyxml is unconstrained.  Any proper YANG tool will accept anyxml
and the tools that know the extension will check the refinement
indicated by the extension. YANG will continue to be used for templates
even if they are moved to private modules and ignored in IETF modules.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 17 Nov 2014 09:40:46 -0800

> any objection to defining an extension suitable to this purpose?

How about this:

module ietf-restconf {

   extension structure { ... Martin provided ext. def }

   // keep all groupings

   rc:structure errors {
       uses errors;
   }

  rc:structure restconf {
       uses restconf;
   }
}

This will use the groupings in this module and let them be reusable
in external modules.  As Martin said there is nothing invalid
about defining a grouping for external reuse. Any XPath can
all be relative to the grouping, so it is reusable as well.

The extension provides a module namespace binding to make Lada happy.
We can reference specific structure statements in the normative text
instead of copying lots of text about XML and JSON encoding rules.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 18 Nov 2014 10:21:26 -0500

>all be relative to the grouping, so it is reusable as well.

Taking a step back, are we really looking at using YANG for
an interface definition language (IDL)?  Would it make sense
to define an IDL module with a full set of IDL extensions?
idl:id, idl:version, idl:request, idl:response, etc. would
be useful.

How does the current groupings reuse hook requests to replies?

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 18 Nov 2014 07:43:48 -0800

> How does the current groupings reuse hook requests to replies?

I am not trying to define an IDL.
The module containing the usage of the extension binds a
YANG module namespace to the structure definition.
Once it has a namespace, semantics, and syntax, it can
be reused.

Our usage in the RFC is to simply have these 3 properties.
The normative text in the RFC says "the foo resource type
is encoded according to the structure "foo" in the "ietf-restconf" module.
The protocol spec using the "structure" extension defines its own
protocol messages.

My logging module can happily import grouping "errors" and use that
in a plain old YANG monitoring module.  There are never any
protocol messages incorrectly exposed as configuration data.

Andy