Subject: [netmod] yang grammar bug

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 Oct 2009 16:15:25 +0200 (CEST)

Hi,

Section 7.17 on extensions allow an extension to have a core YANG
statement as substatement:

   The substatements of an
   extension are defined by the extension, using some mechanism outside
   the scope of this specification.  Syntactically, the substatements
   MUST be core YANG statements, or also defined using "extension"
   statements.  Core YANG statements in extensions MUST follow the
   syntactical rules in Section 12.

But the ABNF grammar does not allow this:

unknown-statement   = prefix ":" identifier [sep string] optsep
                      (";" / "{" *unknown-statement "}")

For example, the following legal YANG is not allowed by the rule
above:

  foo:action my-action {
    input {
      list { ... }
    }
  }

I suggest we fix this by doing:

unknown-statement   = prefix ":" identifier [sep string] optsep
                      (";" / "{" *unknown-statements "}")

unknown-statement2   = [prefix ":"] identifier [sep string] optsep
                      (";" / "{" *unknown-statement2 "}")

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 07 Oct 2009 09:08:20 -0700

>    syntactical rules in Section 12.

I strongly object to this usage of extensions.
I do not think language extensions should be
able to use the YANG language itself to
define constructs outside the language.

Is there some semantics to go along with
the extension statement, or just pure syntax?
Can I rewrite the leaf-stmt if I don't like it,
and use that inside my extensions?  Why not?
Why can't I mix-and-match, or reinvent all
the YANG statements within my extensions?
Are you going to add a section to the draft containing
all the rules for constructs outside the language?

IMO, requiring the YANG compiler to look
for valid YANG constructs inside extensions is
a big change from the original requirements
for parsing extensions -- the parser just has to
skip extensions it doesn't support.

What is the use-case for all this complexity?
What is the standards-value of having such
a complicated mechanism for defining vendor
extensions to the language?

Extensions must use a prefix, so if the YANG
compiler encounters what could be a keyword
without a prefix, then the local module prefix rule
applies, right?  Or is this yet another special
case where prefix usage is defined by the
esoteric context in which it is used?

>                       (";" / "{" *unknown-statement2 "}")

What standards problem does this solve?

> /martin

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 07 Oct 2009 09:22:53 -0700

>                       (";" / "{" *unknown-statement2 "}")

This is not deterministic, and has no standards value
whatsoever.  The YANG syntax is not random.  There is a top-level
construction and every possible legal YANG statement has
a place within this 'syntax tree'.

In your example, the parser is not looking for
the 'input' statement unless the current context is the
rpc-stmt.  Randomly inserting sub-statements into extensions
is not supported by any programming language or any data modeling
language -- because it is nonsense.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 7 Oct 2009 13:37:36 -0400

>define constructs outside the language.

When the extension mechanism was being designed, we determined that
many YANG statements were useful and simple enough to use inside
an extension.  If the parser does not understand the extension,
then the contents of the extension can be easily skipped.  There
is no win in re-inventing an "acme:description" and "acme:reference"
to go inside the acme:foo extension.

>skip extensions it doesn't support.

The parser can skip the contents of extensions it doesn't support.
It will never notice that there's a "reference" statement inside
the acme:foo statement.

>What is the use-case for all this complexity?

The use case is reusing existing statements instead of making <n>
new ones.  There is no complexity in the current solution.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 07 Oct 2009 12:47:18 -0700

> It will never notice that there's a "reference" statement inside

This is true if the contents of the extension use
prefixed keywords.  The current ABNF is correct.
A prefix MUST be used for any keyword
used outside the language.

> new ones.  There is no complexity in the current solution.

Except they are not reused at all.
There is nothing at all in the spec that would
lead to the conclusion that the string 'input'
within an extension is somehow related to the
input-stmt, which is a sub-statement of the rpc-stmt.
There are no syntax rules to follow since the construction
'input-stmt' is not specified anywhere, as a sub-statement
of the extension.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 Oct 2009 09:23:47 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> prefixed keywords.

If the parser does not understand the extension, it can simply skip
any substatements, prefixed or not.  This is what the new ABNF rule
says.

> used outside the language.

Sure, we don't want to change that.  The idea is that an extension
should be able to reuse existing YANG statements.  Here are two
examples:

Some people have talked about "inline rpc" or "action".  The WG said
"not now" and encouraged them to write a draft with a proposal.  Since
YANG is extensible, such a draft would specify a YANG extension:
 
  extension action {
      argument name;
      description "...";
  }

Now, an action will take some parameters, and return a result.  A
parameter can be a leaf, list, choice, container...  And leafs need
types.  If such an extension cannot reuse existing statements, it
would end up duplicating almost all core YANG statements.

Another example is the "complex-type", defined in
draft-linowski-netmod-yang-abstract.  It should be defined as an
extension:

  extension complex-type {
      argument name;
      ...
  }

But in a complex-type, you need leafs, lists, choices, ... and again,
if you cannot reuse core YANG statements, we would have a *third* copy
of almost all core YANG statements in this draft.

NOTE: we do not propose any additions to the extension statement --
which core YANG keywords that are allowed within an extension must be
defined by some mechanism outside of YANG.  An implementation that do
not understand such an extension will still just skip it.

> Except they are not reused at all.

See above.  If we don't allow reuse of core YANG statements, we will
get more complexity b/c every new extension need to copy & paste core
statements.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 8 Oct 2009 09:30:01 +0200

>                       (";" / "{" *unknown-statement2 "}")

I support an ABNF change allowing us to reuse the 'descrition'
statement etc. in extensions.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 08 Oct 2009 00:53:43 -0700

> of almost all core YANG statements in this draft.

The YANG statements do not exist in some buffet-style
random mix-and-match mode.  They are specific constructions
at specific points in a regular grammar.

> get more complexity b/c every new extension need to copy & paste core

I don't want vendor extensions to the language.
IMO, it is not helpful to readers of YANG data models
if the YANG statements are used in uncontrolled,
undocumented, non-standard ways.  This allows every
vendor to create their own version of the YANG
language, and that harms interoperability.

By using a prefix, it is clear that the
vendor extension is not part of YANG.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 08 Oct 2009 11:16:12 -0700

> statement etc. in extensions.

Sec. 6.3.1 clearly specifies that all statements which are not
part of the production 'module-stmt' are language extensions
and they MUST have prefixes on all the keywords.

I think there is also text in the draft that says every unknown
keyword needs a corresponding extension-stmt, or an error will occur.

   stmtend             = ";" / "{" *unknown-statement "}"

These 'unknown-statement' syntax constructions,
which are not defined within YANG, have no semantic
coupling to the YANG syntax.

There is no standards value in letting every vendor define their own
dialect of YANG, picking any combination of sub-statements
they want, redefining whatever they want, etc.

The NETMOD WG decided NOT to use an 'action' or 'method' construct
instead of an 'rpc'.  If a vendor wants that, they can redefine
the 'input-stmt' using some extensions.  Too bad if it takes 1 data
model writer some time.  Use rpc-stmt if you want
to use the YANG standard.

Does an input-stmt for an 'action' have the exact same
semantics as it does for rpc-stmt?  If so, what is the
point of having 'action'? If not, then it only serves
to confuse YANG readers who think they know what input-stmt means.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 Oct 2009 21:20:21 +0200 (CEST)

> and they MUST have prefixes on all the keywords.

Again, no-one wants to change that.

> keyword needs a corresponding extension-stmt, or an error will occur.

Again, no-one wants to change that.

> they want, redefining whatever they want, etc.

This is a straw man.  We simply want to keep the possibility to reuse
existing statement in standard or vendor-specific extensions.

> model writer some time.

You miss the point.  If this WG or someone else wants to define some
new YANG construct, it can be done without updating YANG itself, by
using extensions.  By allowing those extensions access to the core
statements, we can keep the overall complexity down.  'action' and
'complex-type' were just two examples that have been discussed within
this WG.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 08 Oct 2009 12:49:29 -0700

> existing statement in standard or vendor-specific extensions.

I disagree that this is simple or good for the standard.
I think it is much cleaner to use YANG keywords only
within the module-stmt production.  Every 'unknown-statement'
is completely outside the YANG language and standard.

> 'complex-type' were just two examples that have been discussed within

I disagree that it is good for the standard to allow vendors
to use bits and pieces of the YANG syntax in their extensions.
The reason MIBs have standards value is precisely because
every MIB reader has to know just 1 version of SMIv2,
in order to read and understand any MIB module ever written.

I think this aspect of standards usage and
interoperability applies to YANG just as much
as it does to SMIv2.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 Oct 2009 22:08:06 +0200 (CEST)

> interoperability applies to YANG just as much

But it still applies (in the same way it applies to SMIv2).  If you
don't understand an extension, skip it and all its contents.  

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 08 Oct 2009 14:08:52 -0700

> don't understand an extension, skip it and all its contents.  

IMO, this is not helpful for interoperability.
By skipping your 'action' extension and
just reading the standard YANG, a reader would
not understand how to use the conceptual API
and its inherent contract, represented by the
standard YANG statements alone.

Since there are no rules whatsoever for 'unknown-statement',
the YANG reader is not going to have any idea from
vendor to vendor, or platform to platform, of how to
fully understand a YANG module, if some semi-random YANG statements
have been adapted in various ways to apply to ad-hoc,
undocumented, vendor-specific data modeling features.

The YANG reader is the highest priority when it comes
to design choices.  IMO, expecting readers to learn
a different dialect of YANG for potentially every NETCONF product
they use is not a what most people expect from a standard
data-modeling language.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 Oct 2009 23:20:35 +0200 (CEST)

> standard YANG statements alone.

The reader skips the 'action' extension, and all its contents.  Left
is pure standard YANG which the reader understands.

> Since there are no rules whatsoever for 'unknown-statement',

There are rules for unknown-statement.

Wait.  I had a typo in my proposal :(

This is what I wanted to say:

So an unknown-statement can occur as before, but *inside* the
unknown-statement, other unknowns, or core YANG statements can occur.

I am very sorry if this typo is the source of the confusion.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 08 Oct 2009 14:52:32 -0700

> is pure standard YANG which the reader understands.

That is not good enough for a standard.
The 'action' is used instead of an rpc-stmt.
The standard YANG and NETCONF mechanisms are not being extended,
they are being replaced.

When extension-stmt was added to YANG, we were assured that
it would not be mixed with YANG, and the WG agreed
all unknown statements needed to use a prefix.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Thu, 8 Oct 2009 21:52:00 -0400

>all unknown statements needed to use a prefix.

The WG also agreed that reuse of existing statements inside extension
statements made sense, since their use is common and natural.
Forcing extensions to define extension-specific versions of standard
YANG statements is not going to increase readability.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 00:28:37 -0700

> YANG statements is not going to increase readability.

Wow, I've been looking at this standards thing all wrong.
This extension thing is a great idea.  I don't like the
way YANG containers and leafs work, so I can define my
own 'xcontainer', which allows mandatory-stmt,
disallows presence-stmt; and my own 'xleaf',
which allows default and mandatory together,
default and mandatory only work the way I want
them to work, and throws in a few sub-clauses
that leaf should have had, but the WG decided against
them.

The YANG reader can just skip over the xcontainers and xleafs
and just read the rest of the module.

Any you can rewrite the parts of YANG you don't like,
and so can everybody else.  Seems like a great plan.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 9 Oct 2009 03:41:51 -0400

>Wow, I've been looking at this standards thing all wrong.

True.

>This extension thing is a great idea.

True.

>I don't like the
>own ...

Yup, you can roll your own and go your own way, and if no one follows
you, then your models will not be used.

But if there's a real need for something new, someone clueful can
publish a draft detailing this new thing, people will implement it,
and models that use it will be created and used.

The extension thing is a great idea because it allows the language
to grow and change over time.

Yes, we're sure to see lots of silly ideas that no one adopts, and
we're sure to see odd statements that cover odd cases (imagine
adding system-creatable as an extension) that can be ignored by
clients and servers that don't care about these odd cases.

In the end, the community will decide what works and what doesn't.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 07:16:15 -0700

> to grow and change over time.

yeah -- and you don't even need time.
Vendors can just start off right off the bat,
creating their own DML with YANG, while still claiming
that it is a 'standard'.

> In the end, the community will decide what works and what doesn't.

True, and since we are years away from any
standard configuration data models, a vendor
can claim conformance to NETCONF/YANG for many years
without actually implementing anything from the standard. Awesome!

People who want standards with 'plain old standards value' should
stick to SNMP and SMIv2, but they probably can't appreciate
all this 'vendor innovation' anyway.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 9 Oct 2009 10:09:03 -0400

>all this 'vendor innovation' anyway.

I'm not sure you are appreciating the extension work in progress
that Martin already mentioned.  Innovation is not a dirty word.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 08:04:20 -0700

> that Martin already mentioned.  Innovation is not a dirty word.

No, interoperability seems to be the dirty word.
We started this sandbox because there could
not possibly be any standard NETCONF content
with every vendor using their own DML and their own
interpretation of the conceptual NETCONF content layer.

So the solution is to end up with N+1 DMLs and conceptual
models to learn instead of N.  The only advantage is
that the standard content (ietf-*.yang) is forced to
use 'IETF YANG'.  The IETF is just another vendor
with their own solution in this equation.

Operators only need to learn the dialects of YANG
for the vendors they use, of course, and only
take a huge training and re-tooling hit if they
want to change vendors.  This seems rather similar
to the situation we have now with proprietary CLI.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 08:48:38 -0700

> that Martin already mentioned.  Innovation is not a dirty word.

The 'action' extension could easily be a simple
tag in the rpc-stmt instead of a hack replacement for
an inline rpc-stmt:

  rpc foo {
     acme:action;
     acme:calling-point "../some/path";

     input {  ... }
  }

But rpc-stmt is a top-level body-stmt, not a data-def-stmt.

The WG decided not to support rpc-stmt within the data-def-stmt,
so tail-f 'action' is just a way to ignore that WG decision.
Seems like a really weak standard to me, but it's no better
and no worse than the current 100% proprietary solution-space.

As Martin pointed out, within an unknown-statement, YANG statements
do not exist.  There are just some identifiers without prefixes.
There is only the generic YANG syntax:

    [prefix ':'] identifier [string] stmtend

I want this text added to sec. 6.3.1:

   If the YANG compiler does not support a particular extension,
   which appears in a YANG module as an unknown-statement,
   the entire unknown-statement MAY be ignored by the compiler.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 9 Oct 2009 10:01:18 -0700

Hi -

> From: "Phil Shafer" <[email-masked]>
> To: "Andy Bierman" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, October 09, 2009 12:41 AM
> Subject: Re: [netmod] yang grammar bug
...
> In the end, the community will decide what works and what doesn't.

An excellent argument for classifying this stuff as "Experimental"
rather than standards track.  :-(

Randy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 10:35:15 -0700

> rather than standards track.  :-(

I was thinking the same thing.
It seems like the 'IETF version of NETCONF/YANG'
will be the official experiment, which will compete
amongst the vendor experiments for market share and
interoperability.

Maybe Phil is right.
Nothing the IETF has done in 25 years has produced
any meaningful interoperability for NM configuration,
so maybe an Experimental approach will eventually
produce a workable standard.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 9 Oct 2009 14:41:12 -0400

>rather than standards track.  :-(

An excellent argument for classifying specific extensions as
"Experimental", which I think is natural.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 09 Oct 2009 21:38:58 +0200 (CEST)

>    the entire unknown-statement MAY be ignored by the compiler.

Fine with me.  I will add it unless someone objects.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 09 Oct 2009 12:42:23 -0700

> "Experimental", which I think is natural.

You are right.
I called IETF N/Y the 'official experiment'.
One could argue that describes every Proposed Standard RFC.

Andy