Subject: [netmod] extensions and conformance

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 10 Aug 2015 10:14:22 +0200

Hi,

recent discussions show that 6020(bis) text about extensions isn't
sufficiently clear about the scope and semantics of extensions. IMO this
needs to be fixed and so I propose to add the following item to YANG 1.1
issue list. Comments and additional solutions are welcome.

Lada

------------------------------------------------------------------------
* NEW :Yxx: clarify conformance wrt extensions

** Description

   YANG extensions as defined in RFC 6020 have no limits on scope –
   they can possibly modify YANG language, datastore semantics or even
   the NETCONF protocol. However, from the text in RFC 6020 it is
   unclear whether extensions appearing in YANG modules advertised by
   a server are mandatory to implement: "If a YANG compiler does not
   support a particular extension, which appears in a YANG module as
   an unknown-statement (see Section 12), the entire unknown-statement
   MAY be ignored by the compiler."

** Solution Yxx-01
   
   Extensions appearing in the server's model are an integral part of
   the server-client contract. That is, the server MUST implement
   them, and the client SHOULD terminate the session if it doesn't
   implement any of the extensions.

** Solution Yxx-02

   Develop a mechanism for negotiating extensions.

** Solution Yxx-03

   Make extensions optional. This means that extensions won't be
   allowed to change YANG language, NETCONF protocol, and validity of
   datastores and protocol messages.
------------------------------------------------------------------------

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2015 03:17:25 -0700

Hi,

I am strongly against changing the contract on extensions.
They MAY be ignored by any YANG tool. Period.
That means they are far from mandatory.
They are little more than a keyword and a description clause.

There are not even any rules or help determining
which extensions can appear as children of other
extensions.  That indicates that they are not
real statements and not really part of YANG at all.

There are some uses of extensions that fit the modular design,
but not many.  The NACM extensions only affect NACM.
They have no impact whatsoever on any YANG statement.
Yet even these statements should be real YANG statements.
There is really no need to force implementation of NACM
in order to utilize the concept of 'default-deny-all'.
But full implementation of NACM is required to use these
extensions.

Andy

On Mon, Aug 10, 2015 at 1:14 AM, Ladislav Lhotka <[email-masked]> wrote:

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 10 Aug 2015 13:24:50 +0200

> On 10 Aug 2015, at 12:17, Andy Bierman <[email-masked]> wrote:
> 
> Hi,
> 

So do you prefer my proposed solution -02 or -03?

> But full implementation of NACM is required to use these

The descriptions of “default-deny-write” and “default-deny-all” describe normative server behaviour. It is not even clear from 6020 text whether server implementations can ignore these extensions or not and still use the modules where the extensions are present.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2015 08:32:26 -0700

** Solution Yxx-03

This is how we use extensions to tag YANG data models
to drive automation tools.

Your entire problem statement assumes that the consumer
of the extension is a protocol (client and/or server).
IMO this is really bad design to use statements for standards
that are by definition external to the YANG language, as if they
were part of the YANG language.  This is just a hack and
it usually doesn't work well.

For example, when I pointed out that an extension for "ephemeral"
could not be modified with a refine-stmt, you just changed the
subject and ignored that problem.

 Andy

> The descriptions of “default-deny-write” and “default-deny-all” describe
> normative server behaviour. It is not even clear from 6020 text whether
> server implementations can ignore these extensions or not and still use the
> modules where the extensions are present.
>
> Lada
>
> >
> > Andy
> >
> >

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 10 Aug 2015 18:37:32 +0200

> On 10 Aug 2015, at 17:32, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

But that means, for example, that annotations cannot be defined via an extension statement as in draft-ietf-netmod-yang-metadata.

> it usually doesn't work well.

Well, in the past you supported the idea of introducing new XPath functions through extensions, and it is exactly of this sort.

I’d also argue that current wording of 6020 allows a server implementing “ietf-system” and “ietf-netconf-nacm” to ignore the nacm:* extensions and make e.g. container “authentication” writeable by default in non-recovery sessions.

> subject and ignored that problem.

I didn’t ignore that problem, I just didn’t understand why “ephemeral” (or any) extension needs to be refined.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2015 09:46:19 -0700

On Mon, Aug 10, 2015 at 9:37 AM, Ladislav Lhotka <[email-masked]> wrote:

> But that means, for example, that annotations cannot be defined via an
> extension statement as in draft-ietf-netmod-yang-metadata.
>
>

Which means these statements should be real instead of extensions.
If the statements are defining data that is exchanged between
peers in a standard protocol, then YANG extensions are not
appropriate.

> Well, in the past you supported the idea of introducing new XPath
> functions through extensions, and it is exactly of this sort.
>
>
I don't remember the details, but new XPath functions
should be part of the language, not extensions.

> I’d also argue that current wording of 6020 allows a server implementing
> “ietf-system” and “ietf-netconf-nacm” to ignore the nacm:* extensions and
> make e.g. container “authentication” writeable by default in non-recovery
> sessions.
>
> >

Because it is common for data is defined in a grouping without any
config-stmt
(or ephemeral-stmt) and then add these in for each uses-stmt.

Lada
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 10 Aug 2015 19:01:02 +0200

> On 10 Aug 2015, at 18:46, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

I agree, and -03 is my preferred solution, too.

> Because it is common for data is defined in a grouping without any config-stmt

OK, I see.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2015 10:36:27 -0700

> I agree, and -03 is my preferred solution, too.

This would allow "if-feature" to be used so attributes would not
have to be mandatory.

Andy

> > > The descriptions of “default-deny-write” and “default-deny-all”
> describe normative server behaviour. It is not even clear from 6020 text
> whether server implementations can ignore these extensions or not and still
> use the modules where the extensions are present.
> > >
> > > Lada
> > >
> > > >
> > > > Andy
> > > >
> > > >

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 19 Aug 2015 14:49:00 +0200 (CEST)

Hi,

[joining this discussion a bit late]

> I agree, and -03 is my preferred solution, too.

I strongly disagree.  IMO the IETF usage of extensions so far (NACM,
annotations) works well (in the case of NACM proven by multiple
independent implementations), and follows how extensions were intended
to be used, and obviously, how the WG has understood them up until
now.

If the sentence:

   If a YANG compiler does not support a particular extension, which
   appears in a YANG module as an unknown-statement (see Section 12),
   the entire unknown-statement MAY be ignored by the compiler.

in 6020 can be interpreted to mean that an occurance of an extension
is non-normative, we should fix it so that it matches what was
intended and how it is used.

Juergen suggested this replacement text, which I support.  Maybe it
can be improved even more.

       If a YANG parser does not support a particular extension, which
       appears in a YANG module as an unknown-statement (see Section 13),
       the entire unknown-statement MAY be ignored by the parser. Note
       that even in this case the semantics associated with the extension
       still apply (as if they were part of a description statement).

However, I do agree that an extension cannot "undo" semantics defined
by other YANG statements, just as you cannot do that in a description
statement.  For example, this would be illegal:

  leaf foo {
    type int32;
    description "The type is really a string.";
  }

  leaf bar {
    type int32;
    xx:real-type string;
  }

It is also a Very Bad Idea (speaking from experience...) to define an
extension that introduces new data nodes.

The text about "MAY ignore" was meant to capture this.  It's supposed
to be like for a client that doesn't understand an augemntation; it
can skip it.  A client that doesn't understand nacm:default-deny-all
works just fine and can safely ignore it.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 19 Aug 2015 08:32:28 -0700

>        still apply (as if they were part of a description statement).

I strongly disagree with this change.
This would mean that every tool is required to support the
semantics of every extension ever defined.  For example,
we have our own extensions that cause the server to
perform internal tasks certain ways (like ywx:sil-delete-children-first).

According to the text you propose, all servers would have to
provide the behavior embodied in the syntax of this extension
statement?  This extension does not alter any protocol behavior
at all, so it is "well-behaved".

A server MAY skip over the entire extension and completely ignore it.
Other servers are not required to provide the behavior described
in the extension semantics.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 19 Aug 2015 19:49:34 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> semantics of every extension ever defined.

No.  It means that if a server advertises module that uses some
extension to define some behaviour, the server supports that
behavior.  Just as we expect a server to support the text in
description statements.

For example, the nacm: extensions do not apply unless ietf-netconf-acm
is advertised (and nacm is enabled).  I expect most extensions to work
this way.  Another example is if we actually defined i2rs:ephemeral;
this would have no effect unless the "i2rs" capability (whatever that
is) is also advertised.

The text also means that it is perfectly ok for a client to ignore the
extension if it doesn't understand it.  For example, if the client has
no idea what the ephemeral datastore it, it doesn't matter that a node
is marked with i2rs:ephemeral true.

> For example,
> statement?

Yes, all servers that advertise such a module.  B/c as you note below
it is "well-behaved".  So the defintion probably says something like
"if the server implements SIL, then it will delete the children before
it deletes the node itself".

> This extension does not alter any protocol behavior

Yes.  Aren't we in agreement here?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 19 Aug 2015 21:28:43 -0700

> Yes.  Aren't we in agreement here?
No -- I agree with Randy's interpretation of MAY.

However, IMO description-stmts are normative, so if plain text in
the NACM RFC says the nacm YANG extensions MUST be supported,
then I think that makes it mandatory.

YANG conformance does not make it mandatory.

We are in agreement if you mean the RFC or module
containing the YANG extension can define the conformance for the extension.

But I don't agree that statements related to data handling and
datastores should ever be extensions. IMO NACM extensions
should really be YANG 1.1 keywords and the extensions should
be deprecated.

I like YANG extensions -- they are like the minor leagues for YANG
statements. :-)
But once they are stable and there is consensus they are useful, they should
probably be made mandatory by converting them to regular statements.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 10:04:19 +0200 (CEST)

> containing the YANG extension can define the conformance for the extension.

Yes, maybe this is what I am trying to say.  I guess I can't see how
that would be possible if you insist that all extensions always are
truly OPTIONAL.

> should really be YANG 1.1 keywords and the extensions should

I don't understand why you think that.  We agree that it is ok for the
RFC/module for NACM to defined the conformance for the nacm
extensions (and this apparently work in multiple implementations), so
why should these be proper statements?

> probably be made mandatory by converting them to regular statements.

This is probably true in some cases, but I also like the idea of
extensibility and modularity.  It's a feature, not a bug, that we
could define NACM w/o revising YANG.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 10:48:53 +0200

> behavior.  Just as we expect a server to support the text in

This is unclear both from the current text and from Juergen's
proposal. Maybe I am a nitpicker but assume that either (i) server
implementation is completely generated from the data model, or (ii) the
parser in question is simply in the implementor's head. Then I don't
know how the server can implement the semantics of an extension if the
extension is removed while YANG modules containing it are being parsed.

A text like "a client MAY ignore an extension that it doesn't
understand" would be clearer but I don't think it is acceptable if the
extension changes

- semantics of YANG,

- semantics of the protocol,

- schema of the data model (as annotations do).

> this would have no effect unless the "i2rs" capability (whatever that

The nacm: extensions are probably OK because they only give some
additional info about the server behaviour that is independent of
client's support. If the client ignores them, then it may be just
wondering why it cannot read or write some data.

> is marked with i2rs:ephemeral true.

I am not convinced at all about this one. If the server advertises the
"i2rs" capability and modules that use "i2rs:ephemeral" extension, then
a normal NETCONF/RESTCONF client can ignore both (RFC 6241: "Each peer
[...] MUST ignore any capability received from the other peer that it
does not require or does not understand."). Buth what then: can the
client treat nodes tagged with "i2rs:ephemeral" just as standard nodes?

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 12:37:41 +0200 (CEST)

> understand" would be clearer but I don't think it is acceptable if the

Agreed.

> - semantics of the protocol,

Maybe...

> - schema of the data model (as annotations do).

I think it is ok to define an annotation with an extension (as the
draft does).  When designing a feature that is going to use an
annotation it is important to keep in mind that not all clients will
understand the annotation.  So e.g. if the new feature is a "comment",
the server might just blindly return them to all clients, but in the
case of "inactive", the server must not send these attributes to a
client that doesn't ask for them.  (You may argue that even for
comments the client should ask for them, but that is besides the
point).

The bottom line is that you have to be careful when you define a new
extension statement.

> wondering why it cannot read or write some data.

Right.

> client treat nodes tagged with "i2rs:ephemeral" just as standard nodes?

My idea has been that it would be used as:

   leaf my-ephemeral-leaf {
      config false;
      i2rs:ephemeral true;
      ...
   }

which means that if the client doesn't know what to do with
"i2rs:ephemeral", it will ignore it, as just see this as a normal
config false node that can be read with <get>.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 13:06:06 +0200

> On 20 Aug 2015, at 12:37, Martin Bjorklund <[email-masked]> wrote:
> 

For a client that doesn’t understand the annotation (or annotations in general) it is just extra junk that’s not in the data model. This is no different from a new data node type defined through an extension - in fact, in JSON encoding annotations are just special object members. A prudent client may consider such data invalid.

> comments the client should ask for them, but that is besides the

IMO “inactive” data change the NETCONF/RESTCONF protocol. If a client doesn’t see inactive data, it may think it is OK to write its own content to that place, which probably shouldn’t be allowed.

> config false node that can be read with <get>.

It of course depends on the particular usage rules - I assumed ephemeral data would be config=true. But even in your example, if ephemeral data are in a special datastore, they may not be returned with <get>, which can again violate the schema.

I think arbitrary extensions are OK if they are used in a controlled environment, e.g. in single vendor’s server and client software, but in general they can break interoperability unless they are accompanied with a bidirectional client-server negotiation mechanism.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 13:12:42 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> For a client that doesn’t understand the annotation (or annotations in
> general) it is just extra junk that’s not in the data model. This is
> no different from a new data node type defined through an extension -
> in fact, in JSON encoding annotations are just special object
> members. A prudent client may consider such data invalid.
> 
> IMO “inactive” data change the NETCONF/RESTCONF protocol. If a client
> doesn’t see inactive data, it may think it is OK to write its own
> content to that place, which probably shouldn’t be allowed.
> 
> > 
> It of course depends on the particular usage rules - I assumed
> ephemeral data would be config=true. But even in your example, if
> ephemeral data are in a special datastore, they may not be returned
> with <get>, which can again violate the schema.

Right, this idea assumes that ephemeral data *is* returned by <get>.

> I think arbitrary extensions are OK if they are used in a controlled
> environment, e.g. in single vendor’s server and client software, but
> in general they can break interoperability unless they are accompanied
> with a bidirectional client-server negotiation mechanism.

I agree that it is important to think about protocol negotiation
mechanisms, client backwards compatibility etc. when defining
extensions.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 13:23:48 +0200

> On 20 Aug 2015, at 13:12, Martin Bjorklund <[email-masked]> wrote:
> 

But even then it might be difficult to handle situations where one client supports an extension while another doesn’t - the server simply may not be able to behave simultaneously in two different ways. That’s why I think the only option that’s really safe for now is

** Solution Yxx-03

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 13:28:57 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> But even then it might be difficult to handle situations where one
> client supports an extension while another doesn’t - the server
> simply may not be able to behave simultaneously in two different
> ways.

But just b/c there are *some* potential extension that would have
problems, does not mean that *all* extensions should be banned.

> That’s why I think the only option that’s really safe for now
> is
> 
> ** Solution Yxx-03
> 

I think we need explicit text in order to be able to agree on a
solution.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 14:09:41 +0200

> problems, does not mean that *all* extensions should be banned.

My concern is that vendors might misuse extensions for creating silos -
we all remember browser wars, right?

And in the particular case of "annotation", I am not sure whether it is
really OK for a server to advertise an annotation in a module and then
start using it without client's consent.

> I think we need explicit text in order to be able to agree on a

Section 6.3.1:

OLD

NEW

   Extensions MUST NOT affect the following aspects:

   o syntax and semantics of YANG language,

   o management protocols such as NETCONF or RESTCONF,

   o validity of datastores and protocol messages with respect to the
     data model.

   A server advertising modules that define and use an extension MUST
   adhere to the extension's semantics as specified in its definition.

   However, clients MAY ignore any or all extensions appearing in
   modules advertised by the server.

By the way, due to the adopted solution Y49-04, the second paragraph in
sec. 6.3.1 should also be removed.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 15:26:03 +0200 (CEST)

>    o syntax and semantics of YANG language,

I don't know what this means, but it sounds ok...

>    o validity of datastores and protocol messages with respect to the

I am not sure I understand these two either.  I don't think this is
needed.  It's like if we wrote that the text in description statements
MUST NOT "affect management protocols like NETCONF", etc.  If someone
tries to get such an extension standardized, hopefully reviewers will
catch this.

>    adhere to the extension's semantics as specified in its definition.

Ok.

>    modules advertised by the server.

Hmm, I agree with Andy's idea that the definition of an extension
defines the conformance for the extension.

> sec. 6.3.1 should also be removed.

I assume you mean the second paragraph of 6.3.1 in RFC 6020?  That
paragraph is already removed in draft-ietf-netmod-rfc6020bis-06.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 15:43:18 +0200

> On 20 Aug 2015, at 15:26, Martin Bjorklund <[email-masked]> wrote:
> 
> I don't know what this means, but it sounds ok…

Examples: an extension cannot say that 

- a YANG statement may have two arguments,

- a leaf-list may have duplicate entries. 

> tries to get such an extension standardized, hopefully reviewers will

I think we should state *some* restrictions, albeit general. Otherwise authors may be wondering why reviewers reject their great extensions.

With descriptions I also don’t know how to interpret (and restrict) their normative character. For example, would this be OK?

description "This leaf can be configured only on Christmas Day."

> defines the conformance for the extension.

Now I don’t know what this means.

> paragraph is already removed in draft-ietf-netmod-rfc6020bis-06.

OK, I thought I was looking into 6020bis, sorry.

Lada

---

From: Nadeau Thomas <[email-masked]>
Date: Thu, 20 Aug 2015 10:00:39 -0400

> On Aug 20, 2015:9:26 AM, at 9:26 AM, Martin Bjorklund <[email-masked]> wrote:
> 

	I think what you mean is “MUST NOT contradict the syntax or semantics specified in The Yang Language [RFC Reference]”

>>   o management protocols such as NETCONF or RESTCONF,

	This is trickier. We are now in agreement that Yang as a language, is not bound to NETCONF and also RestConf, but it seems the statement above doesn’t buy into that because its coupling the three things together.

	—Tom

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 16:30:26 +0200

> On 20 Aug 2015, at 16:00, Nadeau Thomas <[email-masked]> wrote:
> 
>> 

Yes.

> 	This is trickier. We are now in agreement that Yang as a language, is not bound to NETCONF and also RestConf, but it seems the statement above doesn’t buy into that because its coupling the three…

Well, there is still quite a lot of text in 6020bis that has to do with NETCONF, and some of the existing or proposed extensions clearly affect the protocol(s).

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 20 Aug 2015 07:47:03 -0700

On Thu, Aug 20, 2015 at 6:26 AM, Martin Bjorklund <[email-masked]> wrote:

> >.....

I think this is an important distinction.
RFC 6536 says this extension is mandatory for conformance to NACM.
RFC 6020 says this extension is optional for conformance to YANG.

Andy

---

From: Nadeau Thomas <[email-masked]>
Date: Thu, 20 Aug 2015 10:58:54 -0400

> On Aug 20, 2015:10:30 AM, at 10:30 AM, Ladislav Lhotka <[email-masked]> wrote:
> 
> 

	Yes, I know. We should think seriously about re-aligning that. 

	—Tom

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 20 Aug 2015 17:02:59 +0200

> On 20 Aug 2015, at 16:47, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

This makes sense, the only problem is that the realm of YANG conformance is not precisely defined, and involves protocol considerations.

For RELAX NG it is much easier because it is a schema language and nothing else, so schema annotations are deleted early in the validation procedure.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 20 Aug 2015 08:12:07 -0700

On Thu, Aug 20, 2015 at 7:58 AM, Nadeau Thomas <[email-masked]>
wrote:

> >>      This is trickier. We are now in agreement that Yang as a language,
> is not bound to NETCONF and also RestConf, but it seems the statement above
> doesn’t buy into that because its coupling the three things together.
> >
> > Well, there is still quite a lot of text in 6020bis that has to do with
> NETCONF, and some of the existing or proposed extensions clearly affect the
> protocol(s).
> >
> > Lada
>

This is 2 separate issues.
There is NETCONF-specific text (e.g. all the insert attributes and how they
work in <edit-config>).

Then there is the use of extensions as standard keywords.

IMO...

If (and only if) the extension can be constrained to extra behavior beyond
YANG syntax and semantics, can it be used in an RFC.
E.g NACM defines access-control extensions. The routing module
could define tags that caused a router to treat a route entry a certain way.

But if the keyword is related to datastores, protocol operations, message
encoding,
and cannot be constrained to 1 YANG module, then a regular keyword
should be used instead.  It doesn't save anything to spread keywords
around in YANG modules.  It makes understanding what is in YANG much
more difficult.  Since the "ephemeral" keyword applies to datastores
and could apply to any YANG module, it clearly is not OK as an extension.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 20 Aug 2015 19:50:14 +0200 (CEST)

> > RFC 6020 says this extension is optional for conformance to YANG.

Thanks Andy!  This is a nice summary.

> This makes sense, the only problem is that the realm of YANG
> conformance is not precisely defined, and involves protocol
> considerations.

Agreed.

/martin