Subject: [netmod] allow mandatory nodes in augments

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 18:03:03 +0100

* allow mandatory nodes in augments

** Description

RFC 6020 states in [[http://tools.ietf.org/html/rfc6020#section-7.15][Sec. 7.15]]:

   If the target node is in another module, then nodes added by the
   augmentation MUST NOT be mandatory nodes (see Section 3.1).

This rule was probably based on the assumption that augments will be used for really optional data that can be safely ignored. However, as it turned out, augments have become an essential tool for modular development of YANG data models. In some cases, the inability to define mandatory nodes in augments prevents the data modeller from specifying desired schema constraints.

For instance, [[http://tools.ietf.org/html/draft-ietf-netmod-interfaces-cfg-16#appendix-A][Appendix A]] in "interfaces-cfg" shows how an Ethernet module can be written that augments "if:interface". However, the container "ethernet" cannot define any child leafs as mandatory even though it may be necessary and would be normally done if the "ethernet" container was defined directly in the "ietf-interfaces" module.

** Solution

Remove the rule cited above. 

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 24 Mar 2014 10:33:58 -0700

On Mon, Mar 24, 2014 at 10:03 AM, Ladislav Lhotka <[email-masked]> wrote:

> This rule was probably based on the assumption that augments will be used
> for really optional data that can be safely ignored. However, as it turned
> out, augments have become an essential tool for modular development of YANG
> data models. In some cases, the inability to define mandatory nodes in
> augments prevents the data modeller from specifying desired schema
> constraints.
>
> For instance, [[
> http://tools.ietf.org/html/draft-ietf-netmod-interfaces-cfg-16#appendix-A][AppendixA]] in "interfaces-cfg" shows how an Ethernet module can be written that
> augments "if:interface". However, the container "ethernet" cannot define
> any child leafs as mandatory even though it may be necessary and would be
> normally done if the "ethernet" container was defined directly in the
> "ietf-interfaces" module.
>
> ** Solution
>

In the examples of this sort of augment usage, the augments are in different
modules.  There is no requirement that a server implement all-or-none of
these modules, just because they happen to be published in 1 RFC.

The reason this rule exists is so a server implementation of the augmenting
module
does not break old clients that do not know about this module.  They will
not provide
the mandatory values and edits will fail. (Bonica Principle again).

--
> Ladislav Lhotka, CZ.NIC Labs

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 20:11:38 +0100

> these modules, just because they happen to be published in 1 RFC.

This is not what I am saying.

> 
> The reason this rule exists is so a server implementation of the augmenting module
> does not break old clients that do not know about this module.  They will not provide

I don’t buy this principle. I have already argued that new non-mandatory data in server’s model can lead to unexpected consequences if the client isn’t aware of them, see

http://www.ietf.org/mail-archive/web/netmod/current/msg09296.html

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 24 Mar 2014 20:32:40 +0100 (CET)

Hi,

Added as issue Y26.

/martin

Ladislav Lhotka <[email-masked]> wrote:
> RFC 6020 states in
> [[http://tools.ietf.org/html/rfc6020#section-7.15][Sec. 7.15]]:
> 
> This rule was probably based on the assumption that augments will be used for
> really optional data that can be safely ignored. However, as it turned out,
> augments have become an essential tool for modular development of YANG data
> models. In some cases, the inability to define mandatory nodes in augments
> prevents the data modeller from specifying desired schema constraints.
> 
> For instance,
> [[http://tools.ietf.org/html/draft-ietf-netmod-interfaces-cfg-16#appendix-A][Appendix
> A]] in "interfaces-cfg" shows how an Ethernet module can be written that
> augments "if:interface". However, the container "ethernet" cannot define any
> child leafs as mandatory even though it may be necessary and would be normally
> done if the "ethernet" container was defined directly in the "ietf-interfaces"
> module.
> 
> ** Solution
> 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 24 Mar 2014 12:50:25 -0700

On Mon, Mar 24, 2014 at 12:11 PM, Ladislav Lhotka <[email-masked]> wrote:

> > RFC 6020 states in [[
> http://tools.ietf.org/html/rfc6020#section-7.15][Sec. 7.15]]:
> >
> > This rule was probably based on the assumption that augments will be
> used for really optional data that can be safely ignored. However, as it
> turned out, augments have become an essential tool for modular development
> of YANG data models. In some cases, the inability to define mandatory nodes
> in augments prevents the data modeller from specifying desired schema
> constraints.
> >
> > For instance, [[
> I don't buy this principle. I have already argued that new non-mandatory
> data in server's model can lead to unexpected consequences if the client
> isn't aware of them, see
>

If a new YANG object is defined such that it changes the way an existing
YANG object works, then the new object is broken.

An old client should be able to configure the objects it already knows
and the request should not be rejected because of missing mandatory nodes.

The server can pick reasonable defaults for the new objects without
an old client needing to be re-coded.  A client can use merge instead of
replace to
make sure that new objects do not get deleted. The new objects can be
ignored
for editing purposes.  This is not true if the merge is rejected because of
missing
mandatory nodes.

Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 21:48:12 +0100

On 24 Mar 2014, at 20:50, Andy Bierman <[email-masked]> wrote:

> an old client needing to be re-coded.  A client can use merge instead of replace to
> make sure that new objects do not get deleted. The new objects can be ignored
> for editing purposes.  This is not true if the merge is rejected because of missing
> mandatory nodes.

Let me rephrase the main point of this issue: Significant parts of our core models are defined inside augments, with more to come (e.g., modules for all interface type). These parts cannot have mandatory contents due to that rule from sec. 7.15, which can be a serious limitation because some parameters simply have no reasonable defaults and have to be configured. The possibility of supporting old clients (IMO dubious) does not outweigh this limitation.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 24 Mar 2014 22:14:13 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> > I don’t buy this principle. I have already argued that new non-mandatory data
> > in server’s model can lead to unexpected consequences if the client isn’t
> > aware of them, see
> > 
> Let me rephrase the main point of this issue: Significant parts of our core
> models are defined inside augments, with more to come (e.g., modules for all
> interface type). These parts cannot have mandatory contents due to that rule
> from sec. 7.15, which can be a serious limitation because some parameters
> simply have no reasonable defaults and have to be configured. The possibility
> of supporting old clients (IMO dubious) does not outweigh this limitation.

I agree that the current rule is too limiting - there are cases where
mandatory nodes can be added in a safe way, e.g., if a module for
ethernet parameters defines both the ethernet identity, and augment
nodes based on this ethernet identity.

If we can capture this, I am all for relaxing the rule.  I don't think
just removing the rule is a good idea.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 22:30:10 +0100

> nodes based on this ethernet identity.

Yes, this could be sufficient, though it is not easy to determine what can be considered safe in general.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 24 Mar 2014 14:30:20 -0700

On Mon, Mar 24, 2014 at 1:48 PM, Ladislav Lhotka <[email-masked]> wrote:

> Let me rephrase the main point of this issue: Significant parts of our
> core models are defined inside augments, with more to come (e.g., modules
> for all interface type). These parts cannot have mandatory contents due to
> that rule from sec. 7.15, which can be a serious limitation because some
> parameters simply have no reasonable defaults and have to be configured.
> The possibility of supporting old clients (IMO dubious) does not outweigh
> this limitation.
>
>

If all these augments are in the same YANG module then the augment-stmt is
not
really needed and inline definitions should be used instead.

If they are in different modules, then a server vendor may choose to
implement the augmenting
module in a later release than the base module. It doesn't matter that a
bunch of modules are
published around the same time.  The likely case is that the augmenting
module is over a year later.

It has nothing to do with YANG 1.0 vs. 1.1.
It is a bad idea to break old clients that don't know about the new module.

Note that it is possible to add new nodes, such that the sibling/child nodes
to the augmented nodes are optional, but their descendant nodes are
mandatory.
This should be OK.  An old client that provides just the base objects in a
create or merge will not break if the new mandatory objects are "protected"
via an extra layer:

   container top {
      leaf old-1 { type string; }
   }

NOT OK:

   augment /top {
       leaf from-augment-1 {
         type string;
         mandatory true;
       }
   }

OK:

   augment /top {
       container pcon {
           presence "must exist to make the leaf mandatory";

          leaf from-augment-1 {
             type string;
             mandatory true;
           }
        }
    }

 Lada

Andy

---

From: "Alexander Clemm (alex)" <[email-masked]>
Date: Thu, 27 Mar 2014 03:53:33 +0000

I agree that this type of solution is what is needed.  This has been our experience developing models as well.
The solution as indicated in the example, that the augmenting module itself (and hence "top-level" data nodes) cannot be mandatory, but that in the presence of container data nodes, nodes underneath can become mandatory, makes a lot of sense to me.
--- Alex

From: netmod [mailto:[email-masked]] On Behalf Of Andy Bierman
Sent: Monday, March 24, 2014 2:30 PM
To: Ladislav Lhotka
Cc: [email-masked]
Subject: Re: [netmod] allow mandatory nodes in augments

On Mon, Mar 24, 2014 at 1:48 PM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:

On 24 Mar 2014, at 20:50, Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:

>
>
>
> On Mon, Mar 24, 2014 at 12:11 PM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:
>
> On 24 Mar 2014, at 18:33, Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
>
> >
> >
> >
> > On Mon, Mar 24, 2014 at 10:03 AM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:
> I don't buy this principle. I have already argued that new non-mandatory data in server's model can lead to unexpected consequences if the client isn't aware of them, see
>

If all these augments are in the same YANG module then the augment-stmt is not
really needed and inline definitions should be used instead.

If they are in different modules, then a server vendor may choose to implement the augmenting
module in a later release than the base module. It doesn't matter that a bunch of modules are
published around the same time.  The likely case is that the augmenting module is over a year later.

Note that it is possible to add new nodes, such that the sibling/child nodes
to the augmented nodes are optional, but their descendant nodes are mandatory.
This should be OK.  An old client that provides just the base objects in a
create or merge will not break if the new mandatory objects are "protected"
via an extra layer:

   container top {

NOT OK:

   augment /top {
       leaf from-augment-1 {
         type string;
         mandatory true;
       }
   }

OK:

   augment /top {
       container pcon {

          leaf from-augment-1 {
             type string;
             mandatory true;
           }
        }
    }

Lada

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 27 Mar 2014 08:04:23 +0100 (CET)

Hi Alex,

"Alexander Clemm (alex)" <[email-masked]> wrote:
> I agree that this type of solution is what is needed.  This has been
> our experience developing models as well.
> The solution as indicated in the example, that the augmenting module
> itself (and hence "top-level" data nodes) cannot be mandatory, but
> that in the presence of container data nodes, nodes underneath can
> become mandatory, makes a lot of sense to me.

I don't really understand what you mean.  Do you agree with Lada that
augment should be allowed to add mandatory nodes, or do you agree with
Andy that it is ok to augment e.g. a P-container which defines the
mandatory nodes?

/martin

> On Mon, Mar 24, 2014 at 1:48 PM, Ladislav Lhotka
> <[email-masked]<mailto:[email-masked]>> wrote:
> 
> On 24 Mar 2014, at 20:50, Andy Bierman
> <[email-masked]<mailto:[email-masked]>> wrote:
> 
> >
> >
> >
> > On Mon, Mar 24, 2014 at 12:11 PM, Ladislav Lhotka
> > <[email-masked]<mailto:[email-masked]>> wrote:
> >
> > On 24 Mar 2014, at 18:33, Andy Bierman
> > <[email-masked]<mailto:[email-masked]>> wrote:
> >
> > >
> > >
> > >
> > > On Mon, Mar 24, 2014 at 10:03 AM, Ladislav Lhotka
> > > <[email-masked]<mailto:[email-masked]>> wrote:
> > > used for really optional data that can be safely ignored. However, as
> > > it turned out, augments have become an essential tool for modular
> > > development of YANG data models. In some cases, the inability to
> > > define mandatory nodes in augments prevents the data modeller from
> > > specifying desired schema constraints.
> > >
> > > For instance,
> > > A]] in "interfaces-cfg" shows how an Ethernet module can be written
> > > that augments "if:interface". However, the container "ethernet" cannot
> > > define any child leafs as mandatory even though it may be necessary
> > > and would be normally done if the "ethernet" container was defined
> > > directly in the "ietf-interfaces" module.
> > >
> > > ** Solution
> > >
> > I don't buy this principle. I have already argued that new
> > non-mandatory data in server's model can lead to unexpected
> > consequences if the client isn't aware of them, see
> >
> core models are defined inside augments, with more to come (e.g.,
> modules for all interface type). These parts cannot have mandatory
> contents due to that rule from sec. 7.15, which can be a serious
> limitation because some parameters simply have no reasonable defaults
> and have to be configured. The possibility of supporting old clients
> (IMO dubious) does not outweigh this limitation.
> 
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Mar 2014 00:38:26 -0700

> Andy that it is ok to augment e.g. a P-container which defines the

IMO we cannot break existing clients.
Since it is possible to add the nodes in a way that does not expose the
mandatory
nodes to an "old" create or merge, then that should always be used.
Any create or merge that adds the P container (or list) has to do so
correctly
with the mandatory sub-nodes.

/martin

Andy

> > > > [[
> http://tools.ietf.org/html/draft-ietf-netmod-interfaces-cfg-16#appendix-A][Appendix

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 27 Mar 2014 08:46:30 +0100 (CET)

> IMO we cannot break existing clients.

+1

> with the mandatory sub-nodes.

I agree that this works, and probably is acceptable.  However, as was
demonstrated earlier in this thread, there are situations where it
would be safe to augment a mandatory node, for example if an identity
is defined in the augmenting module, and the augment is made
conditional on that identity.  I don't know if this can be captured in
plain text though.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Mar 2014 07:10:32 -0700

> conditional on that identity.  I don't know if this can be captured in

I am not seeing how this works:

module A {
    container top {
        leaf a { type string; }
    }
}

module B {
   identity X { base XX; }

    leaf flag { type identityref { base XX; } }

    augment /A:top {
         leaf Y {
            when "/B:flag = 'B:X'";
            mandatory true;
            type string;
        }
    }
}

The problem is that it doesn't matter if Y is conditional since /B:flag can
be
set by a new client.  What matters is that an old client can use module A
no matter what is happening with modules that augment module A.

Maybe you have an example that shows how a mandatory node can
be added to /A:top so that an old client can edit /A:top or /A:top/A:aa
and not be impacted by the mandatory sibling of 'aa'.

thanks,
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 27 Mar 2014 15:23:16 +0100

> with the mandatory sub-nodes.

I think this doesn't solve anything. Let's take the example of the "interface" list (which is a very typical use case) and assume the server and client want to use both the base and augmenting module that defines parameters for a particular interface type, say "foo". The client creates an new entry like

"interface": {
  "name": "foo0",
  "type": "foo"
}

and can stop here because all foo parameters are inside a P-container. This may not be acceptable if some of the parameters are really required for a foo interface to work. The P-container makes no difference: the data model is still too loose.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Mar 2014 07:37:53 -0700

On Thu, Mar 27, 2014 at 7:23 AM, Ladislav Lhotka <[email-masked]> wrote:

> I think this doesn't solve anything. Let's take the example of the
> "interface" list (which is a very typical use case) and assume the server
> and client want to use both the base and augmenting module that defines
> parameters for a particular interface type, say "foo". The client creates
> an new entry like
>
> "interface": {
>   "name": "foo0",
>   "type": "foo"
> }
>
> and can stop here because all foo parameters are inside a P-container.
> This may not be acceptable if some of the parameters are really required
> for a foo interface to work. The P-container makes no difference: the data
> model is still too loose.
>
>

It doesn't matter that there might be some parameters that are needed.
They have to be added correctly. They can be set by the server or a new
client.

The rules in sec. 10 are very clear.  YANG is modular.  An old client is not
required in any way to use augmenting modules added after the client was
written.

The P-container wrapper is not itself mandatory so an old client can safely
use
create and merge without error (by ignoring the new P-container).

This is how real software has to be maintained.  Applications get coded
to work with a specific release of a data model. If that code is working and
deployed, then adding a new module in a future release cannot cause the
deployed app to start getting validation errors.

Lada
Andy

> >> > > > modules.  There is no requirement that a server implement
> all-or-none
> >> > > > of
> >> > > > does not break old clients that do not know about this module.
>  They
> >> > > > will not provide
> >> > > an old client needing to be re-coded.  A client can use merge
> instead
> >> > > of replace to

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 27 Mar 2014 16:06:12 +0100

On 27 Mar 2014, at 15:37, Andy Bierman <[email-masked]> wrote:

> They have to be added correctly. They can be set by the server or a new client.

I am talking about a new client, and I assume some items, such as cryptographic data, cannot be set by the server.

> required in any way to use augmenting modules added after the client was written.

This could work in theory if augments were restricted to adding marginal stuff, but as I wrote in the description of this issue, augments do some heavy lifting in the core data models. For example, the routing module is pretty much useless by itself, it has to be augmented by at least one address-family specific module.

> 
> The P-container wrapper is not itself mandatory so an old client can safely use

I don’t get the benefit of this P-container, the problem is still the same: the data model doesn’t bind the client to configure parameters that are in fact mandatory.

> deployed app to start getting validation errors.

This could work if the modules were totally isolated, or carefully designed to support this kind of cherry-picking. But in general it is unsafe: augmenting modules might add defaults that the server will use even if the client isn’t aware of them.

Yes, as a client of an insurance company, I can also sign their contract without reading the fine print …

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Mar 2014 08:35:50 -0700

On Thu, Mar 27, 2014 at 8:06 AM, Ladislav Lhotka <[email-masked]> wrote:

> I am talking about a new client, and I assume some items, such as
> cryptographic data, cannot be set by the server.
>
> >
> This could work in theory if augments were restricted to adding marginal
> stuff, but as I wrote in the description of this issue, augments do some
> heavy lifting in the core data models. For example, the routing module is
> pretty much useless by itself, it has to be augmented by at least one
> address-family specific module.
>
> >
> I don't get the benefit of this P-container, the problem is still the
> same: the data model doesn't bind the client to configure parameters that
> are in fact mandatory.
>

I do not agree that mandatory objects can be added via augments without
breaking the contract specified by the augmented module. You have to up

> This could work if the modules were totally isolated, or carefully
> designed to support this kind of cherry-picking. But in general it is
> unsafe: augmenting modules might add defaults that the server will use even
> if the client isn't aware of them.
>
> Yes, as a client of an insurance company, I can also sign their contract
> without reading the fine print ...
>
>
from RFC 6020, sec. 10:

      New data definition statements may be added if they do not add
      mandatory nodes (Section 3.1) to existing nodes or at the top
      level in a module or submodule, or if they are conditionally
      dependent on a new feature (i.e., have an "if-feature" statement
      that refers to a new feature).

The old client did read the fine print in module A.  New contracts
(augmenting modules) have to add content
in a way that allows the old client to opt-in.

New parameters that augment the interface need to require the conceptual
service or feature to be enabled (create P-container) before any mandatory
parameters
for that feature can be required.

I think the text above prevents any mandatory child nodes to ever be added,
even if it
is done inline in a new module revision.

The if-feature text does not help an old-client/new-server if the module is
updated.
The feature was not in the revision of the module the old client knows.
This is no different than adding mandatory nodes directly. (The if-feature
text
is irrelevant).

It should be OK to add new mandatory child nodes directly to a new module
revision.

The old client only has a contract for module A, rev N.  It should be OK to
add more requirements in a new contract.  Augments is not the same, because
the revision for module A does not change.

Lada
Andy

> > >> > > > In the examples of this sort of augment usage, the augments are
> in
> > >> > > > different
> > >> > > and the request should not be rejected because of missing
> mandatory
> > >> > > nodes.
> > >> > >
> > >> > module in a later release than the base module. It doesn't matter
> that
> > >> > a bunch of modules are
> > >> > Note that it is possible to add new nodes, such that the
> sibling/child
> > >> > nodes
> > >> > This should be OK.  An old client that provides just the base
> objects
> > >> > in a

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 27 Mar 2014 17:04:05 +0100

> in a way that allows the old client to opt-in.

The old client has “signed” the new contract by not closing the session after seeing new server’s hello.

> service or feature to be enabled (create P-container) before any mandatory parameters

The thing is: Every module’s data tree normally starts at the top level. The only way for putting new nodes deeper into the hierarchy of an existing module is to use augments.

Now, the desired modularity and need for supporting other address families in the future dictate that address-family-specific data be factored out of ietf-routing, so augments have to be used.

This all means this “Bonica Principle” leaves me with a crippled data modelling language. Having a P-container in every route is a sheer nonsense.

IMO the requirement of supporting old clients shouldn’t be an absolute priority.

Lada

> 
> I think the text above prevents any mandatory child nodes to ever be added, even if it
> The if-feature text does not help an old-client/new-server if the module is updated.
> This is no different than adding mandatory nodes directly. (The if-feature text
> is irrelevant).
> 
> It should be OK to add new mandatory child nodes directly to a new module revision.
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Mar 2014 09:37:09 -0700

On Thu, Mar 27, 2014 at 9:04 AM, Ladislav Lhotka <[email-masked]> wrote:

> The old client has "signed" the new contract by not closing the session
> after seeing new server's hello.
>
>

It doesn't work that way -- customers get mad when the tools they paid for
stop working when equipment is upgraded.

> The thing is: Every module's data tree normally starts at the top level.
> The only way for putting new nodes deeper into the hierarchy of an existing
> module is to use augments.
>
> Now, the desired modularity and need for supporting other address families
> in the future dictate that address-family-specific data be factored out of
> ietf-routing, so augments have to be used.
>
> This all means this "Bonica Principle" leaves me with a crippled data
> modelling language. Having a P-container in every route is a sheer nonsense.
>
>

It is not crippled.  You do not have to use augment.

YANG allows a new revision of the original module to add a new
mandatory object if there is an if-feature-stmt.

IMO the requirement of supporting old clients shouldn't be an absolute
> priority.
I disagree. You can update the module revision and make a new contract for
a YANG module.
You cannot make these contract changes without updating the module revision.

Andy

> > > >> > > > that augments "if:interface". However, the container
> "ethernet"
> > > >> cannot
> > > >> > > > In the examples of this sort of augment usage, the augments
> are in
> > > >> > > > different
> > > >> > > > The reason this rule exists is so a server implementation of
> the
> > > >> > > > augmenting module
> > > >> > > > does not break old clients that do not know about this
> module.  They
> > > >> > > > will not provide
> > > >> > It is a bad idea to break old clients that don't know about the
> new
> > > >> > module.
> > > >> >

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 28 Mar 2014 16:03:08 +0100

> stop working when equipment is upgraded.

Yes, and blokes who sold them those tools suffer, that’s what I understand. But it is IMO not the complete story.

> It is not crippled.  You do not have to use augment.

What do you mean? Our core data models use augments in several places as a first-class data modelling tool for achieving top-down modularity, not for adding parameters as an afterthought. As far as I can tell, every larger undertaking in YANG does use augments.

> I disagree. You can update the module revision and make a new contract for a YANG module.

One idea - would it be possible to use the packages is a way that guarantees that every package is self-sustainable? This could solve this issue.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Sat, 29 Mar 2014 13:11:22 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> One idea - would it be possible to use the packages is a way that guarantees
> that every package is self-sustainable? This could solve this issue.

So you want to have individually revisioned modules, and an revisioned
package that collects a set of modules, and the server then advertises
the package?

We already have that; use individually revisioned submodules, and let
the module collect the set of submodules, and the server would
advertise the module.  This works today, and it solves your augment
issue.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sat, 29 Mar 2014 09:23:47 -0700

> package that collects a set of modules, and the server then advertises

That is one way to interpret the YANG Conformance draft.
Is it better to represent the API contract as a collection of YANG modules,
or as a collection of conformance packages, where a package usually
represents a service?

> advertise the module.  This works today, and it solves your augment

I just re-read sec. 10.  It is not clear that the main module revision date
MUST be updated if a sub-module is updated (add as new issue?)

   For any published change, a new "revision" statement (Section 7.1.9)
   MUST be included in front of the existing "revision" statements.  If
   there are no existing "revision" statements, then one MUST be added
   to identify the new revision.  Furthermore, any necessary changes
   MUST be applied to any meta-data statements, including the
   "organization" and "contact" statements (Sections 7.1.7, 7.1.8).

I think we should really think through sec. 10 and life-cycle issues for CM
applications.
Sec. 10 is a list of little rules, not a coherent plan for managing CM
life-cycle issues.

We need to examine the design templates that are being used (such as
external augment
of the interface list entry) and figure out if and how they can be properly
used.

  - What is the impact of an optional wrapper container for adding
mandatory nodes?

  - Are servers going to support multiple revisions of a module or just 1?

  - Do all the rules in sec. 10 still make sense, based on experience so
far?

  - Is replication of data structures (to start over) better than
introducing
    incompatible changes in a new revision of existing data structures?

  - Is the reissue of data structures in a new module/namespace (to start
over) better
    than introducing incompatible changes in a new revision of existing
data structures?

 - The current design allows status=obsolete at any time in a new revision.
 Why is this the only
   non-backward compatible change allowed?  Is the plan to set foo=obsolete
and create
   foo1=current? Then foo1=obsolete and foo2=current?

/martin

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 29 Mar 2014 17:26:50 +0100

> advertise the module.  This works today, and it solves your augment

No, it doesn’t. Submodules are not practical for distributed development because with each new submodule one has to change the main module as well.

Look at the routing data model. A client always has to take the base module “ietf-routing” plus at least one address family specific module, such as “ietf-ipv4-unicast-routing”, which augments the base module.

Software packaging systems such as dpkg allow for expressing such dependencies, so it would be nice if YANG packages could do it, too.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 29 Mar 2014 17:36:49 +0100

On 29 Mar 2014, at 17:23, Andy Bierman <[email-masked]> wrote:

> I think we should really think through sec. 10 and life-cycle issues for CM applications.
> Sec. 10 is a list of little rules, not a coherent plan for managing CM life-cycle issues.

First time in this thread I fully agree with you. :-) I think the results could be twofold:

- changes to YANG spec - sec. 10, conformance mechanisms, etc.,

- guidelines for designing larger data models.

Lada

> 
> We need to examine the design templates that are being used (such as external augment
> of the interface list entry) and figure out if and how they can be properly used.
> 
>   - What is the impact of an optional wrapper container for adding mandatory nodes?
> 
>   - Do all the rules in sec. 10 still make sense, based on experience so far?
> 
>   - Is replication of data structures (to start over) better than introducing
>   - Is the reissue of data structures in a new module/namespace (to start over) better
>     than introducing incompatible changes in a new revision of existing data structures?
> 
>  - The current design allows status=obsolete at any time in a new revision.  Why is this the only
>    non-backward compatible change allowed?  Is the plan to set foo=obsolete and create

---

From: Andy Bierman <[email-masked]>
Date: Sat, 29 Mar 2014 16:47:19 -0700

On Sat, Mar 29, 2014 at 9:26 AM, Ladislav Lhotka <[email-masked]> wrote:

> No, it doesn't. Submodules are not practical for distributed development
> because with each new submodule one has to change the main module as well.
>
>
This is a feature, not a bug, if you are an application developer
writing code that works with module A, revision N.

There is no requirement in YANG that multiple modules be implemented
together.  There are no requirements to implement module A
except module A and its imports chain.

There is no way for the app developer to know that a module B
was added later which augments module A, since nothing changed
in module A (or any of its imports) to indicate a change.

YANG is not extensible in the way that you want.
You have to edit the module directly, not augment it from other modules.

> Look at the routing data model. A client always has to take the base
> module "ietf-routing" plus at least one address family specific module,
> such as "ietf-ipv4-unicast-routing", which augments the base module.
>
>

You can only add optional parameters using this design pattern.

> Software packaging systems such as dpkg allow for expressing such
> dependencies, so it would be nice if YANG packages could do it, too.
>
>
dpkg is great, but not perfect.  It will upgrade package A even if there is
no
package available for a dependency (B) yet. It is up to the people
releasing the
packages to make sure all the dependencies are ready before issuing the
update.

We implement something called "SIL bundles" (set of YANG modules +
instrumentation)
so modules can be grouped together for load/unload/upgrade purposes.
NETCONF needs a system-level SW image management module.
I don't think static rules in YANG can really address the problem 100%.

Obviously I am biased, but I think the YANG conformance profiles would
support
the base + augments design pattern quite well.  If a service is really made
up
of N modules, then it is dangerous to manage (load, unload, upgrade,
downgrade)
the modules individually.

Lada
Andy
=

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 30 Mar 2014 10:46:06 +0200

> writing code that works with module A, revision N.

OK, but it is clearly unsuitable for distributed development of modules. Changing ietf-interfaces with every new interface-type-specific module is a no-go.

> in module A (or any of its imports) to indicate a change.

So we could maybe think about a way how to add this information.

> You have to edit the module directly, not augment it from other modules.

But then modularity is gone. Anyway, it is not how it works in practice, several core and other models do use augments in the way I am talking about - examples are ietf-interfaces, ietf-ip, ietf-routing, ietf-snmp, stateless-pf.

> I don't think static rules in YANG can really address the problem 100%.

That's true. Some of these static rules that already exist in YANG were perhaps designed for certain use cases but become CLRs for other common use cases. This IMO includes the rule "no mandatory nodes in augments".

> the modules individually.

That would be great.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 30 Mar 2014 10:21:25 -0700

On Sun, Mar 30, 2014 at 1:46 AM, Ladislav Lhotka <[email-masked]> wrote:

> OK, but it is clearly unsuitable for distributed development of modules.
> Changing ietf-interfaces with every new interface-type-specific module is a
> no-go.
>
>

I do not agree that mandatory node need to be added from external augments.
It would be better not to enable a service just because the YANG module
was loaded.  Or the server can auto-populate the service container and
provide all
the mandatory child parameters.

> So we could maybe think about a way how to add this information.

OK - we are back to sub-modules.  Update/add the sub-module and
then update the main module.

> But then modularity is gone. Anyway, it is not how it works in practice,
> several core and other models do use augments in the way I am talking about
> - examples are ietf-interfaces, ietf-ip, ietf-routing, ietf-snmp,
> stateless-pf.
>

They do not add mandatory nodes, right?

> That's true. Some of these static rules that already exist in YANG were
> perhaps designed for certain use cases but become CLRs for other common use
> cases. This IMO includes the rule "no mandatory nodes in augments".
>
>
There is a built-in assumption that no changes to a data model will ever
be needed that are not backward-compatible.  Since the client cannot
ignore changes easily for CM like it can for monitoring, there needs to be
rules about how to update modules.

> > the modules individually.
I think the YANG conformance draft could be used as a starting point.
Things like "OR expressions" are not supported yet:

   require-module "ietf-ipv4-unicast-routing or  ietf-ipv6-unicast-routing";

This draft does not solve the "update ripple effect".
If module A.1 is used by modules B, C, and D, then the client
does not know for sure what to do when the server starts advertising A.2
instead of A.1.  All the conformance templates that include B, C, and D
need to be verified that they still work with A.2.

This is even worse today for import-by-revision or include-by-revision.
They clearly do not work if the total module set causes the server to load
2 revisions
of any YANG objects (rpc, notification, data). There is no way within
NETCONF
messages to distinguish different versions of the same namespace.
The server has to validate RPC and datastore edit input against 1 module
version, not N.
The server is going to generate 1 version of a notification, not N.
IMO, import and include by revision are broken for these reasons.

Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 30 Mar 2014 20:01:18 +0200

On 30 Mar 2014, at 19:21, Andy Bierman <[email-masked]> wrote:

> was loaded.  Or the server can auto-populate the service container and provide all

How could they, if it is illegal?

Lada

> They clearly do not work if the total module set causes the server to load 2 revisions
> of any YANG objects (rpc, notification, data). There is no way within NETCONF
> The server has to validate RPC and datastore edit input against 1 module version, not N.

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 09:25:28 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> No, it doesn’t. Submodules are not practical for distributed
> development because with each new submodule one has to change the main
> module as well.

If you instead have a package listing all modules, you have to update
the package.  So how is this different?

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 31 Mar 2014 09:27:31 +0200

> the package.  So how is this different?

A package won’t be published as an RFC, I reckon.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 09:31:20 +0200 (CEST)

> A package won’t be published as an RFC, I reckon.

The package is supposedly the conformance requirements.  So we push
the conformance requirements to the vendors?  Each implementation
specifiy its own requirements?

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 31 Mar 2014 09:48:31 +0200

> specifiy its own requirements?

Not necessarily, although even that would be better than no conformance information.

Without considering the IETF workflow, imagine you are a maintainer of the interfaces module and somebody else wants to write a module for a new interface type, perhaps proprietary or experimental. Would you want to update the interfaces module (or package) for every new interface type? 

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 10:26:18 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> Without considering the IETF workflow, imagine you are a maintainer of
> the interfaces module and somebody else wants to write a module for a
> new interface type, perhaps proprietary or experimental. Would you
> want to update the interfaces module (or package) for every new
> interface type?

No, but I am not suggesting that!  I am suggesting that packages don't
solve this issue - they just move the problem.

I think it would be great if we could allow:

  module foo {
    ...
    identity foo;

    augment /xxx:yyy/ {
      when "type = foo:foo";
      leaf my-leaf {
        mandatory true;
        ...
      }
    }
  }

I.e., allow "safe" augmentation of mandatory nodes.

I don't know if we can do this without creating CLRs.

Meanwhile, I think the current approach is good enough - augment a
P-container, in which you put your mandatory nodes.  The *only*
problem with this is that you in some cases may have to introduce a
container that you don't need.  But I am not convinced this is a big
issue.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 31 Mar 2014 10:58:39 +0200

> I don't know if we can do this without creating CLRs.

I think we can. Even now, module X can state in its description that it has to be used together with module Y, or at least one module from a certain set, etc. If it is how module X has been written from the beginning, then there is no issue with old clients. 

> container that you don't need.  But I am not convinced this is a big

Sorry, I still don’t get how a P-container helps. In your example, it means “leaf my-leaf” would be wrapped in such a P-container, right? Then the client can create a valid instance like this:

“yyy”: { 
    “type”: “foo:foo”
}

so “my-leaf”, which is supposed to be mandatory, is missing. How is this different from making “my-leaf” optional in the first place?

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 11:09:42 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> I think we can.

It would be great if you could think about concrete text!

> Even now, module X can state in its description that
> it has to be used together with module Y, or at least one module from
> a certain set, etc. If it is how module X has been written from the
> beginning, then there is no issue with old clients.
> 
> > 
> > 
> Sorry, I still don’t get how a P-container helps. In your example, it
> means “leaf my-leaf” would be wrapped in such a P-container, right?
> Then the client can create a valid instance like this:
> 
> “yyy”: { 
>     “type”: “foo:foo”
> }

Yes, this might also a be problem; the client can create a "meaningless"
empty configuration.

> so “my-leaf”, which is supposed to be mandatory, is missing. How is
> this different from making “my-leaf” optional in the first place?

You do get the behavior you want when the client sets the type, *and*
creates the P-container.  Then all mandatory tests (and more) are
checked.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 31 Mar 2014 11:26:11 +0200

> It would be great if you could think about concrete text!

I’d simply remove the CLR and put a guideline into 6087bis saying something like this:

A supplementary module augmenting another module that has already been in use SHOULD NOT define new mandatory nodes, unless the original module was designed for such an augmentation.

> creates the P-container.  Then all mandatory tests (and more) are

Yes, so it is easier for the client to figure out what is actually mandatory, at the cost of an extra container, which may be an unacceptable overhead, e.g. in every route.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 31 Mar 2014 07:19:57 -0700

On Mon, Mar 31, 2014 at 2:09 AM, Martin Bjorklund <[email-masked]> wrote:

> > >>>>>>>> One idea - would it be possible to use the packages is a way
> that
> > >>>>>>>> guarantees
> > >>>>>> No, it doesn't. Submodules are not practical for distributed
> > >>>> A package won't be published as an RFC, I reckon.
> > >>>

This does not work since module xxx knows nothing of this
module.,  It is the client of the augmented module that
matters -- not the client of the augmenting module.

> It would be great if you could think about concrete text!

The description-stmt "This module xxx MUST be implemented with module A"
requires that module xxx be updated.  This is not acceptable anyway.
The conformance info needs to be machine-readable like it is now.
IMO, the simplest solution is to keep the current conformance model
and not allow any mandatory external augments.

> > Sorry, I still don't get how a P-container helps. In your example, it
> > means "leaf my-leaf" would be wrapped in such a P-container, right?
> > "yyy": {
> >     "type": "foo:foo"
> > so "my-leaf", which is supposed to be mandatory, is missing. How is
> > this different from making "my-leaf" optional in the first place?
>

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 16:26:58 +0200 (CEST)

> matters -- not the client of the augmenting module.

The point is that if the old client creates an entry in xxx, it will never
set the xxx:yyy/type to foo:foo (since it doesn't know about foo), and
since the augment was conditional based on foo:foo, there are no
mandatory nodes in effect.  So this augment is "safe" in the sense
that it doesn't break the old client.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 31 Mar 2014 07:40:01 -0700

On Mon, Mar 31, 2014 at 7:26 AM, Martin Bjorklund <[email-masked]> wrote:

> > > > >>>>>>>> that every package is self-sustainable? This could solve
> this
> > > issue.
> > > > >>>>>>>
> > > > >>>>>> development because with each new submodule one has to change
> the
> > > main
> > > > >>>>>> module as well.
> > > > >>>>>
The 'type' leaf is an independent variable.
It can be set to 'foo:foo' by any client, and when that happens
the mandatory leaf is visible to the old client.
The old client can no longer create  'yyy'.

However, once 'type' is set to 'foo:foo' and if the 'yyy' node
already exists, then the validation will fail unless 'my-leaf' is provided.
But if 'yyy' does not exist yet, the validation for changing 'type' will
succeed,
and an old client can no longer create 'yyy' (because my-leaf is now
mandatory.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 31 Mar 2014 16:52:03 +0200 (CEST)

> The old client can no longer create  'yyy'.

No, 'type' is not indepent; it is in list 'yyy'.  Let's do this with a
more real-world example, from ietf-interfaces:

  container interfaces {
    ...
    list interface {
      key "name";
      leaf type {
        type identityref {
          base interface-type;
        }
      ...
    }
  }

  module ex-ethernet {
     ...
     identity my-ethernet { 
       base if:interface-type;
     }
     augment /if:interfaces/if:interface {
       when "if:type = ex:my-ethernet";
       leaf my-mandatory-leaf {
         mandatory true;
         ...
       }
     }
   }

A client that doesn't know anything about 'my-ethernet' can still
create all other interfaces it knows about.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 31 Mar 2014 08:13:05 -0700

On Mon, Mar 31, 2014 at 7:52 AM, Martin Bjorklund <[email-masked]> wrote:

> > > > > > >>>>>> On 29 Mar 2014, at 13:11, Martin Bjorklund <
> [email-masked]>
> > > > > wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> So you want to have individually revisioned modules, and
> an
> > > > > revisioned
> > > > > > >>>>>>> package that collects a set of modules, and the server
> then
> > > > > advertises
> > > > > > >>>>>>> the package?
> > > > > > >>>>>>>
> > > > > > >>>>>>> We already have that; use individually revisioned
> submodules,
> > > > > and let
> > > > > > >>>>>>> advertise the module.  This works today, and it solves
> your
> > > > > augment
> > > > > > >>>>>>> issue.
> > > > > > >>>>>>
> > > > > > >>>>> If you instead have a package listing all modules, you
> have to
> > > > > update
> > > > > > >>> The package is supposedly the conformance requirements.  So
> we
> > > push
> > > > > > >> the interfaces module and somebody else wants to write a
> module
> > > for a
> > > > > > > No, but I am not suggesting that!  I am suggesting that
> packages
> > > don't
OK - this seems hard to write up.
How does the YANG compiler know that revision X is the
one that introduced the my-ethernet identity?

What if the old client knows about ex-ethernet, in the version
that you describe above, but then the module is changed in a new
version, and 'another-mandatory-leaf' is added?

        leaf another-mandatory-leaf {
          mandatory true;
          type string;
        }
    }

Is this allowed?
The sec. 10 rules do not allow it.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 31 Mar 2014 18:12:34 +0200

> one that introduced the my-ethernet identity?

The server gives the information about its complete data model in hello.

> version, and 'another-mandatory-leaf' is added?

A paranoid client that doesn’t support the exact data model advertised by the server would give up and close the session. Otherwise, it may try some operation that will either succeed or fail with an error message that should give the client sufficient information about the cause.

A much worse scenario is that the operation succeeds but the device does something else than what the client intended - the CLR in question doesn’t address this case at all, and that’s why a security-savvy client should not even try.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 31 Mar 2014 10:06:38 -0700

> ....

The rule would be a lot easier to write if new value the
old client did not know about was part of the list key.

I don't see how this could work for containers, but it
could work for lists.

 Andy

  container interfaces {