Subject: [netmod] Augment a grouping

From: Balazs Lengyel <[email-masked]>
Date: Fri, 20 Jun 2008 11:14:04 +0200

Hello,
Why is it not allowed to augment a grouping? We just came up against a very clear use case 
where that would be needed.

Balazs
-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Jun 2008 07:18:51 -0700

Balazs Lengyel wrote:
> Hello,
> Why is it not allowed to augment a grouping? We just came up against a 
> very clear use case where that would be needed.
> 

I reason not to do it is due to the ripple effect
within all the uses-stmts.

One nice feature with MIBs is that I can extract a MIB module
from an RFC 2 or 10 or 15 years old and I can compile it
to get the same results as when the RFC was published.

If you extend a grouping and it is used already in a published
RFC, then when I compile it 10 years later, and the grouping
has changed since the RFC was published, I end up with
a different conceptual module than the one in the RFC.
That is bad for interoperability.

The solution -- a separate version ID for every import or
construct in the YANG file -- is way worse than the problem.

The current solution in YANG requires a new name:

   grouping foo2 {
      uses foo;

      leaf newleaf { ... }
   }

IMO, stability and interoperability are more important features
to preserve.

> Balazs

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 09:16:18 +0200 (CEST)

Hi,

Andy Bierman <[email-masked]> wrote:
> Balazs Lengyel wrote:
> > Hello,
> > Why is it not allowed to augment a grouping? We just came up against a very clear use case where that would be needed.
> > 
> 

You get exactly the same result with the current YANG mechanisms.  If
the original RFC is augmented 10 years later and you recompile it with
the new augmenting module, you get a different result than w/o the
augmenting module.  This is a feature.

For a given set of modules that uses a grouping, you can get exactly
the same effect as you would get if you were allowed to augment a
grouping, by hunting down all the uses of the grouping, and augment
them directly.  For example:

Given:

    module a {
      grouping foo {
        leaf bar;
      }
    
      container x {
         uses foo;
      }
    
      container y {
         uses foo;
      }
    
    }

Instead of:

    augment /a:foo {  // illegal
       leaf baz;
    }

You would do:

    augment /a:x {
       leaf baz;
    }

    augment /a:y {
       leaf baz;
    }

> The current solution in YANG requires a new name:

I think this is a different issue - this is the upgrade rules for a
grouping.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 23 Jun 2008 09:23:20 +0200

>     augment /a:foo {  // illegal

The question is what "uses foo" means, that is when augments take
effect. In your example, I would expect that "uses foo" in module a
always means the same thing unless the augment /a:foo is in the very
same module. If the augment /a:foo is in a different module b, then I
would say that this can't affect module a. And if you agree with that
(not sure), then the question is how you actually refer to this
augment /a:foo if you want to use the augmented version of foo. So
probably

grouping b:goo {
  uses foo;
  leaf baz;
}

is good enough?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 10:07:50 +0200 (CEST)

Juergen Schoenwaelder <[email-masked]> wrote:
> (not sure)

No.  augment (as it is used currently) is typically external to the
module.  So for example:

   container x {
     leaf y;
   }

always means the same thing in my local module, but 'x' can later be
augmented from some other module.  Same thing when augmenting a
grouping:

  container x {
    uses foo;
  }

> , then the question is how you actually refer to this

You never refer to the augmented grouping.  The whole point with
augment is that you should not have to update the original
definition.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 23 Jun 2008 10:23:20 +0200

> augment is that you should not have to update the original

I understand all that but since groupings are reusable constructs,
they are pretty different from containers. An augment of a container
actually does not change the definition of the container; it just adds
something to be hosted by the augmented container. If I choose so, I
can ignore the augmentation.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 10:36:02 +0200 (CEST)

[...]

> can ignore the augmentation.

The same thing would apply to augment of a grouping.  If grouping
'foo' defined in module 'a' is augmented by module 'b', and a server
implements 'a', but not 'b', then 'foo' (on that server) will not add
the augmented objects from 'b'.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 06:35:32 -0700

> the augmented objects from 'b'.

We almost had it figured out in Philly.
(Now we are back to square one though :-(

The real problem with groupings is understanding the
contents, and therefore the conformance requirements.

Consider 4 modules:

m1: contains grouping G = { leaf1, leaf2 }
m2: uses G
m3: augments grouping G:  G' = G + { leaf3 }
m4: uses G'

There are a couple big problems.
1) locking the 'uses-stmt'

    - When m1 is published it contains 2 leafs.
    - When m2 is published, m1:G contains 2 leafs and the semantics
       of m2 depends on this fact.
    - Later m1 is republished and 'list1' is added to the grouping

    Q: How does a developer compiling m2 (at any time after m2
       is published) import the correct version
       of m1, such that G has 2 child nodes, not 3?
       - If m2 is ever republished, is it required to use the new
         m1:G then, or can it still use the old version of m1:G?

    * Unless this problem is solved, I do not see how a grouping
      can ever be modified (let alone augmented), after it is
      published in an RFC.

2) collecting the augments
    - When m3 augments G with 'leaf3', this is being added to the
      XML namespace for m1.  There is no naming collision protection
      like normal augments.  There is no way to tell in NETCONF PDUs where
      'leaf3' is defined.
    - m2 and m4 will import m1 to access G, but how does m4 know
      to import m3?
        - What if there are multiple uses of G in m4,
          and not all of them should be updated (i.e., m4 wants to
           use G and G')
        - What if the constructs in m2 or m4 which use G are in turn
          used in other constructs or modules?  (e.g., uses within a grouping)
          How does the implementer distinguish G from G' then?

    * Augmenting a grouping is quite complicated.
      IMO we should stick to augmenting accessible objects only for now.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 15:49:36 +0200 (CEST)

> m3: augments grouping G:  G' = G + { leaf3 }

No, m4 uses G.  And m3 just augments G, it does not create a G'.

>        of m1, such that G has 2 child nodes, not 3?

Just like with normal 'augment', the developer knows if the augmenting
module 'm3' is supported, and thus if just the two leaves from m1 should be
implemented, or if the augmented leafs should be implemented as well.

>       can ever be modified (let alone augmented), after it is

I think you're mixing two different concepts here; versioning
vs. augmentations.  I agree with you on the versioning issue though.

>       XML namespace for m1.

No, just like w/ normal augment, 'leaf3' would be defined in m3's namespace.

>       There is no naming collision protection

See above.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 07:37:58 -0700

> No, m4 uses G.  And m3 just augments G, it does not create a G'.

So you are saying that G' can only be local to m3
because the augment of G can only be in m3 and only used in m3?
In that case, I would much rather punt this whole idea
and force the DM writer to use the constructs already
available which can produce the same results.

Combining lots of grouping augments could be a DM reader's nightmare.
I prefer to see it spelled out inline with separate uses-stmts,
instead of hidden in augment-stmts, scattered throughout the module.

> vs. augmentations.  I agree with you on the versioning issue though.

Yes I am mixing 2 similar problems.
Groupings are much different than anything the IETF has ever used
before in a network management data modeling language.

When m2 is published, the exact definitions from m1 it uses needs to be
specified and locked -- regardless of what happens to m1
in the future.  This property of MIBs is not just a feature, it is
a show-stopper MUST HAVE for standards.  A standard cannot have
any 'place-holders' replaced  at compile-time, like a #define in C.
SMIv2 does not allow TEXTUAL-CONVENTIONs to ever be changed in
non-backward-compatible ways.  The OBJECT-TYPE and NOTIFICATION-TYPE
macros are completely enumerated for a given conformance level,
for a given version of the module.  Just because YANG might
have more complex objects to enumerate, does not mean this problem
should be ignored.

It is not as if there is some shortage of unique names for groupings.
If you need a grouping G' that extends G in a module, then just
create a new grouping G2, and start using that instead of G
for new data structures.

> No, just like w/ normal augment, 'leaf3' would be defined in m3's namespace.

This really changes how groupings work.
All the grouping objects are created in the namespace of the uses-stmt.
IMO, groupings are already problematic enough without adding
the complexity of altering the uses-stmt expansion of groupings
from external modules.  You can already do this augmenting
locally in a controlled and tractable manner.

I thought it was verboten to define multiple ways of
doing the same thing in a data modeling language.
(I like that rule -- it means we should be tossing stuff out
of YANG inside of piling on more stuff ;-)

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 11:25:17 -0400

>vs. augmentations.  I agree with you on the versioning issue though.

The problem is that without versioning, augmentation of groupings
is considered dangerous, since ordinary uses can suffer drive-by
augmentation.  But not having augmentation of groupings reduces the
utility of both augmentation and grouping, and forces the use of
hack like grouping names with version number in them.

Which IMHO drives us back to the need for versioned imports.  If I
can control which revision of a module I'm importing, then my
dependency is explicit and changes to the module don't affect me.
This gives the stability and certainty that is needed for modules
to be both fixed and evolving, as they are in the real world.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 09:14:48 -0700

> to be both fixed and evolving, as they are in the real world.

Putting the versioning info in the name is not a hack.
It handles all of the real-world problems with mixing
old and new implementations just fine.

I think multiple concurrent versions of the same module
within the same system or the same agent raises complexity
concerns.  I do not think the IETF has enough experience
with the 'grouping/uses' method of NM definition reuse to
justify this level of complexity.

IMO, the best approach wrt/ IETF standards is to somehow
spell out (in machine-readable form) exactly what is
contained in a particular module version.  This info cannot
be lost as the module changes over time.  SMIv2 uses a separate
conformance section for this purpose, which should be adapted
for YANG.

In my example, if m2 had a conformance section that
spelled out the leafs, like:

module m2 {
   namespace "...";
   prefix m2;
   import m1 { prefix m1; }

   revision 2008-06-23 {
      description "Using new version of grouping G from m1";
   }

   revision 2008-06-01 {
      description "Initial version";
   }

   container foo {
     uses G;
   }

   conformance 2008-06-23 {
     object /foo/leaf1;
     object /foo/leaf2 {
       min-access read-only;
     }
     object /foo/list1/key;
     object /foo/list1/leaf-a;
     object /foo/list1/leaf-b;
   }

   conformance 2008-06-01 {
     object /foo/leaf1;
     object /foo/leaf2 {
       min-access read-only;
     }
   }
}

This way there is only 1 file for each module maintained,
and it is still very clear what is contained in each version.

I know this can get verbose if there are lots of leafs
to enumerate, but so far nobody has standardized more
than 3 NETCONF knobs at a time, so I'm not very concerned
about the burden on readers or writers.  For standards users,
it is really important to know what to expect when a device
claims to support version X of the foo module.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 18:20:48 +0200 (CEST)

> because the augment of G can only be in m3 and only used in m3?

I'm not sure I follow this.  There is no named grouping G'.  For
example:

  module m1 {
    grouping G {
       leaf leaf1;
       leaf leaf2;
    }
  }

  module m2 {
    container x {
      uses m1:G;
    }
    container y {
      container z {
        uses m1:G;
      }
    }
  }
  
  module m3 {
    augment /m1:G {
       leaf leaf3;
    }
  }

If a server implements m2 and m3, you might get:

  <data>
    <y xmlns="m2...">
      <z>
        <leaf1>10</leaf1>
        <leaf2>20</leaf2>
        <leaf3 xmlns="m3...">30</leaf3>
      </z>
    <y>
  </data>

This is exactly the same effect as if m3 was defined like this:

  module m3 {
    augment /m2:x {
      leaf 3;
    }
    augment /m2:y/m2:z {
      leaf 3;
    }
  }

So I don't get what you mean when you talk about G' and how groupings
can evolve over time.

> available which can produce the same results.

augmentation of a grouping solves one problem that is not possible to
solve today.  The 'm3' module can augment m1:G, without having to know
about m2 or any other module that uses G.

> Combining lots of grouping augments could be a DM reader's

Browsing through a list of copy-and-pasted explicit augement of the
data hierarchy might be even worse...

> in the future.

Agreed.  The data tree is well-defined, modulo augment of the data
hierarchy  - this is what we currently have, and it would not change
if we introduce augment of grouping.

> This property of MIBs is not just a feature, it is
> for a given version of the module.

The intention is to do the same for our corresponding statement, which
is 'typedef'.

> create a new grouping G2, and start using that instead of G

Agreed.  But this is a separate isssue; this is about versioning.  You
and I agree on the versioning rules.

> This really changes how groupings work.

No.  This is how augment works.  The grouping/uses stays the same.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 18:26:34 +0200 (CEST)

> hack like grouping names with version number in them.

Again I think the two issues are mixed - adding a version number to
the grouping name solves the versioning problem, but not the
augementation problem.

> to be both fixed and evolving, as they are in the real world.

Some other module would still be able to augment your data tree.

/martin

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 12:33:09 -0400

>Some other module would still be able to augment your data tree.

Yes, but I want that.  What I don't want is for the next version
of a module that I import to add leafs to my data that aren't
implemented.  If I import a module (A) in my module (B), and I code
to today's version of A, when the next version of A appears, no one
knows that my software doesn't implement leafs described in the
modern A.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 09:38:37 -0700

> the grouping name solves the versioning problem, but not the

I don't agree that there is a problem with augment.
I don't agree that the IETF has enough experience with YANG
to reach consensus and sufficiently specify this in
an an inter-operable manner.

> Some other module would still be able to augment your data tree.

Fine.
They can specify the exact data nodes to augment
with the existing augment-stmt.  It is unambiguous
and allows each uses-stmt to be augmented or not,
instead of all-or-none.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 13:06:04 -0400

>Putting the versioning info in the name is not a hack.

The issues with this approach become exposed when you work thru
some more complex cases than are allowed in the space of an email.
But let's try.....

I have a BGP module.  This module defines over three thousand
statements:

    [edit protocols bgp]
    phil@dent# help apropos . | match set | count 
    Count: 3587 lines

This is the number of leaf nodes under the BGP section of the JUNOS
configuration hierarchy.  Certainly many of these are repeated,
since we allow knobs to exist at multiple, more specific levels of
the configuration hiearchy, but this is meant to give you an idea
of the complexity and size of the problem.

As we move to YANG, these repeated nodes will become groupings.
Groupings will use other groupings as required to get the exact
combination of knobs required.

For example, the bgp-afi-options grouping defines two nodes,
prefix-limit and rib-group.  This grouping is used in seven places,
six of which are other groupings.  I traced the references for the
first grouping and found it used in eight places.  If I pick the
first grouping that uses this new grouping, I find it used in three
places which aren't groups.

So it's not a matter of adding version numbers to one grouping, but
to every grouping that uses that grouping.  This leads to YANG that
looks like:

container protocols {
    container bgp {
        grouping bgp-afi-options-v1 {
            container prefix-limit {
                ...
            }
            leaf rib-group {
                ...
            }
        }
        grouping bgp-afi-options-v2 {
            container prefix-limit {
                ...
            }
            leaf rib-group {
                ...
            }
        }
        grouping bgp-afi-options-v3 {
            container prefix-limit {
                ...
            }
            leaf rib-group {
                ...
            }
        }
        grouping bgp-afi-default-v1 {
            use bgp-afi-options-v1;
            ...
        }
        grouping bgp-afi-default-v2 {
            use bgp-afi-options-v2;
            ...
        }
        grouping bgp-afi-default-v3 {
            use bgp-afi-options-v3;
            ...
        }
        grouping bgp-family-v1 {
            ...
            container family {
                container inet {
                    container unicast {
                        use bgp-afi-default-v1;
                        ...
                    }
                    container multicast {
                        use bgp-afi-default-v1;
                        ...
                    }
                    ...
                }
            }
        }
        grouping bgp-family-v2 {
            ...
            container family {
                container inet {
                    container unicast {
                        use bgp-afi-default-v2;
                        ...
                    }
                    container multicast {
                        use bgp-afi-default-v2;
                        ...
                    }
                    ...
                }
            }
        }
        grouping bgp-family-v3 {
            ...
            container family {
                container inet {
                    container unicast {
                        use bgp-afi-default-v3;
                        ...
                    }
                    container multicast {
                        use bgp-afi-default-v3;
                        ...
                    }
                    ...
                }
            }
        }
        grouping bgp-peer-options-1 {
            use bgp-family-v1;
            ...
        }
        grouping bgp-peer-options-2 {
            use bgp-family-v2;
            ...
        }
        grouping bgp-peer-options-3 {
            use bgp-family-v3;
            ...
        }

        use bgp-peer-options-3;
        list group {
            ...
            use bgp-peer-options-3;
            list peer {
                ...
                use bgp-peer-options-3;
            }
        }
    }
}

This is needed because I changed one thing in one leaf in one
grouping.  Imagine this after a few years of a hundred bgp hackers.
Compare it to "import bgp { revision 2008-01-01; }".

History belongs in historical documents.  It doesn't need to be
intermixed with the modern YANG.  It would be like the plot of every
zombie movie ever made.  The volume of dead YANG zombie groupings
limping around would overwhelm our shotgun-wielding heroes.  The
percent of crap you carry inside a module in order to avoid breaking
other people will outweigh and obscure the real YANG you want the
precious reader to understand.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 10:46:44 -0700

>         grouping bgp-peer-options-3 {

This looks clear, unambiguous, and tractable to me.

> Compare it to "import bgp { revision 2008-01-01; }".

So every manager and agent needs to maintain N concurrent
versions of each module, and diff them by hand to figure out
what changed.  Not much better.

> precious reader to understand.

It is not history -- it is standards conformance.
Juergen proposed a modification with conformance-group
definitions like SMIv2 GROUP, which could greatly reduce
the redundancy.

Speaking of history, when SMING WG collapsed in failure,
it was because most of WG, let alone IETF, was not really
on-board with all the changes to SMIv2 that were proposed.
Finally somebody said "My engineers will never bother to learn this",
and the whole thing collapsed, as vendor after vendor agreed
they would rather do nothing than 'learn all this new stuff'.

I do not see much participation in this WG at all so far,
and the possibility of a SMING-repeat seems non-trivial.
IMO, start simple, and make sure there even is a YANG 1.0 published.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 19:48:35 +0200 (CEST)

> knows that my software doesn't implement leafs described in the

Agreed.  augement of grouping will not give you that, just like normal
augment doesn't give you that.  You have full control of this - only
if you implement the augmenting module, your data tree will be
augmented.

/martin

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 14:57:25 -0400

>This looks clear, unambiguous, and tractable to me.

unambiguous?  tractable?  maybe, but even this change of
one leaf makes the problem pretty obvious.  You'd put too
much of the volume of the module into maintenance and not
into clearly describing the current state of the art.

>what changed.  Not much better.

I disagree.  N independently readable, standalone versions
are easier to parse than one soup pot full of everything.
Diff will show you stuff like:

+         grouping bgp-afi-options-v2 {
+             container prefix-limit {
+                 ...
+             }
+             leaf rib-group {
+                 ...
+             }
+         }
...
+         grouping bgp-afi-default-v2 {
+             use bgp-afi-options-v2;
+             ...
+         }
...
+         grouping bgp-family-v2 {
+             ...
+             container family {
+                 container inet {
+                     container unicast {
+                         use bgp-afi-default-v2;
+                         ...
+                     }
+                     container multicast {
+                         use bgp-afi-default-v2;
+                         ...
+                     }
+                     ...
+                 }
+             }
+         }
...
+         grouping bgp-peer-options-2 {
+             use bgp-family-v2;
+             ...
+         }

-         use bgp-peer-options-1;
+         use bgp-peer-options-2;
          list group {
              ...
-             use bgp-peer-options-1;
+             use bgp-peer-options-2;
              list peer {
                  ...
-                 use bgp-peer-options-1;
+                 use bgp-peer-options-2;
              }
          }

This won't show you want's changed.  But if you diffed versions,
you'd get something like:

+        leaf new-thing {
+            ...
+        }

In addition, a YANG-enabled diff application can know the relationship
between the bgp-family grouping over multiple revisions.  If you
turn this into bgp-family-v1/2/3 you lose that continuity.  No one
knows if bgp-family-v2 is new or a new revision.  You end up with
the YANG equivalent of spaghetti code.

YANG tries to make rich data that applications can use.  Being
able to use previous module revisions is a more simple, clear
and consistent mechanism for module evolution.

>and the possibility of a SMING-repeat seems non-trivial.

This is an entirely different discussion.  If you don't think YANG
will be adopted, let's talk about the chief reasons why you think
it won't be.  Versioned imports will surely not be on that list.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 12:51:04 -0700

> and consistent mechanism for module evolution.

What about version mixing?  Is this allowed:

  module m2 (version 2008-06-01) { grouping G {} }

  module m2 (version 2008-06-23) { grouping G { // + new stuff } }

  module m3 {
    import m2 { prefix m2; }

    grouping GG {
      container gg {
        uses G;
      }
    }
  }

  module m4 {

   import m2 {
      prefix m2;
      revision 2008-06-01;
   }

   import m3 { prefix m3; }

   list X {
      uses G;
      uses GG;
   }
}

YANG mid-term 1: Specify the contents of list 'X'. Show your work :-)

> it won't be.  Versioned imports will surely not be on that list.

It might be on that list, if the example I showed above is allowed.
If it is a compiler error, go away and come back when m4 is not
mixing versions of m2:G, then I can live with it.

I prefer to get started with the YANG already defined,
and only do MUST-HAVE changes, if any.  Anything in
the 'nice-to-have' category can wait until YANG 1.1, etc.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 16:14:48 -0400

>YANG mid-term 1: Specify the contents of list 'X'. Show your work :-)

The result is that GG is using the modern G, so X will have an old
G and a new G in GG.

If we flesh your example out a bit:

module m2 {
    revision 2008-06-01 { ... }
    grouping G {
        leaf g-2008-06-01 { type empty; }
     }
}

module m2 {
    revision 2008-06-23 { ... }
    revision 2008-06-01 { ... }
    grouping G { 
        leaf g-2008-06-01 { type empty; }
        leaf g-2008-06-23 { type empty; }
     }
}

module m3 {
    import m2 {
        revision 2008-06-23;
        prefix m2;
    }
    grouping GG {
        container gg {
            uses G;
        }
    }
}

module m4 {
    import m2 {
        prefix m2;
        revision 2008-06-01;
   }

   import m3 { prefix m3; }

   list X {
        uses G;
        uses GG;
   }
}

So the hierarchy would be:

  <X>
    <g-2008-06-01/>  <!-- uses older G from 2008-06-01 -->
    <gg>             <!-- uses GG from 2008-06-23 which uses modern G -->
      <g-2008-06-01/>
      <g-2008-06-23/>
    </gg>
  </X>

>mixing versions of m2:G, then I can live with it.

Your example is really the type of mixing should not be allowed,
since m4 is trying to use an m2 that's older than m3.  This would
mean that m4 was written using out-of-date modules.  A reasonable
rule would be that if you write or rev a module, you MUST move to
fully modern imports.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Jun 2008 22:42:11 +0200 (CEST)

>    import m3 { prefix m3; }

Is this just a typo, or do you suggest that the 'revision' on the
import is optional?

/martin

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 16:46:08 -0400

>Is this just a typo, or do you suggest that the 'revision' on the

typo.  I copied Andy's text and fixed one import but not the other.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 14:02:47 -0700

> Is this just a typo, or do you suggest that the 'revision' on the

I was wondering that myself.
If every import has to have a revision clause, and
every module must have a unique revision statement
for each published version, then it is less of a problem.
The DM writer would be forced to update the 'import m2' in m4
to the latest revision of m2 because 'import m3' collides with it.

It adds some overhead -- every time module 'foo' is changed,
every import in the module will need to be checked and
updated to the latest version.  In an N-writers development
environment, this could become difficult to maintain.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 23 Jun 2008 17:38:56 -0400

>to the latest revision of m2 because 'import m3' collides with it.

Exactly.  A human would need to check and update the module.
And until they did, everything would continue to work as
it did before the new revision was published.

>environment, this could become difficult to maintain.

But it also means the switch to a new revision is gated by someone
making the change.  There's an explicit dependency and an explicit
action is required to change it.

This also gives us more latitude WRT module updating rules, since
we know the DM writers won't be broken by new revisions.  It works
as a safeguard.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Jun 2008 14:53:24 -0700

> action is required to change it.

I know -- that's why it works.

> we know the DM writers won't be broken by new revisions.  It works

yes -- solves the changed-grouping problem, but
the cost is maintaining every version of every module,
instead of just 1 version of every module.  If you encounter
a typo or make a simple bugfix that does not change the
semantics (a clarification) can the bug be fixed in all
affected versions of the module without updating the revision
at all?  It seems you could only make these bug-fixes
in a new latest version.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 24 Jun 2008 13:33:36 +0200 (CEST)

> to the latest revision of m2 because 'import m3' collides with it.

But that won't work.  Suppose we have the following standard modules,
published in rfcs:

module ietf-m1 {

module ietf-m2 {
    import ietf-m1 {
        revision 2008-06-01;
    }
}

module ietf-m3 {
    import ietf-m1 {
        revision 2008-06-23;
    }
}

Now, my m4 module needs to import ietf-m2 and ietf-m3, which means
that I will have dependencies to two versions of ietf-m1.

I don't think we can require that all published modules are
re-published as soon as one common module is updated - as an example,
imagine that a clarification is needed in the inet-types module.

So if we do this, we will have to accept that a server will have to
support multiple versions of the same module.

/martin

---

From: Phil Shafer <[email-masked]>
Date: Tue, 24 Jun 2008 10:23:33 -0400

>support multiple versions of the same module.

Absolutely.  Not the easiest medicine, but IMHO it's what
we need.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 08:21:31 -0700

> Absolutely.  Not the easiest medicine, but IMHO it's what

Is this how CLI works?  It sure isn't how SNMP works.
Are there any agents that support every API version
ever released, or do they support only the latest
release at the time?  I've only worked on the latter,
and don't know of any examples of the former.

I think this complexity is not worth the unproven utility
that the groupings/uses approach to definition reuse
claims to provide.

Can anyone actually think of any sets of objects that
should be defined in a grouping in a stand-alone module,
because they are so reusable for all other modules?
IMO, it is far more likely that the groupings and uses
will be in the same module, in which case the problem
goes away.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 24 Jun 2008 18:25:51 +0200 (CEST)

> will be in the same module, in which case the problem

The problem goes away for non-exported groupings,
i.e. groupings scoped by some other statement than 'module' or
'submodule'.

/martin

---

From: Phil Shafer <[email-masked]>
Date: Tue, 24 Jun 2008 13:19:59 -0400

>will be in the same module, in which case the problem

Are you trying to define away the issue or just looking for a use
case?  Use name is any group of related nodes, say the address,
port, and routing instance of a tcp socket.

But this isn't an isolated issue, since you also have enumerations
which will grow over time.  When I add a new enumeration to a type
that another module uses, how will I know if a device that announces
that module uses the old or new version of that type?  What will
they do when I give them the new enum value?

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 11:24:00 -0700

> port, and routing instance of a tcp socket.

Looking for a use-case, and as usual, you have a good one.

In SNMP-land, even though there are no structures, there
is some experience with reusable data types, like InetAddress.
There was a painful migration from IpAddress to InetAddress,
but at least it was possible because IpAddress was not re-defined.
Instead a new name was used.

Due to the huge lag-time that Juergen mentioned
for getting data models in RFCs updated (e.g., 2 years to never),
import-by-version is not a realistic choice for standards.

IMO, if 'deep indexing' is too complicated for YANG,
than supporting multiple module versions within the
same agent is 10X as complicated.  The interactions
for data-type chaining alone would be a nightmare to code.
Then add in grouping/uses and augments, and
N variants of each callback function, and coding complexity
plus memory requirements goes through the roof.

> they do when I give them the new enum value?

The schema discovery mechanism will include the module version
that contains the type -- this will be the one only version
of that type on the agent.

A manager will always need to be aware of every version
of every module it uses.  That has always been the case,
to deal with N different agents.  My concern is making
the agent deal with N versions, which would be unprecedented.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 24 Jun 2008 15:04:19 -0400

>import-by-version is not a realistic choice for standards.

I see the exact opposite.  Since data models are slow to
update, there needs to be a mechanism to say "Hey, I'm
still using the old one".  Versioned imports give that.

>same agent is 10X as complicated.

This is a question of where the complexity lies.  Deep
indexing puts a cost on the device.  Versioned imports
puts a cost on the application, and the cost is mostly
covered by the fact that you need to grok multiple
modules.  Imagine that we put the revision name as
part of the module name (not that I'm suggesting that
as a real fix).  The code of "import A { revision 2; }"
isn't any worse than "import A-revision-2;".

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 12:40:47 -0700

> still using the old one".  Versioned imports give that.

Conformance statements give you that without requiring
an agent to support N versions of each module.

> isn't any worse than "import A-revision-2;".

I am most concerned with the standards utility of YANG
and NETCONF.  Juergen brought up a critical point:
Vendors do not have change control over standard modules.

If they happen to use 2 incompatible standard modules
(via imports ripple), then they are stuck.  They have
to cut-and-paste one of the (fixed) standard DMs to their own
namespace, to continue using the standard.  That DM
would not be 'standards-based' anymore, so that option
isn't very attractive.

I do not think vendors will implement multiple versions
of the same YANG module in the same agent.  Making this
a requirement would be a show-stopper.  Ignoring it
and letting applications and agents break any time a
version conflict is detected is not going to work either.

This problem may need to be solved with a separate
document that explains how YANG MUST/SHOULD/MAY be used
within IETF documents.  The YANG language does not need
versioned imports.  Let vendors figure this out on their own
with extensions.

In IETF data models there are some changes that must never
be allowed, in order to make the standard usable.
There are some practices (like description clauses) that
are also required for interoperability.

The way YANG is defined now, a standard YANG module does not
need any revision statements, contact info, any description clauses,
and any field value (like the namespace) can change at any time.
I suspect the IESG might have something to say about that
during their review.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 24 Jun 2008 12:50:30 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Phil Shafer" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, June 24, 2008 12:40 PM
> Subject: Re: [netmod] Augment a grouping
...
> an agent to support N versions of each module.

Versioned imports do *not* require an agent to support "N versions
of each module".  They're simply "truth in advertising" about what
an agent *does* support, and what needs to be done to properly
implement a particular definition.

However, I think this discussion does reveal the importance (from
and implementation perspective) of being able to determine whether
two definitions (whether different versions of "the same thing" or not)
are equivalent in some interesting dimension.  That's a critical part
of the factoring process that allows code re-use, as well as setting
the bounds of interoperability.

Randy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 24 Jun 2008 15:55:39 -0400

>an agent to support N versions of each module.

So you see conformance statements that list the value of every
enumeration of every type, so that when a new one is added, apps
can inspect these lists and see that my device's conformance statement
doesn't list the new one?

>a requirement would be a show-stopper.

If that's what the spec says, and what the WG decides, I think it'll
be implemented.  Any technology that is oriented toward networks
and network management that doesn't have the flexibility to allow
specs to evolve is dead.

What we're trying to make with YANG (and NETCONF) is a plumbing
mechanism that one can build one's products around, so that this
isn't code at some far corner of the source code, but a reasonable
way to support the entire product.  YANG-based infrastructure gives
us this.

If YANG modules are dead, we might as well be written them in Latin.
And yes, I'm sure the IESG has a "no Latin" policy.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 13:01:20 -0700

> implement a particular definition.

You are right.
It would not be as clean as complete identifiable versions.
It would be the intersection of the changed imported and augmented
symbols as they ripple from the top module through all
imports, nested imports, submodules, and nested submodules.

Not every construct changes in every release of course,
so it is only an arbitrary subset that one needs to deal with,
which doesn't help the code design at all.

In SNMP, only the typedef is reusable, and allowed to change
only in backward-compatible ways.  So 1 instance
of the fooObject may allow 1..10 and a newer instance
of the fooObject may allow 1..5 instead.  That seems much more
tractable than arbitrary object changes (add, remove, modify)
for any given subtree in the configuration, within the same agent.
Of course, the RPC methods and notifications can have multiple
versions as well.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 24 Jun 2008 22:05:23 +0200 (CEST)

Hi,

Andy Bierman <[email-masked]> wrote:
> of the same YANG module in the same agent.

Although I'm still not convinced that versioned imports is a good
idea, I don't think that this particular issue is a problem.

The agent will have to advertise one version for which it implements
the data definition statements, rpcs and notifications.  But if some
other modules reuse typedefs and groupings, the agent just implements
those types (as it does with other types).

BTW, we I think we should state that a server advertises a module in
the hello message only if it implements data definition statements,
rpc, or notifications from that module.  No need to advertise
e.g. yang-types.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 13:17:04 -0700

> rpc, or notifications from that module.  No need to advertise

I disagree.
If the yang-types module changes over time, and m1 imports yang-types.v1
and m2 imports yang-types.v2, and m3 imports m1 and m2.  Juergen pointed
out that if the 'delta' module is owned by another organization, your
module (e.g., m3) is broken until that organization removes the delta
(e.g., update m1 to use yang-types.v2).

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 24 Jun 2008 22:34:46 +0200 (CEST)

[...]

> (e.g., update m1 to use yang-types.v2).

My point was that it is not broken.  A server that implements m3
would advertise the version it implements (as per above).  The types
used in m1 and m2 are well-defined and part of the normal type-chain
that the server has to implement.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 16:23:39 -0700

> that the server has to implement.

This is kind of an implementation detail, but it seems to
me that if m3 imports yang-types directly, and also imports
m1, that objects expanded from groupings in m1 will be using
yang-types.v1, and objects expanded from groupings directly
in m3 will use yang-types.v2.  This aspect of groupings
is not obvious, and (IMO) complicated to implement.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 24 Jun 2008 23:48:47 -0400

>is not obvious, and (IMO) complicated to implement.

Can you explain why this is more complicated?  It seems like this
is really just a matter of understanding that a module is uniquely
described by the name plus the revision, rather than just the name.
This seems pretty trivial.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 24 Jun 2008 22:34:18 -0700

> This seems pretty trivial.

I don't think it will help anymore to debate implementation details.
The IETF has a lot of experience with SMIv2.  I prefer
the way SMIv2 handles versioning and conformance.
There is one file for each module and you can derive
all the versions from that file, and all the versions
will be backward-compatible with each other.

I guess time will tell if YANG is too complicated,
and there are any deployments or not.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Jun 2008 08:42:20 +0200 (CEST)

> is not obvious, and (IMO) complicated to implement.

I think this would be a natural extension to what's already there in
the draft:

    References from inside the grouping are relative to the scope in
    which the grouping is defined, not where it is used.

/martin