Subject: Re: [netmod] Augmentable Groupings

From: "Linowski, Bernd (EXT-Other - DE/Dusseldorf)" <[email-masked]>
Date: Fri, 19 Sep 2008 15:36:07 +0300

Hi, 

Sorry for the late reply, I was four weeks on vacation ;-)  

> -----Original Message-----
> From: Linowski Bernd (EXT-Other - DE/Dusseldorf) 
> Sent: 15 August, 2008 17:43
> To: [email-masked]; ext Andy Bierman
> Subject: RE: [netmod] Augmentable Groupings
> 
>  
> Hi,
> 
> Andy Bierman [email-masked] wrote:
> > -----Original Message-----
> > From: ext Andy Bierman [mailto:[email-masked]] 
> > Sent: 13 August, 2008 17:18
> > To: Linowski Bernd (EXT-Other - DE/Dusseldorf)
> > Cc: [email-masked]
> > Subject: Re: [netmod] Augmentable Groupings
> > 
> > [email-masked] wrote:
> > > 
> > > Hi,
> > > 
> > > Andy Bierman [email-masked] wrote:
> > >  > Actually groupings are not in the same namespace as objects,
> > >  > so this Xpath is broken.
> > >  >
> > >  >
> > >  >     grouping foo;
> > >  >
> > >  >     container foo;
> > >  >
> > >  > Both of these are legal YANG.
> > > 
> > > That is not in line what is stated in section 7.11 of the
> > > YANG specification:
> > > " If the grouping is defined at the top level of a YANG module or
> > >    submodule, the grouping's identifier MUST be unique within the
> > >    module."
> > > Name "foo" is not unqiue within the module.
> > > 
> > 
> > Here is the text from the draft:
> > 
> > 6.2.1.  Identifiers and their namespaces
> > 
> > ....
> > 
> >     o  All groupings defined within a parent node or at the 
> > top-level of
> >        the module or its submodules share the same grouping 
> identifier
> >        namespace.  This namespace is scoped to the parent 
> > node or module.
> > 
> > ....
> > 
> >     All identifiers defined in a namespace MUST be unique.
> > 
> 
> I agree, there could be an ambiguity as there is a separate grouping 
> identifier namespace. 
> 
> Let me reconsider the options how to address this.
> 

In order to address this issue, I suggest that groupings 
are put in the same namespace as data nodes, rpc's and 
notifications. 
As mentioned in an earlier mail from Andy Bierman, collapsing 
the grouping namespace into the object namespace is in 
practice no problem. Since YANG identifiers are case sensitive
unique naming of objects still remains easy. 
Other advantages of this solution are that a new statement
is not needed and that namespace handling is simplified. 

In summary, introducing augmentable groupings means: 

- A grouping can be the target (node) of
  an augment statement. 
- Augmentation of a grouping becomes only
  effective in case the module containing 
  the augment statement is advertized. 
- The nodes added to the grouping are therefore
  implicitly added to all places the grouping
  is used in the set of modules supported 
  by an agent. 
- The augmented nodes are in the XML namespace
  of the module of the augment statement. 
- Groupings share the same namespace with
  data nodes, rpc's and notification in order
  to address them unambiguously in an augment
  statement. 
- In general, it is not allowed to augment
  mandatory nodes to a grouping. 

I think augmentable groupings are a powerful tool, which has 
different advantages for optimizing reuse but should only be 
utilized if appropriate.

BR,
Bernd

> > 
> > 
> > >  >
> > >  > If I nest the groupings, then what happens to the augment:
> > >  >
> > >  > module C {
> > >  >
> > >  >    grouping new-concepts {
> > >  >       uses concepts;
> > >  >    }
> > >  >
> > >  > Does new-concepts contain the augmentations from concepts?
> > >  >
> > > 
> > > Yes.
> > > 
> > 
> > 
> > Yes, well this makes it an NP-complete problem to resolve all the
> > uses statements in all the modules.
> > 
> > Even if that did not matter (just implementation!), the NETCONF
> > vendors have to deal with a pool of modules for which
> > they have no change control whatsoever.  And what about their own
> > modules already out there in shipping code?
> > 
> > This 'write once, and it shows up everywhere' feature is not
> > actually usable in the operational environment that YANG 
> must address.
> > 
> > 
> > Andy
> 
> 
> 
> 
> 
> 
> > 
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Andy Bierman <[email-masked]>
Date: Fri, 19 Sep 2008 06:10:36 -0700

> Sorry for the late reply, I was four weeks on vacation ;-)  

The YANG namespace issue is trivial, and not the real problem.
As mentioned by several people, this is not how OO works
for derived classes. I do not think YANG should be OO at all,
but if I did, I would want to follow the class hierarchy model.

Specifically, you should create a new class with a new name
if you want to add some new data to an existing class.
This allows all instances of the old class to continue
working as instances of the new class are also created.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 19 Sep 2008 20:32:40 +0200

>   the augment statement is advertized. 

What does "advertized" mean? If you have the NETCONF hello exchange in
mind, then I believe this is broken as long as we assume modular
implementations where some parts of the data model use the augmented
grouping while others do not.

>   is used in the set of modules supported 

See above, I doubt this works in general.

[...]

> different advantages for optimizing reuse but should only be 

I think it would help us if someone can (a) spell out the "different
advantages", (b) how augmentable groupings "optimize reuse", and (c)
when it is "appropriate" to utilize them and when not.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: "Linowski, Bernd (EXT-Other - DE/Dusseldorf)" <[email-masked]>
Date: Tue, 23 Sep 2008 15:21:02 +0300

Hi,

> The YANG namespace issue is trivial, and not the real problem.

I agree with you.

The aim is to make the augmentation of groupings as simple as possible.
So I think using the same namespace for groupings and data nodes, 
is an unambiguous solution for addressing groupings in augment
statements.

> but if I did, I would want to follow the class hierarchy model.

I think augmentable groupings and OO are different things. 
I agree that OO features should be better based on complex types
or class hierarchies.
We should talk on this in the interim meeting if there is interest.

BR,
Bernd

---

From: "Linowski, Bernd (EXT-Other - DE/Dusseldorf)" <[email-masked]>
Date: Tue, 23 Sep 2008 18:18:28 +0300

Hi,

Juergen Schoenwaelder ([email-masked]) wrote:
> What does "advertized" mean? 

The namespace URI of the module that contains
the augment statement is advertized as a capability in the
NETCONF hello-message. 

>If you have the NETCONF hello exchange in

If that is the case, you have to "bite the bullet" and augment
all the places one by one where the additions make sense.
Imo, an augmenting module should only be advertized if
the augmentations of groupings and data nodes it contains
make sense in all other advertized modules.

> when it is "appropriate" to utilize them and when not.

The interesting add-on of augmentable groupings is that the new
elements are available wherever the grouping is used.
An appropriate usage of the augmentable groupings is given imo
if the augmented grouping is defined and used with
"augmentation in mind". 

BR,
Bernd

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 23 Sep 2008 18:15:02 +0200

> if the augmented grouping is defined and used with
"Defining with augmentation in mind"

For me this means that groupings should be explicitly marked as augmentable, so users can be 
prepared for augmentation. This also has the advantage that simple (non-marked) groupings are 
not augmentable, so you don't have to consider the any unintended complications.

grouping interfaceAttributes {
    augmentable true;    // default would be false
    leafs, other content
    ...
}

Balazs

---

From: Andy Bierman <[email-masked]>
Date: Tue, 23 Sep 2008 09:24:02 -0700

> We should talk on this in the interim meeting if there is interest.

I think we have discussed augmentable groupings enough.
I would like to see the issue closed by the WG co-Chairs.

YANG is not OO, so class hierarchies are out of scope.
YANG needs to be used in multi-publisher environments,
and a mechanism that extends any arbitrary grouping in
arbitrary module cannot be properly controlled and restricted.
IMO, it is not implementable either, since an entire set of
modules must be processed 'as if simultaneously'.
(We know how well that worked out for SNMP Set ;-)

It is not consistent with the modular architecture of YANG,
since there is no indication whatsoever within module 'foo'
that it is being extended and modified by module 'bar'.

In every instance, the DM reader (or compiler) can determine
the exact contents of any data structure, from explicit
information within the 'starting module'.  You can follow
imports and includes in a deterministic manner.  This is not
the case with augmented groupings.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 23 Sep 2008 22:26:47 +0200

On Tue, Sep 23, 2008 at 06:18:28PM +0300, Linowski, Bernd (EXT-Other - DE/Dusseldorf) wrote:

> An appropriate usage of the augmentable groupings is given imo if
> the augmented grouping is defined and used with "augmentation in
> mind".

This sounds like magic to me or a circular argument or hand-waving or
a combination of all of this. If we can't write things down in clear
technical terms, then how can we expect that people use it right?

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 23 Sep 2008 22:30:42 +0200

On Tue, Sep 23, 2008 at 06:15:02PM +0200, Balazs Lengyel wrote:

> For me this means that groupings should be explicitly marked as 
> augmentable, so users can be prepared for augmentation. This also has the 
> advantage that simple (non-marked) groupings are not augmentable, so you 
> don't have to consider the any unintended complications.
>

But the underlying question then remains: How does a model writer
determine whether the augmentable bit of a grouping should be on or
off? And if the augmentable bit is on, all the other questions remain
the same - adding a knob does not answer any of those questions...

/js

---

From: "Linowski, Bernd (EXT-Other - DE/Dusseldorf)" <[email-masked]>
Date: Wed, 24 Sep 2008 14:49:51 +0300

Hi Balazs,

Balazs Lengyel [[email-masked]] wrote:
> Sent: 23 September, 2008 18:15
> To: Linowski, Bernd (EXT-Other - DE/Dusseldorf)
> Cc: [email-masked]; [email-masked]
> prepared for augmentation.

Yes, that very much captures what I had in mind.
I also think that making the intent explicit that 
a grouping may be augmented is useful.

> This also has the advantage that 
> simple (non-marked) groupings are 

I agree. And it ensures full backward compatibility with the current 
use of the augment statement.

>     augmentable true;    // default would be false

So in order to further refine the augmentable grouping feature, I
propose
  - to add an "augmentable" keyword which takes a boolean value as
    its only argument and which may appear as sub-statement of grouping.
  - In case the argument value is "true", the grouping may be augmented
  - In case the argument value is "false", the grouping cannot be
     augmented.
  - The default is "false", so groupings without an augmentable
statement
    cannot be augmented.

BR,
Bernd

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 24 Sep 2008 14:07:31 +0200

> use of the augment statement.

There is no official version of YANG out there we have to be backwards
compatible to.
 
>   - The default is "false", so groupings without an augmentable

So now back to the discussion what an augmentable grouping means. Only
when we precisely understand how an augmented grouping works can I
understand whether a bit to turn this on/off is needed and when it
should be turned on/off.

/js

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 24 Sep 2008 14:50:20 +0200

Hello,
When I am defining my data models in some specific situations I already know that I intend a 
part to be augmented/extended.

I also don't see why this is such a dangerous feature because:
- you face most of the same problems with augmenting a lists or containers
- the augmentable statement can totally protect standards
- usually if a node implements and advertises "module augmentor" it will anyway consider all 
the risks.
- you always have the fall back possibility augmentable false; and augment the individual uses 
of the grouping.
- if you are not allowed to augment a grouping with mandatory stuff, all an agent has to do is 
throw away some extra stuff when it receives it

Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 Sep 2008 15:18:33 +0200 (CEST)

Balazs Lengyel <[email-masked]> wrote:
> Hello,
> When I am defining my data models in some specific situations I already know
> that I intend a part to be augmented/extended.
> 

There is one main problem with this proposal.  It is summarized in one
of the first emails on this topic:

http://www.ietf.org/mail-archive/web/netmod/current/msg00784.html

You do NOT have this problem with normal augment.

Adding an 'augementable' flag does not solve this problem.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 24 Sep 2008 15:51:35 +0200

> You do NOT have this problem with normal augment.

If I import a definition from a module that has the original version,
why should I care that someone else also imports that definition and
modifies it in his own module? I will still get the original version.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 24 Sep 2008 15:59:32 +0200

On Wed, Sep 24, 2008 at 02:50:20PM +0200, Balazs Lengyel wrote:
> Hello,
> When I am defining my data models in some specific situations I already 
> know that I intend a part to be augmented/extended.
>
> - usually if a node implements and advertises "module augmentor" it will 
> anyway consider all the risks.
> - you always have the fall back possibility augmentable false; and 
> augment the individual uses of the grouping.
> - if you are not allowed to augment a grouping with mandatory stuff, all 
> an agent has to do is throw away some extra stuff when it receives it

If there are no problems around augmentable groupings, we likely do
not need this bit. If there are problems around augmentable groupings,
then having a bit to turn on/off augmentability does not help solve
the problem.

In other words, the introduction of a knob does not help us much in
figuring out how augmentable groupings would work. This is the IETF
where you can't solve a problem by making it optional.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 Sep 2008 16:10:16 +0200 (CEST)

> modifies it in his own module? I will still get the original version.

Because that's how augment of a grouping would work - if someone
augments a grouping, it affects all other users of that grouping (in a
device which implements the augmenting module).

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 24 Sep 2008 16:21:11 +0200

> device which implements the augmenting module).

Of course! If I import the module with the original definition and then
this definition changes in that module, the same problem appears.

It is more a problem of controlling what really gets imported (which
revision).

Would it help if the augmented definition became part of the augmenting
module namespace?

Module A defines amod:foo.

Module B imports A and augments amod:foo into bmod:foo.

Module C may import just B, then it can only use bmod:foo, or import
also A and then use the original version amod:foo as well.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 Sep 2008 16:28:00 +0200 (CEST)

> Module B imports A and augments amod:foo into bmod:foo.

This can already be done by creating a new grouping:

   module b {
      ...
      import a { prefix a; }
 
      grouping foo {
        uses a:foo;
        leaf b-stuff { ... }
      }
   }

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 24 Sep 2008 16:45:46 +0200

> This can already be done by creating a new grouping:

Oh yes, or even by applying augment here to get inside the grouping
a:foo. To Bernd: isn't this sufficient?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 24 Sep 2008 08:35:44 -0700

> device which implements the augmenting module).

YANG is no different than any other IETF specification
in one regard.  If it is perceived to be too complicated
for the benefit it provides, then it will not be implemented.
The tally of SNMP roadkill is substantial, and the WG would
be naive to think complexity doesn't really matter.

IMO, augmentable groupings provide a feature the IETF cannot
even use, and it is way too complicated to implement.
Show me the running code first, and maybe I will change
my mind.

Current augment is tractable because it operates on specific
nodes in the entire 'MIB tree'.  With augmented groupings you must
process the unspecified "entire module set" (in O(N^^2) time)
to determine the actual contents of a grouping.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 24 Sep 2008 10:24:58 -0700

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, September 24, 2008 7:10 AM
...
...

Maybe it would be helpful to step back and consider what the reasons
for wanting to do "augmentation" are.  I've seen three in the SNMP world.
Forgive the O-O terminology.

(1) to provide a means of implementing something functionally equivalent
to subclassing, in which either all or none of the instances of the superclass
in a given system will also be instances of the subclass.

(2) to provide a means of formally requiring that instances of two classes
within a system exist in a 1:1 relationship.

(3)  To allow standards devlopers to get agreement on a basic subset
capability, providing a hook for other standards (or vendors) to add
additional capabilities subject to the same constraints as in (1) above.

If we separate (2) out as really being a different kind of "beast", this problem
reduces to the one of what netmod's basic metamodel is, and how module
/ definition revision and refinement ares handled.  Until this gets hammered
out, these same arguments will just keep coming up again and again.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 Sep 2008 21:49:52 +0200 (CEST)

Hi,

> in a given system will also be instances of the subclass.

This is what augmentable groupings try to solve.

> within a system exist in a 1:1 relationship.

This can be done with keyrefs in YANG.

> additional capabilities

This is what the current 'augment' mechanism in YANG tries to solve.

> subject to the same constraints as in (1) above.

... but I don't understand this.

/martin

---

From: fanhuaxiang 90002624 <[email-masked]>
Date: Fri, 26 Sep 2008 11:38:41 +0800

Hi,
can anyone explaint it to me why augmentation of grouping raise so many issues augmentation of containers, lists ... do not involve.
e.g. why add a augmentable statement to grouping but not to container, list etc.
> - you face most of the same problems with augmenting a lists or 
> containers- the augmentable statement can totally protect standards
> > off? And if the augmentable bit is on, all the other questions 
> remain> the same - adding a knob does not answer any of those 
> questions...> 
> > /js
> > 
> 
> --

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 26 Sep 2008 07:20:18 +0200

On Fri, Sep 26, 2008 at 11:38:41AM +0800, fanhuaxiang 90002624 wrote:
> Hi,
> can anyone explaint it to me why augmentation of grouping raise so
> many issues augmentation of containers, lists ... do not involve.

Augementations of containers etc is supported and easy since the
augmentation happens in the data tree and thus has a well define
scope. An augmentation of a grouping is open ended and not well scoped
(it is unclear where it applies in data trees) and augmentations of
groupings might lead to name clashes.

In other words, to make augmentations of grouping work, it is
necessary to figure out how to scope them and how to ensure that name
clashes are avoided.

/js

---

From: fanhuaxiang 90002624 <[email-masked]>
Date: Fri, 26 Sep 2008 17:01:35 +0800

Hi,
  Thanks, I see.
  since grouping is different from other statements, why are we expecting the same effect of augment.
  how about that, whenever a grouping is augmented, a new grouping is also created while keep the augmented grouping unchanged?

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 26 Sep 2008 11:56:37 +0200

On Fri, Sep 26, 2008 at 05:01:35PM +0800, fanhuaxiang 90002624 wrote:

> since grouping is different from other statements, why are we
> expecting the same effect of augment.  how about that, whenever a
> grouping is augmented, a new grouping is also created while keep the
> augmented grouping unchanged?

This feature is already supported by YANG.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 26 Sep 2008 12:13:38 -0700

>   how about that, whenever a grouping is augmented, a new grouping is also created while keep the augmented grouping unchanged? 

IMO, it is important that the language is deterministic,
and any module or submodule is 'self-contained', meaning that
a human (or compiler) can start with foo.yang and understand
its contents to determine if they are valid.

I also believe the "all or nothing" augment is totally
useless for the IETF framework which allows modules to
evolve over time, and change control is distributed.

Consider this simple example why augmentable groupings
are not self-compiling:

module A {

    import B { prefix B; }

    augment B:G1 {
      leaf x { type int32; }
    }
}

module B {

   grouping G1 {
     leaf a { type string; }
     leaf b { type string; }
   }
}

module C {

   import B { prefix B; }

   container foo {
     leaf x { type string; }
     uses B:G1;
   }
}

Note that:
   - each individual module would compile just fine
   - there is no indication within module B or C that
     leaf 'x' from module A even exists (normal for augment)
   - when all modules are loaded together into some
     application, then it known that leaf 'x' already exists.
   - since groupings do not preserve namespace identity,
     this is a fatal error
   - normal augment is always deterministic, since at most
     there can be 2 modules involved, and when compiling the
     module with the augment-stmt, its complete validity
     can be determined since the database target is explicit
   - normal augment preserves namespaces, so name collision
     is not an issue anyway

Andy

---

From: fanhuaxiang 90002624 <[email-masked]>
Date: Sat, 27 Sep 2008 09:40:09 +0800

Hi,
comments inline.
> expecting the same effect of augment.
> >   how about that, whenever a grouping is augmented, a new 
> grouping is also created while keep the augmented grouping 
> unchanged? 
> > 
> 
> 
Yeah, C hardly know which grouping to use, original one in B or augmented one in A?
so IMHO, we should preserve the origin of B:G1, and define augmented B:G1 to A:G1 or B:G2(if it is augmented in B), so "uses B:G1" statement is diterministic now.
but in that case, we should design a grouping-specific augment statement.
> 
> 
> Andy
> 
> 
> >> On Fri, Sep 26, 2008 at 11:38:41AM +0800, fanhuaxiang 90002624 
> wrote:>>> Hi,
> >> scope. An augmentation of a grouping is open ended and not well 
> scoped>> (it is unclear where it applies in data trees) and 
> augmentations of
> >> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, 
> Germany>> Fax:   +49 421 200 3103         <http://www.jacobs-
> university.de/>>>

---

From: Andy Bierman <[email-masked]>
Date: Sun, 28 Sep 2008 06:36:30 -0700

> in a given system will also be instances of the subclass.

Is this how OO works?  I think not.
If you want to modify all instances of a class, don't
you have to modify that class definition, or 1 of its ancestors?

Ignoring all the implementation details for augmentable
groupings, this does not seem to be a data modeling practice
that is currently in use.  It seems better suited to the
IRTF than the IETF at this time.

We have lots of experience with vendors augmenting specific
objects in SNMP, and it is quite reasonable to support
that same sort of functionality in NETCONF and YANG.
We have no experience with augmenting arbitrary abstractions,
which in turn can use other groupings as well as augment
other groupings.  If one really thought about all the
new complexity due to the nature of the data-def-stmt,
this feature may not seem so great after all.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 28 Sep 2008 16:45:32 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Sunday, September 28, 2008 6:36 AM
...
...

(1) I'm making no claim that this is "how OO works" or that it is how OO
should work.  It's merely the way SNMP SMI augmentation works,
for better or worse.  This WG needs to be clear on whether, when
it borrows the word, whether it also borrows the baggage that comes
with it.

(2) Configuration management sanity demands that to "modify a 
class definition" in any non-trivial way has to be understood as
creating a new class definition. 

Randy