Subject: [netmod] augmenting imported module

From: Hiroto Shibuya <[email-masked]>
Date: Thu, 6 May 2010 17:04:54 -0400

Hi,

I have recently learned about YANG during my search for data modeling
language for my code generation project outside of NETCONF context.

One particular aspect I liked about YANG was support for augmentation.

--------
4.2.8.  Extending Data Models (augment)

   YANG allows a module to insert additional nodes into data models,
   including both the current module (and its submodules) or an external
   module.
--------

Now, I interpreted this to mean, that when there is published,
complete YANG module, I can import it, augment it with small
modification and use the rest of schema AS IS.  I also intepreted that
the method of doing that is by "import" and "augment".

But while playing around with YANG schema with pyang tool, I noticed
that the language in the draft is not particulary clear about how this
can be achieved.

First it says:

--------
4.1.  Functional Overview
  ...
   A module can import data from other external modules,
--------

But later,

--------
5.1.  Modules and Submodules

   A module uses the "include" statement to include its submodules, and
   the "import" statement to reference external modules.
--------

and then,

--------
7.1.5.  The import Statement

   The "import" statement makes definitions from one module available
   inside another module or submodule.
--------

So, when I "import", I can *refer* to things like "group" and "use" it
in my module, but as far as I can tell, YANG does not provide
mechanism to simply pull in nodes from the external module.  You can
do that by including submodule, but you cannot do that with external
module.

So even if you "augment" a node in the external module, "augment"
statement have no functionality to pull in node from external module
and it simply augment the node in that external module, but have no
effect on the current module.

Since pyang is implemented to this language, when I define a module to
only augment an imported module, pyang will verify everything is ok
but when I tried to output the schema, it output an empty schema.

I did a local patch to pyang to change the semantics of "import" to
actually "include" the contents of the imported module and the result
is what I wanted, but I wondered if this is what is originally meant
by the statement in "4.2.8.  Extending Data Models (augment)".

Hiroto

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 7 May 2010 00:05:53 +0200

> effect on the current module.

Yes, this is how augmentation works. Is section 7.15 not clear enough?
 
> by the statement in "4.2.8.  Extending Data Models (augment)".

No, what you want is to use a grouping and to add to it. This of
course requires that someone designed a reusable grouping. Copying
arbitrary definitions into a new module is not a good idea because
this means the copied definitions live in a new namespace and thus
there is no interoperability. The augmentation mechanism of YANG
preserves interoperability since it avoids copying definitions into a
new namespace.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Hiroto Shibuya <[email-masked]>
Date: Thu, 6 May 2010 18:37:01 -0400

> preserves interoperability since it avoids copying definitions into a
Sounds like I was thinking this backward.

So if I create module2 which is augmenting module1, module2 is actually
modifying schema within module1 and not creating new schema within module2
which augments module1?    So in that situation, if I load module1 and
module2 into a tool to munch on then, it should spit out augmented module1,
and not spit out module2 (which is empty anyway)?

Hiroto

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 07 May 2010 08:24:46 +0200

> effect on the current module.

What you can do is to set up a data model formed by *both* the external
module and the augmenting module. The resulting data tree is then the
union of the trees from both modules and the augmenting applies as well.

> but when I tried to output the schema, it output an empty schema.

The DSDL plugin (the code from the Google Code repository) does what you
want. For example, you can use the yang2dsdl script with multiple
modules to get RELAX NG, Schematron and DSRL schemas for a reply to
<get> as follows:

$ yang2dsdl -t get-reply foo.yang bar.yang

An instance document can also be directly validated using this script.
The details are in the manpage.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Hiroto Shibuya <[email-masked]>
Date: Fri, 7 May 2010 03:07:26 -0400

Hi Lada,

Nice to hear from you.  Thank you very much for your work on pyang.

> union of the trees from both modules and the augmenting applies as well.

So,  is this "data model" something outside the scope of YANG?  In case
module1 import and augment module2, neither module1 nor module2 is the union
of two models (when all module2 does is to augment)?   If that is the case,
what was meant by 4.2.8?

> The details are in the manpage.
I looked at dsdl plugin and after receiving multiple modules from pyang, it
is doing its own walk through of the tree and merging of augmented nodes.
Am I being too lazy to assume that should be handled by the pyang to be in
line with 4.2.8?

Hiroto

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 07 May 2010 09:46:44 +0200

Hiroto Shibuya píše v Pá 07. 05. 2010 v 03:07 -0400:
> 
> So,  is this "data model" something outside the scope of YANG?  In
> case module1 import and augment module2, neither module1 nor module2
> is the union of two models (when all module2 does is to augment)?   If
> that is the case, what was meant by 4.2.8?

Yes, this is supposed to be handled outside of YANG. In NETCONF terms it
means that both modules should be advertized in <hello>. I agree with
you that this part is somewhat confusing.

> I looked at dsdl plugin and after receiving multiple modules from
> pyang, it is doing its own walk through of the tree and merging of
> augmented nodes.    Am I being too lazy to assume that should be
> handled by the pyang to be in line with 4.2.8?  

I am afraid I don't understand what you mean by the last sentence. But
yes, the DSDL plugin grafts the augmenting nodes to the original tree,
because RELAX NG has no such mechanism as 'augment' in YANG. The result
can be demonstrated quite nicely by mapping the "ietf-netconf" module
from 4741bis together with "ietf-netconf-with-defaults", which augments
some RPC operations from the former module:

$ yang2dsdl -t rpc ietf-netconf.yang ietf-netconf-with-defaults.yang

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 7 May 2010 09:53:43 +0200

On Fri, May 07, 2010 at 12:37:01AM +0200, Hiroto Shibuya wrote:
 
> So if I create module2 which is augmenting module1, module2 is
> actually modifying schema within module1 and not creating new schema
> within module2 which augments module1?

No, module 2 is defining something that appears at a place in the
instance tree defined by module 1. Look at the example in the YANG
spec - I think this makes it clear how it works.

> So in that situation, if I load module1 and module2 into a tool to
> munch on then, it should spit out augmented module1, and not spit
> out module2 (which is empty anyway)?

Depends on the muching your tool wants to do. I can only explain how
YANG works.

/js

---

From: Hiroto Shibuya <[email-masked]>
Date: Fri, 7 May 2010 04:38:48 -0400

On Fri, May 7, 2010 at 3:46 AM, Ladislav Lhotka <[email-masked]> wrote:

 Interesting.   I tried

pyang ietf-netconf.yang ietf-netconf-with-defaults.yang -f tree

 and output result did contained properly augmented tree as I expected.

Hiroto

---

From: Hiroto Shibuya <[email-masked]>
Date: Fri, 7 May 2010 04:46:33 -0400

> spec - I think this makes it clear how it works.
Ok, thanks.  Now it is making more sense to me.  It is still mighty
confusing to to state that module2 modifies module1 by importing it.   It is
confusing because you can actually "import" groups from module1 and use it
in module2, so importing is kind of bidirectional and could produce modified
trees in both modules.

Hiroto

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 7 May 2010 11:01:20 +0200

On Fri, May 07, 2010 at 10:46:33AM +0200, Hiroto Shibuya wrote:
 
> confusing to to state that module2 modifies module1 by importing it.
> It is confusing because you can actually "import" groups from
> module1 and use it in module2, so importing is kind of bidirectional
> and could produce modified trees in both modules.

Its not bidirectional. You don't modify a module by importing from it.

Your problem is that you assume that a module exclusively owns the
data tree it creates, but that assumption is not correct in YANG
because we allow augmentations. This might be surprising at first
sight if you are only familiar with schema languages that can't do
this. But it is an incredible feature in order to let vendors extend
standard definition and keeping related data organized closely
together.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 07 May 2010 11:03:44 +0200 (CEST)

> confusing to to state that module2 modifies module1 by importing it.

No, 'import' does not modify the imported module.  In YANG, all
containers and lists can have nodes from other namespaces.  'augment'
defines such nodes.

/martin