Subject: [netmod] definition of "fully expanded YANG"

From: Michael Rehder <[email-masked]>
Date: Wed, 17 Jul 2019 13:03:03 +0000

Has there ever been discussion about defining “fully expanded YANG”, that is a YANG module with all internal and external imports resolved?
This has several uses:
- Base for workable DIFF between versions so one can see all impacts of changes
   I see this as a pretty important part of semantic version management – if you can’t see the net result of changes in a properly modular YANG package, how can you be sure of the version meaning?
- Base for translation/processing of the schema
  There are all sorts other materials one can generate from a YANG (or a fully expanded YIN). If not expanded then this processing is very difficult.

Thanks
Mike
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 17 Jul 2019 15:10:35 +0200

On Wed, Jul 17, 2019 at 01:03:03PM +0000, Michael Rehder wrote:

> Has there ever been discussion about defining “fully expanded YANG”,
> that is a YANG module with all internal and external imports
> resolved?

I wonder what "all internal and external imports resolved" means here
- you mean included inline? I doubt this will be tremendously useful
nor do I know what that will work with augments and the fact that
definitions live in different namespaces and that for example must and
when expressions are namespace qualified.

I believe there are tools that do semantic diffs based on comparing
schema trees internal to a compiler and that sounds to me more robust
than producing "fully expanded YANG".

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Michael Rehder <[email-masked]>
Date: Wed, 17 Jul 2019 13:18:57 +0000

Yes, inlining of everything, including augments. What you'd see in the tree representation.
For example augments makes it particularly impossible for a consumer of the YANG schema to process the schema since the final result.

Some use cases:
- translating to another schema language
- translating to a UI dialog for data entry
- an accurate diff on versions

Model-driven-design means using the "computable specification" for all sorts of things, increasing the ROI of the schema definition.

The diff is a significant issue. As a schema author, I must reliably document all changes.
Without a full expansion, it's easy to not notice that a particular module has changes in its final form.
I don't see how you could verify a semantic version number in a package without this.

> -----Original Message-----
> From: Juergen Schoenwaelder [mailto:[email-masked]]
> Sent: Wednesday, July 17, 2019 9:11 AM
> To: Michael Rehder <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [netmod] definition of "fully expanded YANG"
> 
> > that is a YANG module with all internal and external imports resolved?
> 
> - you mean included inline? I doubt this will be tremendously useful nor do I
> know what that will work with augments and the fact that definitions live in
> different namespaces and that for example must and when expressions are
> namespace qualified.
> 
> I believe there are tools that do semantic diffs based on comparing schema
> trees internal to a compiler and that sounds to me more robust than producing
> "fully expanded YANG".
> 
> /js
> 
> --
> Fax:   +49 421 200 3103
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jac
> obs-
> university.de%2F&amp;data=02%7C01%7CMichael.Rehder%40Amdocs.com%7
> Ce64f1bc5d532478ded2808d70ab844b5%7Cc8eca3ca127646d59d9da0f2a0289
> 20f%7C0%7C0%7C636989658871253899&amp;sdata=US1sJn0ZqC77kiIlqdyzLEtz
> BZdfoo0MCLtG8jqw5w0%3D&amp;reserved=0>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 17 Jul 2019 15:36:33 +0200

> - translating to another schema language

Hi Michael,

If that is interesting to you:
We will spend the weekend on translating between data modeling languages during the IETF105 hackathon as part of the WISHI activity inside T2TRG (sorry for the abbrev overload).  YANG will be among the languages we are looking at.  This is all in the context of the data model convergence activities going on in IoT space, but of course the resulting model translation engines will be useful beyond that.

More info:

http://wishi.space/
https://github.com/t2trg/wishi/wiki/Preparation:-Hackathon-Planning#planned-hacking-topics
https://github.com/t2trg/wishi/wiki/IETF-105-Hackathon

Discussion is on the T2TRG mailing list,
https://www.ietf.org/mailman/listinfo/t2trg

Grüße, Carsten

---

From: Michael Rehder <[email-masked]>
Date: Wed, 17 Jul 2019 13:40:34 +0000

Yes, that is interesting and is exactly from my experience.

I'm sure that each translator will either not support much "replacement" (maybe none at all) or will re-create full-expansion again.

Mik
> From: Carsten Bormann [mailto:[email-masked]]
> Sent: Wednesday, July 17, 2019 9:37 AM
> We will spend the weekend on translating between data modeling languages
> during the IETF105 hackathon as part of the WISHI activity inside T2TRG (sorry
> for the abbrev overload).  YANG will be among the languages we are looking
> at.  This is all in the context of the data model convergence activities going on
> in IoT space, but of course the resulting model translation engines will be useful
> beyond that.
> 
> More info:
> 
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwishi.spac
> e%2F&amp;data=02%7C01%7CMichael.Rehder%40Amdocs.com%7C15271d0a7
> 3d348c798a308d70abbe4ae%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7
> C0%7C636989674421882779&amp;sdata=RpTY8zuzTfa1nOa8sAyBwsZgYpivjvIO
> VKxIRg2CCJM%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co
> m%2Ft2trg%2Fwishi%2Fwiki%2FPreparation%3A-Hackathon-
> Planning%23planned-hacking-
> topics&amp;data=02%7C01%7CMichael.Rehder%40Amdocs.com%7C15271d0a
> 73d348c798a308d70abbe4ae%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%
> 7C0%7C636989674421892775&amp;sdata=oIJx%2FpX%2B9n5kwB7%2BU5F%2F
> H%2FIrQPj9vCsVDUEQ4IJCDE0%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co
> m%2Ft2trg%2Fwishi%2Fwiki%2FIETF-105-
> Hackathon&amp;data=02%7C01%7CMichael.Rehder%40Amdocs.com%7C1527
> 1d0a73d348c798a308d70abbe4ae%7Cc8eca3ca127646d59d9da0f2a028920f%7
> C0%7C0%7C636989674421892775&amp;sdata=bBuZzX9gnlEwsw5PbMZt7puGQ
> GflOEkJiiLtzpnyNag%3D&amp;reserved=0
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.
> org%2Fmailman%2Flistinfo%2Ft2trg&amp;data=02%7C01%7CMichael.Rehder%
> 40Amdocs.com%7C15271d0a73d348c798a308d70abbe4ae%7Cc8eca3ca127646
> d59d9da0f2a028920f%7C0%7C0%7C636989674421892775&amp;sdata=GsnU%
> 2BiTgIdVzbKXcXjRhWJO0nWfjBSakknw4tLL7qRY%3D&amp;reserved=0
> 
> Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 17 Jul 2019 15:54:14 +0200

> Model-driven-design means using the "computable specification" for all sorts of things, increasing the ROI of the schema definition.

It sounds to me like you are interested in what compiler people call
the abstract syntax tree, which is quite a different thing than the
YANG module you start with. The easiest and most robust way to do
translations is likely to simply expand pyang (or any other compiler
of choice).

> I don't see how you could verify a semantic version number in a package without this.

Well, this sounds a bit like a tooling issue. You are likely better
off with a tool that can do a semantic comparison for you instead of
working with "fully expanded YANG" modules and generic diff tools.

/js

---

From: Robert Varga <[email-masked]>
Date: Wed, 17 Jul 2019 16:32:14 +0200

> translations is likely to simply expand pyang (or any other compiler

Depends on how you define the AST (and how many different ASTs are in
your compiler), as .yang file has an AST itself, which you then
cross-reference between modules/submodules, and you arrive at some IR,
from which you derive the effective model with whatever indices/details
your use case requires.

This is certainly possible with OpenDaylight tooling, i.e. arrive at
https://github.com/opendaylight/yangtools/blob/master/yang/yang-model-api/src/main/java/org/opendaylight/yangtools/yang/model/api/EffectiveModelContext.java,
which gives you a per-module tree of statements with all RFC7950
semantic processing applied. It also contains some generally-useful
indices (like schema tree/data tree traversal) as well as references to
original statement declaration (i.e. as it appeared in .yang file, if
applicable).

Regards,
Robert

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 19 Jul 2019 10:36:00 +0000

Hi Michael,

I don’t know about “fully expanded YANG”, but some of the work by the YANG versioning design team might be of interest to you:

draft-verdt-netmod-yang-solutions-01<https://datatracker.ietf.org/doc/draft-verdt-netmod-yang-solutions/> gives an overview of the solution space.
draft-rwilton-netmod-yang-packages-01<https://datatracker.ietf.org/doc/draft-rwilton-netmod-yang-packages/> defines versioned packages (although it needs some updating to accommodate the changes that happened to draft-verdt-netmod-yang-module-versioning-00<https://datatracker.ietf.org/doc/draft-verdt-netmod-yang-module-versioning/>).  A YANG package, if all dependencies are resolved, can be used to define a complete YANG schema (i.e. with module import/include dependencies resolved).  From a definition POV, this might be the “fully expanded YANG” that you are looking for.
However, as others are pointed out, it is effectively tooling that takes a list of YANG modules (perhaps from YANG library, or a YANG package, or other source) and internally constructs a complete schema with dependencies resolved, deviations applied, features handled appropriately.
Thanks,
Rob

From: netmod <[email-masked]> On Behalf Of Michael Rehder
Sent: 17 July 2019 14:03
To: [email-masked]
Subject: [netmod] definition of "fully expanded YANG"

Thanks
Mike
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service