Subject: [netmod] submodules the hidden benefits

From: tom petch <[email-masked]>
Date: Wed, 5 Aug 2020 16:20:17 +0000

Looking at an I-D with endless submodules, I am left thinking

Costs

greater in size
more complex to understand
harder to review
more likely to have undetected errors

Benefits

What am I missing?  I have only ever seen one such module in the IETF that I can recall and it is draft-ietf-idr-bgp-model and AFAICT it would be smaller, simpler, easier to understand if it were done with grouping instead of submodule.

Tom Petch

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 5 Aug 2020 18:48:20 +0200

As a data point, RFC 7407 uses sub-modules. The sub-modules more or
less reflect SNMPv3's modular architecture, the different transports
are for example separated out as sub-modules. I guess we also used
sub-modules back then because we wanted to see how this works in an
IETF context (RFC 7407 was approved/published 6 years ago, we were
still learning how to use YANG at that time).

I personally meanwhile believe that sub-modules add complexity with
little extra value but this view surely is not shared by others.
Anyway, people using them should have a good reason for using them.
Since SNMPv3 has a great modular architecture, it makes sense to
structure the YANG modules accordingly and instead of having many
ietf-snmp-<foo> modules, sub-modules came in handy to have everything
in a single namespace. For technologies that do not have a modular
architecture that can be followed, creating modular sub-modules is
likely not worth the effort.

/js

PS: I am not sure that sub-modules are necessarily harder to
    review. For the crypto YANG modules, we have several intertwined
    modules, they are likely as complex to review as if they were all
    sub-modules sharing a single namespace - or a single module with
    all definitions in one big module.

On Wed, Aug 05, 2020 at 04:20:17PM +0000, tom petch wrote:
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 5 Aug 2020 10:15:20 -0700

> more likely to have undetected errors
How about "confusion due to false sense of modularity"?
People often think submodules work like modules (but they don't).
There think a submodule should "compile" when validated on its own
(that used to be true in YANG 1.0 but not in YANG 1.1).  They think
mix-and-match
include submodule revisions can be selected by the server vendor just like
importing
modules (but that does not work).

> Benefits
>
>
>
> What am I missing?  I have only ever seen one such module in the IETF that
> I can recall and it is draft-ietf-idr-bgp-model and AFAICT it would be
> smaller, simpler, easier to understand if it were done with grouping
> instead of submodule.
>
> Tom Petch
>
>
Andy

---

From: Vladimir Vassilev <[email-masked]>
Date: Wed, 5 Aug 2020 23:22:26 +0200

> little extra value but this view surely is not shared by others.

+1. IMO removing sub-modules from YANG 2.0 should be on the list of 
proposed changes.

/Vladimir

---

From: "Reshad Rahman (rrahman)" <[email-masked]>
Date: Wed, 5 Aug 2020 21:44:47 +0000

Indeed
https://github.com/netmod-wg/yang-next/issues/26

On 2020-08-05, 5:22 PM, "netmod on behalf of Vladimir Vassilev" <[email-masked] on behalf of [email-masked]> wrote:

    /Vladimir

---

From: Mahesh Jethanandani <[email-masked]>
Date: Wed, 5 Aug 2020 15:43:06 -0700

A contrarian view:

I find the use of sub-modules helpful when I want to use separate files to maintain part of the module that is logically separate, while maintaining/restricting the use of them to a single namespace.

The fact that tools have a problem with trying to compile a sub-module can be addressed in the tools themselves.

> On Aug 5, 2020, at 2:44 PM, Reshad Rahman (rrahman) <rrahman=[email-masked]> wrote:
> 
> Indeed

---

From: Jan Lindblad <[email-masked]>
Date: Thu, 6 Aug 2020 11:00:22 +0200

I have to agree with the long list of "Costs" of submodules listed in this thread, and can attest to the brevity of the "Benefits" side. The globally accumulated amount of gray hairs produced by the YANG 1.0 submodule rules is best measured in cubic meters. 

The YANG 1.1 rules are much more in line with industry expectation in my experience, but the fact that the rules differ greatly between the two YANG versions with unchanged syntax and that many tools still do not properly support the YANG 1.1 submodule inclusion rules to this date (despite otherwise boasting YANG 1.1 support for long) is additional salt in a sore world.

One central point that has been missed in the discussion on the merits of modules vs. submodules is the implications this choice has on module versioning. Since versioning happens on the module/namespace level, there is a major difference between releasing 10 modules which are versioned independently, or one module with 10 submodules, which would have a single module version. If this point goes on the "Costs" or "Benefits" side in the book keeping, I'll leave open to interpretation.

/jan

> On 6 Aug 2020, at 00:43, Mahesh Jethanandani <[email-masked]> wrote:
> 
> A contrarian view:
> 
>> On Aug 5, 2020, at 2:44 PM, Reshad Rahman (rrahman) <rrahman=[email-masked] <mailto:rrahman=[email-masked]>> wrote:
>> 
>> Indeed
>> https://github.com/netmod-wg/yang-next/issues/26 <https://github.com/netmod-wg/yang-next/issues/26>
>>

---

From: tom petch <[email-masked]>
Date: Thu, 6 Aug 2020 09:29:01 +0000

Many thanks to all the responses so far; most helpful and much appreciated.  As a point of information, the I-D that triggered this, draft-ietf-idr-bgp-model, is out for, or about to be,  YANG Doctor review so one of you can express a more formal opinon thereon.  (Perhaps it will be the YANG Doctor who gladdened my heart when they said that this grouping is only used once so there is no point in making this a grouping:-).

Again, more for the benefit of the chosen YANG Doctor, the prefix in this I-D are all over the place IMHO.  Setting aside the use of two letter prefix such as  bp: ,bt:.  br:, there are 100 or so places where the prefix will have be to edited - look for example at the use of rpol for routing policy instead of rt-pol.

And I was wrong to say that I had not seen submodules before - I had forgotten about RFC8349.

Tom Petch.

________________________________________
From: netmod <[email-masked]> on behalf of Jan Lindblad <[email-masked]>
Sent: 06 August 2020 10:00

/jan

On 6 Aug 2020, at 00:43, Mahesh Jethanandani <[email-masked]<mailto:[email-masked]>> wrote:

A contrarian view:

On Aug 5, 2020, at 2:44 PM, Reshad Rahman (rrahman) <rrahman=[email-masked]<mailto:rrahman=[email-masked]>> wrote:

Indeed

On 2020-08-05, 5:22 PM, "netmod on behalf of Vladimir Vassilev" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

   /Vladimir

   _______________________________________________
   netmod mailing list
   [email-masked]<mailto:[email-masked]>
   https://www.ietf.org/mailman/listinfo/netmod

_______________________________________________
netmod mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netmod

_______________________________________________
netmod mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netmod

---

From: Vladimir Vassilev <[email-masked]>
Date: Thu, 6 Aug 2020 14:15:09 +0200

On 06/08/2020 00.43, Mahesh Jethanandani wrote:

> A contrarian view:
>
> I find the use of sub-modules helpful when I want to use separate 
> files to maintain part of the module that is logically separate, while 
> maintaining/restricting the use of them to a single namespace.
>
> The fact that tools have a problem with trying to compile a sub-module 
> can be addressed in the tools themselves.

The real problem is that the use of sub-modules complicates the 
interface between the tools. If your tools have to support dynamic 
loading of modules and not only a pre-compiled static schema context 
this problem becomes evident.

/Vladimir

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Fri, 7 Aug 2020 21:40:00 +0000

I agree submodules can cause confusion, but I also agree with Mahesh that they can be useful to partition things for people using the modules. Especially for huge models (e.g. router vendor models). You can jump right to a subsection of the data (a bit like a document with chapters rather than one huge chapter).

It can also be useful for consumers of the models for versioning (when you want to avoid multiple namespaces). If a model is made up of 30 sub-modules, it might be useful to know that the "bgp" part changed while the other parts of the model didn't.

Jason

From: netmod <[email-masked]> On Behalf Of Mahesh Jethanandani
Sent: Wednesday, August 5, 2020 6:43 PM
To: [email-masked]
Subject: Re: [netmod] submodules the hidden benefits

A contrarian view:

Indeed

On 2020-08-05, 5:22 PM, "netmod on behalf of Vladimir Vassilev" <[email-masked] on behalf of [email-masked]<mailto:[email-masked][email-masked]>> wrote:

   /Vladimir

---

From: tom petch <[email-masked]>
Date: Mon, 10 Aug 2020 09:23:07 +0000

From: netmod <[email-masked]> on behalf of Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>
Sent: 07 August 2020 22:40

<tp>

Jason,

Jan commented earlier that versioning happens at the module/namespace level in which case it would seem to me that all submodules will by definition have the same version so you cannot tell which submodules have changed,,

I think too that there is a fallacy in the belief that dividing up something large makes it easier to use, understand etc.  Divide up code by procedure, subroutine etc and if you can provide a parameter or two input, and one or two results output, and all the complexity of data structures, validation, algorithms etc and hidden out of sight, then you have simplified.   But DDL is not like that.  With a grouping or submodule, you need to know the internals, of what objects there are, how they are structured, what semantics they have, pretty much everything inside; it is just harder to find, to reference, to access because it has been wrapped up in something that gets in the way and tucked out of sight in some part of the I-D,
Rather simplification comes from having the right structure in the model, which some WG are good at,

Tom Petch

Jason

A contrarian view:

Indeed

   /Vladimir

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Mon, 10 Aug 2020 20:08:02 +0000

Thanks Tom.

Versioning in current YANG specs and in the new versioning work has independent versioning available for modules and for sub-modules. The module version will always bump when something changes in a submodule, but each submodule has its own revision. So it can help consumers quickly see which parts of a large schema have and haven't changed.

I agree that certain ways of dividing up modules, using groupings, etc can help or hinder readability. But I think that is orthogonal to submodules. YANG authors can make good or bad (*) readability decisions in a schema with submodules or a schema without submodules.  If a schema is divided up at the top level of a tree into submodules, then that is just as readable IMO as a large data model being divided up the same way into individual modules.  i.e. you can have your bgp module, your interfaces module, etc OR you can have your bgp submodule, your interfaces submodule, etc.

(*) good vs bad is often pretty subjective and there are sometimes tradeoffs amongst readability, consistency, maintainability, etc. But again that is all orthogonal to submodules vs modules IMO.

Rgds,
Jason

> -----Original Message-----
> From: tom petch <[email-masked]>
> Sent: Monday, August 10, 2020 5:23 AM
> To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>; Mahesh
> Jethanandani <[email-masked]>; [email-masked]
> I agree submodules can cause confusion, but I also agree with Mahesh that
> they can be useful to partition things for people using the modules. Especially
> for huge models (e.g. router vendor models). You can jump right to a
> subsection of the data (a bit like a document with chapters rather than one
> huge chapter).
> 
> It can also be useful for consumers of the models for versioning (when you
> want to avoid multiple namespaces). If a model is made up of 30 sub-modules,
> it might be useful to know that the "bgp" part changed while the other parts of
> the model didn't.
> 
> <tp>
> 
> Jason,
> 
> Jan commented earlier that versioning happens at the module/namespace level
> in which case it would seem to me that all submodules will by definition have
> the same version so you cannot tell which submodules have changed,,
> 
> I think too that there is a fallacy in the belief that dividing up something large
> makes it easier to use, understand etc.  Divide up code by procedure,
> subroutine etc and if you can provide a parameter or two input, and one or two
> results output, and all the complexity of data structures, validation, algorithms
> etc and hidden out of sight, then you have simplified.   But DDL is not like that.
> With a grouping or submodule, you need to know the internals, of what objects
> there are, how they are structured, what semantics they have, pretty much
> everything inside; it is just harder to find, to reference, to access because it has
> been wrapped up in something that gets in the way and tucked out of sight in
> some part of the I-D,
> I find the use of sub-modules helpful when I want to use separate files to
> maintain part of the module that is logically separate, while
> On Aug 5, 2020, at 2:44 PM, Reshad Rahman (rrahman)
> <rrahman=[email-masked]<mailto:rrahman=[email-masked]
> etf.org>> wrote:
> 
> Indeed
> On 2020-08-05, 5:22 PM, "netmod on behalf of Vladimir Vassilev" <netmod-
> [email-masked] on behalf of vladimir@lightside-
> instruments.com<mailto:netmod-
> [email-masked]%20on%20behalf%20of%20vladimir@lightside-
> instruments.com>> wrote:
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 10 Aug 2020 22:56:38 +0200

There are generic preprocessors such as good old cpp or m4 that can do
wonderful things if there is a need to split large (YANG module) files
into smaller pieces. The question here is what the added benefit is to
build sub-modules into YANG itself. The argument I have heard is that
this allows separate compilation of the sub-modules but even then I am
not sure how well this works in practice and whether it is worth to
save some CPU or IO time on developer machines. Perhaps someone can
post concrete numbers that show the savings that can be obtained...

/js

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2020 14:19:45 -0700

> post concrete numbers that show the savings that can be obtained...
It is not possible in YANG 1.1 to compile a single submodule.
YANG allows the use of identifiers from other submodules without any
include-stmts.
It is not possible to compile (or read) a single submodule without the main
module and
all the correct revisions of the other submodules.

In this way submodules make readability much worse than imported modules.

Andy

> > Versioning in current YANG specs and in the new versioning work has
> independent versioning available for modules and for sub-modules. The
> module version will always bump when something changes in a submodule, but
> each submodule has its own revision. So it can help consumers quickly see
> which parts of a large schema have and haven't changed.
> >
> > I agree that certain ways of dividing up modules, using groupings, etc
> can help or hinder readability. But I think that is orthogonal to
> submodules. YANG authors can make good or bad (*) readability decisions in
> a schema with submodules or a schema without submodules.  If a schema is
> divided up at the top level of a tree into submodules, then that is just as
> readable IMO as a large data model being divided up the same way into
> individual modules.  i.e. you can have your bgp module, your interfaces
> module, etc OR you can have your bgp submodule, your interfaces submodule,
> etc.
> >
> > (*) good vs bad is often pretty subjective and there are sometimes
> tradeoffs amongst readability, consistency, maintainability, etc. But again
> that is all orthogonal to submodules vs modules IMO.
> >
> > Rgds,
> > Jason
> >
> > > From: netmod <[email-masked]> on behalf of Sterne, Jason
> (Nokia -
> > > CA/Ottawa) <[email-masked]>
> > > Rather simplification comes from having the right structure in the
> model, which
> > > some WG are good at,
> > >
> > > Tom Petch
> > >
> > > Jason
> > >
> files to