Subject: [netmod] yang-usage-02; sec 4.6, import by revision

From: Andy Bierman <[email-masked]>
Date: Fri, 11 Dec 2009 07:55:46 -0800

Hi,

There was a request at the WG meeting to add text clarifying
when import/include by revision should be used.
The following text is proposed:

sec 4.6, Header Contents, new para 2 and 3:

  If any modules are imported, then the same prefix value SHOULD
  be used, as defined in the imported module, unless there is
  another imported module already using that prefix.

  If any external typedefs or groupings are used, either imported
  from another module, or included from a submodule, then the
  revision date MUST be used in the import-stmt or include-stmt,
  to identify the specific revision of the external file.

[ed.: extensions, unique-stmt, and XPath references do not need
the revision-date because only the identifier QName is being
referenced, which is never allowed to change.

Currently, not one YANG module I have ever seen has used
import-by-revision, so this guideline is not being followed at all.
The import-without-revision is extremely fragile for groupings,
typedefs, and defaults.  This isn't like SMIv2 at all.]

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 11 Dec 2009 10:52:11 -0500

>  another imported module already using that prefix.

Nit: s/already //  (since the timeline is not important here)
Or use "another module imported using that prefix".

>  to identify the specific revision of the external file.

MUST?  This make import without revision pretty useless.  IMHO
this is a "guidlines" document issue.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 11 Dec 2009 08:27:43 -0800

> this is a "guidlines" document issue.

Only IETF modules have to follow the guidelines.
The RFC 2119 language usage is not based on how much
bother it is to you or anybody else -- it is based
on interoperability.  There is no way to insure that
an external file will not be changed in the future.

The syntactic and semantic properties of a published YANG module
MUST remain fixed, no matter what, for all time.
Import without revision can cause the module to change
any time an external grouping or typedef is changed.

Import without revision is useless to the IETF.
Always has been.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 11 Dec 2009 19:07:42 +0100

> typedefs, and defaults.  This isn't like SMIv2 at all.]

So far, I have not seen any _published_ YANG module - all we have are
moving targets. But, yes, we need proper tools to check for such
things when documents go through the YANG review cycle (I am avoiding
the term "YANG doctors" - would be nice to have a better sounding name
since it is much easier to sell that I spent time on the Security
Directorate than it is to sell time spent as a MIB Doctor).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 11 Dec 2009 10:44:53 -0800

> Directorate than it is to sell time spent as a MIB Doctor).

True, we have no published modules yet.
Maybe the revision dates just need to be added
at the last step before RFC publication.

Your yang-types draft is a good example.
There have been lots of pattern bug-fixes,
and a client would definitely want to know
if the server was using a version with a pattern
that has bugs or not.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 11 Dec 2009 14:48:11 -0500

>if the server was using a version with a pattern

This is a great example, since the moving module is a poor
target to lock in to.  Better to go revision-less until the
yang types is published.  Then you add your revision and
all is well.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 11 Dec 2009 12:49:17 -0800

> yang types is published.  Then you add your revision and

You are right.
The guidelines apply to published modules, which
means RFC, not Internet Draft.  After ietf-yang-types.yang
is published, then an RFC that uses it
(e.g, ietf-netconf-monitoring.yang)
needs to pick a specific revision to import.

IMO, I-Ds should use them as well, if the imported
module is already published, and not a moving target.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 13 Dec 2009 20:33:06 +0100 (CET)

>   to identify the specific revision of the external file.

I don't think it is correct to require import by revision.  If my
module imports ietf-inet-types and uses ipv6-address, why is it
important to lock down the revision?  If a bug fix or clarification is
made in the future, I want my module to work with it.  If we require
import by revision for all ietf modules, we will end up in a situation
where one bug fix ir clarification in one of the base module may
require reublishing all other ietf modules.

If module B imports A (without revision), and there exist multiple
revisions of A, it is of course essential that a client knows which
version is implemented in the server, but that is already covered with
hello advertising (and schema discovery).

I don't even think we should say SHOULD import by revision.  It all
depends on what you are using from a module.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sun, 13 Dec 2009 11:41:47 -0800

> depends on what you are using from a module.

I disagree.
It would be a significant change in IETF OPS-NM standards
to publish an RFC that did not have static contents.

The contents of data models MUST NOT drift over time.
That may seem like a feature to you, but IMO
it makes interoperability more difficult because
the management objects that result from random imports
may not be the same on any 2 implementations.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 13 Dec 2009 20:59:00 +0100 (CET)

> The contents of data models MUST NOT drift over time.

You seem to assume that the contents of the _imported_ module drift
over time.

Are there any examples of problems because of this in (IETF MIBs for)
SNMP?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sun, 13 Dec 2009 12:07:29 -0800

> You seem to assume that the contents of the _imported_ module drift

You seem to assume they won't.

You seem to assume that even if they did, it would be
a good thing for standards-track YANG modules.

> Are there any examples of problems because of this in (IETF MIBs for)

SMIv2 doesn't have groupings, and changes allowed
to TEXTUAL-CONVENTION are extremely limited.
SMIv2 has explicit conformance statements, so it is
possible to define the module revision requirements in a
precise and static manner.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 13 Dec 2009 22:14:31 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> >>> important to lock down the revision?

I would like to know your opinion on this.  I think this is a common
use case.

> You seem to assume they won't.

On the contrary; if an ietf module is not allowed to drift over time
it is safe to use a typedef like ipv6-address without import by
revision.

> a good thing for standards-track YANG modules.

No.

> SMIv2 doesn't have groupings

This question is about typedefs.  If a grouping is imported, I agree
that import-by revision is probably correct in most cases.

> and changes allowed

How is it different from what is allowed with a typedef in YANG?

> precise and static manner.

Can I require that a specific revision of a MIB module with a
TEXTUAL-CONVENTION that I IMPORT be implemented if my module is
implemented?

Are you saying that no, there have not been such problems in SNMP, but
only because of the strict rules?

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 13 Dec 2009 19:44:40 -0800

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Sunday, December 13, 2009 1:14 PM
> Subject: Re: [netmod] yang-usage-02; sec 4.6, import by revision
...
> TEXTUAL-CONVENTION that I IMPORT be implemented if my module is

No, but the rules governing the possible changes to a MIB module make
this unnecessary.
 
> only because of the strict rules?

That's generally the case.

- - -

A comment on the notion of having lots of common definitions in a few
common "library" modules - that was the idea behind ISO 10165-2.

Not recommended.

GDMO's change rules were more flexible than SNMP's, but not as
loose as Yang's.  This was enough to make versioning necessary,
and made "library churn", especially of the supporting ASN.1 modules,
a real problem.  In retrospect, we would have been better off putting
each "typedef" in its own module (which all might still "live" in the
same standard), so that a tweak or bug fix wouldn't
have forced cascading recompiles of everything under the sun.

It's ultimately easier to manage an enormous number of incredibly
simple modules than a smaller number of unwieldy ones.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 14 Dec 2009 09:27:18 +0100 (CET)

> That's generally the case.

Ok, so which rules in YANG make this problematic (rules for typedefs)?

/martin

---

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Mon, 14 Dec 2009 09:51:39 +0100

Yang Experts for YANG expert review?

Bert

---

From: Andy Bierman <[email-masked]>
Date: Mon, 14 Dec 2009 03:55:21 -0800

> Ok, so which rules in YANG make this problematic (rules for typedefs)?

Any change to a machine-readable clause could cause the typedef
definition to change.  Adding a default, changing a pattern,
changing a range, etc.  Even changing the semantics in
the description clause is tightly controlled in SNMP.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 14 Dec 2009 13:09:18 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> definition to change.
> Adding a default

This is not allowed in YANG.

> changing a pattern,
> changing a range, etc.

In SMI, it is legal to expand the value space if the object is an
enumerated integer or bits.  In YANG it is legal to expand the value
space for any type.  Is this difference the reason we need to require
import by revision for typedefs in YANG but not in SMI?

> Even changing the semantics in

Same in YANG.  In fact, the YANG text is copied from RFC 2578 (2579
has different words).

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 14 Dec 2009 08:23:01 -0800

> This is not allowed in YANG.

making YANG default even more of a joke

> Same in YANG.  In fact, the YANG text is copied from RFC 2578 (2579

I wll make the text say groupings MUST have import-by-revision
and typedefs SHOULD have them.  If you know for sure that
the typedef will never be changed, then fine.  Otherwise,
IETF YANG module module writers need to use a revision
date in the published module.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 14 Dec 2009 15:50:50 -0800

Hi -

> Cc: <[email-masked]>; <[email-masked]>
> Sent: Monday, December 14, 2009 4:09 AM
...
> enumerated integer or bits.

Just permitting these limited revisions *has* caused some problems -
they adversely interract with the WRITE-SYNTAX facility from RFC 2580.
If the original version of a module using bits does not use WRITE-SYNTAX
to spell out exactly which values must be supported by a conformant
implementation, a subsequent expansion of the set of BITS will
require that original module to be updated.  Otherwise the semantics
of the original module would effectively have been changed, and
that is verboten.

>  In YANG it is legal to expand the value

It (and the fact that a higher percentage of the YANG-defined information
will be writeable) makes problem more likely to occur.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 15 Dec 2009 23:46:31 +0100 (CET)

> of the original module would effectively have been changed, and

And same for enumerations?  So is it legal or not to add a bit to a
definition in a module which doesn't specifiy a WRITE-SYNTAX?  What
kind of problems has this resulted in?

Hmm - how does this apply to changes in TEXTUAL-CONVENTIONs?

> will be writeable) makes problem more likely to occur.

The idea so far has been that it is ok to expand the value space,
which means that old clients trying to write can still do that, but
they cannot necessarily understand the value read from the server.
Because revisions are advertised, clients can know when this might
happen, and can react accordingly.

If we don't support this, but require both sides to have 100% same
info, we either:

  1.  require import by revision
  2.  do not allow any changes to syntax or semantics (only editorial
      changes) for typedefs and groupings.

A consequence of 2 is that we will se typdefs with embedded
versioning, like:

   typedef ipv6-address { ... }

   typedef ipv6-address-2 { ... }  // bugfixed pattern

   typedef ipv6-address-3 { ... }  // bugfixed pattern
   
... as we have discussed before.

Both these scenarios give ripple effects on published modules - if the
ietf-foo module uses inet:ipv6-address, we probably need to publish a
new revision of ietf-foo as soon as the new ietf-inet-types module is
published.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 15 Dec 2009 14:58:00 -0800

> new revision of ietf-foo as soon as the new ietf-inet-types module is

I think we want to improve on SMIv2 where we can,
especially in this case because there are no formal
conformance mechanisms in YANG that provide the
same 'static contract' as SMIv2.

Contracts are hard enough to understand
when the words are not allowed to change.
It is important (for compliance) that the
contract implied by a YANG module capability URI
is understood the same way by all tools.

IMO, the 2119 term 'SHOULD' is most appropriate
for typedefs because there are corner-cases that
could be exempt from the rule without harm.
The same is true for groupings as well.
If the YANG author(s) can prove to the YANG Experts
and IESG that their module is special, then
import-by-revision doesn't have to be used.

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Dec 2009 00:08:44 +0100 (CET)

> import-by-revision doesn't have to be used.

I am still not convinced that 'SHOULD' is correct, although I think I
agree with the rest of what you're saying - essentially you need to
know why you import with or without revision.

Let's look at a concrete example, the one and only ietf-ipfix-psamp
module.  It imports the following types:

  yang:counter32
  yang:counter64
  yang:date-and-time

  inet:domain-name
  inet:ip-address
  inet:port-number
  inet:uri

Do you think that using any of these types means that they should use
import by revision?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 15 Dec 2009 15:26:19 -0800

> Do you think that using any of these types means that they should use

Yes -- anything with a pattern, or anything
that may ever change in the future.

   typedef RGB {
     type enumeration {
        enum red;
        enum green;
        enum blue.
    }
  }

When the typedef is really simple and cannot
conceptually change, then it is safe to omit the
revision.

An off-line validation tool needs to know
what imports are really used for any given
module.   That alone makes it a SHOULD, IMO.

Most importantly, we want to only allow
intended, and controlled changes to IETF standards.
The entire reason we added import-by-revision was so
it could be used to prevent unintended 'drive-by changes'
(as Phil put it).  Since harm to interoperability is
the normal outcome, not the corner-case, all
standards-track modules SHOULD follow this rule.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Dec 2009 00:45:17 +0100

> Do you think that using any of these types means that they should use

Such examples do not really help since we end up talking about
predictions of the future - if I would be good at that, I would not be
here.

But even in this set of types, I can make a reasonably realistic case
that perhaps the IETF decides that not carrying plain UTF8 in DNS was
a mistake. Fixing this might then impact inet:domain-name. Sure, the
question then for the YANG people would be whether the IETF likes to
change inet:domain-name or create a new inet:utf8-domain-name - again
we can only guess.

Constants that were considered "will never change" have changed in 
the past and as long as the Internet is not replaced by something
completely different, this will continue to be so.

/js

PS: The yang:date-and-time type carries a four digit year and has a
    built-in year 10k problem - at that time there is a clear need for
    a revision ;-)

PS: We have been liberal in the interpretation of SMIv2 rules in order
    to fix bugs and it will at the end remain a judgement call whether
    a bug fix requires a new revision or not.

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Dec 2009 07:44:39 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> predictions of the future

We are discussing guidelines for YANG modules.  In these guidelines,
the proposal is to add a rule that says that modules SHOULD use import
by revision.  In trying to understand how this SHOULD should be
applied, I think looking at a specific use case makes lots of sense.

If we, who created the rule, cannot tell whether it applies or not for
a given module, how can we expect others to do that?

> completely different, this will continue to be so.

Exactly.  Since we can adapt to the future, do we still need those
very strict rules now?

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 15 Dec 2009 23:07:02 -0800

Hi -

> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, December 15, 2009 2:46 PM

It means that a WRITE-SYNTAX has to be added to the conformance
material in the updated module so that the conformance material's
semantics will be identical to those in the original module.  Failure
to do so would change the semantics, and that would be asking for
interoperability trouble.

> Hmm - how does this apply to changes in TEXTUAL-CONVENTIONs?

Same issue.  Making such a change to a TEXTUAL-CONVENTION
can result in the need to add WRITE-SYNTAX to modules that failed
to provide them in their original version.  My recollection is that this
has actually happened in IETF modules.

> > > import by revision for typedefs in YANG but not in SMI?

It's a question of being able to know *exactly* what the statement
"conforms to foo" (where foo is some unit of conformance) means.

> happen, and can react accordingly.

Is it ok for a conformance test to try to write a value from the
expanded space?  What it the unit under test doesn't accept
such a value.  Is it non-conformant?  To what?  I think the answer
is that conformance is evaluated in terms of a specific version
of a module, and that this has to be traced through the entire
heirarchy of definitions used.  Questions like these drove
the SMI rules.  Netconf doesn't need to have the same rules, 
but it does need to be absolutely clear about whatever its rules
are.

> If we don't support this, but require both sides to have 100% same

This is asking about interoperability, rather than evaluating conformance.
Though the two are related, they're not the same.  It's important to
understand both.

> ietf-foo module uses inet:ipv6-address, we probably need to publish a

s/need/want/

> new revision of ietf-foo as soon as the new ietf-inet-types module is

That's why I would lean towards 1.  Though it means more work
(one would have to consciously propagate the "ripples") there are
no surprises, and it better reflects what can happen in multi-vendor
architectures, especially if there is any kind of subagent-like
delegation.  (I don't know whether netconf is going to ever deal
with systems where these issues arise.)

But that's also why I'd lean towards a keeping "library"
definitions one (or some number very close to 1) to
a module, to minimize "colateral damage".

Randy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Dec 2009 09:22:41 +0100

> a given module, how can we expect others to do that?

My interpretation of SHOULD is that if you do not have a strong
argument for not following the SHOULD, you should follow it. Applied
to this case, if we do not have a strong argument otherwise, we import
by revision.

> Exactly.  Since we can adapt to the future, do we still need those

Import by revision is an insurrance against future trouble. Whether an
insurrance is needed always going to be a trade-off decision. A SHOULD
essentially says that if you have no particular strong reason to not
have an insurrance, pick one.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Dec 2009 09:42:46 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> argument for not following the SHOULD, you should follow it.

Yes.

> Applied

I am wondering if 'SHOULD' is correct by trying to apply the rule to
this case.

Personally, I don't think that this module should import
ietf-yang-types or ietf-inet-types by revision.  Suppose we find a bug
in the ipv6 address pattern and fix it.  Now, until we fix and
republish ietf-ipfix-psamp, we force implementations to use the buggy
pattern.

Maybe the guidelines document for ietf modules should be more strict
in how typedefs and groupings are allowed to be changed instead (I'm
not really sure that it is necessary b/c I think we won't make random
changes to these typdefs anyway...)

> have an insurrance, pick one.

I agree there is a trade-off, and I agree that you need to think about
the problem, but I am not convinced that it is correct to give a
guideline that says that import by revision is preferred.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Dec 2009 10:16:24 +0100

> republish ietf-ipfix-psamp, we force implementations to use the buggy

The counter argument of course is that without import by revision, you
simply do not know which version a device implements and which bugs to
reasonably expect to be present. Not sure this makes interoperability
simpler. Anyway, we get here into IETF process details and discussions
we have probably every 2-3 years on how to improve things - so far
without much changes resulting form such discussions. The question
here is whether a republication of ietf-ipfix-psamp is needed or just
of the delta. Are we ready to have another instantiation of this
process discussion?

> changes to these typdefs anyway...)

We do not make random changes, but we do make random errors. The good
news is that the SMIv2 core TCs have stabilized relatively quickly and
thus the problem might in practice not pop up (but if it does, then we
need to be creative). My only concern here is probably that the amount
of technical review the yang data types document has enjoyed may not
have been sufficient.

> guideline that says that import by revision is preferred.

Its an insurrance and whether you need one or not or it bites you is
hard to judge. The only way out could be text suggesting import by
revision and requiring in case import by revision is not needed that
text is added in an appropriate place why it is safe to not use import
by revision. Since at the end, the judgement call depends on what is
imported and how it is being used. Now, requesting such text is easily
written down; implementing the necessary review of this is where the
price for it is.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Dec 2009 10:33:48 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> reasonably expect to be present.

Yes you do - the device advertises all modules it implements.

> Not sure this makes interoperability
> We do not make random changes, but we do make random errors.

And that's why it should be ok to import without revision.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Dec 2009 10:58:33 +0100

> Yes you do - the device advertises all modules it implements.

Assuming the world is simple and everything is monolithic and using
the same revisions of modules they import.

Anyway, I am not fighting strongly in favour of a SHOULD. The point is
more that module authors and reviewers need to understand the
implications and like in the insurrance model, we likely will have
continued discussions on this - unless your point is that import by
revision is not needed at all.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Dec 2009 11:06:34 +0100 (CET)

No.  My point is that I don't think we should use 'SHOULD', and that:

> [...] module authors and reviewers need to understand the
> implications and like in the insurrance model

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Dec 2009 03:02:25 -0800

> No.  My point is that I don't think we should use 'SHOULD', and that:

I think Juergen and Randy have explained why we want
the conformance requirements for a particular revision
of a particular module to be precise, and why SHOULD
is the best default choice.

IMO, SHOULD is very important here, especially since
off-line validation is so important to the WG.  I want to
be able to extract a YANG module from any RFC and
have an extremely high degree of confidence that
every YANG tool in the world will parse the exact same
YANG statements from this extracted module.

Whether changes to the published module are bug fixes,
enhancements, or a rewrite makes no difference.
For conformance purposes, the 'RFC XXXX' version
of the 'foo' module needs to mean exactly the same thing to every
YANG implementation, even off-line validation tools.
Andy