Subject: [netmod] YANG 1.0 and 1.1 integration issues

From: Andy Bierman <[email-masked]>
Date: Mon, 24 Nov 2014 08:10:01 -0800

Hi,

Sorry if this is already covered, but I am wondering if and how
YANG 1.0 modules can co-exist with YANG 1.1 modules.

1) The YANG 1.0 clients are expected to continue using YANG 1.0
modules and ignore all YANG 1.1 modules. Is a YANG 1.0 client
supposed to have a complete API such that all interactions
between 1.0 data models and 1.1 data models can be ignored?

2) It is clearly an error if a 1.0 module imports a 1.1 module, or a 1.0
submodule includes a 1.1 submodule, but what about 1.1 using 1.0?
Does the server enforce 1.0 rules for the imported modules and
included submodules (within the 1.1 context), or does the compiler
promote them and use 1.1 rules for the 1.0 modules/submodules?

3) What happens if a 1.1 module augments definitions in a 1.0 module?
Is this allowed?  This will look like unknown anyxml to 1.0 clients,
since they are unaware of the unadvertised 1.1 modules (and cannot
read YANG 1.1 anyway). Is this the integration plan for augments?

4) In order to convert 1 YANG 1.0 module to 1.1, all the modules
that import it have to be updated as well, then all the modules that
import those modules, etc.  Even if this is done, conversion will
break any 1.0 client that used that module.  How is the IETF
(or anybody with YANG 1.0 modules) supposed to build on the
existing module library and switch to YANG 1.1?  Seems like there
will need to be a flag day where all tools, clients, servers, and modules
are converted at once.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 25 Nov 2014 10:45:01 +0100

> promote them and use 1.1 rules for the 1.0 modules/submodules?

For using imported typedefs and groupings this should be OK, or do we
change semantics somewhere so that 1.0 stuff can be interpreted
differently in the 1.1 context?

> read YANG 1.1 anyway). Is this the integration plan for augments?

I think this kind of augmenting makes the augmented module necessarily 1.1.

> import those modules, etc.  Even if this is done, conversion will

With the existing rules, a module that has been upgraded to 1.1 will
probably have to use a different name, so such imports won't be possible
anyway.

> will need to be a flag day where all tools, clients, servers, and modules

We had this discussion before - IMO it is an illusion to expect that a
client will be able to cherry-pick from the advertised modules, ignore
bumped revisions etc. With all the dependencies that YANG allows
(imports, augments, must/when expressions) a data model must be taken as
a logical whole. That's why I also think it is a mistake to have the
backward compatibility rules in the definition of the language. We are
building a new complex technology and gaining experience along the way,
so we can hardly hope to get everything right at the first attempt.

I think this won't be such a big problem in practice though. In a
concrete network, an upgrade of (some) servers will often require an
upgrade of clients, but the latter is generally much easier than the
former.

Lada

>
>
> Andy
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Tue, 25 Nov 2014 13:29:38 -0800

> differently in the 1.1 context?

This is why it is important that a 1.0 module compile within a 1.1 context
and not only be valid but maintain the same syntax and semantics.

> I think this kind of augmenting makes the augmented module necessarily 1.1.

But the implementation of those modules is 1.0.
See comment above about maintaining same syntax and semantics.

> probably have to use a different name, so such imports won't be possible

So you envision that there will be foo-1.0 and foo-1.1 versions advertised
by the server, with disjoint objects in separate namespaces?
So they would diverge (or else why bother making a 1.1 version?)
and the server will end up with 2 implementations of each module
to maintain on each platform?

That sounds expensive, but it could work.

> upgrade of clients, but the latter is generally much easier than the

Standards often have stricter change rules to follow than non-standard
specifications.  Also, mixing DML versions is different than updating
modules so they do not break old clients.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 25 Nov 2014 22:36:46 +0100 (CET)

> > between 1.0 data models and 1.1 data models can be ignored?

Yes.

> > promote them and use 1.1 rules for the 1.0 modules/submodules?

The 1.0 modules have 1.0 rules and 1.1 modules have 1.1 rules.

> > Is this allowed?

Yes.

> > This will look like unknown anyxml to 1.0 clients,

Yes - this is not different from an augment by an 1.0 module that is
unknown to the client.

> I think this kind of augmenting makes the augmented module necessarily 1.1.

Disagree!

> probably have to use a different name, so such imports won't be possible

Disagree!

> > will need to be a flag day where all tools, clients, servers, and modules

No, but it does make sense to upgrade all such modules in an
implementation in a single upgrade.  But unless we have other reasons
for doing so, it does not seem necessary to update ay ietf-yang-types
just in order to make it 1.1.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 26 Nov 2014 08:40:46 +0100

Martin Bjorklund <[email-masked]> writes:
...

>> I think this kind of augmenting makes the augmented module necessarily 1.1.

Let's say a 1.1 client X configures nodes that are defined in the
augmenting module and follow 1.1 rules. Client Y that is only 1.0
receives this new unknown stuff, or the new stuff appear in its
candidate. What is client Y supposed to do?

>> probably have to use a different name, so such imports won't be possible

In the routing module I use some identity comparisons that we know are
broken. So in the 1.1 version of the module I'd like to use the new
XPath function. Do you think we can continue using the old module name?

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 26 Nov 2014 08:46:26 +0100

> to maintain on each platform?

No, what I envision is that the server will only advertise foo-1.1
version after upgrading, so the old client won't work anyway.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 26 Nov 2014 08:53:13 +0100 (CET)

> candidate. What is client Y supposed to do?

The same as if it receives nodes from an otherwise unknown namespace
(what this is depends on the client app; maybe ignore, maybe store but
do not interpret).

> XPath function. Do you think we can continue using the old module name?

Yes - it is in a new revision of the module.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 26 Nov 2014 09:22:29 +0100

> (what this is depends on the client app; maybe ignore, maybe store but

I don’t see how client Y could possibly edit and then commit such an candidate without knowing the rules. That would be extremely brittle. You are right - it is not much different from the current situation but that’s actually my point about cherry-picking advertised modules.

Another example: a module that a client chooses to ignore may define a new identity that the server is allowed to use as an identityref value of an “old” leaf. This again could/should break the client.  

> Yes - it is in a new revision of the module.

How could a 1.0 client possibly work with XPath expressions it doesn’t understand? What if the new revision contains lists with optional keys, or anydata statements?

Such changes are much more disruptive than those that are forbidden in sec. 10.

I actually agree with you, it *should* be a new revision of the same module because otherwise the set of modules would soon become a terrible mess. However, old clients then simply can’t expect to be able to use a new revision exactly as an old one.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 26 Nov 2014 09:35:07 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> I don’t see how client Y could possibly edit and then commit such an
> candidate without knowing the rules. That would be extremely
> brittle. You are right - it is not much different from the current
> situation but that’s actually my point about cherry-picking advertised
> modules.

This all depend on what the client does.  At the very extreme, the
client might just invoke <get-config> in order to store a backup.
Should this break just b/c one module is YANG 1.1?

As another example, maybe the client is a simple script that sets the
hostname/dns-server/ntp-server on a server from ietf-system.  Should
this break just b/c some other module is YANG 1.1?

Sure, there might be clients that need to fully understand everything,
and that should be fine.

> Another example: a module that a client chooses to ignore may define a
> new identity that the server is allowed to use as an identityref value
> of an “old” leaf. This again could/should break the client.
> 
> > 
> How could a 1.0 client possibly work with XPath expressions it doesn’t
> understand? What if the new revision contains lists with optional
> keys, or anydata statements?
> 
> I actually agree with you, it *should* be a new revision of the same
> module because otherwise the set of modules would soon become a
> terrible mess. However, old clients then simply can’t expect to be
> able to use a new revision exactly as an old one.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 26 Nov 2014 10:03:07 +0100

> this break just b/c some other module is YANG 1.1?

It needn’t, but the client has to know what it is doing. If an augmented leaf is a leafref that refers to that hostname/dns-server/ntp-server, then the configuration will of course be rejected if the simple script changes them. 

Section 10 and other old-client-compatibility rules in RFC 6020 give a false impression that a client can *always* safely ignore stuff it doesn’t understand, new revisions of modules etc., which is dangerous.

On the other hand, the compatibility rules severely limit the options for iterative development of modules.

So IMO it would be better to move the compatibility rules (in the SHOULD form) to 6087bis. Much of the conformance discussion would then also become moot.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 26 Nov 2014 06:49:51 -0800

Hi,

I am not sure you understand how difficult it is to change data
modeling languages
and still maintain the existing code.  It only happened once
for SNMP -- very early on the totally awful SMIv1 was replaced by SMIv2.
SMIv1 was so bad, it was quickly replaced by SMIv2.  All attempts to create
SMIv3 failed spectacularly.

YANG 1.0 is far from awful.  The incremental features added to 1.1
are nice-to-have.  Almost all of them have work-arounds and the
ones that don't are relatively rare (if-feature expressions, XPath
functions, etc)

If YANG 1.1 cannot integrate with YANG 1.0, it will never get deployed.
Existing 1.0 modules need to continue to work.  A massive rewrite of
all existing NETCONF/YANG code seems rather unrealistic.  If the
plan is to require a flag day upgrade, then YANG 1.1 will never fly.
Might as well call it YANG 2.0 if that is the plan.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 26 Nov 2014 07:28:57 -0800

...
> version after upgrading, so the old client won't work anyway.

This is too difficult for commercial deployments, especially by the
time the YANG 1.1 RFC is published, and then updates to standard
modules are published.

>From the charter:

   - All compliant YANG 1.0 modules must be accepted as compliant YANG
1.1 modules.

This bullet shows how little the WG thought about YANG 1.0 co-existence.
At least we understand that no YANG 1.1 modules will be visible or usable
by any YANG 1.0 tools.  But does this bullet assume (like you) that
all 1.0 modules will go away because they were all converted to 1.1?

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Nov 2014 16:47:20 +0100

> all 1.0 modules will go away because they were all converted to 1.1?

The bullet says that all YANG 1.0 modules remain valid with YANG 1.1
and thus there is no need to say upgrade ietf-inet-types.yang. As
usual, there may be corner cases where YANG 1.1 clarifies things that
were unclear in YANG 1.0 and hence for these cases the WG needs to
make judgement calls. But the general direction is clear.

I consider it likely that clients that take full advantage of YANG
(data model driven clients) will pick up YANG 1.1 quickly. If this
assumption turns out to be correct, then I do not really see any
problems. If servers pick up YANG 1.1 faster than data model driven
clients, well then these servers will simply push clients to pick up
YANG 1.1 fast as well. And since clients are generally easier to
upgrade, I am not worried about this either.

But yes, we need to be careful with adoting solutions for issues that
may cause difficulties on the upgrade path. But we need to consider
this in the context of each issue separately (and I think we did in
the past several times). If there are specific solutions where people
think this makes an upgrade difficult, please post the concerns with a
suitable subject line identifying the issue.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 27 Nov 2014 10:11:48 +0100

> ones that don't are relatively rare (if-feature expressions, XPath

In my opinion, we need to be more adaptive at this stage. YANG is not
that widely deployed yet, and if we withhold changes now, it may turn
out to be impossible to do them later. I wasn't there so I cannot really
tell, but this factor could also have contributed to SMIv3 failure.

We can pretend that everything we've done so far is perfect but then in
the long run it might happen that all this YANG business won't be able
to live up to expectations and the effort will be wasted. Then a new
IETF WG may be formed in the next decade to address the same problem
from scratch again.

> Might as well call it YANG 2.0 if that is the plan.

Let's say experience will show that a new mandatory leaf has to be added
to if:interface. Things like this may and will happen. It is no massive
change but our current rules prevent us from adding it to a new
revision of ietf-interfaces. Why?

A YANG 1.1 related example is this: The way how we deal with
identityrefs in XPath expressions is arguably broken because it relies
on a particular assignment of XML namespace prefixes. If we do the right
thing and change such XPath expressions using the new YANG 1.1 XPath
functions, then my reading of sec. 10 in RFC 6020 is that we again can't
make it into a new revision of the old module. This is bad.

Backward compatibility is a good thing but it cannot be an absolute
requirement.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Nov 2014 07:50:43 -0800

> IETF WG may be formed in the next decade to address the same problem

Nobody said perfect.  I said "far from awful".  Not quite the same.

I think 10 years is about right for starting YANG 2.0.
A new WG can start up then and add more features.

> revision of ietf-interfaces. Why?

Because deployments of the existing RFC will break if
new mandatory nodes are added.  Adding new functionality
without breaking old functionality is often complicated.
Look at NETCONF 1.0/1.1 implementations for an example.
Switching framing schemes correctly is complicated, but we do that
so old NETCONF clients continue to work.

> Backward compatibility is a good thing but it cannot be an absolute

Tell that to the vendor who does not want to spend millions
rewriting code that already works.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Nov 2014 07:54:58 -0800

> suitable subject line identifying the issue.

I posted some specific issues. See first email on this thread.

It seems a YANG 1.0 integration plan is not needed because YANG 1.0
will just go away instantly.  I guess vendors will come up with their
own plans if that doesn't work.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 27 Nov 2014 17:28:16 +0100

> without breaking old functionality is often complicated.

So in my example the upgraded server advertises “ietf-interface-bis” module instead of “ietf-interface”. Is this any better for the client?

> so old NETCONF clients continue to work.

Keeping old stuff for compatibility is not without cost (talk to Microsoft guys).

> rewriting code that already works.

I remember hearing similar complaints about Linux because its kernel APIs also change much more often than in commercial systems. Yes, some third party vendors were pissed off along the way, but the result is much better than it could possibly be if compatibility was an absolute priority. I think for complex systems with many contributors this is a better evolution strategy.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 27 Nov 2014 09:00:25 -0800

> So in my example the upgraded server advertises “ietf-interface-bis” module instead of “ietf-interface”. Is this any better for the client?

I do not agree that adding mandatory nodes to an existing data model
is needed.  Why would new mandatory nodes be directly added to the
interface list as child nodes?  The mandatory nodes are going to be
associated with some new protocol, etc.  It is reasonable to
require a P-container or list to contain the mandatory nodes.

> I remember hearing similar complaints about Linux because its kernel APIs also change much more often than in commercial systems. Yes, some third party vendors were pissed off along the way, but th…

Breaking backward compatibility is an engineering trade-off and vendors will
decide between stability and new features (like they always do).
I expect IETF standards track RFCs to be as stable as possible.  I
expect research
and experimentation to be unstable.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 27 Nov 2014 21:37:33 +0100

> own plans if that doesn't work.

OK. I will go through the issues (as technical contributor). But then
you also need to tell me what your goals of starting this thread
are...

Yes.

> promote them and use 1.1 rules for the 1.0 modules/submodules?

It should be fine for a 1.1 module to import a 1.0 module. I do not
see how an import would change the interpretation of a 1.0 module. If
you specific examples where this is not the case, bring them up.

> read YANG 1.1 anyway). Is this the integration plan for augments?

Yes, this is allowed. The 1.0 client will not understand the
augmenting data model. We have the same with anyxml or in NETCONF in
general if different schema languages are used.

> will need to be a flag day where all tools, clients, servers, and
> modules are converted at once.

I do not think we need a flag day. Whether a module is updated depends
on the nature of the modules (there is no immediate benefit of
updating lets say ietf-inet-types.yang to YANG 1.1), it depends how
fast YANG 1.1 clients are deployed, and of course on product specific
decisions. I assume for most of the existing YANG 1.0 instrumentation
code, the upgrade from a YANG 1.0 definition to a YANG 1.1 definition
is a noop.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 28 Nov 2014 09:03:27 -0800

> you also need to tell me what your goals of starting this thread

I want to understand how YANG 1.1 modules can be deployed
in a way that does not disturb any YANG 1.0 implementations.
I do not agree that the migration will be instant, or even fast.

I think you, Martin, and I are no the same page wrt/ updating YANG 1.0
modules only when needed.  They should continue to work even
if YANG 1.1 modules are added to the system.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 28 Nov 2014 18:17:27 +0100

> I think you, Martin, and I are no the same page wrt/ updating YANG 1.0

This is no wonder because you both make a living by developing and supporting management software. One can expect certain bias. :-)

Lada