Subject: [netmod] how to know which module revision an RPC is for?

From: Kent Watsen <[email-masked]>
Date: Tue, 28 Dec 2010 15:59:16 -0800

Sorry if this has been answered elsewhere, but a search didn't reveal it...

Section 4.2.9 of draft-ietf-netmod-yang-13 provides an example whereby an "rpc" statement results in the XML

     <rpc message-id="101"
          xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <activate-software-image xmlns="http://acme.example.com/system">
         <image-name>acmefw-2.3</image-name>
      </activate-software-image>
     </rpc>

     <rpc-reply message-id="101"
                xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <status xmlns="http://acme.example.com/system">
         The image acmefw-2.3 is being installed.
       </status>
     </rpc-reply>

Note that only the namespace is passed (http://acme.example.com/system)

And section 5.6.4.1. says that "A server MUST advertise all revisions of all modules it implements".

   Servers indicate the names of supported modules via the <hello>
   message.  Module namespaces are encoded as the base URI in the
   capability string, and the module name is encoded as the "module"
   parameter to the base URI.

   A server MUST advertise all revisions of all modules it implements.

   For example, this <hello> message advertises one module "syslog".

     <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <capability>
         http://example.com/syslog?module=syslog&revision=2008-04-01
       </capability>
     </hello>

So what if a server implements more than one revision of a module, each defining the same rpc from, but with only a slight variation in output.  Since there is no ability for the client to indicate which module revision their RPC is using, how would the server know which response to send?

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Wed, 29 Dec 2010 09:55:53 -0800

On 12/28/2010 03:59 PM, Kent Watsen wrote:
> Sorry if this has been answered elsewhere, but a search didn’t reveal it…
> Section 4.2.9 of draft-ietf-netmod-yang-13 provides an example whereby an “rpc” statement results in the XML
> <rpc message-id="101"

You can add a leaf to the input for the client to request
a specific version.

Normally, a server implements just the most recent version.

Andy

> And section 5.6.4.1. says that “A server MUST advertise all revisions of all modules it implements”.
> http://example.com/syslog?module=syslog&revision=2008-04-01 <http://example.com/syslog?module=syslog&revision=2008-04-01>
> </capability>
> </hello>
> So what if a server implements more than one revision of a module, each defining the same rpc from, but with only a slight variation in output. Since there is no ability for the client to indicate
> which module revision their RPC is using, how would the server know which response to send?
> Thanks,
> Kent
>
>
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Kent Watsen <[email-masked]>
Date: Wed, 29 Dec 2010 12:04:37 -0800

Hi Andy,

> You can add a leaf to the input for the client to request

I don't understand, can you provide an example?   Oh, wait, do you mean that the payload of the RPC itself would have a <revision> element?  - ugh, do you want to define that in the RFC?   It seems that augmenting the xmlns might be more elegant, perhaps something like:

   <rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <activate-software-image xmlns="http://acme.example.com/system/<revision>">
         <image-name>acmefw-2.3</image-name>
      </activate-software-image>
     </rpc>

Or maybe:

   <rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <activate-software-image xmlns="http://acme.example.com/system&revision=<revision>">
         <image-name>acmefw-2.3</image-name>
      </activate-software-image>
     </rpc>

Or maybe even:

   <rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <activate-software-image xmlns="http://acme.example.com/system" schemaLocation="system-v<revision>.xsd">
         <image-name>acmefw-2.3</image-name>
      </activate-software-image>
     </rpc>

> Normally, a server implements just the most recent version.

But what if the revision breaks compatibility with existing clients?   The current YANG spec does not constrain revisions to be backwards compatible or provide rules enabling clients to skip over unrecognized fields.  An application coded to only support the previous revision will not know how to interoperate with a device that only advertises the new/current revision.

It would be good if we could allow devices to support multiple revisions simultaneously (using something like one of the proposals above) as well as some kind of deprecation/expiration policy to allow clients enough time to gracefully migrate to the new revision before the device drops support for the legacy revisions entirely.

What do you think?

Thanks,
Kent

---

From: Phil Shafer <[email-masked]>
Date: Wed, 29 Dec 2010 14:53:24 -0500

Kent Watsen writes:
>But what if the revision breaks compatibility with existing clients?   The current YANG 
>spec does not constrain revisions to be backwards compatible or provide rules enabling c
>lients to skip over unrecognized fields.  An application coded to only support the previ
>ous revision will not know how to interoperate with a device that only advertises the ne
>w/current revision.

A YANG module is a contract, so breaking between revisions is a bug
in the new revision.  If the module adds new elements, then the old
clients will ignore them.  If the new module deprecates elements,
then both sides can ignore them.  If the new module, say, removes
elements that were previously mandatory, then you have a bug in the
new revision.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 29 Dec 2010 12:39:46 -0800

On 12/29/2010 12:04 PM, Kent Watsen wrote:
> I don't understand, can you provide an example?   Oh, wait, do you mean that the payload of the RPC itself would have a<revision>  element?  - ugh, do you want to define that in the RFC?   It seems that augmenting the xmlns might be more elegant, perhaps something like:
>

This is not something for the standard to worry about.
Usually a server image implements a particular revision of a module,
and advertises just that revision.

There are corner cases where module A and B both import C, but different
versions of C, (e.g. acme-types.yang), so there are 2 versions of C
advertised by the server.

You have a different use-case.
Applications are coded to check for different data structures
in the <rpc-reply> output, based on the module version.  You want to
let the client pick the output version, so the YANG versions align
properly.

You do not need to do this if newer module versions just add to the output.
Optional nodes that go away will look like an unsupported feature.

You could use 'anyxml' in the output, instead of a detailed YANG model.
In this case knowing the exact module version is needed.
This is supported by adding an input parameter, like in your examples.

NETCONF/YANG doesn't really support what you want without changing
the XML namespace each module revision.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Wed, 29 Dec 2010 16:01:20 -0800

> elements that were previously mandatory, then you have a bug in the

The "Updating a Module" section (http://tools.ietf.org/html/rfc6020#section-10) supports most of this.  However, I find the statement:

  "However, changes are not allowed if they have any
   potential to cause interoperability problems between
   a client using an original specification and a server
   using an updated specification."

to be in conflict with the ability to obsolete nodes using the status statement (http://tools.ietf.org/html/rfc6020#section-7.19.2), as a server could cause interoperability problems if it ever stops supporting RPC elements or stops returning RPC-Reply elements.  In fact, it seems that the current definition requires:

   For RPCs:
     - the server MUST support a deprecated nodes forever
     - the server can never obsolete a node

   For RPC Replies:
     - the server MUST continue to return deprecated nodes forever
     - the server can never obsolete a node

Is this true?

Perhaps I'm missing something, as I don't truly understand the purpose of the "obsolete" status as it seems that 1) existing implementations can never obsolete a node (see above) and 2) new implementations wouldn't even bother to support deprecated nodes, so again, no need for an obsolete status.

As I abhor the idea of needing to forever support nodes, it seems that a fix would be to enable a deprecated node to specify a date after which it can be considered obsolete, not to be less than one year from the module's revision date, so as to allow clients enough time to gracefully migrate off of the deprecated node prior to its support being dropped entirely.  What do you think?

Anyway, back to my original question, the conclusion appears to be:

  - it is OK for the server to not know what module-revision a client used to format its RPC message as the server must forever support previous versions of the RPC.

  - it is OK for the client to not be coded to support the current module-revision a server supports as the server must continue to return all previously returned fields and also because the client can safely ignore any unrecognized fields.

  - the ability for the server to return the list of revisions it supports is valuable in that it allows new implementations can indicate support for just the current version and hence never have to support previously deprecated nodes.

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Wed, 29 Dec 2010 16:50:33 -0800

On 12/29/2010 04:01 PM, Kent Watsen wrote:
>> A YANG module is a contract, so breaking between revisions is a bug in the new revision.  If the module adds new elements, then the old clients will ignore them.  If the new module deprecates
>> elements, then both sides can ignore them.  If the new module, say, removes elements that were previously mandatory, then you have a bug in the new revision.
>
>
> "However, changes are not allowed if they have any potential to cause interoperability problems between a client using an original specification and a server using an updated specification."
>
> to be in conflict with the ability to obsolete nodes using the status statement (http://tools.ietf.org/html/rfc6020#section-7.19.2), as a server could cause interoperability problems if it ever
> stops supporting RPC elements or stops returning RPC-Reply elements.  In fact, it seems that the current definition requires:
>
> For RPCs: - the server MUST support a deprecated nodes forever - the server can never obsolete a node
>
> For RPC Replies: - the server MUST continue to return deprecated nodes forever - the server can never obsolete a node
>
> Is this true?
>
>
> Perhaps I'm missing something, as I don't truly understand the purpose of the "obsolete" status as it seems that 1) existing implementations can never obsolete a node (see above) and 2) new
> implementations wouldn't even bother to support deprecated nodes, so again, no need for an obsolete status.
>
> As I abhor the idea of needing to forever support nodes, it seems that a fix would be to enable a deprecated node to specify a date after which it can be considered obsolete, not to be less than
> one year from the module's revision date, so as to allow clients enough time to gracefully migrate off of the deprecated node prior to its support being dropped entirely.  What do you think?
>
>
> - it is OK for the client to not be coded to support the current module-revision a server supports as the server must continue to return all previously returned fields and also because the client
> can safely ignore any unrecognized fields.
>

I don't understand why the server would want to advertise
old versions.  It advertises the version most current at
the time of implementation.  The client should be OK if
the data model was updated properly.

The intent of status obsolete is that the object is removed
from the server.  A mandatory node cannot be removed.  If that
or a key leaf is wrong, the data model has to start over with a
new name.

A client needs to access data nodes by name instead of assuming
the child node list will be exactly a certain way.  Then an old
client will ignore new fields and assume missing optional fields
means the option is not supported.

I think you may have a real concern.
If you have an application that knows about rev 2010-01-01
of the FOO module, then the app might break if the server
stops advertising 2010-01-01 and starts advertising a new version
instead.

Can this client just code to its old notion of the FOO module
and ignore the version difference in the advertisement?
As you pointed out, not if an object is now obsolete.

I think current NETCONF/YANG forces you to keep renaming the RPC,
or add a parameter to request a specific version of the RPC.
There is no other way for the server to know the client wants
an old version.  I do not like adding 'special XML' that signals
the version somehow within the message encoding.

> Thanks, Kent

Andy

---

From: Kent Watsen <[email-masked]>
Date: Thu, 30 Dec 2010 11:05:35 -0800

> the data model was updated properly.

My best guess is that it is so that clients can know that the server implements the since-deprecated nodes.  For instance, assume revision 2011-01-01 defines RPC <foobar>, which is deprecated in 2012-02-02.  A server advertizing both revisions is expected to implement <foobar>, while one only advertising 2012-02-02 is not.

> or a key leaf is wrong, the data model has to start over with a

The rules must be different for RPCs and RPC Replies.  For RPCs, it would be an error if even a non-mandatory node disappeared, as a legacy client could still send it, while for RPC Replies, it would be OK for a non-mandatory node to disappear, since the client shouldn't have been coded to expect it.

Regarding mandatory nodes not being removable, the RFC says otherwise or, more specifically, section 10 says: a "mandatory" statement may be removed or changed from "true" to "false".  I think this statement makes sense for inputs (like RPCs and Config), but not so much for outputs (like RPC Replies and Notifications).  Nonetheless, there it is, which seems to be a mistake.

Starting over with a new name may be required, given the current state of the RFC, but the server still needs to return the mandatory nodes, even if they've been deprecated (see previous email), so this means the server would be returning both the old and the new names (ugh).   We can limit the duration of this suboptimal solution using the explicit expiration policy described previously, but it's still ugly compared to the client passing a namespaces (or a special parameter) in the RPC to indicate which module-revision its using.

> means the option is not supported.

I agree that this is implied by the current versioning strategy defined in the RFC

> As you pointed out, not if an object is now obsolete.

Exactly.  This is the problem.

> the version somehow within the message encoding.

Renaming the RPC seems like a non-starter.  Passing a parameter (or using a namespace) is viable, but I believe that the RFC needs to specify the solution in order to insure interoperability.

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 30 Dec 2010 21:37:57 +0100

On Thu, Dec 30, 2010 at 11:05:35AM -0800, Kent Watsen wrote:

> Renaming the RPC seems like a non-starter.  Passing a parameter (or
> using a namespace) is viable, but I believe that the RFC needs to
> specify the solution in order to insure interoperability.

So what is the big difference between these options?

a) renaming m:foo to m:foo2
b) renaming m:foo to n:foo
c) using m:foo(v=0) vs. m:foo(v=1)

In the SMI world, the guiding principle was that you can make changes
as long as the change is interoperable between old agents/managers and
new agents/managers. This is carried forward in RFC 6020:

   As experience is gained with a module, it may be desirable to revise
   that module.  However, changes are not allowed if they have any
   potential to cause interoperability problems between a client using
   an original specification and a server using an updated
   specification.

If a change is needed where interoperability can't be ensured, you
have to do a) or, if the change is really big, you may consider doing
b).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Kent Watsen <[email-masked]>
Date: Thu, 30 Dec 2010 16:09:49 -0800

>new agents/managers. 

Ay, I misunderstood Andy to mean a new name for a child node to an existing RPC.  Yes, of course, a new RPC could break compatibility with whatever RPC it may have forked off.  Good point.

But how can this be applied to configuration?  - there can only be one configuration (AFAIK), so how can breaks in compatibility ever be supported?  

Other concerns not yet addressed:

1. unclear how "obsolete" status can ever be used (as to do so would break backwards compatibility)
2. missing clarification that removing a "mandatory" statement is only allow for inputs (RPCs and config)
3. missing clarifying statement that clients must skip over unrecognized elements in a response
4. no expiration policy (or are implementations to be forever backwards compatible?)

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Thu, 30 Dec 2010 16:35:17 -0800

Re issue #1:

We better figure out how to use status=obsolete.
Changing the XML namespace or local-name is always possible.
Hopefully we can avoid a cut-and-paste design and come up with
something better.

IMO, the NMS has to be robust, and fail gracefully if a server
advertises a version of a module it does not understand.
It can either use its old version, knowing an object could have
been made obsolete, or fail altogether. (Or use <get-schema>
and stay in synch by compiling modules from the server on-the-fly!)

I think you mentioned a 1 year guideline for how long
an object has to be 'deprecated' before it can transition to 'obsolete'.
That seems like a good idea.

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kent Watsen
Sent: Thursday, December 30, 2010 4:10 PM
To: Juergen Schoenwaelder
Cc: [email-masked]
Subject: Re: [netmod] how to know which module revision an RPC is for?

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 31 Dec 2010 08:28:41 +0100

On Thu, Dec 30, 2010 at 04:09:49PM -0800, Kent Watsen wrote:
 
> But how can this be applied to configuration?  - there can only be
> one configuration (AFAIK), so how can breaks in compatibility ever
> be supported?

It depends on what "break in compatibility" means. If you really do
something radically different, you create a new module which has its
own new name space and this will lead to a break of interoperability.

> 1. unclear how "obsolete" status can ever be used (as to do so would break backwards compatibility)

RFC 6020 says:

   o  "obsolete" means the definition is obsolete and SHOULD NOT be
      implemented and/or can be removed from implementations.

My reading of that is that obsolete definitions simply record the fact
that they were used some time ago - a place holder to track the
history and most importantly to make sure the obsolete name is not
being reused. See also this quote:

   Obsolete definitions MUST NOT be removed from modules since their
   identifiers may still be referenced by other modules.

> 2. missing clarification that removing a "mandatory" statement is only allow for inputs (RPCs and config)

Which text in section 10 of RFC 6020 makes you believe that removing
"mandatory" is allowed?

> 3. missing clarifying statement that clients must skip over unrecognized elements in a response

This might be stated somewhere but I am not sure how to search
efficiently. With vendor augmentations, I guess a client not prepared
to handle unknown elements won't get very far in terms of cross vendor
interoperability.

> 4. no expiration policy (or are implementations to be forever backwards compatible?)

An expiration policy in standards implemented across different vendors
and deployed in rather diverse environments is likely not a very
realistic option. But yes, expiration may work for a single vendor
dealing only with specific product lines.

/js