Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

From: Jan Lindblad <[email-masked]>
Date: Fri, 3 Dec 2021 10:59:12 +0100

Kent, Qiufang, all,

>>> Offline validation of <running> alone IS required
>>> Options:
>>> Clients MUST copy/paste any referenced system configuration into <running>, even though it goes against our objective of avoiding-copy when possible.
>>> Defer work to be a YANG-next effort.
>> 
>> In order to move forward, I would propose working out some more options in this list. I have suggested a few to the authors that I think are better than the two above, but I will leave it to the authors make the call for what they want to bring up for discussion.
> 
> I don’t think the case has been made that *offline* validation of <running> by itself must be valid.  Yes, *online* <running> must be valid, as must <intended>, with the intention of RFC 8342 being that both are validated in one and the same operation by the server  (known to me as an author of RFC 8342).  I understand that implementations may have assumed offline validation of <running> must be valid, I just can’t find the RFC reference that demands it be supported, and without such reference, it falls into an undefined area.  Necessitating offline validation of <running>, if that is what the WG decides, results in solutions for which none seem nice to me, at least I haven’t seen any yet.  Please share any “better options” you have in mind.  

Here you are introducing two concepts that the RFCs (6020, 7950, 8342) are never mentioning: online and offline validation. Then you say that because the RFCs don't talk about these concepts, the behavior is undefined. I strongly disagree. The RFCs talk about validation, and describes the rules for how that happens. These rules always apply, regardless of online, offline or the phase of the moon.

If you think the online and offline validation concepts have merit, I would like to see proper definitions of them, and how you would like to modify the existing RFCs with respect to these concepts. It might also be a good idea to update the proposed draft, which currently does not mention these concepts.

The added value and the core essence of YANG is enabling us to reason about configurations and compute configuration changes without necessarily asking the server if a configuration is valid by "trial and error". This is possible because a YANG model is a detailed and reasonably complete description of the management interface. Introducing changes to YANG that breaks this basic premise would be dumbing down YANG, and I can't see the benefit with this change, or how this would be compatible with YANG 1.0, or 1.1 rules.

I think the ideas behind the system datastore are interesting and in many use cases useful. We just need to introduce them in a way that doesn't break existing, agnostic implementations.

I made some proposals earlier, both on the interim and privately to the draft authors, along these lines:

Option #1
+ We could have a new system datastore that technically is a part of running. Everything in system would show up in running to  clients unaware of the system datastore. Every time the system datastore changes for whatever reason, the running datastore transaction ids (if we manage to get that concept defined), are updated
+ Clients interested to see pure view of the system datastore without anything else in running could issue a get-data towards the system datastore
+ Clients interested to see a pure view of running without any system datastore elements could issue a get-data towards running with an extra flag called 'without-system'
+ Since all system elements are already part of running, clients have no need to copy data between the two datastores

Option #2
+ We could have a system datastore that technically is a part of operational. Everything in system would show up in operational to  clients unaware of the system datastore. The running datastore always maintains referential integrity, i.e. cannot reference the system datastore.
+ Clients interested to see pure view of the system datastore could issue a get-data towards the system datastore
+ Since the system datastore is not part of running, clients can already see a pure view of running without any system datastore elements using a simple get-data. 
+ Clients that are aware of the system datastore and are interested to configure references from running to system elements would issue an edit-data rpc with the additional flag 'resolve-system-references'. This will make the server copy any system elements referenced into running without the client doing so explicitly.
+ Clients unaware of the system datastore would have to include (copy) information from the system datastore to running in order to reference them.

> In the meanwhile, can we discuss the issues around a legacy client failing an offline validation?  In this case, a production deployment must have 1) deployed devices that support <system>, 2) deployed at least one client that supports <system>, and 3) decided to let clients start pushing config using <system>.   While in the pre-production testing phase, it would be quickly discovered that all legacy clients need to be updated.  By the time of going to production, the deployment should have all the clients updated, so it seems that only the forgotten (relatively insignificant) clients might have an offline validation of <running> failure.  Is this a fair assessment?

1) agreed, without devices that support the system datastore, no problem
2,3) well, a "client" in this case could very well be a CLI operator or other management interface. Even in highly automated networks, CLI operators are still common

Your description of the operations environment where the operator has full implementation control of all clients is very different from my field experience. Not only do I believe the situation with multiple, independent clients is the most commonly occurring one in the real world across all server deployments, but I also think the operator usually has limited insight into as well as control over which specific protocol features the clients implement. I therefore think your assessment is fair only for a minority of field situations.

Best Regards,
/jan

---

From: Jürgen Schönwälder <[email-masked]>
Date: Fri, 3 Dec 2021 11:26:10 +0100

> + Clients unaware of the system datastore would have to include (copy) information from the system datastore to running in order to reference them.

Option #3

There is a client on the system that makes changes to running just
like any other remote clients can make changes to running. System
generate config that is not editable explicit config in running goes
straight into the applied config in operational. This does not require
a system datastore (in fact something like a system datastore may
exist as the _logic_ of the system client, the good old example we had
is allocting an unused uid for an account that, once allocated, is
maintained in running).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 3 Dec 2021 03:01:07 -0800

On Fri, Dec 3, 2021 at 2:26 AM Jürgen Schönwälder <
[email-masked]> wrote:

> > + We could have a new system datastore that technically is a part of
> running. Everything in system would show up in running to  clients unaware
> of the system datastore. Every time the system datastore changes for
> whatever reason, the running datastore transaction ids (if we manage to get
> that concept defined), are updated
> > + Clients interested to see pure view of the system datastore without
> anything else in running could issue a get-data towards the system datastore
> > + Clients interested to see a pure view of running without any system
> datastore elements could issue a get-data towards running with an extra
> flag called 'without-system'
> > + Since all system elements are already part of running, clients have no
> need to copy data between the two datastores
> >
> > Option #2
> > + We could have a system datastore that technically is a part of
> operational. Everything in system would show up in operational to  clients
> unaware of the system datastore. The running datastore always maintains
> referential integrity, i.e. cannot reference the system datastore.
> > + Clients interested to see pure view of the system datastore could
> issue a get-data towards the system datastore
> > + Since the system datastore is not part of running, clients can already
> see a pure view of running without any system datastore elements using a
> simple get-data.
> > + Clients that are aware of the system datastore and are interested to
> configure references from running to system elements would issue an
> edit-data rpc with the additional flag 'resolve-system-references'. This
> will make the server copy any system elements referenced into running
> without the client doing so explicitly.
> > + Clients unaware of the system datastore would have to include (copy)
> information from the system datastore to running in order to reference them.
> >
>
> Option #3
>

+1 to option 3.
Consider an implementation that moves all the "hidden system logic" off-box
to a controller, such that the initial config is literally supplied by an
external client.

I have yet to hear a single use-case for creating a <system> datastore.
"The client might want" is not a use-case for standardization.
I do not understand the "pure view". Seems like if this was a real problem
to solve,
then NMDA would have included a system datastore from the start.

/js

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Wed, 8 Dec 2021 22:31:30 +0000

Hi guys,

Andy - about use cases.  Here is a problem we're trying to address:

There are at least several major router implementations that have this concept of "hidden config" (i.e. list entries that can be referenced in a leafref by explicit user config, but those list entries are not returned in a <get-config>).

The server accepts these configurations (i.e. a validate or commit succeeds), but if you actually validate (e.g. with yangLint) the running datastore contents (e.g. fetched using <get-config>) to the YANG model it is not valid. That is against several RFCs referenced in the discussions below.

IMO there isn't any "offline" vs" online" validation that is different. The contents of the running must be valid against the YANG model.  A <get-config> returns the contents of the running.  The server can validate it, or some other entity can validate it, but it must be valid.

In Jan's option #1 the running config could be polluted with 100s or 1000s of lines of unreferenced/unused config. A <get-config> of a system without much config would instead return 100s/1000s of lines. I think that would be very annoying from a usability perspective.

I'm in favor of this aspect of Jan's option #2:  " + Clients unaware of the system datastore would have to include (copy) information from the system datastore to running in order to reference them."

-> Only the list keys of referenced system objects would have to be copied (configured) into the running DS (to resolve leafrefs).  We wouldn't need to copy all the descendant elements inside the referenced object.
-> This copying would only need to be done by operators with a workflow that requires offline validation

Some of this approach is already described in draft-ma-netmod-with-system-01:

"   In all cases, the clients should have control over the configurations
   ,i.e., read-back of <running> should contain only what was explicitly
   set by clients."

"4.3.  Explicit declaration of system configuration

   In addition to modifying system configuration, and adding nodes to
   lists in system configuration as described above, a client can also
   explicitly declare system configuration nodes in <running> with the
   same values as in <system>.  When a client configures a node (list
   entry, leaf, etc) in <running> that matches the same node & value in
   <system>, then that node becomes part of <running>.  A read of
   <running> returns those explicitly configured nodes.

   This explicit configuration of system configuration in <running> can
   be useful, for example, when an operator's workflow requires a client
   or offline tool to see the <running> configuration as valid.  The
   client can explicitly declare (i.e.  configure in <running>) the list
   entries (with at least the keys) for any system configuration list
   entries that are referenced elsewhere in <running>.  The client does
   not necessarily need to declare all the contents of the list entry
   (i.e. the descendant nodes) - only the parts that are required to
   make the <running> appear valid offline.  "

I'm not a fan of option #3. It doesn't allow a mode of operating where a client/OSS has full control over the contents of the <running>.  Some operators will want the master config to be owned up in their client/OSS and push it down. The server should just accept it and the client should be able to do a "read-back" and see exactly what was sent previously.

I think we have a potential solution for this system config that keeps the running valid. But I'm far more worried about configuration templates. I don't see how we can possibly keep <running> valid with config templates. That seems like a major problem to me. But if we ever declare that <running> doesn't have to be valid, and only <intended> has to be valid, then offline tools can never validate (ahead of time) the config they actually download to the server.

Jason

From: netmod <[email-masked]> On Behalf Of Andy Bierman
Sent: Friday, December 3, 2021 6:01 AM
To: Juergen Schoenwaelder <[email-masked]>; Jan Lindblad <[email-masked]>; Kent Watsen <[email-masked]>; maqiufang (A) <maqiufang1=[email-masked]>; [email-masked]
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

On Fri, Dec 3, 2021 at 2:26 AM Jürgen Schönwälder <[email-masked]<mailto:[email-masked]>> wrote:

Option #3

+1 to option 3.
Consider an implementation that moves all the "hidden system logic" off-box
to a controller, such that the initial config is literally supplied by an external client.

I have yet to hear a single use-case for creating a <system> datastore.
"The client might want" is not a use-case for standardization.
I do not understand the "pure view". Seems like if this was a real problem to solve,
then NMDA would have included a system datastore from the start.

/js

Andy

_______________________________________________
netmod mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netmod

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Wed, 8 Dec 2021 22:34:31 +0000

Hi Kent,

I'm not following your "In the meanwhile" thoughts.

Legacy clients are failing offline validation today. If running config has a leafref to system config, and <get-config> doesn't return that system config (which it doesn't in some implementations), then the instance data returned to the client doesn't validate against the YANG model.  These implementations don't have an explicit <system> datastore today (but they do have these internal semi-hidden referenceable list entries).

Jason

From: netmod <[email-masked]> On Behalf Of Kent Watsen
Sent: Monday, November 29, 2021 5:09 PM
To: Jan Lindblad <[email-masked]>
Cc: maqiufang (A) <maqiufang1=[email-masked]>; [email-masked]
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

Hi Jan,

On Nov 23, 2021, at 12:56 PM, Jan Lindblad <[email-masked]<mailto:[email-masked]>> wrote:

Sergio, Qiufang,

Hi Jan,
You correctly wrote:

Then the choices become:

     *   Offline validation of <running> alone is NOT required

        *   Servers internally validate <running> via validating <intended>
        *
SB> but in fact this is what declared, for my understanding, in RFC 8342, for which “validation” is done on “intended” by the server , as also shown in figure 2 of the RFC. Is it needed to change also RFC?

According to RFC 8342, *both* running and intended have to be valid at all times. Section 5.1.3 says:

5.1.3<https://datatracker.ietf.org/doc/html/rfc8342#section-5.1.3>.  The Running Configuration Datastore (<running>)

...

                                                         However,

   <running> MUST always be a valid configuration data tree, as defined

   in Section 8.1 of [RFC7950]<https://datatracker.ietf.org/doc/html/rfc7950#section-8.1>.

Section 8.1 of RFC 7950 was the section I referred to in my previous comment. Section 5.1.4 says:

5.1.4<https://datatracker.ietf.org/doc/html/rfc8342#section-5.1.4>.  The Intended Configuration Datastore (<intended>)

...

   <intended> is tightly coupled to <running>.  Whenever data is written

   to <running>, the server MUST also immediately update and validate

   <intended>.

In my judgement, changing the fundaments of RFC 7950 and 8342 is not going to happen any time soon (for good reason), and there are other (better) options.

     *   Offline validation of <running> alone IS required

        *   Options:

           *   Clients MUST copy/paste any referenced system configuration into <running>, even though it goes against our objective of avoiding-copy when possible.
           *   Defer work to be a YANG-next effort.

Kent // as a contributor

---

From: Andy Bierman <[email-masked]>
Date: Wed, 8 Dec 2021 14:50:07 -0800

On Wed, Dec 8, 2021 at 2:31 PM Sterne, Jason (Nokia - CA/Ottawa) <
[email-masked]> wrote:

> There are at least several major router implementations that have this
> concept of "hidden config" (i.e. list entries that can be referenced in a
> leafref by explicit user config, but those list entries are not returned in
> a <get-config>).
>
>
>

Clearly not in compliance with RFC 7950.

IMO the "enable flag" approach to the general problem, presented by Kent a
couple years ago,
is a much simpler and better solution than a new <system> datastore.
The full set of nodes is in <running>.
A generalized "enable" mechanism causes the resource to be used or not,
where it shows up in <intended> and <operational> if enabled=true.

IMO this fits the original intent of NMDA and does so in a way that requires
the least disruption to current compliant implementations.

Andy

> The server accepts these configurations (i.e. a validate or commit
> succeeds), but if you actually validate (e.g. with yangLint) the running
> datastore contents (e.g. fetched using <get-config>) to the YANG model it
> is not valid. That is against several RFCs referenced in the discussions
> below.
>
>
>
> IMO there isn't any "offline" vs" online" validation that is different.
> The contents of the running must be valid against the YANG model.  A
> <get-config> returns the contents of the running.  The server can validate
> it, or some other entity can validate it, but it must be valid.
>
>
>
> In Jan's option #1 the running config could be polluted with 100s or 1000s
> of lines of unreferenced/unused config. A <get-config> of a system without
> much config would instead return 100s/1000s of lines. I think that would be
> very annoying from a usability perspective.
>
>
>
> I'm in favor of this aspect of Jan's option #2:  " + Clients unaware of
> the system datastore would have to include (copy) information from the
> system datastore to running in order to reference them."
>
>
>
> -> Only the list keys of referenced system objects would have to be copied
> (configured) into the running DS (to resolve leafrefs).  We wouldn't need
> to copy all the descendant elements inside the referenced object.
>
> I'm not a fan of option #3. It doesn't allow a mode of operating where a
> client/OSS has full control over the contents of the <running>.  Some
> operators will want the master config to be owned up in their client/OSS
> and push it down. The server should just accept it and the client should be
> able to do a "read-back" and see exactly what was sent previously.
>
>
>
> I think we have a potential solution for this system config that keeps the
> running valid. But I'm far more worried about configuration templates. I
> don't see how we can possibly keep <running> valid with config templates.
> That seems like a major problem to me. But if we ever declare that
> <running> doesn't have to be valid, and only <intended> has to be valid,
> then offline tools can never validate (ahead of time) the config they
> actually download to the server.
>
>
>
> Jason
>
>
>
>
>
> *From:* netmod <[email-masked]> *On Behalf Of *Andy Bierman
> *Sent:* Friday, December 3, 2021 6:01 AM
> *To:* Juergen Schoenwaelder <[email-masked]>; Jan
> Lindblad <[email-masked]>; Kent Watsen <[email-masked]>; maqiufang (A)
> <maqiufang1=[email-masked]>; [email-masked]
> *Subject:* Re: [netmod] Must offline-validation of <running> alone be
> valid?
>
>
>
>
>
>
>

---

From: "maqiufang (A)" <[email-masked]>
Date: Thu, 9 Dec 2021 13:07:09 +0000

Hi, all

I agree that option #1 will overwhelm a client severely.

Avoid-copying and client-control over <running> written in the introduction of draft-ma-netmod-with-system-00 are actually two competing objectives, if the WG thinks offline-validation of <running> alone IS required,
I think we need to make a compromise. Option #2 may provide a good starting point.

Regarding open #3, it is natural for the clients to believe that what they read back from the server is exactly what they sent to the server.
If there is a "system client" playing a role, this would require some extra handling for the other remote clients and bring them surprise and inconvenience.

Best Regards,
Qiufang Ma

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, December 9, 2021 6:50 AM
To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>
Cc: Juergen Schoenwaelder <[email-masked]>; Jan Lindblad <[email-masked]>; Kent Watsen <[email-masked]>; maqiufang (A) <[email-masked]>; [email-masked]
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

On Wed, Dec 8, 2021 at 2:31 PM Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]<mailto:[email-masked]>> wrote:
Hi guys,

IMO the "enable flag" approach to the general problem, presented by Kent a couple years ago,

Andy

Jason

From: netmod <[email-masked]<mailto:[email-masked]>> On Behalf Of Andy Bierman
Sent: Friday, December 3, 2021 6:01 AM
To: Juergen Schoenwaelder <[email-masked]<mailto:[email-masked]>>; Jan Lindblad <[email-masked]<mailto:[email-masked]>>; Kent Watsen <[email-masked]<mailto:[email-masked]>>; maqiufang (A) <maqiufang1=[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

Option #3

+1 to option 3.

/js

Andy

---

From: Jürgen Schönwälder <[email-masked]>
Date: Thu, 9 Dec 2021 14:46:19 +0100

> If there is a "system client" playing a role, this would require some extra handling for the other remote clients and bring them surprise and inconvenience.

A server accepting and returning non-valid config is also a surprise
and inconvenience.

/js

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Thu, 9 Dec 2021 15:15:24 +0000

> A server accepting and returning non-valid config is also a surprise

Andy made a similar point in his reply but it is currently implemented today and there are some desirable aspects of that functionality. Many major server implementations (at least in the router space) have this concept of hidden config that can be referenced, and support configuration templates.  Both of those mean the server is accepting and returning "non-valid config".

(quotes around "non-valid config" because we're debating what it means to have valid config in this discussion -> i.e. is it ok that running is invalid, as long as intended is valid ?)

Jason

> -----Original Message-----
> From: Jürgen Schönwälder <[email-masked]>
> Sent: Thursday, December 9, 2021 8:46 AM
> To: maqiufang (A) <[email-masked]>
> Cc: Andy Bierman <[email-masked]>; Sterne, Jason (Nokia -
> CA/Ottawa) <[email-masked]>; Jan Lindblad <[email-masked]>;
> Kent Watsen <[email-masked]>; [email-masked]
> > Regarding open #3, it is natural for the clients to believe that what they
> read back from the server is exactly what they sent to the server.
> > If there is a "system client" playing a role, this would require some extra
> handling for the other remote clients and bring them surprise and
> inconvenience.
> >
>

---

From: Jürgen Schönwälder <[email-masked]>
Date: Thu, 9 Dec 2021 16:17:23 +0100

> Andy made a similar point in his reply but it is currently implemented today and there are some desirable aspects of that functionality. Many major server implementations (at least in the router sp…

What defines "many major server implementations"?

/js

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Thu, 9 Dec 2021 15:19:00 +0000

Hi Andy,

I'm not sure I understand Kent's alternative that you're referencing here. Are you talking about something like JUNOS active/inactive configuration annotation ?

Is the "enable" some configurable metadata against data nodes ?

When you say the "full set of nodes in running" are you talking about Jan's option #1 below (but perhaps without the explicit <system> datastore) ?  i.e. all the system config is actually returning from a <get-config> of running ?

Jason

From: Andy Bierman <[email-masked]>
Sent: Wednesday, December 8, 2021 5:50 PM
To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>
Cc: Juergen Schoenwaelder <[email-masked]>; Jan Lindblad <[email-masked]>; Kent Watsen <[email-masked]>; maqiufang (A) <maqiufang1=[email-masked]>; [email-masked]
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

Andy

Jason

Option #3

+1 to option 3.

/js

Andy

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Thu, 9 Dec 2021 16:38:28 +0000

To help illustrate "Many major server implementations (at least in the router space)":

JUNOS and Nokia SR OS have it.

I'm not sure about Cisco - maybe someone who knows IOS-XR well could point out an example of this (or confirm they do/don't have it, or do/don't see a use case given their model-driven architecture).

I assume Huawei must have it given the 3 primary authors are associated with Huawei in the draft.  Maybe they can confirm.

Jason

> Sent: Thursday, December 9, 2021 10:17 AM
> Cc: maqiufang (A) <[email-masked]>; Andy Bierman
> <[email-masked]>; Jan Lindblad <[email-masked]>; Kent Watsen
> <[email-masked]>; [email-masked]
> > Andy made a similar point in his reply but it is currently implemented today
> and there are some desirable aspects of that functionality. Many major
> server implementations (at least in the router space) have this concept of
> hidden config that can be referenced, and support configuration templates.
> Both of those mean the server is accepting and returning "non-valid config".
> >
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 9 Dec 2021 08:47:02 -0800

On Thu, Dec 9, 2021 at 7:19 AM Sterne, Jason (Nokia - CA/Ottawa) <
[email-masked]> wrote:

> Hi Andy,
>
>
>
> I'm not sure I understand Kent's alternative that you're referencing here.
> Are you talking about something like JUNOS active/inactive configuration
> annotation ?
>
>
>

I am not proposing a solution here.
I am referring to RFC 8342 use of the term "inactive", e.g:
https://datatracker.ietf.org/doc/html/rfc8342#section-5

Any solution that overrides MUST requirements in YANG 1.1 requires a new
YANG version.

IMO it is unwise to send the industry the message that YANG 1.1 and NMDA
need to be
replaced and NMDA is an unstable experiment that is not ready for
production networks.
NMDA allows for inactive nodes in <running>. It is trivial to design a data
model
in such a way that it is inactive until a client uses it somehow.

Where in RFC 8342 does it say that <running> config MUST or even SHOULD
contain
only nodes explicitly created by an external client?

> When you say the "full set of nodes in running" are you talking about
> Jan's option #1 below (but perhaps without the explicit <system> datastore)
> ?  i.e. all the system config is actually returning from a <get-config> of
> running ?
>

I am not interested in a new datastore for NMDA. It has enough already.
<running> can have active and inactive nodes in it. There is no need to
create a new datastore and rewrite YANG to use a new datastore.

Andy

>
>
> *From:* Andy Bierman <[email-masked]>
> *Sent:* Wednesday, December 8, 2021 5:50 PM
> *To:* Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>
> *Cc:* Juergen Schoenwaelder <[email-masked]>; Jan

---

From: Kent Watsen <[email-masked]>
Date: Mon, 13 Dec 2021 23:25:46 +0000

Hi Jan,

> Here you are introducing two concepts that the RFCs (6020, 7950, 8342) are never mentioning: online and offline validation. Then you say that because the RFCs don't talk about these concepts, the b…

Look, specs are very simple: stuff is written down in black and white and either it’s there or not.  In this case, the fact remains that there is no document I can point to that says offline validation is a thing.  It is also notable that RFC 8341 say nothing about the fact that clients effected by NACM may not be able to pass validation (it’s not even mentioned).

> If you think the online and offline validation concepts have merit, I would like to see proper definitions of them, and how you would like to modify the existing RFCs with respect to these concepts…

This is part of the discussion we’re having now.  The outcome of which may trigger clarifications to some RFCs.   All fine suggestions, but replace “you” with “folks”, as it’s not on my shoulders to do any of this.

> The added value and the core essence of YANG is enabling us to reason about configurations and compute configuration changes without necessarily asking the server if a configuration is valid by "tr…

I never said offline validation shouldn’t be possible.  Rather, please know that, NACM aside, my understanding of the goal of the draft is that offline validation *is* supported...but it entails clients merging their view of <running> with the server’s <system> before doing the validation (i.e., <running> alone may not be enough, see Subject line).  Good news is that, since <system> is effectively static, no new round-trips are incurred.

> + Clients unaware of the system datastore would have to include (copy) information from the system datastore to running in order to reference them.

I see these as possible fallback solutions.

> 2,3) well, a "client" in this case could very well be a CLI operator or other management interface. Even in highly automated networks, CLI operators are still common

Sure, but there’s no impact to CLI operators as they’re effectively getting server-side validation all the time.  Just saying that CLI-ops don’t seem to be effected by any of this, right?

> Your description of the operations environment where the operator has full implementation control of all clients is very different from my field experience. Not only do I believe the situation with…

Sure, there will be some shops that play it loose, but serious deployments know what is accessing their management network, whether allowed through firewall rules or admin-accounts, it shouldn’t be hard for them to round up a short-list of apps (clients) accessing their management network.

I’m sure the view you obtained via in your years differs from mine.  Please, provide more detail about the management practices of the kinds of shops you have in mind.  Are they providing NaaS (Network as a Service)?

Thanks,
Kent

---

From: Kent Watsen <[email-masked]>
Date: Mon, 13 Dec 2021 23:44:31 +0000

Juergen/Andy,

> is allocting an unused uid for an account that, once allocated, is

Juergen, you mentioned this before.  I don’t recall if there were any responses, but how would this solution address the various concerns that have been raised? 

> to a controller, such that the initial config is literally supplied by an external client.

Andy, IIRC, you +1’ed Juergen’s previous “option #3”, but can you to answer the same question about how solution address concerns?

> then NMDA would have included a system datastore from the start.

Use-cases and issues were discussed at both the Interim and at IETF 112.   I don’t recall you or Juergen attending either, but the 112-recording starts at the 23-minute mark here: https://play.conf.meetecho.com/Playout/?session=IETF112-NETMOD-20211111-1430 <https://play.conf.meetecho.com/Playout/?session=IETF112-NETMOD-20211111-1430>

K.  // as a contributor

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 00:32:30 +0000

Hi Jason,

> I think we have a potential solution for this system config that keeps the running valid. But I'm far more worried about configuration templates. I don't see how we can possibly keep <running> vali…

It isn’t the case that clients can’t offline-validate <running> with templates (or <running> with apparently-dangling refs to <system>-defined nodes), they just have to work for it a little more.  

In the case of templates, they’d have to understand how to perform the template-expansion logic and, in the case of dangling <system>-refs, they have to know how to do the merge-logic.  

In both cases, the general statement is that, if clients want to do offline validation, they need to understand how to “cook” <intended> and validate that instead.

K.

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 00:42:47 +0000

Hi Jason,

> Legacy clients are failing offline validation today. If running config has a leafref to system config, and <get-config> doesn't return that system config (which it doesn't in some implementations),…

You’re correct about that, and I said so before about how, with JUNOS, any ref to a “junos-defaults” node would fail offline-validation of running.  This is already an issue today.

Also to your point, JUNOS has a templating-like mechanism (apply-groups) that clients MUST understand.  The NMS we built at Juniper had to understand the “apply-groups” mechanism just to import config for devices during a new-device onboarding workflow.

K.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Dec 2021 16:46:42 -0800

On Mon, Dec 13, 2021 at 3:44 PM Kent Watsen <[email-masked]> wrote:

> Juergen, you mentioned this before.  I don’t recall if there were any
> responses, but how would this solution address the various concerns that
> have been raised?
>
>
> +1 to option 3.
> Andy, IIRC, you +1’ed Juergen’s previous “option #3”, but can you to
> answer the same question about how solution address concerns?
>
>
I do not have any problem with <running> containing active and inactive
nodes.
That's what has been in place for over 10 years. That's what is written in
NMDA.
I cannot find any RFC text that says <running> has only nodes created by a
client.
I cannot find any RFC text that says system-injected config is special,
especially since
server implementations exist that treat these edits as just another client
(although probably a 'root' user client).

Rewriting NMDA and YANG validation to add a new <system> datastore is
a major change that will impede deployment. The IETF already had 1 "do-over"
because of NMDA.  Not so sure a 2nd do-over is going to go over well.

Andy

> Use-cases and issues were discussed at both the Interim and at IETF 112.
> I don’t recall you or Juergen attending either, but the 112-recording
> starts at the 23-minute mark here:
> https://play.conf.meetecho.com/Playout/?session=IETF112-NETMOD-20211111-1430
>
>
> K.  // as a contributor
>
>
>
>
>
>
>
>
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Dec 2021 17:01:27 -0800

Hi,

On Mon, Dec 13, 2021 at 4:43 PM Kent Watsen <[email-masked]> wrote:

> Legacy clients are failing offline validation today. If running config has
> a leafref to system config, and <get-config> doesn't return that system
> config (which it doesn't in some implementations), then the instance data
> returned to the client doesn't validate against the YANG model.  These
> implementations don't have an explicit <system> datastore today (but they
> do have these internal semi-hidden referenceable list entries).
>
>
>

This is an implementation bug.
YANG validation for configuration data nodes is very clear.
It intentionally does not allow any leafrefs to point at data nodes outside
<running>.
Vendors who follow these YANG rules can return a representation of <running>
that can be validated.

Andy

> You’re correct about that, and I said so before about how, with JUNOS, any
> ref to a “junos-defaults” node would fail offline-validation of running.
> This is already an issue today.
>
> Also to your point, JUNOS has a templating-like mechanism (apply-groups)
> that clients MUST understand.  The NMS we built at Juniper had to
> understand the “apply-groups” mechanism just to import config for devices
> during a new-device onboarding workflow.
>
> K.
>

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 01:31:58 +0000

> On Dec 8, 2021, at 5:50 PM, Andy Bierman <[email-masked]> wrote:
> 

Andy, can you please point to the part in RFC 7950 that says offline validation must be supported?   I believe that this "common understanding” actually lacks a basis, and an equally-valid interoperation is that the <running> must be valid *on the server* vis-a-vis it actually validating <intended>.

> the least disruption to current compliant implementations.

You have the memory of an elephant  ;)

That I-D (conditional-enablement [1]) was mostly about how to support JUNOS’s “inactive” annotation.  I replaced the negative “inactive” with positive “enabled” for readability.  That draft also shined a light on how the “enabled” annotation could be used in firewall pollination for, e.g., 9am-5pm ACL rules.

I guess I’m unclear about the relation to the system-defined config - can you say some more?

[1] https://datatracker.ietf.org/doc/html/draft-kwatsen-conditional-enablement-00 <https://datatracker.ietf.org/doc/html/draft-kwatsen-conditional-enablement-00>

K.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Dec 2021 17:47:56 -0800

On Mon, Dec 13, 2021 at 5:31 PM Kent Watsen <[email-masked]> wrote:

> Andy, can you please point to the part in RFC 7950 that says offline
> validation must be supported?   I believe that this "common understanding”
> actually lacks a basis, and an equally-valid interoperation is that the
> <running> must be valid *on the server* vis-a-vis it actually validating
> <intended>.
>
>
I think sec. 6.4 and sec. 8 are clear enough how validation of the
<running> datastore is done.
Leafrefs in <running> are not allowed to point to a node outside of
<running>.

Andy

> That I-D (conditional-enablement [1]) was mostly about how to support
> JUNOS’s “inactive” annotation.  I replaced the negative “inactive” with
> positive “enabled” for readability.  That draft also shined a light on how
> the “enabled” annotation could be used in firewall pollination for, e.g.,
> 9am-5pm ACL rules.
>
> [1]
> https://datatracker.ietf.org/doc/html/draft-kwatsen-conditional-enablement-00
>
>
> K.
>
>
>
>

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 02:55:15 +0000

Hi Andy,

> I do not have any problem with <running> containing active and inactive nodes.
> That's what has been in place for over 10 years. That's what is written in NMDA.

For posterity, it’s been “in place” only in proprietary implementations.  It would be nice to resurrect the “conditional-enablement” draft to have an RFC for it.

> I cannot find any RFC text that says <running> has only nodes created by a client.

Really?  Interesting.   Still, I know it’s a mantra we’ve held closely for many year, right?

> I cannot find any RFC text that says system-injected config is special, especially since

Very true (and Juergen’s point as well).  I’ve seen this done before.  Unhappy results.

> because of NMDA.  Not so sure a 2nd do-over is going to go over well.

I don’t think a “major” change is being discussed: I think it is, effectively: validate <intended> (not <running>).

And note that Section 5.1.4 says:

   For simple implementations, <running> and <intended> are identical.

I’d guess that *all* implementations are “simple” currently.  When a server introduces a feature (inactive, template, etc.) that demands <intended>, at that time internal logic is switched to "validating <running>" vis-a-vis validating <intended>.  Clients would also have to make that same switch, to the extent they care about offline validation at all.

<putting on flame suit>

As a co-author, I know this was the intention of RFC 8342, which is supported by, for instance:

Section 4.1:

   o  Several implementations have proprietary mechanisms that allow
      clients to store inactive data in <running>.  Inactive data is
      conceptually removed before validation.

   o  Some implementations have proprietary mechanisms that allow
      clients to define configuration templates in <running>.  These
      templates are expanded automatically by the system, and the
      resulting configuration is applied internally.

Section 5:

	<intended>   // subject to validation

Section 5.1.4:

<taking off flame suit>

Of course, some will point to Section 5.1.3:

   However, <running> MUST always be a valid configuration data tree,
   as defined in Section 8.1 of [RFC7950] <https://datatracker.ietf.org/doc/html/rfc7950#section-8.1>.

But it has to be obvious that this is a bug.  For instance,

  - YANG defines a leaf is of type union { uint8 | variable }
  - <running> defines the leaf having value “MAX_FOOBAR” 
    with a template that defines MAX_FOOBAR=1000.
  - so, <running> would be syntactically valid but
    semantically invalid.

Or a more egregious example:

  - YANG defines a "max-elements" value “MAX_FOOBAR”
       - how is this possible when RFC 7950 says max-elements
          Is a positive integer or the string “unbounded”?
       - so now we’re into YANG-next territory as far as
         templates are concerned, but hang with me a little
         while longer…
  - <running> has a template that defines MAX_FOOBAR=3
  - how coulda server validate if <running>’s list contained less
    than max-elements?  Worse, what if the result of another 
    template added more elements to the list?

I may have taken off the flame suit too early  ;)

K.

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 03:03:22 +0000

Hi Andy,

> I think sec. 6.4 and sec. 8 are clear enough how validation of the <running> datastore is done.
> Leafrefs in <running> are not allowed to point to a node outside of <running>.
> 

I don’t discount the “accessible tree” angle.  The focus on if the server can’t validate <running> vis-a-vis <intended>….  

As an author of NMDA, I know that is was/is the goal to let this be true - how else could template ever work, right?

K.

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 03:13:29 +0000

Hi Andy,

> It intentionally does not allow any leafrefs to point at data nodes outside <running>.

lol  “junos-defaults” existed before YANG was a thing  ;)

In either case, 3 of the top-5(?) vendors are doing this, not out of spite, but because business-demands...

Andy, how does YumaWorks handle shared-objects, such as those Jason and I have described?  Do your customers populate them in <running>?  If so, is there an “immutable” flag to prevent delation?

K.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Dec 2021 19:18:11 -0800

On Mon, Dec 13, 2021 at 6:55 PM Kent Watsen <[email-masked]> wrote:

> For posterity, it’s been “in place” only in proprietary implementations.
> It would be nice to resurrect the “conditional-enablement” draft to have an
> RFC for it.
>
>

No. Quite the opposite. The client only gets access to the config after the
device
has created an initial configuration, especially interface configuration.

The text in RFC 7950 is clear:

     The accessible tree depends on where the statement with the XPath

   expression is defined:

   o  If the XPath expression is defined in a substatement to a data
      node that represents configuration, the accessible tree is the
      data in the datastore where the context node exists.  The root
      node has all top-level configuration data nodes in all modules as
      children.

The RFC does not need to discuss offline validation because the constraints
apply to a single datastore.

The original intent (in YANG and NMDA) is that inactive nodes and active
nodes
can coexist in <running>. I see no reason to change NMDA or YANG and abandon
this approach.

IMO distinguishing so-called system config from client config
is not a very compelling operational problem to solve.
Not so sure it is easy to define a system edit vs. a non-system edit.

Andy

> I don’t think a “major” change is being discussed: I think it is,
> effectively: validate <intended> (not <running>).
>
> I’d guess that *all* implementations are “simple” currently.  When a
> server introduces a feature (inactive, template, etc.) that demands
> <intended>, at that time internal logic is switched to "validating
> <running>" vis-a-vis validating <intended>.  Clients would also have to
> make that same switch, to the extent they care about offline validation at
> all.
>
>
>
> <putting on flame suit>
>

---

From: Jürgen Schönwälder <[email-masked]>
Date: Tue, 14 Dec 2021 09:34:45 +0100

> Juergen, you mentioned this before.  I don’t recall if there were any responses, but how would this solution address the various concerns that have been raised? 

Well, while doing the NMDA work, we already acknowledged that there
are proprietary extensions (like templates or inactive nodes) and
hence we moved validation to <intended>. Since people felt it was kind
of hopeless to standardize templates, we accepted that there is vendor
specific magic applied to the config from <running> to become
<intended>. So from this perspective, merging in some system config is
not making things any worse, we already broke the ability to validate
a copy of <running> in an interoperable way in the NMDA work.

If we accept that there is system configuration one can rely on, then
this system configuration obviously flows into <intended>. Whether it
is possible to standardize this, I do not know, we even failed with
inactive in the past.

That said, I am not convinced by the proposed with-system parameter.
If you retrieve <running>, then you should get <running> and not
something that happens to <running> later down the pipeline. If people
want offline validation, then they either retrieve <intended> and work
with that or they have to implement the vendor's magic for merging
system configuration, for dealing with inactive nodes, for expanding
templates, and whatever comes next. This is bad news for everybody who
wants to do validation of config _before_ it is shipped to <running>
since these folks have to implement vendor specific logic. This is
unfortunate but I believe we started accepting this with NMDA and we
accepted it because this is what implementations apparently do...

I still believe that good data model design can avoid many of the
situations where people believe they need to depend on system
configuration. The interface example presented at IETF 112 is a weak
example since we rely on lazy name bindings for interfaces, a
description configured for an interface "foo" is applied to an
interface "foo" if there is one, otherwise it is not applied. You do
not need to know whether the system supplies a "foo" interface to
validate the config. But then I am sure there are examples where non
lazy bindings exist, perhaps hidden deep in some MUST expressions.

/js

---

From: Martin Björklund <[email-masked]>
Date: Tue, 14 Dec 2021 13:14:17 +0100 (CET)

Hi,

> Very true (and Juergen’s point as well).  I’ve seen this done before.

Can you elaborate on this, or send a pointer if it has been documented
somewhere?

I will also add a +1 to letting the system "act as a client" and put
data into running.  This is what we do in our current project.

A few other comments.

The name "system" for this new proposed datastore is perhaps not the
best, since RFC 8342 already defines this term, and associated
semantics.

If the new proposed datastore is supposed to add static data like
"built-in profiles and policies", it is rather limited, compared to
"system config" as defined in RFC 8342.  The reason is that system
config can be injected at various levels in the config hierarchy;
specifically also under user-defined list entries.   And if this is
indeed the use case you (as in all proponents of the proposal) have in
mind, I think the solution with adding this to <running> works.  (But
I am curious to hear your experience of this).

>       resulting configuration is applied internally.

Right, and in both cases, the idea was that <running> contains all
data needed for the transformation into <intended>.  So a client that
wants to do "offline" validation would need the data + the
transformation algorithms.  But no additional data.

/martin

>    as defined in Section 8.1 of [RFC7950]
>    <https://datatracker.ietf.org/doc/html/rfc7950#section-8.1>.
>

---

From: Jan Lindblad <[email-masked]>
Date: Tue, 14 Dec 2021 14:37:52 +0100

Kent, all,

>   - so, <running> would be syntactically valid but

I must confess I raised my eyebrows a little when I saw this. Well, I have often heard server implementors pick some of their least favorite sentences out of an RFC and say that "this is obviously a bug". Still, it's quite another thing when something like that is coming from someone so deeply knowledgeable and immersed in IETF and the WG as Kent. 

Kent, may I ask that you clarify if you do mean what you said, and if you do, if that would be a statement from a contributor or chair?

In my opinion, if the statement above in section 5.1.3 (and you will find similar language in 7950) collides with the template example you gave, may I suggest the possibility that there could be something wrong with that example rather than the RFCs?

/jan

---

From: Jan Lindblad <[email-masked]>
Date: Tue, 14 Dec 2021 15:00:56 +0100

Kent,

> Look, specs are very simple: stuff is written down in black and white and either it’s there or not.  In this case, the fact remains that there is no document I can point to that says offline validation is a thing.  

Absolutely. Today the words written down in the specs are only talking about one kind of validation. 

Offline validation is not a concept I have been using. It's not used in the RFCs. If you think this concept has merit, it would be good if you took the time to define it. And if you also think the RFCs that currently do not use it, should use it, then by all means suggest changes.

> It is also notable that RFC 8341 say nothing about the fact that clients effected by NACM may not be able to pass validation (it’s not even mentioned).

That a client with insufficient privileges may have trouble understanding or controlling a server is no surprise to me. I don't quite see why you think this observation is relevant for the discussion about whether a datastore is valid or not?

> This is part of the discussion we’re having now.  The outcome of which may trigger clarifications to some RFCs.   All fine suggestions, but replace “you” with “folks”, as it’s not on my shoulders t…

Ok, fair enough.

To whom it may concern: Regarding "offline validation" or "online validation", please note that these concepts are yet undefined in any RFCs or drafts (as far as I'm aware). If you intend to use these concepts, please be so kind as to either define or point to definitions of them.

> I never said offline validation shouldn’t be possible.  Rather, please know that, NACM aside, my understanding of the goal of the draft is that offline validation *is* supported...but it entails cl…

Good to hear. My preference would be that the validation rules already established in 6020, 7950 and 8342 stay close to what they are. If we decide to accommodate further use cases, that should be allowed by clients that flag awareness of extended mechanisms. Old, unaware servers and clients should be able to continue staying true to their legacy.

> Sure, but there’s no impact to CLI operators as they’re effectively getting server-side validation all the time.  Just saying that CLI-ops don’t seem to be effected by any of this, right?

Well, I'm not worried about the CLI users getting trouble. I'm worried that CLI users would cause trouble for automation clients, by for example referencing something out of the system datastore. The next time someone does a get-config towards running, it might not be valid any longer, with the RFCs' definition of valid.

Best Regards,
/jan

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 14:53:35 +0000

Hi Jan,

> I must confess I raised my eyebrows a little when I saw this. Well, I have often heard server implementors pick some of their least favorite sentences out of an RFC and say that "this is obviously…

I apologize if I’ve done or said anything to offend you.  Please, let’s keep the discussion on the level.

Regarding raised eyebrows, did you catch that the value “1000” doesn’t fit into a uint8?  The point was/is that validation might miss that error without template expansion.  In this case, pyang/yanglint validation on (unexpanded) <running> would pass, but I don’t think that we would want to settle for that, right?  Again, it was the intention of the authors that validation is moved to <intended>; Rob, Juergen, Martin, and I have all affirmed this understanding recently.  I think that the quoted s5.1.3 line above escaped scrutiny, as well perhaps some text missing in Section 6.1 (to replace or extend “running” with “intended").  I recall that the authors were drawing a fine line bridging being compatible with RFC 7950 and this intention.  I think that an Errata on RFC 8342 may be warranted here, but I’m unsure what it might say just yet.

> Kent, may I ask that you clarify if you do mean what you said, and if you do, if that would be a statement from a contributor or chair?

All my responses on this thread to this point, and in general, unless explicitly called out with an “as co-chair” designation, are from me as a contributor.  I do try to put a “as a contributor” on my messages for brevity, but it is easy to forget sometimes. 

K.

---

From: Jan Lindblad <[email-masked]>
Date: Tue, 14 Dec 2021 16:31:14 +0100

Hi Kent,

> I apologize if I’ve done or said anything to offend you.  Please, let’s keep the discussion on the level.

I'm not offended in any way. It was certainly my intention keep things perfectly level and impersonal. If that's not how it came through, I'm truly sorry.

> Regarding raised eyebrows, did you catch that the value “1000” doesn’t fit into a uint8?

Of course. My raised eyebrow was regarding how you, being both deeply knowledgeable and chair, could dismiss clear statements in RFCs as "obvious that this is a bug." The RFCs are documents that we have agreed. Agreeing is not an easy process, so we should be proud over what we have. While RFCs may have bugs, holes and shortcomings, I think treating RFCs with some respect is in order. We don't want to undermine their authority, or make people believe they can ignore a few rules that are unpleasant to them ("obviously a bug") and still call themselves compliant.

>  The point was/is that validation might miss that error without template expansion.  In this case, pyang/yanglint validation on (unexpanded) <running> would pass, but I don’t think that we would want to settle for that, right?  Again, it was the intention of the authors that validation is moved to <intended>; Rob, Juergen, Martin, and I have all affirmed this understanding recently.  I think that the quoted s5.1.3 line above escaped scrutiny, as well perhaps some text missing in Section 6.1 (to replace or extend “running” with “intended").  I recall that the authors were drawing a fine line bridging being compatible with RFC 7950 and this intention.  I think that an Errata on RFC 8342 may be warranted here, but I’m unsure what it might say just yet.

I agree that your example is at odds with a whole collection of RFCs. But instead of thinking all the RFCs are in error, my conclusion is that the example you give is not supported yet. We can work on making templates usable within the YANG framework, but just as you already recognized above, it doesn't fit in without further work.

> All my responses on this thread to this point, and in general, unless explicitly called out with an “as co-chair” designation, are from me as a contributor.  I do try to put a “as a contributor” on…

Thanks Kent. I do appreciate all you do for the WG.

/jan

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 15:48:42 +0000

Hi Jan,

> That a client with insufficient privileges may have trouble understanding or controlling a server is no surprise to me. I don't quite see why you think this observation is relevant for the discussi…

It’s related to if offline validation can succeed or not.  Effectively, all but the "recovery session” clients will not be able to offline-validate an update to <running> prior to pushing config to the server.  That RFC 8341 doesn’t even mention this fact suggests that “validation” is a concept that only manifests in servers.  Just pointing to evidence of precedence is all.

Of course, I strongly support client-validation prior to push.  My recommendation is that clients, when interacting with an NMDA-server, learn how to “cook” <intended> and validate that instead of just <running>.

The question is what to do about legacy clients?  

First, we need to discuss what an “aware” (i.e., not “legacy”) client is.  It seems that “awareness" would have to be more than just understanding that <intended> exists; the client would also have to understand each “cooking” step (pruning inactive nodes, expanding templates, merging system nodes, etc.) the server supports.  There would need to be a client- compatibility check such that, if the client doesn’t understand all of the steps, then it must NOT proceed (aside: this reminds me of the “critical” extension that Lada and I discussed before).  Actually, we might consider reversing the handshake and instead have the client present to the server a list of all the “critical” things it understands and for the server to effectively drop the client if it’s missing anything, since clients cannot predict what new critical things may arise, there could be no false-positives.  Warning, we’re deep into NETCONF-next and RESTCONF-next territory here.

It might turn out that the server, once upgraded to support “next” stuff, can ONLY interact with next-clients.  If so, there would be never be a mix of legacy and not-legacy clients, and thus this entire hybrid-client compatibility concern disappears.

K. // contributor

---

From: Jürgen Schönwälder <[email-masked]>
Date: Tue, 14 Dec 2021 17:20:02 +0100

> transformation algorithms.  But no additional data.

Having to know proprietary transformation algorithms really kills the
idea of interoperable offline validation. It does not really get any
worse if transformation algorithms merge in additional definitions.

/js

---

From: Jan Lindblad <[email-masked]>
Date: Tue, 14 Dec 2021 18:10:36 +0100

Kent, all,

> It’s related to if offline validation can succeed or not.  Effectively, all but the "recovery session” clients will not be able to offline-validate an update to <running> prior to pushing config to the server.

I'm not sure why you say so. In my field experience, NACM is seldom, if ever, an issue for validation. Clients may have a full or partial view of the server, but even when partial, I can't recall seeing a case where that subset has been broken from a validity standpoint. Whoever is responsible for the deployment makes sure the client sees what it needs to see, and that that in itself is a consistent, valid (subset of) running.

>  That RFC 8341 doesn’t even mention this fact suggests that “validation” is a concept that only manifests in servers.  Just pointing to evidence of precedence is all.

I disagree :-) I don't think this is something we have agreed, and I don't see the point with this stance. In what way is the world a better place if we take this "suggestion" seriously?

> Of course, I strongly support client-validation prior to push.  My recommendation is that clients, when interacting with an NMDA-server, learn how to “cook” <intended> and validate that instead of…

Great. While it sounds very nice to be able to validate both running and intended properly, establishing agreement on one particular industry wide recipe for how to cook running into intended sounds hard, and at the very least some way off. Even if we would agree on the cooking, I still think running needs to remain valid(*). Running is the interface that clients can interact with, and they should have a clear definition of what the contract looks like without spending an hour in the kitchen.

*) Validity, as defined today, concerns running. It does not involve resolving templates etc. While intended should also be validated, as mentioned in 8342, there are no clear rules for how that validation should happen. Clearly it would be possible for a client to come up with a config that is valid in running, but gibberish in intended. The server is, already today, free to reject such a config. That this final part of the validation cannot happen in running is no argument for not validating running, and allowing running to violate the YANG constraints defined for it.

> First, we need to discuss what an “aware” (i.e., not “legacy”) client is.  It seems that “awareness" would have to be more than just understanding that <intended> exists; the client would also have…

Yes, if we agree to let legacy clients and servers behave as specified in the RFCs, we could come up with very creative things for "aware" systems. Some of that might be in *-next territory, but much of it could also happen today, as long as we have mutual consent from the participants.

> It might turn out that the server, once upgraded to support “next” stuff, can ONLY interact with next-clients.  If so, there would be never be a mix of legacy and not-legacy clients, and thus this…

Even if I think the industry usually goes for evolution rather than revolution, I'm fine with that. As long as we get some value return for the work, and it doesn't break what's out there already, it's worth discussing.

/jan

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 Dec 2021 19:43:47 +0000

> worse if transformation algorithms merge in additional definitions.

Of the three transform algs under discussion (pruning inactive, merging system, and expanding templates), only the last may be proprietary and, even then, nothing is stopping IETF from standardizing one or a few well known templating mechanisms.

K.

---

From: Jürgen Schönwälder <[email-masked]>
Date: Tue, 14 Dec 2021 23:29:00 +0100

> Of the three transform algs under discussion (pruning inactive, merging system, and expanding templates), only the last may be proprietary and, even then, nothing is stopping IETF from standardizin…

I doubt that existing implementations will converge to a standard
solution (which will take years to define); it seems the window of
opportunity for standards in this space is closed.

/js

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 Dec 2021 10:47:15 -0800

On Tue, Dec 14, 2021 at 2:29 PM Jürgen Schönwälder <
[email-masked]> wrote:

> > Of the three transform algs under discussion (pruning inactive, merging
> system, and expanding templates), only the last may be proprietary and,
> even then, nothing is stopping IETF from standardizing one or a few well
> known templating mechanisms.
> >
>

I am still looking for a meaningful problem statement that addresses
operational problems.
Working on architecture for the sake of elegance is not interesting to me.

The problem seems to be that the WG incorrectly assumed that  all these
unspecified proprietary mechanisms
could be represented as YANG data, conforming to specific YANG modules.
This data MUST always be valid
in both <running> and <intended>. Clearly it was a mistake to assume that
these proprietary mechanisms
could be ignored for the purpose of YANG validation.

It should be obvious that if this data cannot be properly represented in
<running>
then moving the data to <system> does not help at all.  In fact, it makes
the problem
even worse, because the protocol operations are designed to work on 1
datastore at a time.
Offline validation based on multiple retrievals will require long-lived
locks, or just be wrong
due to time skew.

I do not understand the use-case for a leafref that points at instances
that do not exist in <running>.
It seems like a simple "enabled" leaf allows the unused instance to exist
in <running> but be disabled.
YANG validation is constrained to a single datastore. It would be very
disruptive to change this now.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Fri, 17 Dec 2021 15:11:05 +0000

Andy, et. al., 

> No. Quite the opposite.  <snip>

There was a brouhaha back when I proposed the "keystore” draft have an “action” called “generate-private-key” that would insert the generated key into <running>.  Claims were made by prominent members of this list that it’s bad form for anything but a client to edit <running>.  

As a result, extensive effort was spent defining a mechanism enabling the generated key to be returned in the RPC-reply in an encrypted form (such that only the server that generated the key could decrypt it), all so the client could immediately return it to the server via a config push in order to preserve the sanctity of client read-backs.

If current claims were true then, why didn’t someone just say it’s okay since the server is acting like a client under the hood?

K.

---

From: Martin Björklund <[email-masked]>
Date: Fri, 17 Dec 2021 17:45:17 +0100 (CET)

Hi,

Kent Watsen <[email-masked]> wrote:
> There was a brouhaha back when I proposed the "keystore” draft have an
> “action” called “generate-private-key” that would insert the generated
> key into <running>.  Claims were made by prominent members of this
> list that it’s bad form for anything but a client to edit <running>.

The problem with an action that is supposed to modify the running
config is that it also has to be prepared to handle systems with
<candidate>, handle locks etc.  And if you don't have <candidate> you
may want to add the private-key together with other data in one go;
this is not possible if it was added by an action.

For the purpose of adding "built-in list instances" (which seems to be
the use case for the proposed solution), I think the factory-default
datastore can be used.  (this is actually better than the server
"acting as a client").

/martin

> 
> As a result, extensive effort was spent defining a mechanism enabling
> the generated key to be returned in the RPC-reply in an encrypted form
> (such that only the server that generated the key could decrypt it),
> all so the client could immediately return it to the server via a
> config push in order to preserve the sanctity of client read-backs.
> 
> If current claims were true then, why didn’t someone just say it’s
> okay since the server is acting like a client under the hood?
> 
> K.
>

---

From: Kent Watsen <[email-masked]>
Date: Fri, 17 Dec 2021 17:27:51 +0000

> this is not possible if it was added by an action.

If the RPC/action backend were a client, then that client would be subject to locks/etc. too and, if unable to acquire after some timeout amount of time, could return an RPC-error, right?  But, again, I thought the hesitation surrounded client read backs, perhaps I misunderstood at the time...

> datastore can be used.  (this is actually better than the server

Two issues:

1) those nodes need to be immutable.  (See separate thread with “immutable” in Subject line)
2) there are many hundreds of such objects in JUNOS.  It would be a lot of clutter in <running>.

K.

---

From: Andy Bierman <[email-masked]>
Date: Fri, 17 Dec 2021 10:05:56 -0800

On Fri, Dec 17, 2021 at 7:11 AM Kent Watsen <[email-masked]> wrote:

> “action” called “generate-private-key” that would insert the generated key
> into <running>.  Claims were made by prominent members of this list that
> it’s bad form for anything but a client to edit <running>.
>
> As a result, extensive effort was spent defining a mechanism enabling the
> generated key to be returned in the RPC-reply in an encrypted form (such
> that only the server that generated the key could decrypt it), all so the
> client could immediately return it to the server via a config push in order
> to preserve the sanctity of client read-backs.
>
> If current claims were true then, why didn’t someone just say it’s okay
> since the server is acting like a client under the hood?
>

Maybe not a lot of non-security people were following the thread.
IMO a better design would have been to introduce a "secure-NV-storage"
concept.
The keys are never exposed. Only labels are exposed and the client can
store a reference to the key in <running>.
Maybe Juergen proposed this idea.

YANG-based management assumes that the conceptual API for a YANG device
can be described by all the [module, revision, features, deviations] tuples
(YANG library).
There was an original assumption that <running> could contain all the
information to
describe this "real API".

The original design was too simplistic so NMDA introduced <intended> and
<operational> datastores.
Now <running> no longer represents the "real API". It now contains some or
all of the information required
to produce the real API, which is now deemed to be <intended>.

Not one of the mechanisms to transform <running> into <intended> is
standardized.
Now <running> + ??? is a combination of the real API and the "proprietary
setup API".

It has never been true that <running> is always valid. In hindsight, that
MUST is really a SHOULD, post-NMDA.
Inactive nodes cannot be counted against constraints (e.g. must,
min/max-elements, unique).
Constraints on config templates do not address the needed constraints on
the expanded data.

IMO it is too late "fix" <running> by placing any restrictions on the
contents or who can edit it.
NMDA (wisely) did not do it.  A "system edit" is difficult to describe,
especially in a controller-driven bootstrap framework.  The immuable issue
is just hidden access control, so tag data nodes with new metadata to
expose that.

Andy

---

From: "maqiufang (A)" <[email-masked]>
Date: Tue, 21 Dec 2021 13:08:56 +0000

Hi, Andy, all

From: Andy Bierman [mailto:[email-masked]]
Sent: Saturday, December 18, 2021 2:06 AM
To: Kent Watsen <[email-masked]>
Cc: maqiufang (A) <[email-masked]>; [email-masked]
Subject: Re: [netmod] Must offline-validation of <running> alone be valid?

On Fri, Dec 17, 2021 at 7:11 AM Kent Watsen <[email-masked]<mailto:[email-masked]>> wrote:
Andy, et. al.,

No. Quite the opposite.  <snip>
[Qiufang Ma] Kent and Jason said that the “shared objects” is the primary <system> configuration use cases. I do agree. In Huawei’s implementation, The server also generates hundreds of system data objects(service templates, predefined policies, etc). Physical resource related configuration is really only a small part of that.
And as you mentioned, there is no RFC text says <running> have only nodes created by a client. Actually, in our implementation, we do allow system configuration to be populated into <running> automatically when it is generated, because copy/paste for reference is painful. That is, all the system configuration is really a part of <running>. But our customers do complain that when retrieving <running> they usually find a large number of configuration data objects not defined by themselves, most of which are irrelevant to their own configurations and they have no idea where they are come from.
I also tried to bring this idea to the IETF(https://datatracker.ietf.org/doc/html/draft-ma-netconf-with-system-02#section-3.1) and would like to collect some more feedback, it seems that the WG thinks we want the client to fully own the running datastore, the server should never auto-populates <running> unless asked by the clients.

Best Regards,
Qiufang Ma

Maybe not a lot of non-security people were following the thread.
IMO a better design would have been to introduce a "secure-NV-storage" concept.
The keys are never exposed. Only labels are exposed and the client can store a reference to the key in <running>.
Maybe Juergen proposed this idea.

YANG-based management assumes that the conceptual API for a YANG device
can be described by all the [module, revision, features, deviations] tuples (YANG library).
There was an original assumption that <running> could contain all the information to
describe this "real API".

The original design was too simplistic so NMDA introduced <intended> and <operational> datastores.
Now <running> no longer represents the "real API". It now contains some or all of the information required
to produce the real API, which is now deemed to be <intended>.

Not one of the mechanisms to transform <running> into <intended> is standardized.
Now <running> + ??? is a combination of the real API and the "proprietary setup API".

It has never been true that <running> is always valid. In hindsight, that MUST is really a SHOULD, post-NMDA.
Inactive nodes cannot be counted against constraints (e.g. must, min/max-elements, unique).
Constraints on config templates do not address the needed constraints on the expanded data.

IMO it is too late "fix" <running> by placing any restrictions on the contents or who can edit it.
NMDA (wisely) did not do it.  A "system edit" is difficult to describe,
especially in a controller-driven bootstrap framework.  The immuable issue
is just hidden access control, so tag data nodes with new metadata to expose that.

K.

Andy