Subject: Re: [Netconf] [OPSAWG] config persistence (was Re: Read-write access in VMM-MIB)

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 29 May 2014 13:37:29 +0200

Hi,

I am moving this discussion to the NETCONF mailing list.

Andy Bierman <[email-masked]> writes:
...
>
> I don't agree NETCONF/YANG persistence is ill-defined.
> It is not uniform. There are a few variants, identified by
> capability URIs.

I'd say persistence is at least under-defined. For example, it doesn't seem to be clear from 6241 whether candidate is persistent.

Lada

>
> The XMLCONF design team (pre-4741) tried very hard to
> get the router vendors to agree on a single transaction model
> but it was way too big a change, and (of course) each vendor
> thought their way was better, so the standard should use that.
> We ended up with 3 configuration datastores (candidate, running, startup).
> The transaction models that can be used with these datastores
> are compatible with specific router CLI architectures.
>
> Two particular vendors do not agree on how to recover
> from the "unreachable" problem.  If the operator makes
> config changes that break the connectivity to the device,
> then the device has to recover somehow.
>
>   M1: do not persist the changes until they are proven to
>         be correct.  Force the device to reboot with the saved
>         (known good) config if things go bad.
>
>   M2: persist the changes right away.  Pick some timeout
>         and if the client does not follow up with a confirmation
>         before the timeout, automatically revert to the version
>         that was running before the transaction started.
>
> RFC 6241 could be more clear on the motivation for the
> capabilities and operations that support various transaction models.
> Maybe that is what you meant by 'ill-defined'.
>
>
> Andy
>
> So the two objects controlling Notifications seem clearly configuration,
>> whether or not the setting of them persists or is lost on reboot.  You
>> could create a YANG module which controls the setting of these two
>> objects for SNMP usage but I think that even the IESG might see that
>> that is an unusual approach, as opposed to making them read-write in
>> SMI!
>>
>> When I first read the draft IESG statement, it was unclear to me whether
>> or not the authors fully understood what they had written.
>>
>> Tom Petch
>>
>> > (*1) http://www.ietf.org/iesg/statement/writable-mib-module.html
>> > (*2) http://tools.ietf.org/html/draft-ietf-opsawg-vmm-mib-00
>> >
>> > Hirochika
>> >
>> >
>> > On May 27, 2014, at 3:55 AM, joel jaeggli <[email-masked]> wrote:
>> >
>> > > yeah if you want to discuss it with the ops/management ADs and the
>> > > chairs that's fine.
>> > >
>> > > I don't think there's a reason to involve the whole iesg.
>> > >
>> > > Thanks
>> > > joel
>> > >
>> > > On 5/26/14, 10:48 AM, Hirochika Asai wrote:
>> > >> js and Joel,
>> > >>
>> > >> Thank you for your comments.
>> > >>
>> > >> I agree that the IESG statement seems to be talking about
>> configuration,
>> > >> but I cannot definitely say that the objects I listed in my
>> previous E-mail
>> > >> are out of the IESG statement's scope.
>> > >>
>> > >> I think it would be better to move this issue to the IESG, but I
>> don't keep
>> > >> up the procedure.  May I, as an author of the draft, send an E-mail
>> > >> stating this issue to [email-masked], CCing WG? Or ask WG chairs to
>> > >> handle it?
>> > >>
>> > >> Thank you.
>> > >> Hirochika
>> > >>
>> > >>
>> > >> On May 27, 2014, at 1:24 AM, joel jaeggli <[email-masked]> wrote:
>> > >>
>> > >>> On 5/26/14, 9:20 AM, Juergen Schoenwaelder wrote:
>> > >>>> On Mon, May 26, 2014 at 08:42:47AM -0700, joel jaeggli wrote:
>> > >>>>> On 5/26/14, 2:31 AM, Juergen Schoenwaelder wrote:
>> > >>>>>> Asai,
>> > >>>>>>
>> > >>>>>> the IESG statement is here:
>> > >>>>>>
>> > >>>>>> http://www.ietf.org/iesg/statement/writable-mib-module.html
>> > >>>>>>
>> > >>>>>> My reading is that it specifically talks about configuration.
>> While
>> > >>>>>> the discussion started with "lets ban all write access", it may
>> be
>> > >>>>>> important to note that the final statement does not say this.
>> Hence,
>> > >>>>>> I am not sure we have to remove the MAX-ACCESS read-write.
>> > >>>>>
>> > >>>>> some of the vm options do cause me existential peril. The
>> remaining
>> > >>>>> one's however do not. so I think Juergen's assessment  is a
>> correct one.
>> > >>>>> The statement seems to be serving it's purpose!
>> >>>>
>> > >>>> Joel, can you please decrypt your message so that it becomes
>> perhaps
>> > >>>> actionable?
>> > >>>
>> > >>> I'm agreeing with you.
>> > >>>
>> > >>>> /js
>> > --
>> > Hirochika Asai <[email-masked]>, The University of Tokyo
>> >
>> > _______________________________________________
>> > OPSAWG mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/opsawg
>>
>> _______________________________________________
>> OPSAWG mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/opsawg
>>
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 29 May 2014 05:31:03 -0700

On Thu, May 29, 2014 at 4:37 AM, Ladislav Lhotka <[email-masked]> wrote:

> I'd say persistence is at least under-defined. For example, it doesn't
> seem to be clear from 6241 whether candidate is persistent.
>
>
OK

sec 8.3.5.2 says:

   When a client fails with outstanding changes to the candidate
   configuration, recovery can be difficult.  To facilitate easy
   recovery, any outstanding changes are discarded when the lock is
   released, whether explicitly with the <unlock> operation or
   implicitly from session failure.

IMO it is odd that candidate is cleaned up only if it was locked.
If a client edits candidate and then closes the session somehow the changes
are left behind.
The RFC is silent about the contents of candidate at boot-time. Our server
always boots
with the candidate and running having the same contents.  Otherwise the
candidate is
dirty at boot-time and cannot be locked by any session until a
discard-changes is invoked.
There is also no explicit requirement to boot with any particular candidate
contents, so
booting with the contents of running is compliant.

Andy

> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 29 May 2014 15:16:31 +0200

On 29 May 2014, at 14:31, Andy Bierman <[email-masked]> wrote:

> If a client edits candidate and then closes the session somehow the changes are left behind.
> The RFC is silent about the contents of candidate at boot-time. Our server always boots
> with the candidate and running having the same contents.  Otherwise the candidate is
> dirty at boot-time and cannot be locked by any session until a discard-changes is invoked.
> There is also no explicit requirement to boot with any particular candidate contents, so

I think an alternative interpretation of the 6241 text is that candidate is a persistent staging area that survives reboots and is only reverted to running after discard-changes. So a client can close a session, then later open a new one and start editing where he left off. 

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 29 May 2014 08:35:42 -0700

On Thu, May 29, 2014 at 6:16 AM, Ladislav Lhotka <[email-masked]> wrote:

> I think an alternative interpretation of the 6241 text is that candidate
> is a persistent staging area that survives reboots and is only reverted to
> running after discard-changes. So a client can close a session, then later
> open a new one and start editing where he left off.
>
>

There is no text that says anything about the initial state of candidate.
Does any known implementation behave this way?
(Note that the shared candidate is a horrible place to toss some edits
and come back later to finish up.  The shared candidate is not really
intended to left unlocked so partial edits from various clients can
accumulate.)

Lada
Andy

---

From: t.petch <[email-masked]>
Date: Fri, 30 May 2014 11:34:10 +0100

----- Original Message -----
From: "Andy Bierman" <[email-masked]>
To: "Ladislav Lhotka" <[email-masked]>
Cc: "Netconf" <[email-masked]>
Sent: Thursday, May 29, 2014 1:31 PM

Andy

I am happy to characterise it as under-defined rather that ill-defined,
but I think that the RFC are lacking.   We did have this discussion a
few months back, and my point then was that if the only configuration
datastore is running, then the assumption by implementers was that any
change to it would immediately be persistent.  Seems reasonable, but not
specified anywhere, and since running then has the opposite semantics
with startup alongside it, well, that makes it less clearly defined, for
me.

As Lada points out, that is not the only such under-definition.

Tom Petch

> > >> > (*1)
> > >> > >>>> Joel, can you please decrypt your message so that it
becomes

------------------------------------------------------------------------
--------

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 30 May 2014 13:11:08 +0200

> As Lada points out, that is not the only such under-definition.

Another thing that puzzles me:

Let’s say a client edits the value of parameter X in running. The new value is immediately projected into operational state. Then later the state value gets changed by some means external to NETCONF. In what event, if any, will the value of X be reset to the value that’s configured in running?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 07:17:31 -0700

> with startup alongside it, well, that makes it less clearly defined, for

So you are saying you did not understand RFC 6241, sec.  8.7?
Is there something unclear in this section? It says:

   The device supports separate running and startup configuration
   datastores.  The startup configuration is loaded by the device when
   it boots.  Operations that affect the running configuration will not
   be automatically copied to the startup configuration.  An explicit
   <copy-config> operation from the <running> to the <startup> is used
   to update the startup configuration to the current contents of the
   running configuration.  NETCONF protocol operations refer to the
   startup datastore using the <startup> element.

The client checks for the "distinct startup" capability.
How is this unspecified?  Did you miss this section?

> As Lada points out, that is not the only such under-definition.
The RFC does not say anything about High Availability features.
If the server wants to persist state information so it can recover
from a hot-swap faster, then that is a fine extra feature, but
not required at all by the standard.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 07:30:23 -0700

On Fri, May 30, 2014 at 4:11 AM, Ladislav Lhotka <[email-masked]> wrote:

> Let's say a client edits the value of parameter X in running. The new
> value is immediately projected into operational state. Then later the state
> value gets changed by some means external to NETCONF. In what event, if
> any, will the value of X be reset to the value that's configured in running?
>
>
If state variables and config variables interact, then the description-stmt
should explain this data-model specific behavior.

Lada

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 07:44:30 -0700

> with startup alongside it, well, that makes it less clearly defined, for

OK -- you are talking about a server that does not advertise :startup.
We used to call that "mirror mode", but that is not in the RFC.

Since no NETCONF implementation goes back to factory-default
config on every reboot, it seems vendors understand that persistence
of config data is required in real products, if not in the RFC.

This text in 1.4 says persistence is an implementation-specific feature:

   Note that the NETCONF protocol is focused on the information required
   to get the device into its desired running state.  The inclusion of
   other important, persistent data is implementation specific.  For
   example, user files and databases are not treated as configuration
   data by the NETCONF protocol.

   For example, if a local database of user authentication data is
   stored on the device, it is an implementation-dependent matter
   whether it is included in configuration data.

This text in 8.4.1 specifies mandatory persistence for confirmed commit.

   If the device reboots for any reason before the confirm timeout
   expires, the server MUST restore the configuration to its state
    before the confirmed commit was issued.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 30 May 2014 16:58:38 +0200

> should explain this data-model specific behavior.

They needn’t interact within the NETCONF realm. For example: configured static routes appear in a RIB, but then some of the routes are deleted from the RIB via I2RS. So the RIB and config are out of sync.
Will the configured routes ever be re-installed, e.g. after running changes again?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 08:06:56 -0700

On Fri, May 30, 2014 at 7:58 AM, Ladislav Lhotka <[email-masked]> wrote:

> They needn't interact within the NETCONF realm. For example: configured
> static routes appear in a RIB, but then some of the routes are deleted from
> the RIB via I2RS. So the RIB and config are out of sync.
>From my understanding of I2RS architecture, the local config could
be assigned a priority. It is likely an implementation detail how the
system re-installs config that was replaced by state, and then that state
changed
or was removed by I2RS.  If vendors or other standards add new behavior
to the system outside the scope of NETCONF, then they should specify
how they interact with the configuration.

Lada

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 30 May 2014 17:09:02 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> > Let’s say a client edits the value of parameter X in running. The new value
> > is immediately projected into operational state. Then later the state value
> > gets changed by some means external to NETCONF. In what event, if any, will
> > the value of X be reset to the value that’s configured in running?
> > 
> > 
> They needn’t interact within the NETCONF realm. For example: configured static
> routes appear in a RIB, but then some of the routes are deleted from the RIB
> via I2RS. So the RIB and config are out of sync.

As Andy already wrote, there is not a single generic answer.  In this
particular case, it is up to I2RS to clearly define.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 30 May 2014 17:35:16 +0200

> particular case, it is up to I2RS to clearly define.

I2RS was just an example, my question simply is: When is the running configuration (or parts of it) applied/re-applied? On a Unix system configured in init scripts it is usually clear - e.g. any time

/etc/init.d/xxxx reload

is run. NETCONF has no such explicit push mechanism, so I wonder whether/when this might happen anyway.
The spec says nothing about it, presumably because running was considered to be always in sync with operational parameters.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 08:49:55 -0700

On Fri, May 30, 2014 at 8:35 AM, Ladislav Lhotka <[email-masked]> wrote:

> value
> >>> the value of X be reset to the value that's configured in running?
> >>>
> >>>
> static
> >> Will the configured routes ever be re-installed, e.g. after running
> changes
> >> again?
> >
> I2RS was just an example, my question simply is: When is the running
> configuration (or parts of it) applied/re-applied? On a Unix system
> configured in init scripts it is usually clear - e.g. any time
>
> /etc/init.d/xxxx reload
>
> The spec says nothing about it, presumably because running was considered
> to be always in sync with operational parameters.
>
>
I think NETCONF is explicit about the semantics of the running
configuration.
The system MUST be aware of the desired state of the system as embodied
in the running datastore.  The system uses that information, along with all
sorts of other information, to perform its functions.  The resulting
operational
state will be data model and implementation specific.

Lada
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 30 May 2014 18:10:07 +0200

On 30 May 2014, at 17:49, Andy Bierman <[email-masked]> wrote:

> I think NETCONF is explicit about the semantics of the running configuration.
> sorts of other information, to perform its functions.  The resulting operational

My company implemented NETCONF on OpenWRT so that UCI scripts essentially act as the running config. The way how UCI works is that “uci commit” may generate a native configuration file for a certain service. A user can nonetheless use vi to edit the native configuration file directly, so the UCI config gets out of sync and will be resynchronised only after next “uci commit” for that service.

Are you saying this is not a compliant NETCONF implementation?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 30 May 2014 09:30:53 -0700

On Fri, May 30, 2014 at 9:10 AM, Ladislav Lhotka <[email-masked]> wrote:

> > >>> Let's say a client edits the value of parameter X in running. The
> new value
> > >> They needn't interact within the NETCONF realm. For example:
> configured static
> > The spec says nothing about it, presumably because running was
> considered to be always in sync with operational parameters.
> >
> >
> My company implemented NETCONF on OpenWRT so that UCI scripts essentially
> act as the running config. The way how UCI works is that "uci commit" may
> generate a native configuration file for a certain service. A user can
> nonetheless use vi to edit the native configuration file directly, so the
> UCI config gets out of sync and will be resynchronised only after next "uci
> commit" for that service.
>
If a client requests contents from the running configuration datastore via
NETCONF protocol operations, then your implementation is required to
return the correct values.  Other protocols can change the running config
outside the scope of NETCONF.  If so, the server still has to return the
correct <rpc-reply>.  (There could be latency, e.g. the server polls the
last-modified timestamp on the file in your implementation.)

Lada

Andy

---

From: t.petch <[email-masked]>
Date: Fri, 30 May 2014 17:46:33 +0100

----- Original Message -----
From: "Andy Bierman" <[email-masked]>
To: "t.petch" <[email-masked]>
Cc: "Ladislav Lhotka" <[email-masked]>; "Netconf" <[email-masked]>
Sent: Friday, May 30, 2014 3:17 PM
<snip>

Andy

And when there is no startup configuration (because it is optional)?  We
did agree a year ago that that condition was not specified in an RFC,
although implementations seem to be consistent about it.  That is the
scenario I am referring to

Tom Petch

> > > If a client edits candidate and then closes the session somehow
the
> > changes
> > > are left behind.
> > > The RFC is silent about the contents of candidate at boot-time.
Our
> > server
> > > always boots
> > > with the candidate and running having the same contents.
Otherwise
> > the
> > > > > We ended up with 3 configuration datastores (candidate,
running,
> > > > startup).
> > > > >> whether or not the setting of them persists or is lost on
reboot.
> > You
> > > > >> could create a YANG module which controls the setting of
these
> > two
> > > > >> objects for SNMP usage but I think that even the IESG might
see
> > that
> > > > >> that is an unusual approach, as opposed to making them
read-write
> > in
> > > > >> SMI!
> > > > >>
> > > > >> When I first read the draft IESG statement, it was unclear to
me
> > whether
> > > > >> > (*2)
http://tools.ietf.org/html/draft-ietf-opsawg-vmm-mib-00
> > > > >> >
> > > > >> > Hirochika
> > > > >> >
> > > > >> >
> > > > >> > On May 27, 2014, at 3:55 AM, joel jaeggli
<[email-masked]>
> > wrote:
> > > > >> >
> > > > >> > > yeah if you want to discuss it with the ops/management
ADs
> > and the
> > > > >> > >> I think it would be better to move this issue to the
IESG,
> > but I
> > > > >> don't keep
> > > > >> > >> up the procedure.  May I, as an author of the draft,
send an
> > E-mail
> > > > >> > >> On May 27, 2014, at 1:24 AM, joel jaeggli
<[email-masked]>
> > > > wrote:
> > > > >> > >>>>>> the discussion started with "lets ban all write
access",
> > it may
> > > > >> be
> > > > >> > >>>>>> important to note that the final statement does not
say
> > this.
> > > > >> > >>>>> one's however do not. so I think Juergen's assessment
is
> > a

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 30 May 2014 20:19:03 +0200

> outside the scope of NETCONF.  If so, the server still has to return the

Other protocols, even if they care (some don’t), may not able to do it if running is locked.

Lada

---

From: "Alberto Gonzalez Prieto (albertgo)" <[email-masked]>
Date: Fri, 30 May 2014 23:23:12 +0000

I understand the text above identifies running datastore with "desired state" (vs. operational state ?)

>From RFC 6241:
"running configuration datastore: A configuration datastore holding
      the complete configuration currently active on the device."
"A configuration datastore is defined as the complete set of configuration data that
   is required to get a device from its initial default state into a desired operational state."

I could not find a definition of "active configuration" in the RFC.
But considering the second snippet, I understand that, in the absence of stimuli, the contents of the running datastore and the operational state must be eventually consistent.
Is this correct?
That would mean that if the desired state would stop being operational (say this is caused by some permanent failure) or could not be made operational, the running datastore contents should change.

Thanks,

Alberto

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 31 May 2014 11:29:15 +0200

> But considering the second snippet, I understand that, in the absence of stimuli, the contents of the running datastore and the operational state must be eventually consistent.

I don’t think this is correct, unless eventually means “sometimes, maybe". If it was true, the dual config and state values as defined in ietf-ip and ietf-routing would not be needed.

Lada