Subject: [Netconf] Is there a problem with confirmed commits?

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 12:50:38 +0000

Hi,

No one seems to be responding to my email and proposed erratum around 
the subject of confirmed commits (apart from Martin), but I would really 
like to know it I am missing something here. As far as I can tell, 
session termination during a confirmed commit leads to unpredictable 
behaviour and I would like to know whether anyone is using confirmed 
commits and how (if at all) they address the issues outlined below. My 
assumptions are that locks are used and :writable-running is not 
supported.

If the <candidate> and <running> configuration datastores are locked to 
prevent concurrent access, and a confirmed commit sequence is 
interrupted by the session terminating, the locks will automatically be 
released but the server MUST NOT accept a lock on <running> from any 
session if another session has an ongoing confirmed <commit>. 
Consequently, after session termination no client can acquire a <lock> 
on <running>, not even the one that initiated the confirmed <commit>, 
until after the confirmed <commit> has timed out. However, if the 
confirmed <commit> included the <persist> parameter, the original client 
could still issue a <commit> using the persist-id to complete the 
sequence prior to the timeout, even without a lock.

Of course, the problem now is the race for the new lock on <candidate>. 
If the original client is successful then all is good. But if a new 
client locks <candidate> before the timeout on the confirmed commit, 
whether or not they precede <lock> with <discard-changes>, <candidate> 
will be the same as <running> and the new client will pick up everything 
from the previous session. However, the client won’t be able to lock 
<running> until after the timeout, at which point <running> reverts but 
<candidate> still represents the previous session. If the client tries 
to lock <candidate> after the timeout, <running> will have reverted and 
the lock will only be granted after a <discard-changes> which will cause 
the <candidate> to revert. So, depending on when the lock on <candidate> 
occurs relative to the confirmed commit timeout, the client could be 
editing <candidate> in one of two states. Further, before the timeout on 
the confirmed commit, even if the new client has locked candidate, the 
original client could still issue a confirming commit (they don’t need a 
lock on <candidate> to do so) which would persistently commit any edits 
made by the new client. NOTE: it is not the use of the persist-id that 
introduces this behaviour; a new client would have the same problem even 
if a confirmed commit was not intended to persist beyond a session 
termination.

If the server also supports the :startup capability then, if the session 
termination was due to the server rebooting, the behaviour above would 
be further complicated by <running> now containing the configuration 
from the <startup> configuration datastore.

Am I right?

Jonathan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 14 Jan 2019 14:23:42 +0100 (CET)

"Jonathan Hansford" <[email-masked]> wrote:
> the subject of confirmed commits (apart from Martin), but I would
> really like to know it I am missing something here. As far as I can
> tell, session termination during a confirmed commit leads to
> unpredictable behaviour and I would like to know whether anyone is
> using confirmed commits and how (if at all) they address the issues
> outlined below. My assumptions are that locks are used and
> :writable-running is not supported.
> 
> If the <candidate> and <running> configuration datastores are locked
> to prevent concurrent access, and a confirmed commit sequence is
> interrupted by the session terminating, the locks will automatically
> be released but the server MUST NOT accept a lock on <running> from
> any session if another session has an ongoing confirmed
> <commit>. Consequently, after session termination no client can
> acquire a <lock> on <running>, not even the one that initiated the
> confirmed <commit>, until after the confirmed <commit> has timed
> out. However, if the confirmed <commit> included the <persist>
> parameter, the original client could still issue a <commit> using the
> persist-id to complete the sequence prior to the timeout, even without
> a lock.
> 

According to 7.5, a lock is not granted if:

      *  The target configuration is <candidate>, it has already been
         modified, and these changes have not been committed or rolled
         back.

      *  The target configuration is <running>, and another NETCONF
         session has an ongoing confirmed commit (Section 8.4).

So I think that it will not be possible to take the lock on either
datastore in this situation.

/martin

> If the original client is successful then all is
> good. But if a new client locks <candidate> before the timeout on the
> confirmed commit, whether or not they precede <lock> with
> <discard-changes>, <candidate> will be the same as <running> and the
> new client will pick up everything from the previous session. However,
> the client won’t be able to lock <running> until after the timeout, at
> which point <running> reverts but <candidate> still represents the
> previous session. If the client tries to lock <candidate> after the
> timeout, <running> will have reverted and the lock will only be
> granted after a <discard-changes> which will cause the <candidate> to
> revert. So, depending on when the lock on <candidate> occurs relative
> to the confirmed commit timeout, the client could be editing
> <candidate> in one of two states. Further, before the timeout on the
> confirmed commit, even if the new client has locked candidate, the
> original client could still issue a confirming commit (they don’t need
> a lock on <candidate> to do so) which would persistently commit any
> edits made by the new client. NOTE: it is not the use of the
> persist-id that introduces this behaviour; a new client would have the
> same problem even if a confirmed commit was not intended to persist
> beyond a session termination.
> 
> If the server also supports the :startup capability then, if the
> session termination was due to the server rebooting, the behaviour
> above would be further complicated by <running> now containing the
> configuration from the <startup> configuration datastore.
> 
> Am I right?
> 
> Jonathan
> 
> ---

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 14:50:56 +0100

Hi,

I have not yet understood where you see a problem. In general,
<candidate/> contains arbitrary stuff and hence it is the client's
responsibility to clear any arbitrary stuff found in <candidate/>
after obtaining a lock. If does not really matter whether there has
been a failed confirmed commit before or something else. I think the
general safe pattern is:

lock(candidate)
discard_changes()
push_whatever_needed()
commit()
unlock(candidate)

If you do a confirmed commit and the session disappears, then the lock
will disappear as well. But I do not think this creates a race
condition, or I am just not yet seeing it. Perhaps it helps to write
down the sequence of actions that leads to a race.

/js

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 14:47:14 +0000

RFC 6241, Section 8.2.4.1 states, 'If the running or candidate 
configuration is currently locked by a different session, the <commit> 
operation MUST fail with an <error-tag> value of "in-use".' So if the 
new client acquires a lock on <candidate>, the original client cannot 
<commit> to <running>. So that fixes one of the issues below.

Section 8.3.5.2 states, 'When a client fails with outstanding changes to 
the candidate configuration, recovery can be difficult.  To facilitate 
easy recovery, any outstanding changes are discarded when the lock is 
released, whether explicitly with the <unlock> operation or implicitly 
from session failure.' So this means <candidate> would revert. Section 
8.4.1 also states, 'If the device reboots for any reason before the 
confirm timeout expires, the server MUST restore the configuration to 
its state before the confirmed commit was issued.' However, earlier in 
Section 8.4.1 it states, 'If the session issuing the confirmed commit is 
terminated for any reason before the confirm timeout expires, the server 
MUST restore the configuration to its state before the confirmed commit 
was issued, unless the confirmed commit also included a <persist> 
element.', Section 8.4.5.1 states the persist parameter makes 'the 
confirmed commit survive a session termination, and set a token on the 
ongoing confirmed commit' and its description in Appendix C states, 
'This parameter is used to make a confirmed commit persistent.  A 
persistent confirmed commit is not aborted if the NETCONF session 
terminates.  The only way to abort a persistent confirmed commit is to 
let the timer expire, or to use the <cancel-commit> operation.'

So it is the persist-id that is the cause of all the problems, not just 
the use of confirmed commit as I previously stated. But is there a way 
to resolve the outstanding issues around the use of the persist-id?

Thanks

------ Original Message ------
From: "Jonathan Hansford" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 14/01/2019 12:50:38
Subject: [Netconf] Is there a problem with confirmed commits?

>confirmed <commit> included the <persist> parameter, the original 
>client could still issue a <commit> using the persist-id to complete 
>the sequence prior to the timeout, even without a lock.
>
>will be the same as <running> and the new client will pick up 
>everything from the previous session. However, the client won’t be able 
>to lock <running> until after the timeout, at which point <running> 
>reverts but <candidate> still represents the previous session. If the 
>client tries to lock <candidate> after the timeout, <running> will have 
>reverted and the lock will only be granted after a <discard-changes> 
>which will cause the <candidate> to revert. So, depending on when the 
>lock on <candidate> occurs relative to the confirmed commit timeout, 
>the client could be editing <candidate> in one of two states. Further, 
>before the timeout on the confirmed commit, even if the new client has 
>locked candidate, the original client could still issue a confirming 
>commit (they don’t need a lock on <candidate> to do so) which would 
>persistently commit any edits made by the new client. NOTE: it is not 
>the use of the persist-id that introduces this behaviour; a new client 
>would have the same problem even if a confirmed commit was not intended 
>to persist beyond a session termination.
>
>--------------------------------------------------------------------------------
>Avast logo
><https://www.avast.com/antivirus>
>www.avast.com <https://www.avast.com/antivirus>
>
>
><#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

---

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 15:51:34 +0100

Dear Jonathan,

can you please explain how we get into a race condition? What is the
sequence of actions the clients involved have to go through to arrive
at a race?

/js

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 15:14:02 +0000

If a persistent confirmed commit has not timed out, the running 
configuration datastore will be the same as the candidate and 
<discard-changes> won't change its contents. Any edit of candidate will 
be based on the configuration resulting from the persistent confirmed 
commit.

If the persistent confirmed commit has timed out, the running 
configuration datastore will have reverted and <discard-changes> will 
change candidate. Any edit of candidate in this case will be based on 
the configuration prior to the start of the persistent confirmed commit.

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 15:16:47 +0000

The candidate can be locked because the changes have been committed, 
just not confirmed.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 16:40:26 +0100

It seems the <candidate> datastore should not be allowed to be used as
long as a persistent confirmed commit is still ongoing. I leave it to
Martin to check whether this is said somewhere or an omission.

In general, an application can't assume that <candidate> contains
anything sensible. Hence, the proper way is to lock <candidate> and
then to make sure it contains something sensible, i.e., issuing a
discard_changes. And I think implementations should not allow an
application to obtain a lock on <candidate> while a commit is active.
The text on page 45 already says:

      A lock MUST NOT be granted if any of the following conditions is
      true:

      [...]

I think this covers the case of an ongoing but not completed
persistent confirmed commit, no?

/js

On Mon, Jan 14, 2019 at 03:14:02PM +0000, Jonathan Hansford wrote:
> <discard-changes> won't change its contents. Any edit of candidate will be
> based on the configuration resulting from the persistent confirmed commit.
> 
> If the persistent confirmed commit has timed out, the running configuration
> datastore will have reverted and <discard-changes> will change candidate.
> Any edit of candidate in this case will be based on the configuration prior
> to the start of the persistent confirmed commit.
>

---

From: Robert Wilton <[email-masked]>
Date: Mon, 14 Jan 2019 15:48:58 +0000

Hi Juergen,

On 14/01/2019 15:40, Juergen Schoenwaelder wrote:
> discard_changes.

But the text that you quote below states that a client cannot acquire a 
lock on candidate if it contains any changes.  Doesn't this implies that 
discard_changes after acquiring the lock should be unnecessary?

Thanks,
Rob

>   And I think implementations should not allow an

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 16:01:59 +0000

I think the text you refer to on page 45 is open to interpretation. If a 
persistent confirmed commit has occurred then the changes have been 
committed (albeit in a confirmed commit), and the candidate and running 
configuration datastores are the same (unless and until the confirmed 
commit times out). The text depends on the definition of 'modified' (and 
how the server detects it) and 'committed'.

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Jan 2019 16:05:03 +0000

And maybe a discard_changes before the lock is better behaviour (or is 
there something that prevents that?) to ensure the lock will succeed.

------ Original Message ------
From: "Robert Wilton" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "Jonathan Hansford" <[email-masked]>
Sent: 14/01/2019 15:48:58
Subject: Re: [Netconf] Is there a problem with confirmed commits?

>But the text that you quote below states that a client cannot acquire a lock on candidate if it contains any changes.  Doesn't this implies that discard_changes after acquiring the lock should be unnecessary?
>
>Thanks,
>Rob
>
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 17:19:19 +0100

Someone can modify candidate between your discard_changes() and
lock().  I have to check whether lock requires <candidate/> to match
<running/>.

/js

On Mon, Jan 14, 2019 at 04:05:03PM +0000, Jonathan Hansford wrote:
> And maybe a discard_changes before the lock is better behaviour (or is there
> something that prevents that?) to ensure the lock will succeed.
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 17:25:32 +0100

For me, a confirmed commit is not complete if it can still timeout.

/js

On Mon, Jan 14, 2019 at 04:01:59PM +0000, Jonathan Hansford wrote:
> configuration datastores are the same (unless and until the confirmed commit
> times out). The text depends on the definition of 'modified' (and how the
> server detects it) and 'committed'.
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 14 Jan 2019 08:57:06 -0800

> For me, a confirmed commit is not complete if it can still timeout.
It is hard to imagine how a confirmed-commit could be considered complete
if the server is still waiting for the confirming commit.

> /js

Andy

> commit
> configuration
> > > >  From: "Juergen Schoenwaelder" <[email-masked]
> >
> everything

---

From: <[email-masked]>
Date: Mon, 14 Jan 2019 20:07:26 -0000

So what you are both saying is that a commit isn’t a commit if it is a confirmed commit that hasn’t been followed by a confirming commit. That being the case, it would be helpful if, rather than trying to come up with a form of words that clarifies the second bullet on page 45, the third bullet was modified to state ‘The target configuration is <candidate> or <running>, and another NETCONF session has an ongoing confirmed commit (Section 8.4).’

In summary:

*	If a non-persistent confirmed commit suffers a session termination, any locks are released and the <running> configuration datastore rolls back to its state prior to the confirmed commit, or the first confirmed commit in a sequence, being issued.
*	If a persistent confirmed commit suffers a session termination, any locks are released but the <running> configuration datastore does not roll back until the confirmed commit times out. In the meantime, neither the <candidate> nor the <running> configuration datastore can be locked by any session, but any session that knows the persist-id can prevent the rollback on the timeout by issuing a confirming commit which will irrevocably commit the <candidate> configuration datastore to <running>.

Is that everyone’s understanding of how non-persistent and persistent confirmed commits work?

Just as a footnote, if the cause of the session termination was the device rebooting and the :startup capability was supported, the <running> configuration datastore will contain whatever was in the <startup> configuration datastore. However, a confirming commit on a persistent confirmed commit would still commit the <candidate> configuration datastore to <running>. If the purpose of the confirmed commit was to test the new configuration prior to making it permanent, it would therefore make sense to copy the <candidate> configuration datastore to the <startup> configuration datastore prior to the confirmed commit.

From: Andy Bierman <[email-masked]> 
Sent: 14 January 2019 16:57
To: Juergen Schoenwaelder <[email-masked]>; Jonathan Hansford <[email-masked]>; [email-masked]
Subject: Re: [Netconf] Is there a problem with confirmed commits?

On Mon, Jan 14, 2019 at 8:25 AM Juergen Schoenwaelder <[email-masked] <mailto:[email-masked]> > wrote:

 

/js

Andy

On Mon, Jan 14, 2019 at 04:01:59PM +0000, Jonathan Hansford wrote:
> From: "Juergen Schoenwaelder" <[email-masked] <mailto:[email-masked]> >
> To: "Jonathan Hansford" <[email-masked] <mailto:[email-masked]> >
> Cc: "[email-masked] <mailto:[email-masked]> " <[email-masked] <mailto:[email-masked]> >
> > >  From: "Juergen Schoenwaelder" <[email-masked] <mailto:[email-masked]> >
> > >  To: "Jonathan Hansford" <[email-masked] <mailto:[email-masked]> >
> > >  Cc: "[email-masked] <mailto:[email-masked]> " <[email-masked] <mailto:[email-masked]> >
> > >  > >  [email-masked] <mailto:[email-masked]> 

_______________________________________________
Netconf mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/netconf

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 14 Jan 2019 23:32:21 +0100

> Just as a footnote, if the cause of the session termination was the device rebooting and the :startup capability was supported, the <running> configuration datastore will contain whatever was in th…

My understanding of a confirmed commit is:

    copy condidate to running and wait for a confirmation that
    everything is OK; if no confirmation is received within a certain
    time window, rollback to the original state of <running>.

The situation you describe is a bit unclear for me; when does the
device reboot, after the confirmed commit has been issued but before
<candidate> was copied to <running>? That would be a rare situation
and I have some doubts that implementations after the reboot would
start the confirmed commit or they would not execute a reboot before
copying <candidate> to <running>.

/js

---

From: "Jonathan Hansford" <[email-masked]>
Date: Tue, 15 Jan 2019 10:02:09 +0000

The issue I am trying to address is one where the original session is 
terminated. This could occur, for example, if the device's IP address is 
changed by the confirmed commit. I have also worked with devices in the 
past where certain configuration changes caused the device to reboot. In 
both circumstances the <candidate> has been copied to <running> but the 
commit hasn't been confirmed. Consequently there is still the need to 
issue a confirming commit which, according to my reading of the RFC, 
would copy <candidate> to <running> again. Clearly if <candidate> hasn't 
changed the server can choose not to make any changes to <running> but, 
if another client has in the meantime updated <candidate>, the 
confirming commit should (by my reading) copy those changes over.

>From the last few emails on this subject it seems the persist-id can be 
considered a de facto lock on both <candidate> and <running> that is 
released on a confirming <commit>, a timeout on the confirmed <commit> 
or a <cancel-commit>. Unlike other locks, this "lock" could be shared 
between clients by sharing the persist-id. Is that a fair summary?

------ Original Message ------
From: "Juergen Schoenwaelder" <[email-masked]>
To: [email-masked]
Cc: "'Andy Bierman'" <[email-masked]>; [email-masked]
Sent: 14/01/2019 22:32:21
Subject: Re: [Netconf] Is there a problem with confirmed commits?

---

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 15 Jan 2019 11:14:09 +0100

On Tue, Jan 15, 2019 at 10:02:09AM +0000, Jonathan Hansford wrote:
> changed by the confirmed commit. I have also worked with devices in the past
> where certain configuration changes caused the device to reboot. In both
> circumstances the <candidate> has been copied to <running> but the commit
> hasn't been confirmed. Consequently there is still the need to issue a
> confirming commit which, according to my reading of the RFC, would copy
> <candidate> to <running> again. Clearly if <candidate> hasn't changed the
> server can choose not to make any changes to <running> but, if another
> client has in the meantime updated <candidate>, the confirming commit should
> (by my reading) copy those changes over.

Why copy again? The flow would be:

  upon confirmed-commit:
     copy <candidate> to <running>
     reinitialize what needs to be reinitialized
     wait for the confirmation
     if (timeout) rollback <running>

> released on a confirming <commit>, a timeout on the confirmed <commit> or a
> <cancel-commit>. Unlike other locks, this "lock" could be shared between
> clients by sharing the persist-id. Is that a fair summary?

Yes, this may be one way to look at it.

/js

---

From: "Jonathan Hansford" <[email-masked]>
Date: Tue, 15 Jan 2019 10:36:02 +0000

>      if (timeout) rollback <running>
Section 8.4.1 states, 'A confirmed <commit> operation MUST be reverted 
if a confirming commit is not issued within the timeout period (by 
default 600 seconds = 10 minutes). The confirming commit is a <commit> 
operation without the <confirmed> parameter... the confirming commit ... 
MAY introduce additional changes to the configuration' So the 
confirmation comes in the form of a confirming commit which is not 
described as being different to any other commit, other than it 
completes a confirmed commit.

Section 8.3.4.1 states, 'The <commit> operation instructs the device to 
implement the configuration data contained in the candidate 
configuration.' So a confirming commit has to copy to <running> any 
changes, if any, that have been made to <candidate> since the previous 
confirmed commit.

---

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 15 Jan 2019 11:59:35 +0100

> changes, if any, that have been made to <candidate> since the previous

Since the data has already been copied and the server is just waiting
for the confirming commit to finish the confirmed commit, there is no
need to copy data again.

/js

---

From: "Jonathan Hansford" <[email-masked]>
Date: Tue, 15 Jan 2019 11:10:19 +0000

>for the confirming commit to finish the confirmed commit, there is no
But that is not what the RFC suggests and there is nothing to stop 
changes being introduced to <candidate>. That is why Section 8.4.1 
states, 'the confirming commit ... MAY introduce additional changes to 
the configuration'. So if <candidate> has changed, those changes need to 
be committed.

---