Subject: [Netconf] <lock> in a server that supports both writable-running and candidate

From: "Sterne, Jason (Nokia - CA)" <[email-masked]>
Date: Thu, 14 Jul 2016 19:03:16 +0000

Hi all,

I'm looking for opinions on how a client & server might interact for the <lock> RPC when a server supports both writable-running and candidate DS.

I suspect this is an unusual situation -> does anyone out there work with or have a server implementation that supports both ?

The spirit of a <lock> is to reserve exclusive access to alter the configuration of the device, and to prevent other clients (or management interfaces) from altering the config in the meantime.  I realize that in theory the locks are "per-DS" (i.e. independant) but I think that might be too simplistic for a system with both writable-running and candidate.

In a server that supports both writable-running & candidate, it seems to make sense that a <lock> of the running DS would also cause the server to implicitly lock the candidate (or at least reject a lock request of the candidate).  Otherwise (i.e. accepting a subsequent lock of the candidate) the client may assume since it has the candidate lock then it has the exclusive rights to the config and can happily edit & commit the config.  It would be confusing to error on the commit (due to locking) after having accepted a lock.

Similarly for the reverse situation -> if a client takes a candidate <lock> then couldn't it assume that nobody can subsequently block their commit by taking a lock of the running DS ?

It seems like the server should manage this interaction and avoid giving an "OK" to clients taking the two locks on the two datastores.

On the other hand this could all be left up to the clients to deal with.  Clients would have to see that a server has both a writeable-running and a candidate DS and take a lock of both.   But that would require all clients to have that behavior (instead of the server that supports the 2 DSes dealing with this).  It also has race conditions where the client takes 1 lock but someone else gets the second lock.

Opinions appreciated.
Regards,
Jason

---

From: Andy Bierman <[email-masked]>
Date: Thu, 14 Jul 2016 12:13:41 -0700

Hi,

IMO NETCONF locks are not well designed at all.
For example, you are not even allowed to lock multiple datastores at once.
You are forced to lock 1 at a time, and if 2 clients are doing this at the
same time,
it can lead to a deadlock between them (Each 1 holding 1 lock waiting for
the other)

The shared candidate is sort of a disaster but if you want to use it
then you will need to invent your own "git update".  The reason nobody
implements
:candidate and :writable-running together is because there is no way to
synch
the candidate if running has been changed underneath it.

A concurrent write database is not "first writer wins, everybody else blows
up".

Andy

On Thu, Jul 14, 2016 at 12:03 PM, Sterne, Jason (Nokia - CA) <
[email-masked]> wrote:

> Hi all,
>
> I’m looking for opinions on how a client & server might interact for the
> <lock> RPC when a server supports both writable-running and candidate DS.
>
> I suspect this is an unusual situation -> does anyone out there work with
> or have a server implementation that supports both ?
>
> The spirit of a <lock> is to reserve exclusive access to alter the
> configuration of the device, and to prevent other clients (or management
> interfaces) from altering the config in the meantime.  I realize that in
> theory the locks are “per-DS” (i.e. independant) but I think that might be
> too simplistic for a system with both writable-running and candidate.
>
> In a server that supports both writable-running & candidate, it seems to
> make sense that a <lock> of the running DS would also cause the server to
> implicitly lock the candidate (or at least reject a lock request of the
> candidate).  Otherwise (i.e. accepting a subsequent lock of the candidate)
> the client may assume since it has the candidate lock then it has the
> exclusive rights to the config and can happily edit & commit the config.
> It would be confusing to error on the commit (due to locking) after having
> accepted a lock.
>
> Similarly for the reverse situation -> if a client takes a candidate
> <lock> then couldn’t it assume that nobody can subsequently block their
> commit by taking a lock of the running DS ?
>
> It seems like the server should manage this interaction and avoid giving
> an “OK” to clients taking the two locks on the two datastores.
>
> On the other hand this could all be left up to the clients to deal with.
> Clients would have to see that a server has both a writeable-running and a
> candidate DS and take a lock of both.   But that would require all clients
> to have that behavior (instead of the server that supports the 2 DSes
> dealing with this).  It also has race conditions where the client takes 1
> lock but someone else gets the second lock.
>
> Opinions appreciated.
> Regards,
> Jason
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>
>

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 14 Jul 2016 13:00:35 -0700

I see clients acquiring lock on the candidate config but not on running. You are right that it does not prevent, say RESTCONF or even CLI from updating the running-config directly.

A workflow for what should be happening is:

- acquire lock on candidate and running (can create deadlock as Andy mentions)
- edit the candidate config
- validate candidate config
- commit candidate to running
- release lock on candidate and running

> On Jul 14, 2016, at 12:03 PM, Sterne, Jason (Nokia - CA) <[email-masked]> wrote:
> 
> Hi all,
>  
> I’m looking for opinions on how a client & server might interact for the <lock> RPC when a server supports both writable-running and candidate DS.  
>  
> The spirit of a <lock> is to reserve exclusive access to alter the configuration of the device, and to prevent other clients (or management interfaces) from altering the config in the meantime.  I realize that in theory the locks are “per-DS” (i.e. independant) but I think that might be too simplistic for a system with both writable-running and candidate.
>  
> Similarly for the reverse situation -> if a client takes a candidate <lock> then couldn’t it assume that nobody can subsequently block their commit by taking a lock of the running DS ?
>  
> It seems like the server should manage this interaction and avoid giving an “OK” to clients taking the two locks on the two datastores.
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/netconf <https://www.ietf.org/mailman/listinfo/netconf>
Mahesh Jethanandani
[email-masked]

---

From: "Sterne, Jason (Nokia - CA)" <[email-masked]>
Date: Thu, 14 Jul 2016 20:30:16 +0000

Thx Andy.  BTW -> This isn’t a purely academic question.  Nokia SR OS supports both writable-running and candidate.  And yes it is complex to manage.  The two datastores can’t be simply treated independently and some locking/coherency mechanisms are needed between them.   But I think some of the same challenges exist for implementations that do “private” candidates (in CLI or NETCONF).

I agree with the deadlock (that was my last point).  That is one reason why it may make more sense to implicitly grab both locks on the server side when the 1st one is asked for (i.e. reject the 2nd client’s lock request) instead of leaving this to the client side.

But despite the challenges/disaster do you think it makes sense (and may almost be necessary) for the server to effectively grant both locks when either single one is taken ?

Jason

From: Netconf [mailto:[email-masked]] On Behalf Of Andy Bierman
Sent: Thursday, July 14, 2016 15:14
To: Sterne, Jason (Nokia - CA)
Cc: Netconf
Subject: Re: [Netconf] <lock> in a server that supports both writable-running and candidate

Hi,

IMO NETCONF locks are not well designed at all.
For example, you are not even allowed to lock multiple datastores at once.
You are forced to lock 1 at a time, and if 2 clients are doing this at the same time,
it can lead to a deadlock between them (Each 1 holding 1 lock waiting for the other)

The shared candidate is sort of a disaster but if you want to use it
then you will need to invent your own "git update".  The reason nobody implements
:candidate and :writable-running together is because there is no way to synch
the candidate if running has been changed underneath it.

A concurrent write database is not "first writer wins, everybody else blows up".

Andy

On Thu, Jul 14, 2016 at 12:03 PM, Sterne, Jason (Nokia - CA) <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

Opinions appreciated.
Regards,
Jason

_______________________________________________
Netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf

---

From: "Sterne, Jason (Nokia - CA)" <[email-masked]>
Date: Thu, 14 Jul 2016 20:34:04 +0000

Hi Mahesh,

I’m not sure I understand your first statement.  Can you explain further ?  Are you saying that on a system that supports both writable-running and candidate you think clients may just take the lock on the candidate, do their updated, commit and then release the lock ?   I think that is reasonable/possible.

As for preventing other mgmnt. interfaces from interfering -> I think RFC6241 is reasonably clear that should be prevented:

Additionally, the system
      will ensure that these locked configuration resources will not be
      modified by other non-NETCONF management operations such as SNMP
      and CLI.

I agree with the general workflow but the question is whether the client should explicitly get both locks, or whether the server side should implicitly take both locks (i.e. prevent separate modifications/lock of the other DS).  It isn’t obvious but I lean towards the server side handling this situation (especially since the combination of writeable-running and candidate support is somewhat rare).

Jason

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Thursday, July 14, 2016 16:01
To: Sterne, Jason (Nokia - CA)
Cc: Netconf
Subject: Re: [Netconf] <lock> in a server that supports both writable-running and candidate

On Jul 14, 2016, at 12:03 PM, Sterne, Jason (Nokia - CA) <[email-masked]<mailto:[email-masked]>> wrote:

Hi all,

Opinions appreciated.
Regards,
Jason

Mahesh Jethanandani
[email-masked]<mailto:[email-masked]>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 14 Jul 2016 14:12:57 -0700

Hi,

I do not like the server taking out locks on behalf of a client.
The client does not know this is happening and will not release the locks.

In general (in traditional NM) the server NEVER guesses about the
NMS intentions.  The server does what it is told.

The server should not create NP containers if default-operation=none.
If the client wanted that, it would pick 'merge'.

Andy

On Thu, Jul 14, 2016 at 1:30 PM, Sterne, Jason (Nokia - CA) <
[email-masked]> wrote:

> Thx Andy.  BTW -> This isn’t a purely academic question.  Nokia SR OS
> supports both writable-running and candidate.  And yes it is complex to
> manage.  The two datastores can’t be simply treated independently and some
> locking/coherency mechanisms are needed between them.   But I think some of
> the same challenges exist for implementations that do “private” candidates
> (in CLI or NETCONF).
>
>
>
> I agree with the deadlock (that was my last point).  That is one reason
> why it may make more sense to implicitly grab both locks on the server side
> when the 1st one is asked for (i.e. reject the 2nd client’s lock request)
> instead of leaving this to the client side.
>
>
>
> But despite the challenges/disaster do you think it makes sense (and may
> almost be necessary) for the server to effectively grant both locks when
> either single one is taken ?
>
>
>
> Jason
>
>
>
>
>
> *From:* Netconf [mailto:[email-masked]] *On Behalf Of *Andy
> Bierman
> *Sent:* Thursday, July 14, 2016 15:14
> *To:* Sterne, Jason (Nokia - CA)
> *Cc:* Netconf
> *Subject:* Re: [Netconf] <lock> in a server that supports both
> writable-running and candidate
>
>
>
> Hi,
>
>
>

---

From: "Sterne, Jason (Nokia - CA)" <[email-masked]>
Date: Thu, 14 Jul 2016 21:41:29 +0000

Thx.

The approach I was proposing was not that the server takes the lock explicitly but that it takes it implicitly.  It would reject an attempt by a client to take the other lock, but not require the client to release any 2nd lock (since it didn’t take it).   The implicit lock on the 2nd DS would be automatically released (just like it was automatically taken).

I think that is more in the spirit of what the client is trying to tell the server.  The client (by taking the <candidate> lock for example) wants to make sure it has exclusive access to modify the config.   The alternative is that the client gets a lock and then could suddenly be blocked (by another lock) from updating the config.  That seems to defeat the purpose/use of the locks.

Regards,
Jason

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, July 14, 2016 17:13
To: Sterne, Jason (Nokia - CA)
Cc: Netconf
Subject: Re: [Netconf] <lock> in a server that supports both writable-running and candidate

Hi,

Andy

On Thu, Jul 14, 2016 at 1:30 PM, Sterne, Jason (Nokia - CA) <[email-masked]<mailto:[email-masked]>> wrote:
Thx Andy.  BTW -> This isn’t a purely academic question.  Nokia SR OS supports both writable-running and candidate.  And yes it is complex to manage.  The two datastores can’t be simply treated independently and some locking/coherency mechanisms are needed between them.   But I think some of the same challenges exist for implementations that do “private” candidates (in CLI or NETCONF).

Jason

From: Netconf [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Andy Bierman
Sent: Thursday, July 14, 2016 15:14
To: Sterne, Jason (Nokia - CA)
Cc: Netconf
Subject: Re: [Netconf] <lock> in a server that supports both writable-running and candidate

Hi,

Andy

Opinions appreciated.
Regards,
Jason

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 14 Jul 2016 14:53:27 -0700

Jason,

> On Jul 14, 2016, at 1:34 PM, Sterne, Jason (Nokia - CA) <[email-masked]> wrote:
> 
> Hi Mahesh,
>  

What I was stating was a statement of fact. Here is a typical transaction that I see. Notice, no effort to lock the running configuration.

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="2">
  <lock>
    <target>
      <candidate/>
    </target>
  </lock>
</rpc>

<rpc-reply message-id=“2"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="3">
  <edit-config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
    <target>
      <candidate/>
    </target>
    <test-option>test-then-set</test-option>
    <error-option>rollback-on-error</error-option>
    <config>
      … config stuff
    </config>
  </edit-config>
</rpc>

<rpc-reply message-id="3"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="4">
  <commit>
    <confirmed/>
  </commit>
</rpc>

<rpc-reply message-id="4"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="5">
  <commit/>
</rpc>

<rpc-reply message-id="5"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="6">
  <unlock>
    <target>
      <candidate/>
    </target>
  </unlock>
</rpc>

<rpc-reply message-id="6"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="7">
  <close-session/>
</rpc>

> On Jul 14, 2016, at 12:03 PM, Sterne, Jason (Nokia - CA) <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi all,
>  
> [email-masked] <mailto:[email-masked]>
Mahesh Jethanandani
[email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Thu, 14 Jul 2016 15:07:25 -0700

On Thu, Jul 14, 2016 at 2:53 PM, Mahesh Jethanandani <
[email-masked]> wrote:

> I’m not sure I understand your first statement.  Can you explain further
> ?  Are you saying that on a system that supports both writable-running and
> candidate you think clients may just take the lock on the candidate, do
> their updated, commit and then release the lock ?   I think that is
> reasonable/possible.
>
>
> What I was stating was a statement of fact. Here is a typical transaction
> that I see. Notice, no effort to lock the running configuration.
>
>

>From RFC 6241, 8.2.4.1: <commit>

         If the running or candidate configuration is currently locked
         by a different session, the <commit> operation MUST fail with
         an <error-tag> value of "in-use".

Andy

> As for preventing other mgmnt. interfaces from interfering -> I think
> RFC6241 is reasonably clear that should be prevented:
>
> Additionally, the system
> I agree with the general workflow but the question is whether the client
> should explicitly get both locks, or whether the server side should
> implicitly take both locks (i.e. prevent separate modifications/lock of the
> other DS).  It isn’t obvious but I lean towards the server side handling
> this situation (especially since the combination of writeable-running and
> candidate support is somewhat rare).
>
> Jason
>
> *From:* Mahesh Jethanandani [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Thursday, July 14, 2016 16:01
> I see clients acquiring lock on the candidate config but not on running.
> You are right that it does not prevent, say RESTCONF or even CLI from
> updating the running-config directly.
>

---

From: "Sterne, Jason (Nokia - CA)" <[email-masked]>
Date: Fri, 15 Jul 2016 18:48:04 +0000

Hi Mahesh,

“Typical transaction” with what NC client and what NC server ?

Did the NC server advertise both writable-running and candidate capabilities ?

If a server only advertises candidate (and not writable-running) then I’d only expect the client to ever lock the candidate.

Jason

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Thursday, July 14, 2016 17:53
To: Sterne, Jason (Nokia - CA)
Cc: Netconf
Subject: Re: [Netconf] <lock> in a server that supports both writable-running and candidate

Jason,

On Jul 14, 2016, at 1:34 PM, Sterne, Jason (Nokia - CA) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mahesh,

Additionally, the system

Jason

Hi all,

Opinions appreciated.
Regards,
Jason

Mahesh Jethanandani

Mahesh Jethanandani

---

From: Andy Bierman <[email-masked]>
Date: Fri, 15 Jul 2016 12:06:25 -0700

On Fri, Jul 15, 2016 at 11:48 AM, Sterne, Jason (Nokia - CA) <
[email-masked]> wrote:

> If a server only advertises candidate (and not writable-running) then I’d
> only expect the client to ever lock the candidate.
>
>
>

No features are required to lock running.
The running datastore must exist.
It can be locked for reading by another client.
(e.g., the client wants a stable snapshot of the config)

The section on <commit> clearly states that the commit will
fail if another session has locked the candidate or running datastores.

Andy

> *From:* Mahesh Jethanandani [mailto:[email-masked]]
> *Sent:* Thursday, July 14, 2016 17:53