Subject: [Netconf] Candidate config for big multiuser nodes - is it usable?

From: Balazs Lengyel <[email-masked]>
Date: Wed, 8 May 2013 16:26:44 +0200

Hello,
How do you use candidate config with big, multiuser nodes?
In these big nodes you can not lock the full running configuration for 
any reasonable time, as other operators might want to configure 
different parts of the nodes configuration.
The normal operation would be
1) copy running to candidate
2) partially lock the running config (and the candidate config) to cover 
my interesting parts
3) edit the candidate
4) commit the candidate
5) release the locks

There are multiple problems with this:
A) if someone else updates another part of the configuration with 
perfectly OK and independent configuration, the commit operation 
overwrites these changes. While the locks might be partial, but the 
commit according to the  RFC is global. This effectively rules out using 
the candidate configuration if you must allow parallel configuration by 
multiple users.
Is this true?
What do you do in your implementation to avoid this problem?
On the CLI configure does not have to be exclusive, but do you have 
something similar on Netconf?

B) If you don't have access rights to parts of the configuration you 
will not be allowed to commit. True?
regards Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
System Manager
ECN: 831 7320                        Tel: +36-1-437-7320
Mobile: +36-70-330-7909              email: [email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Wed, 8 May 2013 08:17:40 -0700

Hi,

The shared candidate config has known problems.
The main problem is that a shared candidate
doesn't work well with multiple writers.

The candidate works well when a writer takes out global locks and holds
them until the [confirmed] commit is done.  Then the next writer can
do the same.  This probably isn't the definition of "shared" you had in
mind.

Concurrent transactions are way outside the scope of the standard.

Andy

On Wed, May 8, 2013 at 7:26 AM, Balazs Lengyel
<[email-masked]>wrote:

> In these big nodes you can not lock the full running configuration for any
> reasonable time, as other operators might want to configure different parts
> of the nodes configuration.
> perfectly OK and independent configuration, the commit operation overwrites
> these changes. While the locks might be partial, but the commit according
> to the  RFC is global. This effectively rules out using the candidate
> configuration if you must allow parallel configuration by multiple users.
> Is this true?
> B) If you don't have access rights to parts of the configuration you will
> not be allowed to commit. True?
> regards Balazs
>
> --
> ______________________________**_________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/netconf<https://www.ietf.org/mailman/listinfo/netconf>
>

---

From: Kent Watsen <[email-masked]>
Date: Wed, 8 May 2013 18:22:38 +0000

Hi Balazs,

Another solution is to define a capability for private candidate datastores (per-session, not per-user).  This is what a couple Juniper products do.  Recall, NETCONF is an extensible protocol, so Ericsson can define a proprietary capability for this.  Alternatively, if interested, I'd be willing to co-author an I-D with you, maybe it would get picked up by the WG…

This is essentially the same as transactions Andy mentioned, only how it's presented seems to be different.  I like private candidate datastores more only because it's more like existing NETCONF – even :confirmed-commit carries over…

Thanks,
Kent

From: Andy Bierman <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, May 8, 2013 11:17 AM
To: Balazs Lengyel <[email-masked]<mailto:[email-masked]>>
Cc: NetConf <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Netconf] Candidate config for big multiuser nodes - is it usable?

Hi,

The candidate works well when a writer takes out global locks and holds
them until the [confirmed] commit is done.  Then the next writer can
do the same.  This probably isn't the definition of "shared" you had in mind.

Andy

On Wed, May 8, 2013 at 7:26 AM, Balazs Lengyel <[email-masked]<mailto:[email-masked]>> wrote:
Hello,
How do you use candidate config with big, multiuser nodes?
In these big nodes you can not lock the full running configuration for any reasonable time, as other operators might want to configure different parts of the nodes configuration.
The normal operation would be
1) copy running to candidate
2) partially lock the running config (and the candidate config) to cover my interesting parts
3) edit the candidate
4) commit the candidate
5) release the locks

There are multiple problems with this:
A) if someone else updates another part of the configuration with perfectly OK and independent configuration, the commit operation overwrites these changes. While the locks might be partial, but the commit according to the  RFC is global. This effectively rules out using the candidate configuration if you must allow parallel configuration by multiple users.
Is this true?
What do you do in your implementation to avoid this problem?
On the CLI configure does not have to be exclusive, but do you have something similar on Netconf?

B) If you don't have access rights to parts of the configuration you will not be allowed to commit. True?
regards Balazs

--
Balazs Lengyel                       Ericsson Hungary Ltd.
System Manager
ECN: 831 7320                        Tel: +36-1-437-7320
Mobile: +36-70-330-7909              email: [email-masked]<mailto:[email-masked]>

_______________________________________________
Netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf

---

From: Wes Hardaker <[email-masked]>
Date: Mon, 13 May 2013 12:38:29 -0700

Kent Watsen <[email-masked]> writes:

> Another solution is to define a capability for private candidate
> datastores (per-session, not per-user).  This is what a couple Juniper
> products do.  Recall, NETCONF is an extensible protocol, so Ericsson
> can define a proprietary capability for this.  Alternatively, if
> interested, I'd be willing to co-author an I-D with you, maybe it
> would get picked up by the WG…

That's definitely better in terms of making sure you don't stomp each
other during editing, but it still lets you accidentally wipe out
changes that other operators made because of when you started.  Consider
the following time table:

  | time | operator 1                  | operator 2                          |
  |------+-----------------------------+-------------------------------------|
  | T0   | copies running to O1-config |                                     |
  | T1   |                             | copies running to O2-config         |
  | T2   | Modifies O1-config.foo      | Modifies O2-config.bar              |
  | T3   | Commits O1-config           |                                     |
  | T4   |                             | Commits O2-config                   |
  |      |                             | (which drops the O1-config.foo mod) |

Netconf's commit/copy mechanisms between global copies aren't safe for
multi-user use.  So make sure you lock running during the copy-away and
back.  You need some mechanism to signal that you need to be the only
writer for a while.  But that should work as long as you're noting doing
so many config changes at once that users will need to wait a long time.

Note, though, that with the copy-to/from-URL URL feature you can already
have separate candidate configs essentially.  You don't need a
vendor-feature to achieve it (though you may want to anyway).

-- 
Wes Hardaker
Parsons

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 May 2013 13:45:24 -0700

>   | time | operator 1                  | operator 2

There are ways for the server to figure out that this is not a collision.
If they both wrote config.bar, that would be a collision.  The transaction
commit procedure does not exactly follow RFC 6241 or else the deletion
of edits from other clients would occur.  It's basically the same except
the server keeps track of the changed data, etc.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Mon, 13 May 2013 22:42:02 +0000

From:  Andy Bierman <[email-masked]>
> the server keeps track of the changed data, etc.

Exactly, the <commit> from the 2nd client could return an <rpc-error>
indicating that a conflict was detected, so the 2nd client can retry after
resynchronizing itself to the device's new configuration.

Thanks,
Kent

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 14 May 2013 06:27:23 -0700

> resynchronizing itself to the device's new configuration.

But that's not something that is standardized.  And it throws into
question whether operator1 actually wanted to revert operator2's changes
in the first place.

Yes, there are many hacks you can (and maybe some do) put in place to
detect, track, etc these changes.  But how is a management station
framework supposed to be written based on the protocol itself?  Is it
supposed to detect that this device is a FooBar version 2, which does
conflict detection?  I sure hope those features are advertised in the
hello exchange.

This just smells like a need for standardization before it gets out of
hand if different companies are taking different approaches about what
to do in the case of conflicts.
-- 
Wes Hardaker
Parsons

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 May 2013 06:48:30 -0700

> question whether operator1 actually wanted to revert operator2's changes

If operator1 wants to delete /foo, then an explicit edit
within their transaction is required.  The reason shared candidate
is useless for multiple writers is that there are implicit actions
all over the place.
  - a missing node is an implicit delete
  - an unchanged user-ordered list becomes an implicit
    request to re-order the list
  - a commit operation implicitly acts as the confirming commit
    for another operator's changes

Collision detection works better if you compare only explicit changes.

> conflict detection?  I sure hope those features are advertised in the

If a request fails, the client gets an <rpc-error> back.
It's not as if the handful of standard operations and data
in NETCONF somehow eliminated the need for vendor-specific features.

> to do in the case of conflicts.

I don't see any problem with vendors figuring out what works best.
Concurrent transactions are fairly worthless if the minute you start
using them, first writer wins and everything else is rejected.
The NETCONF WG is busy working on other things anyway.

--

Andy

---

From: Xiang Li <[email-masked]>
Date: Tue, 14 May 2013 09:03:53 -0500

On 5/14/2013 8:27 AM, Wes Hardaker wrote:
> Kent Watsen<[email-masked]>  writes:
>
agreed
> to do in the case of conflicts.
+1

Without this clarified/standardized,   a client must first lock 
candidate during the whole
process of editing candidate/committing, otherwise things won't work as 
expected when
there are multiple clients competing editing. That might well work in 
the real word
and perhaps it is the intended way? Because RFC6241 says:

The candidate configuration can be shared among multiple sessions.
    Unless a client has specific information that the candidate
    configuration is not shared, it MUST assume that other sessions are
    able to modify the candidate configuration at the same time.  It is
    therefore prudent for a client to lock the candidate configuration
    before modifying it.

--Xiang

---

From: Xiang Li <[email-masked]>
Date: Tue, 14 May 2013 09:11:13 -0500

On 5/14/2013 8:48 AM, Andy Bierman wrote:
>
>
> On Tue, May 14, 2013 at 6:27 AM, Wes Hardaker <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Kent Watsen <[email-masked] <mailto:[email-masked]>> writes:
>

I could not  find any texts in RFC6241 that require a server to do 
collision
detection.

We sure can make NETCONF server to do version control like CVS/SVN/Git but
that's not how the current NETCONF is define , and whether we need that is
IMO debatable.

+1

--Xiang

> https://www.ietf.org/

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 May 2013 07:22:22 -0700

On Tue, May 14, 2013 at 7:11 AM, Xiang Li <[email-masked]> wrote:

> On Tue, May 14, 2013 at 6:27 AM, Wes Hardaker <[email-masked]>wrote:
>
We are talking about proprietary features, so not surprising that
there is no text in RFC 6241 about it.

It works like any other proprietary NETCONF extension.
If the client understands the advertised capability and knows
how to use it, then what's the problem?

> that's not how the current NETCONF is define , and whether we need that is
These are implementation details.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Tue, 14 May 2013 17:24:32 +0000

As Andy notes, this is a proprietary extension in Juniper's implementation to get around some of the issues in NETCONF's shared candidate model.  There is no standard yet, but I'm willing to submit an I-D and see if there is any WG interest…

Thanks,
Kent

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 31 May 2013 15:42:34 +0200

Hello kent,

Sorry for droping ut for so long.

I Ericsson we are interested in this problem. As I know Tailf also
has some kind of solution for concurent editing my multiple parties.

regards Balazs

On 2013-05-14 19:24, Kent Watsen wrote:

As Andy notes, this is a proprietary extension in Juniper's
implementation to get around some of the issues in NETCONF's
shared candidate model.  There is no standard yet, but I'm
willing to submit an I-D and see if there is any WG interest…

Thanks,

Kent