Subject: [Netconf] partial-lock and the candidate config

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Apr 2009 09:59:17 -0700

Hi,

When a partial-lock is granted, it affects all the
operations [2.4.1, para 2; 2.5, para 1]

sec 2.1.1.3:
   - missing this sentence from previous section, which
     also applies the a shared candidate:

    This scenario assumes, that the global lock operation from [NETCONF]
    is not used.

The partial-lock operation leads to a deadlock scenario
that Wes has described many times wrt/ global locking.
Except with partial-locking, the recovery is much more severe.

If session A has access rights and a partial-lock
on /foo and session B has access rights and a partial-lock
on /bar, then neither one of them can issue a commit,
and neither one can issue a discard-changes.

The candidate config is dead-locked until one session
releases all its partial-locks and one set of edits
to the candidate config are 'undone'.

But what if that doesn't happen?
What if session A issues a kill-session on the other
one instead?  Or session B terminates for any reason?

The partial-locks for /bar will be released OK,
but the discard-changes will fail because of the active
partial-lock on /foo, owned by session A.

So session A is left with one option.
Issue a discard-changes (which reverts /bar -- but
session A has no access rights to /bar?!)
It is not allowed to alter /bar, so it has to toss
out its own changes to /foo, and start over.
And session A has no idea what else changed in the candidate.
It just knows its discard-changes and commit requests
are failing.

I'm not convinced this is better than just holding a
global lock for the duration of an edit to
the candidate config.

Andy

---

From: WashamFan <[email-masked]>
Date: Tue, 14 Apr 2009 11:18:37 +0800

>  and neither one can issue a discard-changes.

Yep. That is where deadlock happens, and the reason is
absence of partial-[op], IMO. candidate datastore is mostly
a one-session-at-a-time basis.

>  It just knows its discard-changes and commit requests

With recovery, I think it is more an access control issue than
partial-lock issue. 
Given example:

1) session B takes out a global lock, and then terminates
by any reason or releases the lock without <commit>. It leads to
oustanding changes on /bar.

2) session A takes out a global lock( I am not sure whether a session
without a global access can acquire a global lock, but I assume it can),
for the same token you mentioned, session A will fail <commit> and
<discard-changes> as no access right to /bar.

So can we say anyone without access rights to the whole candidate
datastore would encounter some trouble issuing <commit> and <discard
-changes>?

washam

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Apr 2009 00:10:20 -0700

> a one-session-at-a-time basis.

It is not a matter of opinion.
It is simple engineering.

> oustanding changes on /bar.

why is there any problem with global lock?
If only one session has altered the candidate,
then its discard-changes (triggered by the dropped
session) will work fine.  Please re-read all the relevant
text in partial-lock and 4741.

> datastore would encounter some trouble issuing <commit> and <discard

I think analysis of the current text is quite simple.
The partial-lock operation does not provide any actual
protection for multi-session writes to the candidate config.
This is database-201 basic stuff here.

Andy

---

From: WashamFan <[email-masked]>
Date: Tue, 14 Apr 2009 15:52:17 +0800

Hi,
 
>  text in partial-lock and 4741.

What the example is expected to express is, whatever is used, global
lock or partial lock, recovery issue will raise.

When only one session at a time accesses candidate, it will probably still deal
with 'undone' edits left by another sessions, if it didn't have enough rights,
<discard-changes> and <commit> will fail, too. Am I misunderstand?

washam

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 14 Apr 2009 10:37:10 +0200 (CEST)

> oustanding changes on /bar.

No, b/c when B terminates with an outstanding lock, the canididate is
reverted, see 8.3.5.2 of RFC 4741.

> <discard-changes> as no access right to /bar.

Also, a session cannot take a global lock if there are any
outstanding, uncommitted modifications to the candidate (see 7.5 of
RFC 4741).

But your point can be made w/o any locks at all.  Suppose session A
modifies /foo in the candidate w/o a lock.  Next, B modifies /bar in
the candidate w/o a lock.  Then B terminates.  A will not be able to
commit or discard-changes.  The solution is to use locks.

If the conclusion is that b/c of this partial-lock should not be
allowed on the candidate, maybe we should also disallow edit-config of
the candidate unless the global lock is set?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Apr 2009 02:04:22 -0700

> the candidate unless the global lock is set?

IMO, the conclusion is that partial-locking on the candidate
works just as badly as no locking at all.  That does not
lead to any logical deduction about global locking.
That works on the candidate just fine.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 14 Apr 2009 11:08:45 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> >> partial-lock issue. Given example:
> >>
> works just as badly as no locking at all.

Ok, so should we disallow both partial-lock and no-lock of candidate?

> That does not

Yes, global lock on candidate works fine.

/martin

---

From: WashamFan <[email-masked]>
Date: Tue, 14 Apr 2009 17:54:46 +0800

>  Ok, so should we disallow both partial-lock and no-lock of candidate?

Or we just state clearly how difficult recovery would be when partial-lock and
no-lock apply to candidate, and recommend global lock is always used to
candidate?

washam

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Apr 2009 07:03:08 -0700

> Ok, so should we disallow both partial-lock and no-lock of candidate?

No.
We do not mandate that locking be used or how it is used.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Apr 2009 14:18:31 -0700

> no-lock apply to candidate, and recommend global lock is always used to

I have a solution proposal, but it requires that we
plan out some NETCONF architecture, build out
some core infrastructure, and horrible stuff
like that.

Way back when I realized partial-lock and <candidate> config
don't get along very well, I proposed a "group lock".
This of course assumes there is some /groups data structure
somewhere, to assign user names to groups.  That sure would
be useful for access control, though.

After that (there's more?!) the global <lock> operation
needs to be augmented to allow a <group> parameter.
Then, any session associated with a user name that belongs to
the group that owns the lock can issue edit-config,
commit, discard-changes, unlock, partial-lock, and
partial-unlock.  The sessions can use partial-lock
if they want, but the global lock is needed to keep
out all the sessions that are not part of
the 'co-operating managers' in this use-case.

E.g.,

   <rpc>
     <lock>
       <target><candidate/></target>
       <group>grp:admin</group>
     </lock>
   </rpc>

IMO, this is a lot simpler than explicit XPath locks,
and it requires hardly any YANG to implement:

module groups {

   namespace foo;
   prefix grp;
   ...

   typedef NetconfGroup {
     description "A NETCONF group identifier.";
     type identityref {
        base netconfGroups;
     }
   }

   identity netconfGroups {
     description "NETCONF groups base";
   }

   identity admin {
     description "example group definition";
     base netconfGroups;
   }

   container groups {
     presence "If empty, then no groups are configured.";

     list group {
        description "One NETCONF group definition.";
        key groupId;
        leaf groupId {
          description "Identity naming this entry.";
          type NetconfGroup;
        }
        leaf-list userName {
           description
             "User name string belonging to this group";
           type string {
             length "1..max";   // may want restrictions
           }
        }
     }
   }
}

NETCONF access control and co-operative locking could use
the same data structures, assuring that alignment and
simplicity are maintained.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 14 Apr 2009 23:02:17 -0400

>a one-session-at-a-time basis.

Even for writable-running, partial locks are problematic.  The
assumption seems to be that an application will grab a lock for the
length of the application's lifetime, not a short-term more atomic
lifespan.

This means that simple tasks like config archival are not possible,
since an archival app can't grab a lock on the entire config if a
long-standing partial lock exists.  Without a lock, the database
can continue to change under a reader, so any archive would not be
guaranteed to be consistent.

The same problem exists (with more serious consequences) if two
apps lock different parts of the config on a box that has
distinct-startup, where either app doing a copy-config from running
to startup would never be sure that the config isn't being changed.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Apr 2009 23:27:07 -0700

> to startup would never be sure that the config isn't being changed.

But at least this partial/global lock overlap is
controllable by the operator.  'lock time' is just
another resource to manage, like bandwidth.

My concern is 'transaction framing'.
With the candidate config, there is no 'lock, edit, commit or discard'
transaction-like procedure for partial-lock, like there is for global
lock.

It's like trying to take a photo of your family in
a crowded public square.  No matter how hard you
try, there are all these strangers (not paying attention
to you) walking through your shot.  Doesn't bother
them at all, but you never get to take your photo.

(If not using the partial-lock at all produces the same results
as if partial-locking is used correctly, then the partial lock
doesn't actually work.)

Andy

---

From: WashamFan <[email-masked]>
Date: Wed, 15 Apr 2009 17:36:44 +0800

>  simplicity are maintained.

Users in a group could share a lock, right? And all the users are
expected to be cooperating with each other to achieve a final goal,
right? But there is no nested lock to constraint the behavior of a single
user within the group, so how to cope with edits issued by different
users simultaneously, are we introducing an additional mechainsm to
ensure users in a group will edit in a reasonable order? It seems not
enough convincing to me.

washam

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Apr 2009 07:40:14 -0700

> ensure users in a group will edit in a reasonable order? It seems not

The co-operating managers using the candidate config could
use partial-locking within the global group lock if they wanted.
IMO, this is mostly pointless, since the entire candidate
is committed at once, so if the co-operating managers step
on each other, there is a bug in their code.

The agent does not have to ensure anything here wrt/
order or keeping the managers from making mistakes.
They are using the scratchpad database.  It can be
inspected, validated, and altered before the commit.

Locking is available for managers to use, misuse, or ignore.
Responsibility for coordinated database edits lies entirely
with the application(s), not the agent.

Andy

---

From: WashamFan <[email-masked]>
Date: Thu, 16 Apr 2009 10:50:14 +0800

Hi,

>  on each other, there is a bug in their code.

OK. A partial lock operation MUST fail if any session owns the
global lock ( I assume global group lock is still a global lock). So
how to use partial-locking within the global group lock according
to current text?

lock is to pretect one session from intervention of another one
during a (short) period of time. If multiple sessions share a global
group lock at the same time (even partial lock is allowed), difficult
recovery is still there.

Some issues are unclear about the proposal, such as, 
how to define a group? 
Are all users within a group has the same or different privilege?
Are you also introduing the concept of 'role' versus 'group'?
who to grab a group lock, any user or specific one within the group? 
Who to release a group lock, any user or the one who locked before?
But, maybe, there may be a draft about it some day.

so far, we see benefits from 1) applying lock to a single session. And
you see benifits from applying lock to 2) multiple users with multiple 
sessions. Anyone see benefits from 3) applying lock to a single user
with multiple sessions? Maybe 3) is a special case of 2) ;)

washam

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Apr 2009 20:10:57 -0700

> how to use partial-locking within the global group lock according

OK -- then I wouldn't change that, since using a partial-lock
within a shared global lock isn't very useful anyway.

> group lock at the same time (even partial lock is allowed), difficult

Why?
This assumes some problems with access-control?
If not, then discard-changes is the recovery mechanism, as usual.

> Some issues are unclear about the proposal, such as, 

with <edit-config>, using the YANG model above. e.g.:

   <groups>
     <group>
        <groupId>grp:admin</groupId>
        <userName>fred</userName>
        <userName>barney</userName>
     </group>
     <group>
        <groupId>acme:acme-admin</groupId>
        <userName>fred</userName>
        <userName>wilma</userName>
        <userName>betty</userName>
     </group>
   </groups>

> Are all users within a group has the same or different privilege?

that's an out-of-scope access control question.

> Are you also introduing the concept of 'role' versus 'group'?

no -- I don't actually see the word 'role' anywhere.

> who to grab a group lock, any user or specific one within the group? 

anyone -- good point: it is an error to request
a group lock if your user name is not in the specified group.
It is totally out of scope how user names get discovered
from the transport layer for each session.

> But, maybe, there may be a draft about it some day.

anyone in the group can release the lock.
maybe a draft someday.  I am more interested
in getting access control, a system group,
an interfaces table, and content like that first.

Andy

---

From: WashamFan <[email-masked]>
Date: Thu, 16 Apr 2009 14:18:02 +0800

>  that's an out-of-scope access control question.

Imagine session A with only access right to /foo and session B with only access
right to /bar are in the same group. session A has not enough authority to
discard 'undone' changes left by session B. That is why I ask are all users
within a group has the same privilege, and say difficult recovery is still there.
When a single session owns a global lock, it can always discard-changes made
by itself, but when a group of sessions share a global lock, difficult recovery arise
again.

washam

---

From: Andy Bierman <[email-masked]>
Date: Sun, 19 Apr 2009 09:13:14 -0700

> by itself, but when a group of sessions share a global lock, difficult recovery arise

This deadlock scenario (first described by Wes) has always been
a possibility for the candidate config.  The only way to prevent
it is to use the global lock operation.  Partial-lock does not
address this problem at all.  (When this work was first chartered,
I strongly approved because I thought it was going to address
the issues raised by Wes.)

You are correct in pointing out that the group lock really depends
on the access control model used in conjunction with locking.
If the ACM assigns access rights to groups instead of directly
to users, then this group lock will work fine.

There is no standard access control. There are no valid
assumptions one can make about how a compliant NETCONF agent
MUST behave wrt/ access control.  Within the NETCONF standard,
if you get a session opened, you have access to everything
and so does everybody else. This candidate config
deadlock is not possible, according to the standard.

Andy