Subject: [netmod] yang-09; 7.19.5 when-stmt

From: Andy Bierman <[email-masked]>
Date: Wed, 09 Dec 2009 10:59:07 -0800

Hi,

Sec. 7.19.4 para 3 does not specify the context of the when-stmt
for a uses-stmt.

I am very concerned about the inherent complexity due to
server implementation of the when-stmt constraints.
The reason must-stmt is not a problem is that it cannot
be inherited from uses or augment statements.

This extreme example show that understanding the
final object definition, and understanding which
when-stmt will be evaluated in which XPath context
can be very difficult.  Implementing the XPath in
a bullet-proof manner is somewhat challenging as well.

    grouping A {
      leaf a1 {
        when "/top-leaf > 4";
        type string;
      }
    }

    grouping B {
      uses A {
        when "../b1";
      }
      leaf b1 { type string; }
      container b2;
    }

    leaf top-leaf {
      type int32;
    }

    container foo {
      uses B {
        when "b2/bar = 'fred'";
        augment b2 {
          when "/top-leaf < 3";
          leaf bar {
            when "../a1 == 'barney'";
            type leafref {
               path "../../top-leaf";
            }
          }
        }
      }
    }

IMO, the when-stmt should be removed from 'uses' and only allowed
for 'augment'.  This would improve interoperability and preserve
the real use-case from SNMP (SPARSE-AUGMENTS).  The complexity
of the ripple effect is not worth the corner-case benefits it
may provide.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 09 Dec 2009 22:00:04 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> for 'augment'.

Just a quick clarifying question.  Do you want to remove 'when' from
'uses' or do you want 'when' to be allowed only in 'augment'?  Note
that you can have 'when' under any container/leaf/...

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 09 Dec 2009 13:33:56 -0800

> that you can have 'when' under any container/leaf/...

Just remove it as a sub-statement of uses-stmt.
This removes the complexity from N levels of inherited uses-when
conditions, with up to N different context nodes.

With this change, we will have all the flexibility
that is really needed, and a constrained XPath model.
Given that when-stmt is evaluated top-down because
if the parent doesn't exist, then no need to check further,
there are at most 2 when-stmts, both with the same context
node as the target node.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 9 Dec 2009 17:41:52 -0500

>conditions, with up to N different context nodes.

I see the uses/when as a key ingredient for the reusability
of groupings.  Please don't remove it.  I don't see it as
adding any sgnificant complexity, since "when" is already
there and the implementations must already deal with it.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 09 Dec 2009 15:32:35 -0800

Since this is so easy, why don't you write the text that
is required to specify the context node for uses-stmt?
This must be done in order to leave uses-when in the draft.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 08:41:43 +0100

> a bullet-proof manner is somewhat challenging as well.

I second this. If an implementation is about to parse an XML document,
it should already know the schema for the document. With 'when', in
order to know whether a given element must, may or mustn't appear in the
document, the document must be parsed first. I believe this is a
substrate for race conditions and should be avoided.

That's why I proposed that grammatical constraints be declared as
independent of any XPath evaluations in 'when' and 'must'. That is, the
datastore schema must be derivable a priori from YANG modules and active
features and must not depend on the datastore contents. 

In my view, the grammar (schema) is formed by all data node definition
statements ('leaf', 'container' etc.) and also by 'mandatory',
'min/maxElements' and 'presence'.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 09:30:27 +0100

On Thu, Dec 10, 2009 at 08:41:43AM +0100, Ladislav Lhotka wrote:
> Andy Bierman p????e v St 09. 12. 2009 v 10:59 -0800:
> > Hi,
> > 

You are trying to convert semantic constraints expressed in XPATH
statements into document grammar rules, which I think is the wrong
approach.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 10 Dec 2009 09:38:14 +0100 (CET)

> This must be done in order to leave uses-when in the draft.

when in uses has the same context node as when in choice, i.e. its
parent.

So we should extend the text in 7.19.5, bullet 2:

OLD:

   o  If the "when" statement is a child of a "choice" or "case"
      statement, then the context node is the closest ancestor node to
      the "choice" or "case" node which is also a data node.

NEW:

   o If the "when" statement is a child of a "uses", "choice", or
     "case" statement, then the context node is the closest ancestor
     node to the "uses", "choice", or "case" node which is also a
     data node.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 09:57:46 +0100

> statements into document grammar rules, which I think is the wrong

Which of the data node definition statements, 'mandatory',
'min/maxElements' and 'presence' uses XPATH constraints?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 10:00:48 +0100

> 'min/maxElements' and 'presence' uses XPATH constraints?

What is the problem with these? The subject is about the 'when-stmt',
no?

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 10:17:45 +0100

> What is the problem with these? The subject is about the 'when-stmt',

Yes, but I want exactly the opposite of what you wrote, namely that
'when' statement must not cancel or relativize any of the above
grammatical statements. For example:

container x {
    leaf y {
        type uint8;
    }
    leaf z {
        mandatory true;
        when "../y > 0";
        type empty;
    }
}

I claim that "<x><y>0</y></x>" is not a valid document, because the
"mandatory true" constraint is not satisfied (parent <x> exists but <z>
doesn't).

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 10:22:32 +0100

On Thu, Dec 10, 2009 at 10:17:45AM +0100, Ladislav Lhotka wrote:
> Juergen Schoenwaelder p????e v ??t 10. 12. 2009 v 10:00 +0100:

Yes - and where is the problem?

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 10:31:50 +0100

> Yes - and where is the problem?

The problem is that Martin does not agree with it.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 10 Dec 2009 10:41:03 +0100 (CET)

> The problem is that Martin does not agree with it.

The current text says, in 8.2 Hierarchy of Constraints:

   "must", "mandatory", "min-elements", and "max-elements" constraints
   are not enforced if the parent node has a "when" or "if-feature"
   property that is not satisfied on the current device.

So, in this case, since the when expression is false, leaf z is not
mandatory.

Mandatory and min/max elements are also semantical constraints, not
grammatical.  These constraints need to be satisfied in a valid
configuration only.  It is perfectly legal for a mandatory leaf to be
absent from the candidate.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 10:52:21 +0100

>    property that is not satisfied on the current device.

Yes, I object to this section, too, but it doesn't apply to the present
example anyway: it is not the parent node that has the 'when' statement.

On the other hand, sec. 7.6.4 says (third bullet):

   o  Otherwise, the leaf MUST exist if the ancestor node exists in the
      data tree.

I see no ifs or whens here.

> absent from the candidate.

I don't agree with this interpretation.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 10:53:48 +0100

> On the other hand, sec. 7.6.4 says (third bullet):

7.6.5, sorry.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 10:55:38 +0100

On Thu, Dec 10, 2009 at 10:41:03AM +0100, Martin Bjorklund wrote:
> > Juergen Schoenwaelder p????e v ??t 10. 12. 2009 v 10:22 +0100:

[Lets leave out candidate since candidate since this scratch buffer
 can hold pretty arbitrary stuff anyway.]

There seems to be a difference between 'if-feature' and 'when' -
'if-feature' is a global property of a session (features are announced
via capabilities during session startup - lets ignore the capability
change discussion for now) while 'when' expressions usually refer to
the content of a datastore and I can see that this indeed makes it
difficult to translate things to other schema languages (except by
dropping all constraints and making everything optional).

I think I do understand the problem now - I do not know what the right
answer is though.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 10 Dec 2009 11:06:05 +0100

> dropping all constraints and making everything optional).

Absolutely, this is exactly my point.

> I think I do understand the problem now - I do not know what the right

I propose these changes:

**** Section 3.1

     OLD
     A mandatory node is one of:

     NEW
     A data node is called mandatory if it satisfies one of the
     following grammatical constraints:

**** Section 8 [New text at the beginning - before 8.1]

     YANG modules specify three types of constraints:

     1. Grammatical constraints define the hierarchical structure
        (schema) of the data tree and ordinality or co-occurence
        conditions on all data nodes.

     2. Data type constraints define the permitted set of values for
        the contents of all leaf nodes.

     3. Semantic constraints use XPath expression [XPath] for defining
        any number of semantic rules and relations among data nodes in
        the entire configuration datastore.

     Grammatical constraints are primary and MUST be unconditionally
     satisfied. Semantic constraints MAY assume that all grammatical
     constraints hold. On the other hand, semantic constraints cannot
     override grammatical constraints - both types are verified
     independently. Therefore, module authors should construct the
     semantic conditions carefully in order to prevent any conflicts
     resulting from contradictory conditions in grammatical and
     semantic constraints.

     Semantic constraints are always verified against the conceptual
     data tree, which is a given configuration datastore complemented
     by all applicable default values that are defined by the YANG
     data model .

**** Section 8.2

     Use it only for 'if-feature' not for 'when'.

**** Section 8.3.3

     OLD
     When datastore processing is complete, the final contents MUST
     obey all validation constraints.

     NEW
     When datastore processing is complete, the final contents MUST
     obey all grammatical, data type and semantic constraints.

     ADD [as the first bullet item]
     o The data tree MUST satisfy all grammatical constraints.

**** Section 7.9.15

     OLD
     The "when" statement makes its parent data definition statement
     conditional.  The node defined by the parent data definition
     statement is only valid when the condition specified by the
     "when" statement is satisfied.  The statement's argument is an
     XPath expression, which is used to formally specify this
     condition.  If the XPath expression conceptually evaluates to
     "true" for a particular instance, then the node defined by the
     parent data definition statement is valid, otherwise it is not.

     NEW
     The "when" statement makes the data node defined by the parent
     statement semantically conditional, which means that the node may
     be present in a valid configuration only if condition specified
     by the "when" statement is satisfied.  The statement's argument
     is an XPath expression, which is used to formally specify this
     condition.  If the XPath expression conceptually evaluates to
     "true" for a particular instance, then the data node defined by
     the parent statement may be present in the instance, otherwise it
     must not be there.

     Note that 'when' statements do not override grammatical
     ordinality of data nodes, which is determined using the
     rules in Section 3.1.

**** Section 7.5.3
     [This subsection should probably be moved to sec. 7.19 because
     'must' is a common statement.]

     OLD
     The "must" statement, which is optional, takes as an argument a
     string which contains an XPath expression.  It is used to
     formally declare a constraint on valid data.  The constraint is
     enforced according to the rules in Section 8.

     NEW
     The "must" statement, which is optional, takes as an argument a
     string which contains an XPath expression.  It is used to
     formally declare a semantic constraint on valid data.  The
     constraint is enforced according to the rules in Section 8.

     Note that semantic constraints expressed by 'must' statements do
     not override any grammatical constraints.

**** Section 7.5.5

     OLD
     The "presence" statement assigns a meaning to the presence of a
     container in the data tree.  It takes as an argument a string
     which contains a textual description of what the node's presence
     means.

     NEW
     The "presence" statement assigns a meaning to the presence of a
     container in the data tree and makes the container optional
     (grammatical constraint).  It takes as an argument a string
     which contains a textual description of what the node's presence
     means.

**** Section 7.6.5

     OLD
     The "mandatory" statement, which is optional, takes as an
     argument the string "true" or "false", and puts a constraint on
     valid data.  If not specified, the default is "false".

     NEW
     The "mandatory" statement, which is optional, takes as an
     argument the string "true" or "false", and puts a grammatical
     constraint on valid data.  If not specified, the default is
     "false".

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 10 Dec 2009 07:31:42 -0800

> I don't agree with this interpretation.

But the candidate is part of NETCONF.
It is just a scratchpad, not a real database.
The YANG constraints apply to the running config for data nodes.

The server must ignore when-stmt in the candidate,
because the spec says it applies to a valid configuration.
By design, that excludes the candidate config.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 10 Dec 2009 08:02:24 -0800

>      node to the "uses", "choice", or "case" node which is also a

This is fine.

For data nodes inside the candidate config,
the when-stmt is completely ignored (except for <validate>).
It only applies to data nodes in a valid configuration,
e.g., applied during the <commit> operation.

The text is not very clear about this point.
False 'when' nodes get deleted during the commit,
not after every <edit-config> to candidate.
This is different than choice-stmt, which causes
a case-stmt to be removed if another case is selected.

The order that when-stmts are evaluated will influence the
outcome of the validation process.  It is rather difficult
to evaluate an arbitrary set of when-stmts all at once.
Expect servers to use a different implementation strategy,
such as a top-down, left-to-right sequence.

I never got an explanation why the description-stmt and
reference-stmt are useful for must-stmt but not when-stmt.
It looks like sloppy work to me, but if there's some
reason these properties apply to 1 kind of XPath expression
but not another one, it would be good to note that in the draft.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 10 Dec 2009 18:20:41 +0100 (CET)

> a case-stmt to be removed if another case is selected.

Actually, the draft says, in section 8.3.2:

   During <edit-config> processing:

   o  If the NETCONF operation creates data nodes under a "choice", any
      existing nodes from other "case" branches are deleted by the
      server.

   o  If the NETCONF operation modifies a data node such that any node's
      "when" expression becomes false, then that node is deleted by the
      server.

So when is like choice in this regard.

> but not another one, it would be good to note that in the draft.

I think we can add these substatemnts to 'when'.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 10 Dec 2009 09:45:40 -0800

> So when is like choice in this regard.

oops -- I even coded it this way (!)
I confused it with deleting empty NP containers,
which I assume is OK to wait until the commit to delete.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 20:12:33 +0100

>       "when" expression becomes false, then that node is deleted by the

This is pretty ambiguous - which node is deleted? Anyway, does the
delete overwrite all other constraints like mandatory or rendering
other must expressions invalid? Or does a server has to detect such
situations before doing the delete and fail the request because of
this?

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 10 Dec 2009 20:46:28 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> This is pretty ambiguous - which node is deleted?

The node with the when expression.

> Anyway, does the
> other must expressions invalid?

Section 8.2 says:

   "must", "mandatory",
   "min-elements", and "max-elements" constraints are not enforced if
   the parent node has a "when" or "if-feature" property that is not
   satisfied on the current device.

> Or does a server has to detect such

Such an auto-delete is handled just like an explict delete by a
client.  If it modifies running, and would result in an invalid
config, it is not executed.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 10 Dec 2009 12:03:56 -0800

> situations before doing the delete and fail the request because of

Any node with a when-stmt (or augment/uses inherited when-stmt(s))
that evaluates to false.  When you delete such a node, it is possible
that a different when-stmt is now false, so there is a ripple effect
to handle in the code.  Of course, this is done before the semantic
constraint validation (must/min/max/unique), since the deleted nodes
may affect those results.

Since the augment and uses 'objects' are not supposedly in the
database, using their 'parent' as the context node is somewhat
confusing, but it is just an implementation detail.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 21:19:59 +0100

> The node with the when expression.

That what I assumed but we should be clear, that is
s/that node/the node with the when expression/
 
> config, it is not executed.

May I assume "it" here means the "request to modify a node such that
other node's "when" expressions became false that lead to an
auto-remove that caused other must constraints to become invalid.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 10 Dec 2009 21:21:41 +0100

> may affect those results.

OK

> confusing, but it is just an implementation detail.

Not sure I follow - where are the augment and uses 'objects' if not in
the datastore?

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 10 Dec 2009 21:47:57 +0100 (CET)

> s/that node/the node with the when expression/

Ok, fixed.

> auto-remove that caused other must constraints to become invalid.

"it" means "an operation".

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 10 Dec 2009 13:14:24 -0800

> Not sure I follow - where are the augment and uses 'objects' if not in

They are in the datastore in the implementation.
But there can never be an instance of these objects,
like with a container or leaf.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 13 Dec 2009 22:28:23 +0100 (CET)

Hi,

> I propose these changes:

[...]

It seems you propose two things:

  1.  Introduce the term "grammatical constraint" and classify
      mandatory and presence as grammatical contraints.

  2.  Say that "when" MUST NOT be present on "mandatory" nodes.

You say that a grammatical constraint MUST be unconditionally
satisfied.  This is not correct.   They do not need to be satisfied in
the candidate or in an edit-config PDU.

As for 2, if that is what we want to do, we should say that
explicitly.  But if we do that, we take away the possibility to do
data models like:

  leaf-list version {
    type enumeratoin {
      enum v1;
      enum v2c;
      enum v3;
    }
  }

  leaf engine-id {
    when "../version = 'v3'";
    mandatory true;
    ...
  }

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 14 Dec 2009 16:20:19 +0100

>       mandatory and presence as grammatical contraints.

Yes, because they can be determined only from the YANG module(s) and
advertised features and put to the schema, unlike 'when' and 'must'.
 
>   2.  Say that "when" MUST NOT be present on "mandatory" nodes.

Such a CLR is not necessary. There can be no race condition or
interoperability problem if someone does so. Right, it doesn't seem to
make much sense, but the semantic constraints can be screwed up in many
ways - in contrary to the grammatical constraints.
 
> the candidate or in an edit-config PDU.

The grammar/schema of candidate or edit-config is certainly different
from that of running or startup.

But actually by "unconditional" I mean that such a constraint is
independent of the instance document and cannot be overriden by the
semantic constraints. Of course, a better adjective may be used.

>     when "../version = 'v3'";

No, we don't:

    must "engine-id and version = 'v3' or "
         + "version != 'v3' and not(engine-id)";
    leaf engine-id {
       ...
    }

(engine-id is not mandatory here)

I understand 'when' only as kind of syntactic sugar that can always be
replaced by a 'must' expression (perhaps more complicated, as it is
here).

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Mon, 14 Dec 2009 10:29:28 -0500

>advertised features and put to the schema, unlike 'when' and 'must'.

I don't follow your meaning here.  Can you elaborate?

>replaced by a 'must' expression (perhaps more complicated, as it is

The "when" version carries more information that your "must" version
and this information is vital to the smooth flow of the apps that will
manipulate the config data.

With the "when" version, my app can "grey out" the engine-id field
unless the version field is set to "v3".  If the user sets version
to "v3", my app can know that engine-id is now mandatory so the
user needs to provide a value.  If the user sets version to "v2",
my app knows that the engine-id field should not be set and should
not be sent to the server.  Instead of giving the user a copy of
the must's error message, the app helps the user avoid the error
completely.

So saying that "when" is equivalent to "must" is not accurate.
While both can describe the same constraints, using the appropriate
mechanism will help tools channel users into better decisions and
promote better data hygiene, reducing errors and making the world
a better place.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 14 Dec 2009 17:19:12 +0100

> I don't follow your meaning here.  Can you elaborate?

With mandatory and presence the occurrence constraint is known a priori
and required for any instance document that conforms to a particular
schema. With 'when' and 'must' the app may first have to parse the
instance document and check some values in it in order to determine the
schema.
 
> manipulate the config data.

Sounds like marketing. :-) What is this "more information"?

> the must's error message, the app helps the user avoid the error

I fail to see why the app couldn't do the same based on the 'must'.
(I am not sure it is a good idea to let the server remove a part of the
database after the manager changes a single value somewhere, but that's
another story.)

> promote better data hygiene, reducing errors and making the world

If 'when' were allowed to override the grammatical constraints, one
would no more be able to write any schema that would validate all
documents without false positives or negatives. Having a strict
schema/grammar is essential exactly for the "better data hygiene".

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Mon, 14 Dec 2009 13:26:32 -0500

>instance document and check some values in it in order to determine the

I find your use of "schema" here confusing.  The schema is static
information.  It defines the hierarchy and containment.  Mandatory,
must, and when define constraints on the data modeled by that schema.
"must" does not change the schema.

>Sounds like marketing. :-) What is this "more information"?

I followed this claim with a paragraph explaining the information
being carried, what can be done with it, and why I think it is
important.  If you want marketing, I'll add a claim that it's
up to 20 times better or more.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 15 Dec 2009 10:18:49 +0100

> "must" does not change the schema.

Sorry, what I wrote was really confusing, the last sentence should have
been:

With 'when' and 'must', the app may first have to parse the
instance document and check some values in it in order to determine the
value of the XPath expression and this evaluation should not affect the
schema.

We are differing in the classification of 'mandatory' (and 'presence')
but in fact it affects all data node definition statements, too, e.g.

 container foo {
    when "../../bar = 42";
    ...
 }

If I understand your interpretation of 'when', element "foo" must not be
present if ../../bar != 42, so the app even doesn't know a priori the
hierarchy.
  
> up to 20 times better or more.

I still don't see the part of the information that you claim is missing
in the 'must' statement. It says exactly this: engine-id is present if
and only if version = 'v3'.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C