Subject: [netmod] system-creatable

From: Martin Bjorklund <[email-masked]>
Date: Mon, 12 Oct 2009 18:33:46 +0200 (CEST)

Hi,

Here is proposed text for a new statement "system-creatable"
(a.k.a. "assigned-by system").

7.6.6.  The leaf's system-creatable Statement

   The "system-creatable" statement, which is optional, specifies
   whether the server is permitted to create instances of a leaf.  Such
   leafs are used when the device needs to record persistent values that
   cannot be known until the device validates the configuration.  The
   argument is one of the strings "true" or "false".  If not specified,
   the default is "false".

   If "system-creatable" is "true", the server MAY create an instance of
   the leaf in the data tree, if it doesn't exist, in the following
   cases.  The behavior depends on the leaf's closest ancestor node in
   the schema tree which is not a non-presence container:

   o  If no such ancestor exists in the schema tree, the server MAY
      create an instance of the leaf if it doesn't exist.

   o  Otherwise, if this ancestor is a case node, the server MAY create
      an instance of the leaf in the data tree if any node from the case
      exists in the data tree, or if the case node is the choice's
      default case, and no nodes from any other case exist in the data
      tree.

   o  Otherwise, the server MAY create an instance of the leaf in the
      data tree if the ancestor node exists in the data tree.

   The server MUST NOT create the leaf if the leaf already exists or if
   "system-creatable" is "false".

   A system-creatable leaf is created by the server after validation
   (see Section 8.3.3), if the validation succeeds.  Since the leaf
   creation occurs after validation, constraints (see ^constraints) will
   not be able to access the new leaf value.

   If "system-creatable" is "true", the leaf MUST NOT be mandatory, and
   it MUST NOT have a default value.

   This statement is ignored if the leaf represents state data, RPC
   input or output parameters, or notification content.

And our favorite use-case added as an example to 7.6.9:

  The following example illustrates how "system-creatable" can be used:

     list users {
         key "name";
         leaf name {
             type string;
         }
         leaf uid {
             type uint32;
             system-creatable true;
         }
     }

   With the following <edit-config>, the server assigns an appropriate
   value for the "uid" leaf for the new user:

     <rpc message-id="101"
          xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
          xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
       <edit-config>
         <target>
           <running/>
         </target>
         <config>
           <system xmlns="http://example.com/schema/config">
             <user nc:operation="create">
               <name>barney</name>
             </user>
           </system>
         </config>
       </edit-config>
     </rpc>

And some FAQ entries:

{{ faq: Why is system-creatable leaf creation after validation?
The constraints defined in the YANG module are meant to be
useful in both the client and server contexts, allowing clients
to check the configuration they are about to send to a server.
If the constraints were allowed to see system-creatable leafs,
only the server could enforce constraints.
}}

{{ faq: Why can't a system-creatable leaf have a default value?
A default value tells the client and server what value will be
used if the value does not exist in the configuration.  If the
client sends data with no default value, then the client knows
the server will use this default value, and the only value the
server can assign such a leaf would be the default value.
Since both sides already know the value, having the system
create a leaf with a fixed value has no benefit, and so is
not permitted.
}}

{{ faq: Why can't a system-creatable leaf be mandatory?
The constraints defined in the YANG module are meant to be
useful in both the client and server contexts, allowing clients
to check the configuration they are about to send to a server.
If the a leaf could be marked mandatory but the system creates
it, then the client cannot validate the configuration before
sending it to the server.
}}

/martin

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 12:47:01 -0400

Thank you.  That looks good and useful to me.
Joel

Martin Bjorklund wrote:
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Phil Shafer <[email-masked]>
Date: Mon, 12 Oct 2009 12:55:19 -0400

Very nice.....

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 10:18:21 -0700

>    not be able to access the new leaf value.

I do not agree with this CLR or the interpretation in the FAQ below.
This changes the meaning of a 'valid database'.  All the YANG constraints
and all the semantics defined in description-stmts refer to
the validity of the running configuration, which is post-system-create.

The 'pre-system-create' is just another incomplete candidate database,
of no real interest since it is an implementation detail.

>    it MUST NOT have a default value.

I don't get these CLRs either.

Mandatory true means the server must pick some value.
It turns the MAY into a MUST.  That is useful.

Default and system-creatable do not really help,
but don't hurt either.  This could happen via refine
or deviations, and it should be OK.  It just means
the system will create a leaf with the specified default.
The system-creatable property is redundant, not an error.
Mandatory and default are not allowed together, but otherwise
these are OK for system-creatable.

> sending it to the server.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 12 Oct 2009 13:29:35 -0400

>the system will create a leaf with the specified default.

Why is this useful?  Having the system assign a fixed default
is not interesting, and forces us toward the "report-all"
extra noisy mode.

>these are OK for system-creatable.

For the same reasons we don't allow mandatory and default (having
to set it to a value when the value is coded in the YANG module),
we don't allow SC and either.  It makes no sense and adds nothing
to the model.

The big issue here is that we don't want to make constraints that
are not enforcable in the client context, and we don't want to make
two classes of constraints, some for the server, some for the client.
A client should know that the config it is sending is valid, to the
extent possible.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 11:21:42 -0700

> is not interesting, and forces us toward the "report-all"

I do not see the connection to the optional
report-all mode, which is a retrieval operation,
not an edit operation.

> we don't allow SC and either.  It makes no sense and adds nothing

A fatal error for a no-op condition
is rather extreme.  There are so many little CLRs
in YANG on Day One, I can't imagine how many there
will be in 5 years.

> A client should know that the config it is sending is valid, to the

Who is 'we'?
IMO, the constraints apply to the operational state which YANG
completely ignores.

   leaf minimum-mtu {
      system-creatable true;
      type uint32;
   }

   leaf big-packet-feature {
      type boolean;
      when "/minimum-mtu >= 9000";
   }

Do you really think the YANG data modeler intends for the
big-packet-feature to only be available if the client sets
/minimum-mtu?  IMO, the data modeler intends the
feature to be available any time the minimum MTU is >= 9000,
even if the system sets the MTU.

I think having the constraints resolve to 'false' any time
the server creates a leaf referenced in any validation
statements (instead of the client) is a huge bug, not a feature at all.

It goes against the agreement reached on the conference call
that all YANG validation refers to all config, plus server
created nodes.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 12 Oct 2009 20:29:09 +0200

> (a.k.a. "assigned-by system").

Didn't the WG agree on feature freeze in Stockholm? Also, I thought the
conference call was intended to close up the issues around mandatory,
default etc. but this has the potential to open the can of worms again.
So I don't agree with this addition.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 12 Oct 2009 20:32:16 +0200

On Mon, Oct 12, 2009 at 08:29:09PM +0200, Ladislav Lhotka wrote:
> Martin Bjorklund p????e v Po 12. 10. 2009 v 18:33 +0200:
> > Hi,
> > 

So can you please tell us how to close the issue without any addition
or change? Make a concrete proposal please.

Thanks,

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 12 Oct 2009 20:40:53 +0200

> or change? Make a concrete proposal please.

I thought the issue was closed by the conclusions of the conf call and
my understanding was that nothing essential would change except certain
clarifications. Is this new feature so badly needed for YANG 1.0?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 12 Oct 2009 20:45:35 +0200

On Mon, Oct 12, 2009 at 08:40:53PM +0200, Ladislav Lhotka wrote:
> Juergen Schoenwaelder p????e v Po 12. 10. 2009 v 20:32 +0200:

I think it is part of the resolution of an issue that kept us from
moving forward for quite a while. Again, if you know some other
solution, please tell us what it is. Concrete text appreciated.

/js

---

From: Phil Shafer <[email-masked]>
Date: Mon, 12 Oct 2009 15:00:23 -0400

>> A client should know that the config it is sending is valid, to the

Apologies, I should only speak for myself.

I do not want to make constraints that are not enforcable in the
client context.  And I do not want to make two classes of constraints,
some which the client can enforce, and some which it cannot.

>IMO, the constraints apply to the operational state which YANG

NETCONF (and YANG) are definitely more concerned with
configuration than operational data, but I don't see
where this applies to "system-creatable", which is fully
configuration.

Martin,
  Can you please add a line about SC being "config true" only?

>even if the system sets the MTU.

IMHO this is a bad constraint, since the client cannot hope
to enforce it.  It will never know if the config that it
sends is going to work, even if you fix the leading slash.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 12 Oct 2009 22:08:10 +0200 (CEST)

>   Can you please add a line about SC being "config true" only?

The proposed text says:

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 12 Oct 2009 22:21:13 +0200

> solution, please tell us what it is. Concrete text appreciated.

Solution to exactly what? I don't know what the pressing issue is.

On the other hand, Andy is right that this system-creatable introduces a
new phase of configuration life cycle (pre- and post-system-create) and
new way of fiddling with the configuration infoset besides defaults.
This shouldn't be taken so lightly, especially two weeks before I-D
submission deadline and past WG deadline.

Lada

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 16:30:19 -0400

Actually, rather than introducing a new phase, I think the definition of 
System createable makes clear certain things that were otherwise quite 
muddy.

To focus on the question of the interaction with MUST, by marking things 
system-creatable, and defining the rules for interaction with MUST 
statements in the model, we avoid problems.

Without this definition, you can have MUST statements in the model such 
that a client can not determine whether a config is valid, and therefore 
will have no clue what is wrong when he tries to commit it, and it 
fails.  Having "MUSTs" depend upon optional things is fine, as long as 
the MUST is written to allow for the case of non-existence.  But if the 
system may fill in the value, then the MUST may well fail, even though 
there is no defined default, and the user has not specified an illegal 
value.
And under our current setup, when the config fails, the "system defined" 
values would still not be in the non-committed config, so the user would 
have no clue as to what is wrong.

With system-creatable, and the rules as Martin wrote them, we get 
predictable and comprehensible behavior.  That seems a significant 
improvement.

This also gets us out of the confusion about whether the system is 
allowed to unilaterally over-ride the "default" value from a default 
clause.  It isn't.  But up until now we have had smart people read the 
documents and conclude that such a behavior is allowed.  While other 
smart people have thought it was prohibited.  Clarifying the what 
behavior is allowed is important.

Yours,
Joel

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 12 Oct 2009 23:02:02 +0200

> statements in the model, we avoid problems.

I am not happy with these rules, if I understand them correctly.
Extending Martin's example:

     list users {
-->      unique uid;
         key "name";
         leaf name {
             type string;
         }
         leaf uid {
             type uint32;
             system-creatable true;
         }
     }

Is there any guarantee for uniqueness of "uid" (or possibly 'must'
constraints involving "uid") if the text says that
"Since the leaf creation occurs after validation, constraints (see
^constraints) will not be able to access the new leaf value."?

> that a client can not determine whether a config is valid, and therefore 

I don't see how this could happen, provided that I know what "config"
exactly is - and system-creatable makes this notion even more fuzzy.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 15:23:45 -0700

> exactly is - and system-creatable makes this notion even more fuzzy.

I agree with you.
The 'unique' constraint in your example, and the 'when' constraint
in my example, show that the data modeler wants to write validation
rules which are independent of client/server origin status.
Whether you call that operational state or configuration, it
still ends up meaning the constraint applies to the running
system, not to the client-provided configuration data.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 18:51:53 -0400

Let us suppose that we allow such rules.
What is the poor user of netconf / netmod supposed to do when he gets an 
error?  The value producing the error does not even appear in his 
configuration.

For the unique uid case, one can argue that the system is misbehaving.
But for the mtu example, the user gets an error because the mtu is too 
small, but there is no mtu in the configuration.
And, if we allow arbitrary MUST statements referencing system-creatable 
objects, then the set of possible confusions is, I think,  much larger 
than the set of "valuable items" that may be enabled.

At least, if we decide to allow this, the use of the system-createable 
tag provides a hint as to where to look for the problem.  Although how 
one would look for it is beyond me.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 16:21:44 -0700

> error?  The value producing the error does not even appear in his

This is incorrect.
If the YANG config-stmt value is 'true',
then the node appears in the configuration.

> one would look for it is beyond me.

I would not object to this new construct if
it worked in an implementation-neutral manner, and
did not bring back the server-decides-what-validate-means
problems.

But this proposal specifies when the system-assigned nodes are created,
(but not default nodes for some reason) and that is an implementation issue,
not a standards-issue.

I prefer to allow the SNMP create-and-wait model for all nodes.
The client creates an entry with createAndWait RowStatus and the server
will fill in all the defaults.  The client can then edit, destroy,
or activate the entry.

The YANG proposal is that the server MUST NOT let the client
know what values it will use for server-assigned nodes.
I do not think this is helpful to operators or applications.

Note that Lada's addition to the example below is critical.
Unix systems usually require a unique UID,
and it does not matter if you provided an explicit UID
to /usr/sbin/useradd or just took the next available UID.
The 'unique' constraint applies to all values, not just
explicitly selected values.

A validation model that treats server-created nodes
as if they do not exist is not useful, or representative
of real use-cases.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 19:45:50 -0400

The question is when they (server created nodes) come into existence.

Clearly, when the user is doing a put to a non-running config, until 
they do a commit, they better not see anything except what the config 
started with, and what they wrote to it.  It seems to me that the system 
should not be dynamically modifying things at that point in the process.

So, at that point, the user can not see what values the system will 
assign to system-creatable nodes.  Heck, the system probably does not 
know until the whole config is there, and you try to commit it.

(There is no argument that if one looks in running after the commit, the 
config true nodes, including the system-creatable ones, have to be there.)

Now, the user does a commit.
The system has to perform two operations.  It must create any 
system-createable nodes it is going to create, and it must validate the 
configuration.
You seem to be suggesting that if the commit fails, and if the system 
created any nodes, those system created nodes must now be in the 
candidate configuration that the user was editing.  That seems fairly 
radical, and an implementation constraint all its own.
But those created nodes can not be in running, because the commit 
failed, so there better be no change to running.

So where is the information that the user can look at to tell him about 
the error?  Given that the information appears to be absent, the only 
conclusion I can come to is that the non-present information MUST NOT 
cause an error.  Hence, there can not be "MUST" statements pointing at it.

It may be possible to define things so that one can point must 
statements at system-createable nodes, but the evaluation can only 
consider nodes that exist in the candidate configuration, and will 
exclude any nodes that the system is going to create.  But that does not 
solve the problem you raise, and it is very messy.  So I would prefer 
not to go there.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 17:06:54 -0700

> The question is when they (server created nodes) come into existence.

The current proposal breaks a fundamental property of NETCONF:

The <validate> operation should work the same
whether applied to the candidate database or the running database.

The point of the <validate> operation is to find out in
advance whether a <commit> will work or not.  If <validate>
and <commit> work on different conceptual databases, then
<validate> is useless.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 21:30:14 -0400

Actually, as far as I cna tell, the current proposal preserves that 
exact property.
If instead we insist that system-created nodes have to get instantiated 
before validation during a commit, then <validate> will give different 
answers than <commit>.

There are alternatives.
We can pretend that there are no system-createable nodes.  That is 
counter-factual, but we could try it.
We could insist that <commi> and <validate> both must cause all 
system-creatable nodes to be created in the candidate.  This has 
multiple negative consequences.  THe simplest problem is that those 
nodes are now present, so if the user adjusts other values that would 
change the system created values, those changes will not have their 
proper side-effects, since the system has created the createables already.
We can ignore the issue and pretend that what we had before was consist 
and met your stated rule, when it didn't.
Or we can stick with the rule that actually meets your statement, which 
is the one that Martin wrote.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 19:33:15 -0700

> before validation during a commit, then <validate> will give different

<validate> MUST NOT give different answers than <commit>,
except for resource-denied or in-use type of errors.
Certainly not different schema validation errors.

> is the one that Martin wrote.

I don't understand 'meets your statement'

What Martin wrote does not even allow the client to know
what server-created values are going to be used, prior to
the commit.

The constraints in a YANG module are useless if they only
apply to client-provided values.  In the conf-call, we agreed
the constraints applied to the entire conceptual database.
In the YANG document, it clearly states that server-created
nodes (which are not the YANG default value) are part of the database.

If not, then IMO YANG constraints are useless.
They force the operator to set every conceivable leaf
manually that may ever be used in a validation rule somehow.
IMO that is worthless, and we are better off not standardizing
this aspect of a NETCONF database at all.

Just like 'with-defaults' capability, we can eventually identify different
validation models with a new capability, and let the server declare
which one they use.

I agree with Lada that this new 'system-creatable' should not
be added.  IMO, there is no consensus on one implementation strategy,
or one concept of validation, mandatory, default, or when/if
a data node instance even exists.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 12 Oct 2009 22:48:48 -0400

So, assume that we don't do system-createable.
Assume that we allow, as that would leave us, must expression pointing 
at optional nodes that the system may decide to fill in.

Yes, those filled in nodes will appear in running, if the commit succeeds.

Yes, the values to be used are part of the conceptual database, which 
the must expressions can reference, given that we currently allow that.

But that does not mean that those nodes are created in candidate.  In 
fact, in the case of a failing commit it may not be possible to create 
all of the nodes.  The problem may well evince itself as the inability 
to determine what some node should be set to (since even if the must 
expressions are consistent across teh configured values, it is still 
perfectly possible that the combination of musts, configured values, and 
system created values, creates an inconsistency for a must.  It oughtn't 
but I can not tell a modeller how to write things to guarantee it can 
not happen.)

And the current document certainly does not specify that system create 
values that would be created during commit need to be back-filled into 
candidate.

So how, under the current setup, would a user diagnose a problem caused 
by a must inconsistency between his configured values and a 
system-created value?  The system-created statement did not create the 
problem.

I agree with you about what the constraints ought to be.

But to meet those constraints requires either outlawing system creation, 
or making sure that must statements do not reference system-created values.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Mon, 12 Oct 2009 19:57:06 -0700

> or change? Make a concrete proposal please.

Using our favorite example, the unix users table:
the authoritative source for validation is /etc/passwd,
which is really operational data, ince it is rarely
edited directly by humans.

The UID field does not record whether it was auto-selected
or sysadmin-selected.  That is irrelevant to the uniqueness test.

I do not think there is consensus on 1 implementation approach,
and this work is too new and experimental to assume that the WG
knows for sure.

It is possible for an implementation to make
all the server-created nodes visible in the candidate and running config,
and still remember that they were server-created.
That is just an implementation detail.

IMO it is useful to know what the server-created values are going to
be, by examining the candidate.  One should not be forbidden by the standard
from implementing this feature.

There is no rule or text in YANG that says how a server applies nodes
from candidate to running, or how they are saved in NV-storage.
Hopefully the server skips all non-changes, but
maybe not.  Again, this is just an implementation detail.

I agree with Lada that we were better off
with the ambiguous mandatory-stmt, rather
than the changes to the validation model that
are being proposed.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 13 Oct 2009 07:25:47 +0200

> edited directly by humans.

Is this an attempt to come up with a new definition of operational
state data? If yes, I disagree with it.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Oct 2009 09:26:04 +0200 (CEST)

> But this proposal specifies when the system-assigned nodes are created,

Yes it does, otherwise it cannot be implementation-neutral, which you
also want it to be.  If we don't specify when and under which
circumstances s-c nodes are created by the server, different
implementations will behave differently.  And this was the problem
with the old text, without s-c.  We agreed that adding a s-c statement
would make it clear that the server cannot spontaneously create nodes
as it wants to.  With this statement, the client knows what to expect
from a server.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 13 Oct 2009 01:42:01 -0700

> as it wants to.  With this statement, the client knows what to expect

The validation model you propose does not support
very many use cases.

You are trying to claim that a system-creatable MUST be
unknown until commit-time, and that is simply false.
The server created value may be algorithmic, and
not a dynamic default based on config-time state data at all.
That is 1 possibility out of many.

A server implementation may have callback code that allows
instrumentation several chances to instantiate a config=true
node.  There is no value in forcing the server to wait
as long as possible before revealing that choice (if any).

At the end of the day, you are trying to claim that if
the server sets the node it is not config, but if the
client sets it, it is config.  I don't agree with that.
The current YANG spec does not agree with that.
If config=true, then it is config, whether you
like it or not.

A server-created config=true node that the client is not allowed
to see, and is outside the validation model, is broken.
It should not be part of the standard.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Oct 2009 11:05:38 +0200 (CEST)

> unknown until commit-time, and that is simply false.

No, I'm not claiming that.  

> not a dynamic default based on config-time state data at all.

Of course.

I don't care how or when the server decides *what* to put in a s-c
leaf.  I do care that it is clear *when* it puts it there.

> That is 1 possibility out of many.

Yes, but I don't think we should over-specify this statement.  We
don't need 10 flavors of system-createable.  I think we agreed that it
will be used in rare occasions.  The most important problem this
statement solves is that it becomes clear when a server is allowed to
create config nodes.

> as long as possible before revealing that choice (if any).

As Joel explained, there is value in this - the client can do off-line
validation before it sends data to the server.  If it needs
server-created data to do its validation, it can never validate it
before sending it.

Yes, this statement limits what a server is allowed to do, but that
was the whole purpose of adding it!  With the current text,
interoperability would suffer precisely because it was not clear
if/when/how servers could create config data.

> client sets it, it is config.

Absolutely not!!  I have never said or stated that.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 13 Oct 2009 11:47:55 +0200

> than the changes to the validation model that

Actually, the important information that the "uid" leaf is mandatory
(meaning that the system needs it for proper operation) is lost in

         leaf uid {
             type uint32;
             system-creatable true;
         }

If "uid" is not provided by the client when a new user is being created,
the server MAY create the instance - or MAY NOT, which results in an
error.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Tue, 13 Oct 2009 03:42:42 -0700

> server-created data to do its validation, it can never validate it

This is flawed logic.
The client validation is incomplete and will
always be incomplete for dynamic defaults that must
be calculated at commit time.  This corner-case should
not break algorithmic or static (but platform-specific) defaults.
These far outnumber the commit-time state data type of defaults anyway.

If YANG cannot model the user ID problem correctly,
then it should not be part of the standard.
Leave it up to vendors to solve this problem
over time, and standardize it when a solution that actually
works is available.

> Absolutely not!!  I have never said or stated that.

The text states that a server-created node is not part
of the validation checks, but if the client set the
node to the exact same value as the server, then
it would be part of the validation checks.
That is broken.

Your proposal doesn't even work for your
favorite use-case -- the user id field.
The unique constraint clearly applies to all values
in the real world, yet the model doesn't work
that way at all.  As Lada pointed out,
the server MAY create a node, which is
not the correct model for this example either.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Oct 2009 12:38:57 +0200 (CEST)

> (meaning that the system needs it for proper operation) is lost in

True, but again I don't think we should over-specify this.  You can
always specify behavior in the description clause.  

> the server MAY create the instance - or MAY NOT, which results in an

If the server didn't create an instance, it probably won't signal an
error either :)

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 13 Oct 2009 13:04:26 +0200

> always specify behavior in the description clause.

Over-specify? Occurrence constraints are essential in all XML schema
languages. I could argue in the same way - system-creatable could be
specified in the description clause.

> If the server didn't create an instance, it probably won't signal an

This system-creatable statement adds some new semantics but breaks the
existing one. Currently I am able to validate an entire configuration
against quite a strict schema and this won't be possible anymore.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Oct 2009 13:19:12 +0200 (CEST)

> specified in the description clause.

That's what I argued for!  But after the endless confusion on the
list, the WG decided that it would be better to clarify the intended
behavior of server created nodes with a new statement (and I agree
with that decision).

> against quite a strict schema and this won't be possible anymore.

I disagree.  With the new statement, a client can validate an entire
configuration before sending it to the server.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 13 Oct 2009 13:53:17 +0200

> behavior of server created nodes with a new statement (and I agree

Where is this WG decision documented? I know this issue popped up within
the huge thread on defaults, but I have always assumed it would be
addressed together with the config-statistics-operational data issues,
and in any case after YANG 1.0. If I am not wrong, four people (Andy,
Tom, Washam and myself) have expressed the opinion that it is not
critically important to know whether a given leaf value was (or can be)
created by the client(s) or server.

> configuration before sending it to the server.

I am talking about server-side validation, e.g. at commit time where the
server makes sure that all required configuration data are present and
satisfy all datatype and other constraints. In your example, a
configuration where the "uid" is not present for some items will be
found valid - currently it won't if the "uid" leaf is declared as
mandatory. So it is a step back in my view.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 08:59:09 -0400

Andy Bierman writes:
>apply to client-provided values.

So we need to add a sentence that say "When the system creates
a value for a leaf, the value MUST NOT break any constraints
in the YANG module.".  Seems fairly simple.

Instead of our past pattern of rat holing and debating without
listening, can we try to make productive progress by proposing
concrete solutions?

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 13 Oct 2009 15:23:06 +0200

> listening, can we try to make productive progress by proposing

Sure: postpone this after YANG 1.0.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 09:10:37 -0400

>specified in the description clause.

The occurance for "uid" is 0..1, since a valid config does not
need it.  The server MAY support a value and the description
may detail that the server SHOULD supply a value, but the
important parts are:

(a) the client knows that it doesn't need to supply a value, and
(b) the client knows that if it doesn't supply a value, it needs
to re-fetch the config to accurate know the value the system created.

The fact was the system will always assign a uid does not change
(a) or (b).

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 09:14:47 -0400

Andy Bierman writes:
>be calculated at commit time.

I disagree.  If your config validation depends on unknowns that can
only be calculated at commit time, then you have no faith that the
config that committed last time will be able to commit this time.
Validation cannot depend on state data, dynamic defaults, installed
FRUs, or other items that can change from one commit to the next.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 09:18:26 -0400

>to see, and is outside the validation model, is broken.

I don't understand where you read that "client is not allowed
to see" s-c leafs.  The client can only see them after they
are created, but after that they are part of the configuration
and will persist with the configuration.

Are you saying we need a sentence that specifies that s-c leafs
are created in the candidate before the commit operation copies
it over to running?  This I can agree with.  s-c leaf should
persist in the candidate so they are consistent between commits.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 09:25:12 -0400

>edited directly by humans.

On modern unix systems, /etc/passwd is read-only data (never edited
directly by humans) generated from /etc/master.passwd (which is
edited directly by humans via "vipw").

But if you are running your user database from your NETCONF-based
configuration data, then the NETCONF database is the authoritative
source for this config data.

You could have your NETCONF server proxy for /etc/master.passwd,
using that file as the database, in which case edit-configs to
running will add records to master.passwd, in which case the NETCONF
database is master.passwd and is still the authoritative source for
this config data.

But I don't follow how this is "really operational data", since
one way of another, the config data isn't operational data.

Or are you arguing for "config maybe;"?  ;^)

Thanks,
 Phil

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 13 Oct 2009 09:51:57 -0400

If we want the s-c leaves created in the candidate, when a commit 
succeeds, then we better say that.

I think this is actually a good idea.  One can make it more complicated 
by demanding that such nodes be marked as "system-created and subject to 
recalculation", but frankly I think we are better of not trying to cover 
that case.

Where I get into difficulty is if we try to mandate that system created 
nodes are created even if a commit or verify fails.  Without a valid 
config, those nodes do not seem to have meaningful values.

Yours,
Joel

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Oct 2009 16:01:36 +0200 (CEST)

> in the YANG module.".  Seems fairly simple.

I think this would be a good addition.

So Ladislav's example:

     list users {
-->      unique uid;
         key "name";
         leaf name {
             type string;
         }
         leaf uid {
             type uint32;
             system-creatable true;
         }
     }

would be valid, and any uid created by the server MUST be unique.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 13 Oct 2009 08:11:57 -0700

> would be valid, and any uid created by the server MUST be unique.

Any users entry created by anyone MUST have a unique uid
field.  That is why all config=true nodes need to appear in
the candidate.  If the server created uid=502, but doesn't
make that visible, then the client adds a new entry
and explicits sets '502', that is a commit-time error.

If the client cannot use <get-config> + inspection, or
<validate> to make a decision about what uid value to
provide, then the system-creatable leaf is not manageable
by the client.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 13 Oct 2009 11:20:24 -0400

But when do they appear?
Although we did not say it, requiring that system-created nodes appear 
in candidate configs after a commit seems to follow from basic principles.

But should they appear in candidate after a successful validate?  I 
think folks would find that surprising.

And even more surprising would be if they appeared after an unsuccessful 
commit or unsuccessful validate, since there is a good chance that they 
would be wrong, but would then be set to those wrong values and not 
updated by the system as the rest of the config is corrected.
-- But herein is the problem.  As currently written, these 
non-instantiated nodes can affect the validity of the configuration.

Just to complete the list,  they certainly can not appear before a 
commit or validate, since there is simply not enough information to 
create them.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Tue, 13 Oct 2009 08:59:32 -0700

> in candidate configs after a commit seems to follow from basic principles.

The commit is the transfer of the candidate to running,
so maybe you mean the edit-config that creates the
parent and/or sibling nodes (NP container, choice)
that would cause a system-creatable leaf to be created?

When to create the system node is data-model
and implementation dependent.

IMO, this corner-case used to set all the rules
in the proposal does not even follow the current
rules.

A config=true constraint is prevented by CLRs from
referencing config=false nodes.  So a server leaf that
cannot pick a valid value until commit-time is using
criteria based on state data, not other config=true nodes.
This is broken.  It should not be allowed in YANG.

Therefore, a server should be capable of picking some value
for each leaf, at edit-config time.  The value may change,
if other config=true nodes change before the
commit operation, but that is OK.

IMO, this would fix system-creatable.

Also, mandatory=true is still valid with s-c=true,
because the server MAY (not MUST) create a node.
So mandatory=true means that the commit-time constraint
is still valid.  The server does not have to provide a
value (as I said before).  Instead, the commit will fail
because the client did not make sure a value exists.

> commit or validate, since there is simply not enough information to

Why wouldn't there be enough information to create them?
There is the same data available to the client, and the
client is expected to pick a value.  Why can't the
server do the same thing?

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 13 Oct 2009 13:55:25 -0400

>config, those nodes do not seem to have meaningful values.

Yup, this is a touchy case.  Discarding these s-c leafs if commit
fails seems good on the surface, but I'll admit that I do not do
this in my code.  Once the commit progresses to the point that an
s-c leaf is created, it persists.  Perhaps not ideal, but certainly
simple.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 13 Oct 2009 12:12:09 -0700

> s-c leaf is created, it persists.  Perhaps not ideal, but certainly

I don't think anybody is proposing that.
Originally I thought that might work, but it doesn't.
However, mandatory=true means the same thing it meant before,
The node must exist in a valid configuration, and if not,
the commit will fail.

It is fine with me if the server MAY create a system-creatable
node (in candidate) any time from the edit-config to the commit, so it
allows all implementation strategies.

That means that if the server is last to add nodes, then they must not
trigger any constraint violations, as Martin pointed out.
In this case, the client will not see these new nodes until after the commit,
as Joel pointed out.

The text about the server MUST create s-c node at commit time is a CLR.
Just as before, the client cannot count on the state of
every candidate implementation to change the same way.
YANG only describes a valid configuration -- the contents
of the running config after the commit.

Andy

---

From: WashamFan <[email-masked]>
Date: Wed, 14 Oct 2009 14:28:54 +0800

>  created by the client(s) or server.

My understanding of system-creatable behavior is, the server as a proxy
creates s-c leafs on behalf of the client. there are two players manipulating
the config datastore, the client and the server. So the validation by client
side does not equal to validation by server side. validation by client side is
to validate input by the client to the config datastore. validation by server
side is to validate the config datastore that reflects inputs by the client
and the server when s-c leaf created. The s-c leafs should be treated in
the same way as other leafs when validating the config datastore.

I agreed with Andy that s-c leafs should be reflected in candidate datastore
before <commit>. the client should get all configs visible whether it provides itself, 
the server provides on behalf of it or defaults it can assume via documents.

I am wondering how any netconf implementations handle uid case in reality.

washam
washam

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 14 Oct 2009 11:36:51 +0200

> I think this would be a good addition.

So what about the sentences about validation in your original text?

My reading is that s-c leafs are exempt from validation.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 14 Oct 2009 12:08:29 +0200 (CEST)

> My reading is that s-c leafs are exempt from validation.

They are not yet created when a user does <validate> or during the
validation that occurs during <commit>, so they are not there.  That
doesn't mean that the leaf itself in general is exempt from
validation - the normal rules apply for this leaf just as for any
other leaf (e.g. if it has a value, its "must" expressions are
evaluated).

The extra sentence suggested by Phil just says that the server MUST
NOT create a value that will make the config invalid.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 04:11:51 -0700

> the server provides on behalf of it or defaults it can assume via documents.

It is OK if the s-c leaf shows up during the edit-config
or the commit.  As long as the server does not trigger
any constraint violations, then it can add leafs during the
commit.

> I am wondering how any netconf implementations handle uid case in reality.

In general, the central stack will never add s-c leafs
because they are not derived from YANG files.
But instrumentation callbacks can add leafs.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 14 Oct 2009 13:17:36 +0200

> other leaf (e.g. if it has a value, its "must" expressions are

Is there any reason for having these values NOT filled in at commit
time? This means that commit-time schema-based validation is rendered
impossible and thus decreases the utility of DSDL mapping.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 14 Oct 2009 13:22:13 +0200 (CEST)

> impossible and thus decreases the utility of DSDL mapping.

If they are filled in for validation at <commit> they also have to be
filled in at <validate>.  And having config data appear as a side
effect of <validate> seems weird.  Also having config data appear as a
side effect of a failed <commit> seems weird.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 14 Oct 2009 14:11:48 +0200

On Wed, Oct 14, 2009 at 12:08:29PM +0200, Martin Bjorklund wrote:
> > Martin Bjorklund p????e v ??t 13. 10. 2009 v 16:01 +0200:

Let me try to see whether I understand what the text is saying:

1) Both, the config without the s-c leafs filled in as well as the
   config which has the s-c leafs filled, have to be valid when I do
   an edit-config on <running>. 

   (Or does this interact with the test-option parameter?)

2) The config without the s-c leafs filled in has to be valid when
   doing a validate on <candidate>.

3) The config with the sc-leafs filled in has to be valid after doing
   a commit on <candidate>.

What about other operations, such as copy-config? It might help to
have all the relevant cases clearly spelled out...

/js

PS: I better do not ask how this interacts with partial-locks, e.g.
    a partial lock on an s-c leaf. ;-)

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 09:34:20 -0400

>   an edit-config on <running>. 

True.

>   (Or does this interact with the test-option parameter?)

No.

>   doing a validate on <candidate>.

True.

>3) The config with the sc-leafs filled in has to be valid after doing

True.

>have all the relevant cases clearly spelled out...

Copy-config should not change data, so it should not create s-c leafs.

>    a partial lock on an s-c leaf. ;-)

If an s-c leaf does not exist, it cannot be locked.  If it exists,
it's normal data.

We do need to say something about creating locks when the parent
is locked by the current session (that it is allowed, even though
the "system" is creating it) and when the parent is locked by
someone else (that it is not allowed).

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 06:58:39 -0700

>    an edit-config on <running>. 

Yes -- every edit to running is supposed to leave the
running config valid, but it is not a MUST.
The edit-config continue-on-error mode makes this unclear.

>    (Or does this interact with the test-option parameter?)

Yes -- if test-then-set, the runing config should remain
unchanged if there are going to be any problems.
Again, it is not clear if resource-denied or in-use
could occur after validation.

>    doing a validate on <candidate>.

Nothing ever has to be valid in candidate.
It is an incremental scratchpad.  It only
needs to be valid at commit-time.

> 3) The config with the sc-leafs filled in has to be valid after doing

yes

A client should expect that the server MAY create leafs
as a side-effect of the edit-config or the commit.
This is data-model and implementation-specific.
The server instrumentation may be written to
fill in some leafs right away and some during
the commit.

For the uid example, it makes sense
to wait until the commit to actually allocate a user ID,
instead of a more complicated temp-reserve design.

For a platform-specific static default (not YANG-documented as
a deviation or a refinement), the instrumentation might
just plug in the value right away.

> have all the relevant cases clearly spelled out...

YANG just needs to describe the properties of
a valid NETCONF database.  The NETCONF server is
expected to maintain a valid running config.
Other than that, it is unclear what the standard
needs to specify (e.g., url, startup structure).

>     a partial lock on an s-c leaf. ;-)

Excellent point.
All XPath (YANG, select, partial-lock, proprietary RPCs)
needs to operate on the same conceptual XML instance document.
All nodes in use (client, default, s-c) exist in this conceptual document.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 09:43:13 -0400

>allows all implementation strategies.

This would be ambiguous about the results.  Consider:

    <edit-config>
      ...
      <config>
        <user>
          <name>one</name>
        </user>
        <user>
          <name>two</name>
          <uid>2000</uid>
        </user>
      </config>
    <edit-config>

If the system starts numbering user uids at 2000 (as JUNOS does),
then the results of generating uids could be an error if the user
"one" is assigned uid 2000 when the </user> tag is seen.  A similar
scenario exists for multiple <edit-config>s before a <commit-config>,
but this is probably less interesting.

How much do we care about this case?  How worried are we over the
interoperability differences caused by allowing the vendor to choose
when s-c leafs are created?  (Does it seem like we're on odd sides
of this one?)

>of the running config after the commit.

I want the constraints in YANG to be applicable inside the client,
so outgoing config can be validated.  This gives s-c leaf creation
a post-validation timeframe.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 07:24:06 -0700

> when s-c leafs are created?  (Does it seem like we're on odd sides

This is data-model specific:

   leaf asset-tag {
      type string;
      system-creatable true;
      // the server will use its default service tag string
      // built into the hardware, if this leaf is not set
   }

In this example, the server knows right away what value
it will use, and if the instrumentation callback creates
the asset-tag leaf during the commit, then so what?
What problem does that cause?  None.  It is just
an implementation choice, made for a specific leaf.

> a post-validation timeframe.

  leaf foo {
     must "../bar = 3";
     type string;
  }

  leaf bar {
     type uint32;
     system-creatable true;
  }

I want a validation model that is independent of
how a data node is created.

All validation breaks in your design.
Even if the server sets 'bar' to 3,
the 'foo' leaf will still be invalid
because the must-expr will evaluate to false
if the 'bar' leaf is not present.

The model imposes a false requirement -- that the
client set both 'foo' and 'bar' -- even though this is
not the intent of the data modeler or required by the protocol.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 14 Oct 2009 16:35:08 +0200

> Copy-config should not change data, so it should not create s-c leafs.

Quoting RFC 4741:

     <rpc message-id="101"
          xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
       <copy-config>
         <target>
           <running/>
         </target>
         <source>
           <url>https://[email-masked]:passphrase/cfg/new.txt</url>
         </source>
       </copy-config>
     </rpc>

Even if the URL points to a local file, the local file might not
contain all s-c instances, e.g. leave the uid out. So you are saying
such a copy will fail?

/js

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 10:21:56 -0400

>      // built into the hardware, if this leaf is not set

How is a hardware-based asset-tag is configuration data?  What
happens when you swap the hardware?   This is a bad use case.

>if the 'bar' leaf is not present.

This would be a bad data model, since it's modeling a constraint
that cannot be enforced.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 10:25:20 -0400

>contain all s-c instances, e.g. leave the uid out. So you are saying

Nope, my bad.  I was thinking about candidate, not running.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 08:25:46 -0700

> happens when you swap the hardware?   This is a bad use case.

The HW is soldered on the MB and cannot be replaced.

What problems are caused by allowing the server to
create a leaf in candidate?

Perhaps the problem is that we 100% disagree on the
validation model.  There are plenty of people
who do not want the validation rules to work if
the client sets a value, and fail if the server sets a value.

There is no way either camp is going to
convince the other that they are right.

> This would be a bad data model, since it's modeling a constraint

In your implementation maybe.
There is no way that operators are going to
explicitly set every node that is part of
a YANG constraint, or fail every commit if they don't.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 14 Oct 2009 17:44:41 +0200

> the client sets a value, and fail if the server sets a value.

My understanding is that both will succeed because the schema will be
lax for such s-c leafs.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 14 Oct 2009 18:20:16 +0200 (CEST)

> the client sets a value, and fail if the server sets a value.

I have to ask.  Who wants the opposite?

Remember that Phil suggested:

  "When the system creates a value for a leaf, the value MUST NOT
  break any constraints in the YANG module.".

which I agreed with.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 09:21:21 -0700

> My understanding is that both will succeed because the schema will be

What does lax mean here?
How are you going to enforce must "../foo = 3"
or any XPath that references a s-c node?
Evaluating ../foo --> null --> false will
cause the commit to fail.  Ignoring it
it the DSDL schema is going to produce
a model that does not represent the author's intentions.

I don't understand the fascination with server-created
vs. client-created, or default vs. non-default value.
I don't understand why a server-created leaf exists
for validation purposes if it contains the YANG default,
but it doesn't exist if it contains a non-default value.

I do not accept the problem statement that YANG and
NETCONF should focus on the client-provided instructions
instead of the desired state of the running system.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 14 Oct 2009 12:23:39 -0400

This highlights the risk I see with having s-c leaves appear in 
candidate when a commit fails.
(Note for clarity, I can live with the consequences, and if it will get 
us off the dime, lets just write it all down and move forward.  But we 
need to be aware of them.)

Suppose instead of the backplane specific tag Andy was referring to, it 
was indeed a board specific tag.  (Please excuse the improper and sloppy 
YANG.  I think it still gets the point of this discussion across.)

dontainer declaration of some kind {
    leaf board-select { description "which board" }
    leaf assignable-board-property {
       type string;
       system-creatable true;
       // if not set, will be set from the board hardware
    }
    leaf board-max-ports {
       type int;
       system-creatable true;
       // the maximum number of ports
       //      that the config wants to use
       // the system will fill this in from the board
       //  if not specified.
       // config will fail if the number exceeds the hardware limit
    }
    list of some kind {
      leaf port {
         type int;
         must expression that checks against part board-max-ports
      }
      leaf port-properties {...}
   }

So the user does a set against candidate.  The information provided by 
the user is a board-select, and a set of port setups.
When the user does a commit, at some point the system fills in the 
assignable-board-property and board-max-ports.

On the one hand, the suggested must check is impossible if we prohibit 
must expressions from pointing at system-createable (in this case 
system-created) fields.

Now suppose that the commit fails.  The user looks, and realizes that 
they pointed at the wrong board.  So they do a further edit, just 
changing the board-select field.  And do another commit.
1) If the system did set board-max-ports in the candidate, then it is 
still set.  So the config will fail even though the new board has enough 
ports.
2) Suppose there was no board-max-ports and no MUST.  The first config 
fails for some other reason.  The assignable-board-property will still 
be sest to the property from the first board, even though it is clearly 
the second board property that the user intends.

Note that tagging things system-createable makes it easier to discuss 
the problem, but is not actually the source of the difficulty.  The same 
confusion about when things are created, with what values, existed before.

Thank you,
Joel

Andy Bierman wrote:
...
...

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 12:04:13 -0400

>The HW is soldered on the MB and cannot be replaced.

If you swap the MB, should restoring an archived config give the
new MB the asset tag of the old MB?  I'm assuming not, so this
isn't config data.

>There are plenty of people

My guess is that the number of people who what validation rules to
work on the client and fail on the server is zero.  Another strawman?

>a YANG constraint, or fail every commit if they don't.

The model's constraints will need to be written in a way that avoid
scenarios like this.  Your foo/bar model does not, so it's a bad
data model.

Thanks,
 Phil

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 14 Oct 2009 12:32:08 -0400

Every time I read that statement, I stop and stare at it.
I can not prove it, but I strongly suspect that the constraint as 
defined is unmeetable.  That is, one can not define a data model that 
allows for real hardware constraints (i.e. it fails if you ask the 
hardware to do something impossible), that also requires all s-c nodes 
to meet all constraints.
In the abstract, there can be an s-c node N that is defined by must 
expressions to have properties p1 relative to a node N1 and P2 relative 
to node N2, such that there are values V1 and V2 of nodes N1 and N2 that 
are mutually fine (no must expression is violated by setting N1 to V1 
and N2 to V2), but there is no value V for N is ch that P1(V, V1) and 
P2(V, V2).
If I try to write a specific example, I am sure one would say "don't do 
that".  But I strongly suspect complex indirect examples are quite 
possible, and hard for the modeler to avoid.

Yours,
Joel

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 14 Oct 2009 18:32:30 +0200

> > My understanding is that both will succeed because the schema will be

I mean that the leaf will be optional from the schema POV.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 09:48:43 -0700

> I mean that the leaf will be optional from the schema POV.

Then your machine-generated implementation of the
model will be incorrect.  The author used
a must statement.  There is nothing optional
about that.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 14 Oct 2009 18:52:37 +0200 (CEST)

> Every time I read that statement, I stop and stare at it.

There are a couple of cases.

1.  If there is a must expression on the s-c node itself, it will not
    be evaluated until the node exists, so when the server creates a
    value, it must pick a value that doesn't break the must expression
    (or not pick a value)

2.  If there is a must expression that refers to a s-c node and
    requires the s-c leaf to exist, then it essentially makes the s-c
    mandatory instead (requiring the client to set it).

3.  If there is a constraint (must or unique) that refers to the s-c
    node that can handle that the s-c node does not exist, the server
    must pick a value that doesn't break the constraint (or not pick a
    value).

> I can not prove it, but I strongly suspect that the constraint as defined is
> unmeetable.  That is, one can not define a data model that allows for real
> hardware constraints (i.e. it fails if you ask the hardware to do something
> impossible), that also requires all s-c nodes to meet all constraints.
> In the abstract, there can be an s-c node N that is defined by must expressions
> to have properties p1 relative to a node N1 and P2 relative to node N2, such
> that there are values V1 and V2 of nodes N1 and N2 that are mutually fine (no
> must expression is violated by setting N1 to V1 and N2 to V2), but there is no
> value V for N is ch that P1(V, V1) and P2(V, V2).
> If I try to write a specific example, I am sure one would say "don't do that".
> But I strongly suspect complex indirect examples are quite possible, and hard
> for the modeler to avoid.

But is this s-c specific?  It seems the same thing can happen with a
normal leaf.

/martin

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 14 Oct 2009 13:02:29 -0400

Below...

>> Every time I read that statement, I stop and stare at it.
... description elided ...
> But is this s-c specific?  It seems the same thing can happen with a

The difference between the s-c case and the normal config case is that 
in the normal case, if the user produces an internally inconsistent 
config, then the user gets an error, and the user gets the task of 
figuring out how he wants to fix the problem.

If the inconsistency is such that the system can not create a value for 
the node, and the system considers it needs the node (it can not be 
mandatory in the config, because that would mean the user had to create 
it), then the requirements have become unmeetable.

It can be argued, and maybe should be argued, that this has nothing to 
do with tagging something system-createable.  That we always allowed the 
system to create nodes, and that we therefore already had this 
constraint.  The one difference with the statement is that up till now, 
it seemed to me the system was allowed to say "invalid config" when it 
could not create the node it needed with a sensible value.  The 
statement from Phil about requiring s-c nodes to meet must constraints 
seems to say that the system will not do that.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 10:09:11 -0700

leaf foo {
     must "../bar = 3";
     type string;
  }

  leaf bar {
     type uint32;
     system-creatable true;
  }

Is this legal YANG or not?
Ignoring the must-expr sometimes is not an option.
Either the s-c value (bar) exists for validation purposes,
or this is a fatal error in the YANG compiler.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 12:53:09 -0400

>it), then the requirements have become unmeetable.

Should we have the rule that must constraints cannot refer to s-c leafs?

Also: we need a rule that says s-c leafs cannot be keys.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 10:51:13 -0700

> Should we have the rule that must constraints cannot refer to s-c leafs?

Yes.
You realize this opens up the garbage-XML issue again.
If the compiler can tell a node in an XPath expr
references an s-c leaf, then it certainly knows
whether the node is a real leaf or not.

What about when-stmt, unique, min/max elements in a leaf-list?
These are also illegal.

> Also: we need a rule that says s-c leafs cannot be keys.

Yes -- this is all non-optimal, but at least predicable.
It will make people think carefully about using s-c leafs.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 14 Oct 2009 20:30:42 +0200

> a must statement.  There is nothing optional

Yes, this is true, so Schematron generated from such a YANG module would
fail.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 11:42:44 -0700

> Yes, this is true, so Schematron generated from such a YANG module would

So maybe a good compromise is to make s/c leafs outside
the validation model, and it is a YANG syntax error if
any validation constraint statements reference an s-c leaf.

That way, if the client provides the s-c leaf, it doesn't
change any validation constraints, because that s-c leaf
couldn't be part of any constraints in the first place.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 14:58:25 -0400

>It will make people think carefully about using s-c leafs.

Which is good, since s-c leafs should be one in a million.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Oct 2009 12:28:10 -0700

> Which is good, since s-c leafs should be one in a million.

What about dynamic defaults?
I agree this is not saved in NV-store,
as a different default may apply every time.
This may be more than 1 in a million.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 14 Oct 2009 15:35:54 -0400

>This may be more than 1 in a million.

Sure, but they aren't config data.  The dynamic default is really
more for the operational "twin" leaf than the config "twin".  In
fact, I could claim that there were no dynamic defaults for config
data, that these are just the dynamic value of the operational
"twin" leaf.   Not an important claim, but another way to look
at the issue.

Thanks,
 Phil

---

From: Kent Watsen <[email-masked]>
Date: Wed, 14 Oct 2009 14:31:25 -0700

> Which is good, since s-c leafs should be one in a million.

True, and maybe this is the problem - the current s-c definition is very narrow, I wonder if there is a more useful/general construct...

The current definition is limited to values that the server may fill in if the user doesn't, but there are other cases when the server will create nodes - for instance:

  - automatically creating an "encrypted-password" when user supplies
    "clear-text" password

  - automatically terminating firewall rules due to a policy that says
    all such rules should be terminal (like a commit script)

  - automatically populating entries in a list of signatures when a
    new signature-database is referenced via config (not an op cmd).
    Note: assume this can not be state data as it needs to be referenced
    by instance-identifiers

And, of course, this shouldn't limit the server to just creations, but also allow for modifications, and deletions.

IMO, marking nodes as server-creatable isn't nearly as useful as having a general mechanism indicating for all server modifications.  An easy way to implement this is to enable <edit-config>/<commit>/<copy-config> RPC replies to include at least a something-got-modified flag or, better, list out all modified xpaths.  A more complicated solution is to also model these semantics in the YANG model, which cover most cases, but still allow the RPC Reply to return xpaths for the few unmodelable cases (commit scripts)

What do you think? - maybe this just nees to be looked at a differently?

Applying this logic to the uid example, we have:

      list users {
          key "name";
          unique "uid";
          leaf name {
              type string;
              mandatory true;
          }
          leaf uid {
              type uint32;
          }
      }

The uid leaf is optional and, when not specified by the user, MAY be filled in by the server (or rejected) but, if it is filled in, it MUST be unique and the RPC Reply will indicate that the leaf was modified.  

  - this looks perfect to me, but I'm with YANG describing the config model,
    not the operational model, camp

Thanks,
Kent

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 16 Oct 2009 19:59:48 -0700

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Sent: Monday, October 12, 2009 9:33 AM
> Subject: [netmod] system-creatable
...
>    If "system-creatable" is "true", the server MAY create an instance of

Replace "MAY" with "MUST".  Otherwise, interoperability will suffer.

>    o  If no such ancestor exists in the schema tree, the server MAY

Replace "MAY" with "MUST".

>    o  Otherwise, if this ancestor is a case node, the server MAY create

Replace "MAY" with "MUST".

>       data tree if the ancestor node exists in the data tree.

Replace "MAY" with "MUST".

As I see it, the only difference between system-creatable and "default"
is that a default value is explicitly given in the model, and is consequently
in some sense redundant when given in response to a retrieval operation.

Randy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 16 Oct 2009 23:04:32 -0400

I do not think the intend was that if something is marked 
system-createable, then the system is required to ensure that it exists.
Rather, it was a notification to the client that if the client cares 
about this node, then after doing a config commit (and maybe after doing 
a verify, we need to sort that out), the client should check to see if 
the system has created a value for the node.

I do not believe that there is any interoperability problem if the 
system does not create the node.  After all, if the system does need a 
value for the node, then the system will create it.

Yours,
Joel

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 16 Oct 2009 20:24:01 -0700

Hi -

> From: "Joel M. Halpern" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, October 16, 2009 8:04 PM
> Subject: Re: [netmod] system-creatable
>

The proposed text does not say so.

It merely says "MAY".  The only hint of support for your
reading is "Such leafs are used when the device needs to
record persistent values that cannot be known until the device
validates the configuration."  That text, however, only provides
rationale, and does not require devices that "need" values to
actually create them.

If your reading is what is intended, the text needs to be more
explicit.

Randy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 16 Oct 2009 23:43:03 -0400

I can well believe that the text needs to be more explicit.
Yours,
Joel

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 19 Oct 2009 18:42:35 +0200 (CEST)

> If your reading is what is intended, the text needs to be more

Yes, what Joel wrote is the intention.  It says that the server MAY
create the leaf.  I do not understand what text you want to add.  "The
server MAY create the leaf, if it is needed by the server"?

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 19 Oct 2009 11:20:12 -0700

Hi -

> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Monday, October 19, 2009 9:42 AM
...
> server MAY create the leaf, if it is needed by the server"?

I understand the case for bits like user-id to be assignable by the
system.  I agree that there is value in identifying those bits in an
information model, so that clients will be able to know which bits
necessary for a successful configuration will (not merely "might")
be generated by the server if not supplied by the client.

What I have a bit of trouble with is the very idea that a model would
specify bits of information that may or may not be needed by the server.
At first glance, such things would seem to be symptomatic of a badly
designed model, so I'd appreciate a use case.  All the ones I can
think of are like user-id; for the system to fail to supply a value would
be distinctly unhelpful, and not the kind of behaviour we'd want to go
out of our way to support.

The "MAY" language is of no help to the client - it provides no guarantee
that the server will supply a value if the client omits one from the configuration.

Randy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Oct 2009 11:47:09 -0700

> server MAY create the leaf, if it is needed by the server"?

The s-c statement is ignored by the server, like mandatory
for /rpc/output leafs.  If the server violates the rule, it
must be a bug.

On the client side, knowing s-c=false says the server will
not provide a value.  Knowing s-c=true says the server may
provide a value.  Not that useful.

This new statement doesn't actually change anything
for the client, other than the possibility of a warning
message, as Juergen suggested.

The s-c property doesn't actually impact the 'constraints
ripple effect' (mandatory leaf X w/must-stmt forces every
leaf in the must-stmt to be mandatory, if X is set).
If a leaf is part of a constraint, the client
must provide the relevant 'associated nodes'.
The s-c statement doesn't change that a bit.

IMO, the behavior may be OK (but may be problematic
wrt/ corner-cases), but the clause doesn't seem
to help the client at all.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 19 Oct 2009 21:11:14 +0200 (CEST)

> provide a value.  Not that useful.

I think it is useful, b/c it tells the client which leafs may be
created by the server.  It knows that other leafs will not be created
by the server.

So if the alternative is to do nothing, what can a client expect from
a conformant server?  Is the server allowed to do anything?

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 19 Oct 2009 12:21:32 -0700

Hi -

> To: <[email-masked]>
> Cc: <[email-masked]>; <[email-masked]>
> Sent: Monday, October 19, 2009 12:11 PM

That's a strawman.  Another alternative is to use MUST rather than
MAY.  Then the client would know exactly what to expect, and the
requirements for the server would be unambiguous.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 19 Oct 2009 21:32:20 +0200 (CEST)

"Randy Presuhn" <[email-masked]> wrote:
> That's a strawman.

I don't think it is - people *have* suggested that we should not add
s-c, and I really want to know what people think should be allowed.
Kent talked about thir commit scripts, and AFAIK, these scripts can do
anything.  If we don't add s-c, I think we should add text that
describes what a server is allowed to do, and what a client can
expect.

> Another alternative is to use MUST rather than

Yes.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Oct 2009 12:46:47 -0700

> requirements for the server would be unambiguous.

The problem with MUST is that is turns every s-c=true
leaf into mandatory=true.

At best, you could say server MUST create the leaf,
or MUST return the appropriate <rpc-error>,

I prefer to leave mandatory=true meaning the client
is responsible for making sure the leaf exists.
That could be one of:
   A - create the leaf
   B - hope that the server will create the leaf

Robust client applications will probably not choose plan B.

That's how we got here.
Sometimes mandatory=true means (B) will work.

IMO, usually not a problem (*).  The client is still responsible,
so use plan B at your own risk.  The server is responsible
for only adding leafs that keep the config valid.

(* == data model design will impact this problem.
Top-level leafs are kind of an exception.
Usually, the client MUST create the container parent,
list parent, or case siblings to make the s-c problem relevant.
In these cases, I don't see any problem if the server
fills in valid leafs during the commit.  It probably
is a problem wrt/ offline validation, but 1 of many.)

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 19 Oct 2009 15:32:55 -0400

>requirements for the server would be unambiguous.

If it is "MUST" the client will have to refetch the config to know
the final config.

If it is "MAY" the client will have to refetch the config to know
the final config.

I'm missing why it needs to be MUST.  What does this solve?  If the
server does not create it, the client's behavior is the same.

The idea of having edit-config and/or commit-config return a list
of created, changed, and deleted nodes seem like a better answer.

I'm thinking we should leave s-c until we get some operational
experience.  We may end up with a different answer, like a module
that augments the replies for edit-config and commit-config.

Thanks,
 Phil

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 19 Oct 2009 13:08:33 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> Sent: Monday, October 19, 2009 12:46 PM
...
> leaf into mandatory=true.

No.  Mandatory=true means the *client* is required to supply
the information.  The text I proposed does not have that effect.
The MAY->MUST changes proposed only affected what was
required of the *server*, not what was required of the client.

(though s-c only makes sense for information that is in some
more general sense "required", if not in the technical "mandatory=true"
sense)  Why on earth would one want a model to specify that the
server should create information that it doesn't need?
 
> At best, you could say server MUST create the leaf,

Yes, that is what the text changes I proposed did.

> or MUST return the appropriate <rpc-error>,

This should only happen under pathological circumstances,
like ENOMEM, not on an implementation whim.
 
> is responsible for making sure the leaf exists.

No one is proposing that we change that.

> Robust client applications will probably not choose plan B.

What I am arguing for is that s-c should, instead of your "plan B",
be *required* to create such leaves when the client omits the data.
 
> Sometimes mandatory=true means (B) will work.

Let's *not* confuse s-c with mandatory=true.
 
> for only adding leafs that keep the config valid.

s-c (as I'd like to see it) is a way of identifying the bits which the
server *must* create to meet these requirements, rather than
a merely advisory facility that says "oh, by the way, if you don't
supply this bit, some servers might make up a value for you
if they feel like it at the moment."

> is a problem wrt/ offline validation, but 1 of many.)

Lots of things are problems wrt/offline validation.  I don't see
s-c bits as making offline validation any harder than it already is.

Randy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 19 Oct 2009 16:13:51 -0400

There seem to be two related but distinct needs:
1) Randy, you seem to be asking for a marking that tells the sesrver 
what it must create.
2) The proposed definition basically says "client, if for any reason 
your operation depends upon knowing the value of this node, then after 
the commit you better fetch the node, because the system may well have 
chosen to create a value for this."

I find (2) to be useful.  I understand what a management system can do 
because of that marking.  It can be argued that it is not extremely 
useful because many management systems will fetch the config after a 
committ anyway.

On the other hand, I do not understand who is helped by definition (1). 
  The managed device does not really care.  It either needs the 
information, in whcihc case it was alreayd prepared to create it, or it 
does not need the information, in which case it may well have no clue as 
to what value to use in this node it has been ordered to create.
The client does not appear to be helped either.  It is exceedingly rare 
that the client requires a node to exist, but does not care about its value.

Somewhat confused by teh discussion,
Joel

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 19 Oct 2009 13:22:37 -0700

Hi -

> From: "Phil Shafer" <[email-masked]>
> Sent: Monday, October 19, 2009 12:32 PM

If it's a MUST, then, in our user-id example, no client needs to
have user-id allocation logic.  If its a MAY, then all clients will
need user-id allocation logic, in order to accommodate the servers
that don't.

...
> that augments the replies for edit-config and commit-config.

There are a bunch of cases, especially in provisioning, where
a client only needs to allocate / reserve a "foo", without
much or any concern about which one is to be allocated.  Once
allocated, however, for purposes of configuration backup/restore
and versioning, which one got allocated matters a great deal,
since the "which one" may be information that has meaning and
gets propagated beyond the immediate control of the configuration
management system.

I think this could be turned into an argument for a "reserve" or
"allocate" operation, but I wouldn't want to do it without having
a coherent metamodel and object lifecycle to support it.  For the
same reason (lack of metamodel & lifecycle) I can agree with
deferring s-c.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 19 Oct 2009 13:39:50 -0700

Hi -

> Sent: Monday, October 19, 2009 1:13 PM

> 1) Randy, you seem to be asking for a marking that tells the sesrver 

It also tells the client what it can depend on the server to create if
the client does not supply a value.

> useful because many management systems will fetch the config after a 

But (2) is already true for everything anyway, since bits of configuration
data may have been put in place by other clients, as well as the server
itself.

>From an *interoperability* perspective it does matter.  Can a client
rely on a server to supply a c-s value?  With the MAY language, it cannot.

> that the client requires a node to exist, but does not care about its value.

The client may care a great deal about the value, but doesn't care how
the server arrives at it.  Group IDs, User IDs, MAC addresses (writeable
on some systems, but normally it's preferable to let the system use what
it reads from hardware) SNMP Engine IDs (may be computed from local
information, but architecturally they're *administrative*, and should be
considered configuration data) backup media from a "scratch" pool -
just a few examples.

Randy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Oct 2009 13:59:21 -0700

> that augments the replies for edit-config and commit-config.

I agree with you (for a change :-)

One might even use notifications to inform the client(s)
that the running config has changed.  There are lots of
factors that might make the s-c leaf a non-problem.

Experience is knowing which theoretical problems
were not real problems, and which real problems
were totally missed by the theories.  Let's write some
standard YANG modules first, (ACM anyone?) and see
what works and what doesn't.

Andy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 19 Oct 2009 17:05:15 -0400

Part of my problem may be a failure of imagination.
I have trouble imagining a field for which
the data model requires that the field exist
the system can reliably know the value
and the client can not reasonably be expected to know the value.

The userid case is not actually a good example of this, because the 
system may not need userids.  As something the system may create, if it 
needs them, it makes sense to me.  As a global requirement that the user 
model MUST have valid userids, I don't follow it.  (Requiring that IDs, 
if they are used, be unique, makes sense because good security practice 
says to avoid confusing users.)

We can give up on all notion of knowing what the system may modify, and 
take the view that since multi-manager means that you can never know 
anyway, the client effectively must refresh his model before beginning 
any operation, and after any commit if operation is continuing.  But 
folks have repeatedly said that thye do not want clients to have to 
fetch everything in sight.  (There are meaningful alternatives for 
dealing with multi-manager, but they will not cope with system-created 
very well.)

Yours,
Joel

Randy Presuhn wrote:
>> Sent: Monday, October 19, 2009 1:30 PM
>> I do not follow your reasoning about the relationship between whether 
>> S-C is a must and the user-id example.
>>
>> If user-id is mandatory, then the client better be prepared to create it.
> 
> It is not mandatory, at least not in the Yang sense.
> 
>> If user-id is optional, then clients do not have to create it if they 
>> don't want to.
> 
> Agreed.
> 
>> If user-id is optional and system-createable,  then the client is not 
>> required to create it, but the client should be aware that the system 
>> may choose to assign it a value, for example, if it uses userids.
> 
> replace "may" with "MUST".
>  
>> I suppose we could add the mixed case of a leaf declared as both 
>> mandatory and system-createable, to indicate that the system is required 
>> to create it if the client does not.  The verbiage would be pretty 
>> strange, but at least we don't use the optional case that I think is useful.
> 
> I don't want to go there.
> 
> I think the confusion may come from the shorthand.  When I say
> "MAY" and "MUST" here, I am referring to the replacement of
> the "MAY" in the proposed text with "MUST"  The "MAYs" in the
> proposed text all had to do with *server* requirements, not
> client requirements.
> 
> Randy
> 
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 19 Oct 2009 17:34:19 -0400

If we want to punt, then we need to explicitly say:
"Warning, the contents of running after the commit may be different from 
those of candidate before the commit.  Any client expecting to perform 
continued operations on the configuration is advised to perform a 
get-config operation after a commit."

At that point, the system can do whatever it wants, and the client has 
been warned that unexpected things may happen.  That still leaves some 
very odd corner cases about constraints that reference optional leaves 
that the system chooses to fill in.  But, well, if we want to punt then 
we punt.

Yours,
Joel

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Oct 2009 14:47:28 -0700

> that the system chooses to fill in.  But, well, if we want to punt then

I do not think leaving out the s-c-stmt is punting.
There was never any intent to make s-c=true a MUST,
as Randy wants.  The original intent was to make sure
the server MUST NOT create particular leafs.
The description-stmt can still cover this case.

If the client edits the running config directly,
then this is more of a problem than with candidate -> commit,
since a get-config is needed after every edit-config.

As Randy pointed out, arbitrary, non-discoverable optional behavior
is useless to the client.   Yet, I see no basis to
pick universal MUST/SHOULD/MAY language here at all.  It seems
data-model dependent to me.  (description-stmt!)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 20 Oct 2009 10:14:33 +0200 (CEST)

> The description-stmt can still cover this case.

So, instead of marking the leafs that the client might have to
refetch, you want to do it the other way around?

> data-model dependent to me.  (description-stmt!)

Do we agree it is data-model dependent (not implementation dependent)?

We should optmize for the common case.  So if we don't want to
introduce s-c, we could use the description statement for this, but be
explicit about it in the spec.  I.e. say that the server MUST NOT
create/delete/modify any config data unless dictated by the data model
(in the description statement).

/martin

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 20 Oct 2009 11:57:03 +0200

Hello,
Some general comments about system creatable from a late comer:
- system-created should be part of the validation e.g. unique, must, ...
- can you mark a leaf-list as system-creatable? I assume yes
- mandatory should be allowed for system-created as Andy proposed, meaning the system MUST 
create the leaf

- Ericsson is already using system-created list entries all over our models (OK today they are 
not YANG models, but the concept is the same). I mentally visualize this them as created by an 
autoconfig system using the NETCONF interface internally. This is not the best method, but OK

Balazs

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 20 Oct 2009 12:21:01 +0200

> (in the description statement).
So auto-configuration would not be allowed? I assume you are allowed to put a management app on 
your node, that can use the Netconf interface to configure the node itself.

Also as we are in a multi-user environment, the configuration can be changed between operations 
by another user, so such a MUST NOT CHANGE rules won't protect you really.

Balazs

---

From: Phil Shafer <[email-masked]>
Date: Tue, 20 Oct 2009 09:14:51 -0400

>autoconfig system using the NETCONF interface internally. This is not the best method, b

Are your list entries more "initial" or "default" configuration
than "s-c" config?  Are they merely the initial contents of the
configuration, or will they be system created at various points
in the lifespan of the device?

In JUNOS, we have "factory default" config that is loaded when
the box boots up with no config, but I consider this distinct
from "s-c" leafs, since "s-c" can appear during an edit-config
or commit-config operation.

Thanks,
 Phil

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 20 Oct 2009 15:56:24 +0200

Hello Phil,
Most of them are factory default/initial, but in our system any list ("managed object") can be 
marked as system created, meaning that it will be created once it's parent list (object) is in 
place.
E.g. When a 155 Mbit SDH interface is created an SDH VC-4 path layer is automatically created 
within.

Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320                  email: [email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Oct 2009 08:46:54 -0700

> - system-created should be part of the validation e.g. unique, must, ...

This seems to be a data-model dependent issue.
Validation still works, but if the server has
not created the node yet, then the candidate config will
not reflect what will be transferred to running
during the commit.  Problem?  Yes.  CLI Reality?  Yes.

> - mandatory should be allowed for system-created as Andy proposed,
> meaning the system MUST create the leaf
> 

My first interpretation of s-c mandatory was wrong.
I now think it still applies, but it means the client
is still responsible, not the server (server still MAY, not MUST).

If what you and Randy want is important, then it shows
that the s-c property is an enumeration, not a boolean.

   system-creatable (required, optional, not-allowed);

> - Ericsson is already using system-created list entries all over our
> models (OK today they are not YANG models, but the concept is the same).
> I mentally visualize this them as created by an autoconfig system using
> the NETCONF interface internally. This is not the best method, but OK
> 

In implementation, the module probably checks what explicit
config is already provided, and if none, sets up the hard-wired
initial settings.  This 'startup mode' may be an area of
future standardization, but it is far outside the scope
of current NETCONF specs.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 20 Oct 2009 10:14:13 -0700

Hi -

> To: "Balazs Lengyel" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, October 20, 2009 8:46 AM
....
...

To dispell any confusion about "what Randy wants"...

*If* there is to be a system-creatable property, it should
behave like this:  if a client fails to supply a value, then
the server MUST supply one, otherwise the creation
request fails.  In this sense it shares some similarity
with "mandatory", but should not be confused with it.

I think the enumeration/boolean question is the tip of the
proverbial iceberg that is the underlying object/configuration
lifecycle model we don't have.  Others' points about pre-configuration,
factory defaults, auto-configuration, and what happens in
multi-manager situations make that clear.

>From the perspective of a client, "optional" is not always
distinguishable from not-allowed.  Consider the multi-manager
situation.  (But hopefully this pathological case would be
avoided by partial locking and selective commits.  :-) * 0.5

However, the divergence in underlying assumptions about lifecycle
leads me to think that we'll end up not doing this property until
we develop a successor to yang and netconf.  It's too early to
nail it down now, and by the time we have enough experience
with yang models to come to agreement, there will be too much
diversity in deployed systems to permit standardization.

Randy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Oct 2009 11:14:42 -0700

> diversity in deployed systems to permit standardization.

Hmmm...
I don't think we need to wait that long to make progress.
The problem with the IETF these days is that there is
no sense of urgency to get anything done.

NETCONF/YANG will keep morphing via capabilities as we proceed.
There is never going to be one-size-fits-all, and that's
why N/Y has a chance.

<rant>
IMO, we have already wasted at least a year on corner-case
features that will only affect the most advanced usage.
Even still, there are years of 'advanced work' yet to
be done.

The NETCONF/NETMOD WGs will have to somehow convince
other WGs, vendors, and SDOs that N/Y is a better solution
that the current 'XML BCP' (write an XSD schema and create
your own protocol, or just copy the XML file to the device).

The first step in convincing anybody to use N/Y is to finish it.
Create a stable platform and build on it over time.  All
NETCONF work is currently held up, waiting for YANG to be published.
</rant>

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 21 Oct 2009 09:57:24 +0200

> If it is "MAY" the client will have to refetch the config to know

But with MAY, what happens if neither the client nor the server provides
a value? If the configuration parameter is needed for server operation
(and I guess most are), this would result in an error condition.

> that augments the replies for edit-config and commit-config.

I agree.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 21 Oct 2009 10:23:30 +0200

> by another user, so such a MUST NOT CHANGE rules won't protect you really.

Right, that's why I think the guarantees that s-c can potentially offer
are quite weak. So I'd say we shouldn't optimize prematurely.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 21 Oct 2009 10:56:29 +0200

Hello,
If we use system-created as opposed to system-creatable this helps, as you know that the system 
will definitevly create the value unless provided by the client. The manager does not need to 
create it.

One possible way to mark this is to use mandatory, but maybe "system-supplies-default" is better.
Balazs

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 21 Oct 2009 11:00:36 +0200 (CEST)

> are quite weak. So I'd say we shouldn't optimize prematurely.

I think there is a rather big difference.  When the server creates s-c
leafs, it does it as part of a client operation (commit or
edit-config), _even if the client has a lock_.  Anything that implements
changes as a client (even if it is a "built-in client" executing on
the device), cannot make any changes if there is a lock.   Marking
leafs as s-c tells the client what to expect also when it has the
lock.

/martin

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 21 Oct 2009 11:58:20 +0200

On 10/21/09 11:00, Martin Bjorklund wrote:
> Ladislav Lhotka<[email-masked]>  wrote:

Yes.

Also s-c can also create stuff _during_ an operation of another client while the built-in 
client is not allowed to do that.

I agree s-c is more powerful. Personally I would like to see a broad s-c; for leafs, 
containers, lists, leaf-list, with modifications (like Kent). But I can accept if that is YANG 
2.0 and Ericsson will just have some deviations/extensions.

Balazs

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 21 Oct 2009 11:58:57 +0200

Hello,
Is it proposed to allow s-c for leaf-lists?
Balazs

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 21 Oct 2009 13:00:28 +0200

> leafs as s-c tells the client what to expect also when it has the

Yes, I understand this, but does it really make a difference? Consider
this typical scenario:

lock
perform an operation
unlock

If the server runs a "post-operation" script immediately after the
unlock and that script adds any non-s-c values, how does it help the
client to know that the new leafs were NOT declared as s-c in the data
model?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: "tom.petch" <[email-masked]>
Date: Wed, 21 Oct 2009 14:38:21 +0200

----- Original Message -----
From: "Joel M. Halpern" <[email-masked]>
Sent: Monday, October 19, 2009 11:05 PM

Joel

Gerhard said that this was a use case for IPFIX at the back end of
August last year.  As one of the few people to have used YANG in
anger, I take that as a clear requirement.

I disagree with the current formulation of system-creatable on this
(I want MUST and not MAY), on restricting the system to be able
to create only nodes with system-creatable (that for me came from
nowhere) and with the leaf only existing after validation and so not
being available to any of the extra logic that YANG adds:-(

So as currently formulated, I find it of no additional value.
Tom Petch

---

From: Phil Shafer <[email-masked]>
Date: Wed, 21 Oct 2009 09:59:08 -0400

>(and I guess most are), this would result in an error condition.

If the server know that it the leaf must have a value, then
it will do The Right Thing and provide it.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 21 Oct 2009 16:43:28 +0200

> it will do The Right Thing and provide it.

How does the client know that the server knows? :-) The subsystem in the
server that operates on configuration data based on client requests
needn't be integrated with the subsystem that reads the configuration
and modifies server behaviour. The role of the former can be just to
make sure that the commited configuration is valid according to the data
model.

So I guess this boils down to the question whether the configuration
without the s-c leaf is valid or not.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 21 Oct 2009 09:09:46 -0700

> without the s-c leaf is valid or not.

This is why the mandatory-stmt is still valid,
even for leafs that have the s-c=true property.

The client knows that mandatory=true means it is responsible
for making sure the leaf exists.  The client cannot rely on
the server to create the leaf.  The description-stmt
should clarify the s-c property.  There may be inherent
complexity in the data model that requires human comprehension.
(YANG isn't going to solve that problem ;-)

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 21 Oct 2009 12:02:50 -0400

>the server to create the leaf.

These two paragraphs seem to read opposite.  If the client cannot
rely on the server to create the leaf, then it must create the leaf
itself, which means s-c leafs and mandatory can't coexist.

My view is that mandatory and s-c do not make sense, since the
client needs to be able to validate the config before sending it
to the server, and this combination is not enforcable on the client.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 21 Oct 2009 09:46:51 -0700

> to the server, and this combination is not enforcable on the client.

But mandatoty-stmt is not defined
the way you want.  It just means the
node must exist.  If the client is aware
that a particular server will create a particular
s-c leaf, then why force it to send the node?

The s-c=true is just a MAY create, not a MUST create.
That leaves a lot of fuzzy ground.

I think we need to drop s-c and move on.
We do not agree that one very narrow implementation
strategy is necessary or sufficient.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 21 Oct 2009 12:46:26 -0400

>s-c leaf, then why force it to send the node?

Mandatory means the node must exist in a valid config, so
both the client and server can enforce this.  The new
text for s-c leafs requires that s-c leafs not be mandatory
so that clients and servers can continue to enforce the
mandatory constraint.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 21 Oct 2009 11:05:57 -0700

Phil Shafer wrote:
> both the client and server can enforce this. 

huh?

the definition of mandatory-stmt does not
say anything about the client MUST create the value.

   leaf client-must-always-set {
      type string;
      mandatory true;
      system-creatable false;
   }

   leaf client-set-depends-on-data-model {
     type string;
     mandatory true;;
     system-creatable true;
   }

 The new
> so that clients and servers can continue to enforce the

That isn't true.
Since s-c=true means server MAY create,
the client (offline) validation for all nodes without a default-stmt,
regardless of the mandatory-stmt, is not reliable.

I do not see how these s-c leafs can be left outside part or all
of the validation model, especially since they become 'normal' leafs
once created.  This is not a workable solution.

This s-c statement is too narrow and creates too many CLRs
to be included in YANG 1.0.

Andy