Subject: [netmod] validation of default up to server

From: Andy Bierman <[email-masked]>
Date: Fri, 18 Sep 2009 13:37:00 -0700

Hi,

7.6.5.  XML Mapping Rules

   A leaf node is encoded as an XML element.  The element's name is the
   leaf's identifier, and its XML namespace is the module's XML
   namespace.

   The value of the leaf node is encoded to XML according to the type,
   and sent as character data in the element.

   A NETCONF server that replies to a <get> or <get-config> request MAY
   choose not to send the leaf element if its value is the default
   value.

According to this normative text in 7.6.5, it is clear that
a server MAY choose to create node tree values for YANG default
leafs.

This is the rationale for explicit-mode according to Phil.
A server MAY choose to create leafs with the default value,
so the validation rules for leafref and i-i may or may not work,
depending on whether the server chooses to instantiate
particular nodes, even if they have the default value.

I do not understand why it is important that a default node NOT
exist in the database.  The spec already says the server can
do whatever it wants wrt/ returning leafs to the client.

The validation model is broken because some validation
is done with defaults, and some is not, and every server
gets to choose on their own what an 'instance' means.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Sat, 19 Sep 2009 10:09:37 -0400

>a server MAY choose to create node tree values for YANG default

Yes, this was done for two scenarios:
(a) the value was explicitly set to the a value which is the default
(b) the server is auto-populating the config database with
    default values even when they are not being explicitly set

>This is the rationale for explicit-mode according to Phil.

(a) is me ("explicit"), (b) is you ("report-all").  The spec gives
the flexibility to work both ways or the "trim" way.

>do whatever it wants wrt/ returning leafs to the client.

This is old ground.  Are you really not understanding or are you
just upset that it doesn't force everyone to work the way you want?

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Sat, 19 Sep 2009 09:12:32 -0700

>     default values even when they are not being explicitly set

There is no mention of explicitly set anywhere in the
YANG draft.  Any node MAY be created by the
server -- even a default leaf.

> just upset that it doesn't force everyone to work the way you want?

It is not old ground.
The validation model is broken because leafref and i-i treat a
leaf with a default value as non-existent according to Martin.

But I think he is wrong, and the text lets me implement
this on the server any way I want.  If I want a node
with a default value to exist, that is OK for the server to
treat the default as a node.

The client and data modeler still have no idea what to
expect from any given leaf on any given server, but so what, right?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 19 Sep 2009 09:43:26 -0700

> just upset that it doesn't force everyone to work the way you want?

If you would just address the new issues that I keep finding (not you),
or if the co-Chairs ever announced any thread conclusions, then this
process would not take so long.

It is fine with me if the co-Chairs want to record
a strong objection in the doc-write-up, and move on.
The APPs A-D may or may not care about these issues.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 21 Sep 2009 10:16:22 +0200 (CEST)

> server -- even a default leaf.

I think we agreed that we should introduce a new statement
'system-assignable' (was: assigned-by system), which controls this.
With this statement, it should become clear that only nodes marked as
system-assignable true MAY be created by the server.  I have an action
point to provide some initial text for this, to be further discussed
on the ML of course.

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 21 Sep 2009 12:51:34 -0700

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Monday, September 21, 2009 1:16 AM
> Subject: Re: [netmod] validation of default up to server
...
> system-assignable true MAY be created by the server.
...

This doesn't sound right.  Rephrasing what I think you wrote, you'll
see why I think it isn't right:

Nodes not marked as system-assignable MUST NOT be created
by the server.

Wouldn't this exclude defaults specified by the model?  It seems to
me that defaults specified by the model are a special case of 'system-assignable'.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 21 Sep 2009 22:40:09 +0200 (CEST)

> me that defaults specified by the model are a special case of

The intention is that defaults specified by the model are not created
by the server, i.e. they are not part of the database.  This is the
"explicit" model for handling defaults.

However, since we thought that it would not be possible to agree on
one default handling method, the spec allows a server to send back the
default values in a reply to <get-config>, so that the "trim" and
"all" methods can be implemented as well.

Do you want to say that default values MAY be assigned by the system?

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 21 Sep 2009 14:12:09 -0700

Hi -

> To: <[email-masked]>
> Cc: <[email-masked]>
> Sent: Monday, September 21, 2009 1:40 PM
...

No.  I'd phrase it more like: "if a value is not supplied by the client, the
default value specified in the model MUST be assigned by the system."

Stepping back a bit, if I had to use this stuff, here's how I'd like it to work:

    - for purposes of consistency checks and conditional retrieval (e.g.
      using xpath to find "interesting" parts of the configuration, etc.)
      the system must behave as though "default" values actually exist,
      regardless of how they are (not) represented in the internal data store;

    - it must be possible to retrieve the complete set of values actually being used
      as configuration data, regardless of whether explicitly set by the user
      or not, and regardless of how they are (not) represented in the internal
      data store;

    - for some applications it is desirable to be able to retrieve only the subset of
      configuration data  which has been explicitly configured, along with those
      configuration values  supplied by the server which are not specified in
      machine-readable form in the model.  Since this requires an implementation
      to remember how a bit of configuration came to have its value, I'd consider
      this to be an extra-cost (but perhaps high-value) option, rather than a basic
      mode of operation.

Randy

---

From: Kent Watsen <[email-masked]>
Date: Mon, 21 Sep 2009 18:27:33 -0700

A couple related things we considered:

1. fetching config should include a flag to indicate if defaults should be returned.  If defaults are returned, then they should be marked with an attribute (i.e. default="true") so an app can differentiate them

2. marking nodes as server-assignable is a start, but we took it a step further and created schema that described which operations would assign a node.  We call these "implicit-changes" and also implemented a mechanism whereby the <rpc-reply> can provide an XPATH to each implicitly modified node.  This way a app knows which tree nodes it needs to re-fetch

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Mon, 21 Sep 2009 19:56:55 -0700

> 1. fetching config should include a flag to indicate if defaults should be returned.  If defaults are returned, then they should be marked with an attribute (i.e. default="true") so an app can diff…

Phil says that Juniper customers do not want to
see the defaults.  You are saying that you want
the defaults, plus some meta-data flagging the defaults.

The issue with adding meta-data is whether the client
will deal with it properly.  There is an expectation
that the XML returned in <get-config> can be used directly
in a <copy-config> or <edit-config>, back to the same server.

> 2. marking nodes as server-assignable is a start, but we took it a step further and created schema that described which operations would assign a node.  We call these "implicit-changes" and also im…

I've seen an interesting CableLabs document (CM-SP-EQAM-PMI-I01-081209)
that identifies which CRUD operations are allowed for every parameter.

I prefer an algorithmic approach for YANG, if possible, but
this basic idea is still much better than ad-hoc description statements.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 22 Sep 2009 09:37:52 +0200 (CEST)

>       regardless of how they are (not) represented in the internal data store;

Ok.

>       or not, and regardless of how they are (not) represented in the internal

Ok.

>       machine-readable form in the model.

Ok.

>       Since this requires an
>       implementation
>       to remember how a bit of configuration came to have its value,

This is implementation specific.  In an implementation which doesn't
store the defaults, there is no extra bit to consider.

It seems we agree on the functionality needed.  But we disagree how it
should be specified.

The problem with always sending all defaults, is that save / restore
will not produce the same result in an implementation that does not
store defaults, since the <get-config> will include all default, and
the <edit-config> (or <copy-config>) will explicitly set all
defaults.

The approach taken in the YANG spec is the opposite.  It allows both
types (actually all three as defined in with-defaults) of
implementations, and the with-defaults I-D allows the client to
control this behavior further.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 22 Sep 2009 07:25:57 -0700

> It seems we agree on the functionality needed.  But we disagree how it

In the conceptual database model, a leaf containing the default value
exists in all implementations.  If there is any value in use at all,
regardless of how it was created, then a 'report-all' retrieval
request will return this leaf.  It is not an implementation-specific
concept at all.  Every server could generate this bit.

> the <edit-config> (or <copy-config>) will explicitly set all

I hope we are not back to Strawman #37:  The only choices are
never send the defaults are always send the defaults, so
better to choose 'never'.

The solution almost 2 years in development by the NETCONF WG
is 'with-defaults'.   If all servers supported 'report-all',
then the problem is solved.  The NETMOD WG is ignored chartered
work almost done in the NETCONF WG, and talking about starting
over in NETCONF with new protocol operations.  This must be
a new definition of 'synergy' I am not familiar with.

> control this behavior further.

YANG allows every NETCONF server to decide what each YANG module means,
how it will be validated, and what NETCONF operations will actually
work.  A leafref or i-i can never be set
safely to any value, based on the YANG module, because
a server can provide a different default
to every single session if it wants, via the deviation-stmt.
This is not usable as a standard.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Tue, 22 Sep 2009 10:45:50 -0700

> > 1. fetching config should include a flag to indicate if defaults should
> be returned.  If defaults are returned, then they should be marked with an
> attribute (i.e. default="true") so an app can differentiate them
> >
> 

Sorry if I was a bit ambiguous, but I was referring to the SOAP interface for our mgt-app; though I believe the same logic applies to the NetConf returned by the device.  Point being is to give the clients a choice, but having meta-data flagging the defaults, when returned, is key

Essentially, some clients are smart and able/prefer to figure the impact of defaults for themselves, saving a large amount of bandwidth and then there are the dumb/thin clients that need to have the data-sourcing information provided in the response

> in a <copy-config> or <edit-config>, back to the same server.
 
Maybe the spec could satisfy this expectation by stating that the NetConf server would ignore nodes marked with the "default" attribute in <copy-config> and <edit-config> operations?

> > 2. marking nodes as server-assignable is a start, but we took it a step
> further and created schema that described which operations would assign a
> node.  We call these "implicit-changes" and also implemented a mechanism
> whereby the <rpc-reply> can provide an XPATH to each implicitly modified
> node.  This way a app knows which tree nodes it needs to re-fetch
> >
> 

OK, FWIW, here is more detail on what we do (note: we're using XSD for all this):

  - in the <appinfo> of the source-node whose modification would trigger an
    implicit change, we can have the following, which indicates how a server
    might modify other nodes in the config for various <edit-config>
    operations:

      <xs:element name="implicit-changes" minOccurs="0">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="implicit-change" maxOccurs="unbounded">
              <xs:complexType>
                <xs:sequence>

                  <!-- NETCONF <edit-config> operations      -->
                  <xs:element name="source-operation"/>
                    <xs:simpleType>
                      <xs:restriction base="xs:string">
                        <xs:enumeration value="merge"/>
                        <xs:enumeration value="replace"/>
                        <xs:enumeration value="create"/>
                        <xs:enumeration value="delete"/>
                      </xs:restriction>
                    </xs:simpleType>
                  </xs:element>

                  <!-- NETCONF <edit-config> operations       -->
                  <!-- Optional if same as <source-operation> -->
                  <xs:element name="target-operation" minOccurs="0"/>
                    <xs:simpleType>
                      <xs:restriction base="xs:string">
                        <xs:enumeration value="merge"/>
                        <xs:enumeration value="replace"/>
                        <xs:enumeration value="create"/>
                        <xs:enumeration value="delete"/>
                      </xs:restriction>
                    </xs:simpleType>
                  </xs:element>

                  <!-- The XPath(s) of target(s) of the      -->
                  <!-- implicit change                       -->
                  <xs:element name="path" type="xpath-value"/>

                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Tue, 22 Sep 2009 11:21:30 -0700

> Sorry if I was a bit ambiguous, but I was referring to the SOAP interface for our mgt-app; though I believe the same logic applies to the NetConf returned by the device.  Point being is to give the…

Well, the current YANG proposal allows NETCONF servers to
always leave out default nodes, let alone extra meta-data
describing how each value was added to the conceptual database.

> Maybe the spec could satisfy this expectation by stating that the NetConf server would ignore nodes marked with the "default" attribute in <copy-config> and <edit-config> operations?

IMO, this is a bit overkill, but not unreasonable to have in
the standard somehow.  I was thinking of qualified XML attributes
stuffed into specific data nodes (in PDUs), like we already
do with the nc:operation, yang:insert, yang:key, and yang:value attributes.

E.g, yang:is-default="true"

  <xs:attribute name="is-boolean" type="xs:boolean" default="false" />

>     might modify other nodes in the config for various <edit-config>

So what PDUs would include an instance document conforming
to this XSD?  Does the <implicit-changes> element appear
in NETCONF PDUs?  IT is unclear how this XSD is used.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Wed, 23 Sep 2009 14:10:13 -0700

>   <xs:attribute name="is-boolean" type="xs:boolean" default="false" />

Right, I think we're saying the same thing.  For instance:

<get-config>
   <yang:expand/>  // client requests server to also return defaults
</get-config>      // PS: I don't care what the flag is called

RPC Reply:

  ----SNIP----
  <interfaces>
    <interface>
      <mtu>1400<mtu>  // user has explicitly configured MTU = 1400
      ...
    </interface>
    <interface>
      <mtu>1500<mtu>  // user has explicitly configured MTU = 1500
      ...
    </interface>
    <interface>
      <mtu is-default="true">1500<mtu>  // no user value, default used
      ...
    </interface>
  </interfaces>
  ----SNIP----

> in NETCONF PDUs?  IT is unclear how this XSD is used.

No PDUs would contain this appinfo, it is only in our config schema so the app can determine which config node(s) would need to be re-imported thru static analysis.  Anyway, the point is that we can do better than just marking nodes as <server-assignable> - it's more useful to detail when/how they are server-modified

That said, we also defined a ":implicit-change" capability to extend the <get-config> rpc-reply (using <rpc-error> with severity="warning") to dynamically return a list of nodes implicitly changed by the server.  Thus it is up to the device team if they want to model it or calculate it at runtime

Thanks,
Kent