Subject: [netmod] order of children

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 18 Nov 2009 10:56:27 +0100

Hi,

I have two questions regarding order of children:

1. Sec. 7.15.2 states:

   When a node is augmented, the augmenting child nodes are encoded as
   subelements to the augmented node, in any order.

   Is this true also inside RPC parameters?

2. container cont { 
      leaf-list foo { ... }
      leaf bar { ... }
   }

   If these definitions are not inside RPC parameters, is the following
   a valid order in an instance XML?

   <cont>
     <foo>...</foo>
     <bar>...</bar>
     <foo>...</foo>
   </cont>

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: WashamFan <[email-masked]>
Date: Wed, 18 Nov 2009 19:50:15 +0800

>    Is this true also inside RPC parameters?

This is true according to the text you quoted. And I think
it should be true.

>    a valid order in an instance XML?

I don't think so. foo should have preceded not surrounded
bar, IMO.

washam

> Lada
> 
> -- 
> Ladislav Lhotka, CESNET
> PGP Key ID: E74E8C0C
> 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 18 Nov 2009 13:21:06 +0100

> This is true according to the text you quoted. And I think

Since there is the requirement on fixed element order in RPC parameters,
I'd assume the fixed order is important in all cases, but with an
augment it won't be fixed anymore.

> I don't think so. foo should have preceded not surrounded

Sec. 7.5.7:
   If the container defines RPC input or output parameters, the 
   subelements are encoded in the same order as they are defined
   within the container statement. Otherwise, the subelements
   are encoded in any order.

As I understand it, all three ordering options are possible:
'foo foo bar', 'bar foo foo' and also 'foo bar foo'. Only the relative
order of list items may be important if the leaf-list is ordered-by
user.

I am happy with this, just asking whether this liberal interpretation
was really intended - it has implications for the DSDL mapping.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 18 Nov 2009 08:57:56 -0800

> was really intended - it has implications for the DSDL mapping.

I do not understand why we need any XML ordering CLRs.
They exist for the benefit of 2 specific implementations
of the <config> parameter to the edit-config operation.

IMO, the server MAY require RPC order, or
it MAY accept RPC nodes out of order.
Same for key leafs within in the PDU.
This is better than the current MUST terminology.

A server implementation other than the 2 that need
this CLR should not be forced to treat out-of-order
nodes (according to YANG CLRs) as errors.  This violates
the Postel Principle.

A client needs to be aware that a server MAY treat
out-of-YANG-order as an error, and a conservative
client will always send in YANG order, since all servers
MUST accept that.

> Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 18 Nov 2009 18:38:03 +0100

> client will always send in YANG order, since all servers

So what do you want the spec to state? The order of subelements is
relevant for both client->server and server->client communication.

I am personally in favor of allowing arbitrary order everywhere but IMO
the rules must be clear. Saying that clients MAY send children in any
order and the server MAY treat it as an error doesn't seem very useful.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 18 Nov 2009 09:48:30 -0800

> order and the server MAY treat it as an error doesn't seem very useful.

That isn't what I am suggesting.

    The server MAY reject <rpc> requests that are not in
    the YANG specified order.

This is much more robust than:

    The server MUST reject <rpc> requests that are not in
    the YANG specified order.

The only reason we have this CLR is to allow a particular
implementation strategy.  Both CLRs allow it, but the MUST
mandates a particular implementation strategy, and that
goes against IETF tradition (and perhaps policy).
If the server can accept XML elements in any order,
then that seems better than forcing it to check for
error conditions that are not really errors at all.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 18 Nov 2009 11:21:40 -0800

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Ladislav Lhotka" <[email-masked]>
> Cc: "NETMOD Working Group" <[email-masked]>
> Sent: Wednesday, November 18, 2009 9:48 AM
> Subject: Re: [netmod] order of children
>
> Ladislav Lhotka wrote:

When a specification uses a "MUST", it should be because doing otherwise
won't work reliably.  If servers MAY reject <rpc> requests that are
not in the YANG-specified order, then clients *MUST* send <rpc> requests
in the YANG-specified order to ensure interoperability.  The net effect
of all this is that even though servers wouldn't be required to enforce
ordering, a conformant client would be required to abide by it.

If we permit clients to mess with the ordering, while permitting
servers to enforce it, interoperability will suffer.

Randy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 18 Nov 2009 12:21:52 -0800

> servers to enforce it, interoperability will suffer.

Be conservative in what you send.
Be liberal in what you accept.
This has worked out pretty well so far.

So what is the CLR?
The client MUST send top-level children of the rpc-operation
node in exact order, but all children of those nodes
may appear in any order?

  <rpc>
    <rpc-operation>
       <exact-order1>
          <any-order1>7</any-order1>
          <any-order2>fred</any-order2>
       </exact-order1>
       <exact-order2>barney</exact-order2>
    </rpc-operation>
  </rpc>

So a server needs code because it MUST support arbitrary order at
XML layer 4+, and also MUST enforce an exact order at XML layer < 4?
Is that the CLR the WG wants?  I don't understand
why this is a MUST requirement in the protocol.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 18 Nov 2009 21:41:54 +0100 (CET)

> node in exact order, but all children of those nodes

Can't we stick with the current CLR that in RPCs, all elements are
sent in the order defined?

The problem is, as Lada pointed out, what to do with (multiply)
augmented nodes:

  rpc foo {
    input {
      container x {
        leaf a { ... }
        leaf b { ... }
      }
    }
  }

  augment /foo/input/x {
    leaf c { ... }
  }

  augment /foo/input/x {
    leaf d { ... }
  }

'a' and 'b' must be sent in that order, and we could say that any
augmented nodes are sent after the other nodes, in any order, so this
is ok:

  <foo>
    <x>
      <a>...</a>
      <b>...</b>
      <d>...</d>
      <c>...</c>
    </x>
  </foo>

/martin

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 18 Nov 2009 13:04:56 -0800

Hi -

> To: "Randy Presuhn" <[email-masked]>
> Sent: Wednesday, November 18, 2009 12:21 PM
...
> This has worked out pretty well so far.

We agree on this.

This is the principal that effectively turns a
server's "MAY" ("liberal in what you accept")
into a client's "MUST" ("conservative in what you send")
if interoperability is to be maintained.

If the WG believes there is value in permitting, but
not requiring, server implementations to ignore
ordering here, then the logical consequence of that
decision is to REQUIRE that clients to send only
ordered stuff.

I'm not saying that I like this.  Options and flexibility
come at a cost, and should provide commensurate benefits.
Presumably the WG has concluded that some benefit is provided
by permitting servers to accept "disordered" <rpc> requests,
even though a robust client should never send them if it is
going to interoperate with servers that *do* happen to enforce
the order.

Randy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 18 Nov 2009 15:46:53 -0500

>sent in the order defined?

Isn't that a CLR?  What is the benefit from enforcing this?  IMHO
the only ordering with a benefit is the "keys first" one, since
this means buffering is not required.  Beyond that, is there a
strong reason that order should matter?

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 18 Nov 2009 22:13:32 +0100 (CET)

> strong reason that order should matter?

There is a buffering issue in copy-config with inline config.  If you
send the target *after* the inline config, the server has to buffer
the entire config before it knows which datastore it will go to.

/martin

---

From: Phil Shafer <[email-masked]>
Date: Wed, 18 Nov 2009 17:36:59 -0500

>the entire config before it knows which datastore it will go to.

Ah, cool.  Then I'm fine w/ ordering rpc elements.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 18 Nov 2009 16:22:33 -0800

> the entire config before it knows which datastore it will go to.

I think the CLR is under-specified, and at the same time,
it is over-reaching.

As written, it seems to say that all descendant nodes
of the rpc-operation are ordered.  The contents of
the <config> node in edit-config/copy-config
is an exception because it is type anyxml.

This is where the CLR really falls apart.
The server is expected to check for ordering
within the anyxml node and make sure keys are first, etc.,
even though the YANG definition of the config node
is an opaque 'anyxml' subtree.

Augmenting nodes are unordered because the order depends
on lots of implementation details.  Top-level
nodes contained in submodules are also implementation ordered.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 19 Nov 2009 08:06:36 +0100

> augmented nodes are sent after the other nodes, in any order, so this

Looks good. How about the other issue - interleaving list items with
other sibling elements?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Nov 2009 08:42:55 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> of the rpc-operation are ordered.

Yes.

> The contents of

Yes.

> This is where the CLR really falls apart.

I disagree.  The draft doesn't say anything about order within anyxml
(obvsiously; since it is opaque).  If all YANG did was to specify rpcs
we would be done here.  But YANG goes further, and specifies the
contents of the data stores.  This content has additional requirements
on keys and ordering, and the server is expected to check for this.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Nov 2009 08:49:11 +0100 (CET)

> Looks good. How about the other issue - interleaving list items with

I think we discussed this at some point (see the thread starting at
http://www.ietf.org/mail-archive/web/netmod/current/msg01867.html).

I'm fine either way.  Currently there is no text that says that you
cannot interleave (leaf-)list entries with other siblings.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 19 Nov 2009 09:31:35 +0100

> cannot interleave (leaf-)list entries with other siblings.

OK, do we all agree that interleaving is allowed? If so, it should be
explicitly noted in sections 7.7.6 and 7.8.5, otherwise some
implementors may be taken by surprise. Something like this can be added
as the second paragraph in both sections:

The elements representing list items MUST appear in the order specified
by the user if the (leaf-)list is "ordered-by user", otherwise the order
is implementation-dependent. In both cases, the elements representing
list items MAY be interleaved with other sibling elements. 

Lada
  
-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 19 Nov 2009 09:45:58 +0100

Hi,

> http://www.ietf.org/mail-archive/web/netmod/current/msg01867.html).

Another issue that was discussed in this thread and that is perhaps
worth mentioning in the YANG spec (or NETCONF?) is the persistence of
element order, namely:

1. If the client does a copy-config and then retrieves the same 
   datastore, the order of elements MAY be different.

2. If the same get(-config) operation is performed twice on the same 
   datastore with the same contents, the order of elements MUST be 
   identical.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 19 Nov 2009 04:14:14 -0800

> on keys and ordering, and the server is expected to check for this.

There is no basis in XML for these hacks.
They are just CLRs specific to YANG.
YANG does not specify any order within an anyxml node.

There is only hand-waving that get you
from anyxml to an ordered database.  As you pointed out,
the standard ignores how an opaque XML blob is
recognized as specific YANG database subtrees.

Yet you want CLRs on the order of arbitrary sub-trees
within an arbitrary XML blob.  This is a new class of CLR.
How are you going to specify the CLR precisely enough so it
can be implemented consistently? Are you going to say
that anyxml nodes named 'config' and 'data' are special in YANG?
What criteria does an implementor use to determine that
a particular anyxml blob is really the top-level root
of a YANG database?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Nov 2009 13:31:59 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> within an arbitrary XML blob.

No.  I never said that.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 19 Nov 2009 05:24:43 -0800

>> within an arbitrary XML blob.

How does the 'keys first' rule apply to a
node nested within an 'anyxml'?  In order to do that,
the server must ignore the rules for anyxml
and decide 'this XML blob is a list'.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Nov 2009 14:31:48 +0100 (CET)

> and decide 'this XML blob is a list'.

Yes, the server treats the anyxml in edit-config different than other
anyxml.  Ordering is just one aspect.  But it also checks the
structure and type of leafs etc.  The whole point of YANG is to
describe the contents of the data store, so the server better have
special handling for this anyxml.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 19 Nov 2009 06:06:37 -0800

> special handling for this anyxml.

I use an extension called ncx:root to deal with this problem,
instead of hardwiring the QName of the special node.

http://www.netconfcentral.org/modules/ncx/2009-06-12#root.455

I don't think the draft makes it clear that some special
mechanism is needed to override YANG anyxml semantics.
It seems to me the DSDL mapping will not make it clear
to applications that a node in an rpc/input is supposed
to be a config root, and therefore the only valid child
nodes are top-level YANG module nodes advertised by the server.

How does an application know that an anyxml blob called <filter>
really is just an XML blob, but another one called <config>
is expected to be structured as a config root?
(A: description-stmt, if any)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Nov 2009 15:09:37 +0100 (CET)

> (A: description-stmt, if any)

Yes, if any.  The YANG draft already talks about how data nodes can be
modified with edit-config.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 19 Nov 2009 17:47:37 +0100

> is an opaque 'anyxml' subtree.

I understand this 'anyxml' as a placeholder for the content layer to
step in and define the structure of this blob. It is pretty much like
the layering of network protocols: for IP, everything after byte 20 is
opaque data, but TCP/UDP then provides more structure.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 19 Nov 2009 09:40:06 -0800

> opaque data, but TCP/UDP then provides more structure.

I think YANG leans the other way -- saying that a server
does not have to support must/when 'inside' an anyxml node.
Maybe the placeholder/layering text like your paragraph
above needs to be added to the section on anyxml

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 19 Nov 2009 11:11:56 -0800

Hi -

> From: "Ladislav Lhotka" <[email-masked]>
> To: "Martin Bjorklund" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, November 19, 2009 12:45 AM
> Subject: [netmod] persistence of element order
...
>    datastore with the same contents, the order of elements MUST be 

Be careful what you ask for.  :-)  This hinges on what you mean by
"the same contents".  Consider the case of a system with three
datastores I'll call A, B, C for simplicity.  B and C are "the same"
in that they have exactly the same information, differing only in the
order of some non-ordered elements, due to the way in which they
were created.  Client X does a copy-config B->A.  Client X does
a get of A, and gets something that looks exactly like B.  Client Y
does a copy-config C-> A.  Client Y does a get of A and gets something
that looks exactly like C.  So far, everyone is happy, and someone
observing both conversations will say the implementation didn't
exercise the "MAY" in (1) above.  Now client X does another
get of A.  If (2) above is to be honored, the server is in trouble.

Randy

---

From: Kent Watsen <[email-masked]>
Date: Thu, 19 Nov 2009 13:10:15 -0800

> opaque data, but TCP/UDP then provides more structure.

This won't always be the case.  For instance, JUNOS uses some anyxml nodes for undocumented/hidden features.  Only customers with explicit knowledge of the feature are able to configure it.  In this case, the application is never provided the schema to understand the blob.

Maybe the solution here is to let the data-model define if order matters - like the difference between xs:sequence and xs:all...

Kent

---

From: WashamFan <[email-masked]>
Date: Fri, 20 Nov 2009 09:55:50 +0800

>  get of A.  If (2) above is to be honored, the server is in trouble.

I'm afraid (2) above didn't mean this. datastore A when client
X does the first get is NOT "with the same content" as datastore A
when client X does the second get. (2) above wanted to say the datastore
content should not be changed during the two identical operation performance, IMO.

washam

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 20 Nov 2009 10:21:49 +0100 (CET)

> list items MAY be interleaved with other sibling elements. 

Unless anyone objects, I will add this text to 7.7.6 (leaf-list
version) and 7.8.5 (list version):

  The XML elements representing leaf-list entries MUST appear in the
  order specified by the user if the leaf-list is "ordered-by user",
  otherwise the order is implementation-dependent.  The XML elements
  representing leaf-list entries MAY be interleaved with other sibling
  elements.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Nov 2009 03:30:37 -0800

>   representing leaf-list entries MAY be interleaved with other sibling

Isn't it possible for lists to be interleaved as well?

Does this impact the CLR you recently added about case members
appearing in exact order?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 20 Nov 2009 12:35:06 +0100 (CET)

> Isn't it possible for lists to be interleaved as well?

Yes, that's 7.8.5, list version.  Same text as above with s/leaf-//g

> appearing in exact order?

No, that was for rpc input / output.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Nov 2009 08:11:11 -0800

> No, that was for rpc input / output.

But what if a list or leaf-list is a descendant of input-stmt
or output-stmt?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Nov 2009 21:50:03 +0100 (CET)

> But what if a list or leaf-list is a descendant of input-stmt

Does this work?

  elements, unless the leaf-list defines RPC input or output
  parameters.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Nov 2009 13:49:36 -0800

>   elements, unless the leaf-list defines RPC input or output

The problem is that <config> within edit-config and copy-config
are RPC input parameters.  Other than the optional <url>
parameter, there is no other way to specify configuration data
in NETCONF.

It seems to me that ALL rpc/input descendant nodes
are strictly ordered, and the 'MAY be interleaved'
only applies to remote <url> based configs.

Even though the client MUST send the correct order
to be safe, there is no normative text that says the
server MUST NOT accept interleaved RPC input parameters.

IMO, the CLR needs to be simple and consistent.
All rpc/input descendant nodes are ordered,
according to the spec.  The text should say
the client MUST NOT interleave, and that gives a server
the ability to refuse out-of-order rpc/input subtrees.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Nov 2009 22:55:55 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> are RPC input parameters.

1.  Are you suggesting that this would not be a problem if we didn't
    have a YANG model in 4741bis?

2.  Even with the YANG model in 4741bis, the config is defined as an
    anyxml, and the draft does not say anything at all about encoding
    within anyxml.

So is this really a problem?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Nov 2009 17:06:07 -0800

>     have a YANG model in 4741bis?

No -- it has to do with any <rpc>, not just the
ones in ietf-netconf.yang.

> So is this really a problem?

When can the client interleave list or leaf-list entries
with other siblings?  Is it in any message the client
sends to the server except <rpc>? (which is just <hello>)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Nov 2009 10:05:16 +0100 (CET)

> sends to the server except <rpc>? (which is just <hello>)

It cannot interleave if the node is a descendant of the rpc statement,
possibly via uses, or if it is augmented into a descendant of the rpc
statement.

As soon as the rpc contains anyxml, there are no rules for the
encoding of the anyxml contents in general.

But for the anyxml in edit-config and copy-config in
ietf-netconf.yang, the encoding rules for data nodes apply, which
means that nodes may be interleaved.

/martin