Subject: Re: [netmod] prefixes in groupings example

From: Martin Bjorklund <[email-masked]>
Date: Sat, 01 Aug 2009 10:07:05 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> Martin Bjorklund wrote:
> > Andy Bierman <[email-masked]> wrote:
> >> This text does not fully address the example I posted (footest2.yang).
> >>
> >> The problem is more complicated than just must/when expressions.
> >>
> >> If the local prefix is not used (the normal use-case), then
> >> the module with the uses-stmt will be used to find
> >> identities and types, and resolve the default-stmt.
> >> This is broken.
> > 
> > I agree that this would be broken.  But that is not how it works.  I
> > was a bit unclear in my quote of the proposed text -- the text was
> > from section 6.4.1, the XPath context, which applies to must/when/path
> > only.  Below I have attached the complete text for this section.  The
> > rules for non-xpath statements are the same as before.
> > 
> 
> how does this apply to types and identityrefs, and defaults?

It does not change.

[...]

> Before, all of these statements that use QNames will use the
> module that defined the statement as the local module,
> to resolve the missing prefix .

Yes.

> Now these are all changing too, right?

No.

> Are you saying that some prefixes (or missing prefixes)
> inside a grouping will mean one thing, and mean another
> thing just inside must/when statements?

They already do.  In YANG statements, the prefix refers to a module
(or even a particular version of a module).  In XPath, the prefix
refers to an XML namespaces.

For a module designer, the simple rule of thumb is "do not use
prefixes for local references" (which also saves typing and makes the
module easier to read).

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 01 Aug 2009 13:37:12 +0200

> refers to an XML namespaces.

Or putting it differently: names in XPath expressions refer to nodes in
the instance data tree, other names refer to "schema objects" (nodes,
typedefs, groupings, identities, ...).

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Sat, 01 Aug 2009 08:11:01 -0700

>> Now these are all changing too, right?

Well -- this is broken.
Why is it OK for a prefix in a default-stmt to do
the wrong thing, but not a must-stmt?
IMO, your proposal is extremely fragile
and impossible for off-the-shelf tools to implement.

It is also broken, since the exact same QName used
in a default-stmt (my-identity in the example)
will be treated differently in a must or when statement.

Can you explain why you think only the prefixes in
an XPath expression are a problem, but not other
uses of the local prefix?

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 01 Aug 2009 18:16:48 +0200

> and impossible for off-the-shelf tools to implement.

Why? The implementation just has to be careful about the context in
which QNames (actually only those without any prefix) are used.

> will be treated differently in a must or when statement.

This is because the use of prefixes is overloaded. QNames inside XPath
expressions point to qualitatively different things than those outside.

> uses of the local prefix?

All prefixes (be they in XPath or not, in a grouping or not) are
resolved in the module where they appear. Only the handling of names
without prefixes differs because we want the late binding for names of
data nodes in groupings. This late binding of course must not happen for
names schema objects. However, these two cases are easily separable on
the basis of the statement in which they appear.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Sat, 01 Aug 2009 09:38:07 -0700

> the basis of the statement in which they appear.

Look at the example I posted.
The ambiguity for no-prefix resolution
exists everywhere.  Your proposal will
only make things worse by creating special rules
for some statements, while leaving other statements
in the same grouping broken.

The prefixes are not special because they are in must/when
expressions.  These are NOT XML prefixes at all.  They
are within a YANG module, and they are using YANG prefixes,
not XML prefixes.

The mapping from YANG module namespace to XML namespace is
an implementation detail that applies to every
part of YANG, not just must/when statements.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sat, 01 Aug 2009 21:49:01 +0200 (CEST)

> are within a YANG module, and they are using YANG prefixes,

No they *are* different in XPath than in ordinary YANG statements.  In
XPath they are used to map to an XML namespace, in YANG statements
they are used to map to a (specific version of a) YANG module.

At this point I am not sure I understand your objection.  Do you think
that there are technical problems with the solution?  If so, what are
they?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sat, 01 Aug 2009 15:20:15 -0700

> they are used to map to a (specific version of a) YANG module.

That seems irrelevant to me.

> that there are technical problems with the solution?  If so, what are

must and when statements are not the only YANG constructs
that can be defined within a grouping that can have an implied prefix.

mod 1: prefix foo1;

    grouping ggg {
       container test-top {
         leaf bar { type uint8; }
         leaf baz {
             must "../goo = 'my-identity' and . > 42";
             type MyType;
         }
         leaf goo {
            type identityref {
               base MyBase;
            }
            default my-identity;
         }
      }
   }

mod 2: prefix foo;
   container one {
        uses foo1:ggg;
    }
}

In this example, the must expression will reference
the foo:my-identity definition, but the default
references the foo1:my-identity.  This is not obvious at all.

The type-stmt references the foo1:MyType typedef for leaf 'goo',
but in XPath the leaf 'goo' will use the other definition of MyType,
which is a boolean, not an int32.  This seems broken to me.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 02 Aug 2009 13:10:13 +0200

Hi,

we should strictly distinguish between data node identifiers that live
in a "target XML namespace" and become names of XML elements in instance
documents, and names of "schema things" that essentialy belong to a YANG
module - schema nodes, types, groupings, identities.

Only the former - if they appear in a grouping and have no prefix - may
be subject to the late namespace binding.

It is IMO a bit unfortunate that these two namespaces are handled as
one. At least we now don't use XPath for addressing schema nodes (in
'augment' and 'refine').

The identityrefs are really very tricky and in XPath expressions they
seem to be almost unusable since they appear as string literals. It is a
technical problem although it wasn't introduced by the new text. IMO the
only sensible way out may be a custom XPath function identityref()
performing the type casting. I guess we are stretching poor XPath too
far here.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Sun, 02 Aug 2009 07:12:36 -0700

> be subject to the late namespace binding.

why is that, exactly?

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 02 Aug 2009 17:48:30 +0200

> > be subject to the late namespace binding.

That's my understanding of how groupings are supposed to work. Is it
insufficient?

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Sun, 02 Aug 2009 09:57:56 -0700

> That's my understanding of how groupings are supposed to work. Is it

yes.  It is not my understanding of how the default prefix works.
IMO, it is extremely non-obvious that, within a YANG file,
the same QNames will mean different things in various contexts.

IMO, one of the best tests of a good modeling language is how easy or difficult
it is for an author to mess up and inadvertently do the wrong thing.
The 'astonishment factor' in YANG is fairly high, and about to get worse.

Andy

---

From: "tom.petch" <[email-masked]>
Date: Wed, 5 Aug 2009 17:31:15 +0200

----- Original Message -----
From: "Ladislav Lhotka" <[email-masked]>
To: "Andy Bierman" <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, August 02, 2009 1:10 PM

> module - schema nodes, types, groupings, identities.

Disagree: there is something going on here, but I have yet to get my mind around
what it is.

Schema nodes include list, leaf-list, list etc and I think that their identifier
structure strictly parallels that of the data tree, the obvious qualification
being the use of keys to identify items in a list (OID fragment, if you know
SMI).

I think that 'XML namespace', although used in the I-D, is the wrong term.
Rather an identifier has local and global parts, and the global part is a URI
and must be unique, not just within YANG but within URI used as XML namespaces
etc.  But calling it an XML namespace, as the I-D does, seems
to add nothing of value; it isn't a namespace.

As the I-D says, there are multiple identifier namespaces, one per parent node,
and we have no separate identifier for each of those identifier namespaces; they
all come under the namespace URI, which is bound to the module with the
namespace statement, without differentiation.

And that is equally true for schema tree and data tree.

So the issue, which may now be resolved, is of what happens to identifiers
without the prefix which implies a global part, and when is that default or
explicit prefix turned into a global part.  I think the characterisation that a
prefix sometimes refers to a module, and sometimes to an XML namespace, is
wrong; for me, it always refers to a URI, a global part.

Early and late provision of the default prefix (not binding, to me that is of
prefix to URI), I am comfortable with.

Tom Petch

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 05 Aug 2009 09:51:39 -0700

> wrong; for me, it always refers to a URI, a global part.

nice summary

> prefix to URI), I am comfortable with.

I can live with it -- I just want it clear to users.

We have seen plenty of evidence that the 'distributed'
nature and sheer size of the NETCONF RFC has led to
lots of interoperability problems.

Nobody is going to guess that no-local-prefix means
different things for different statements, but only
in groupings.

Astonishment:fragility is 1:1.  We should know better by now.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 05 Aug 2009 13:54:12 -0700

> Disagree: there is something going on here, but I have yet to get my mind around

the following YANG shows that this problem
is not related to the XML namespace in an XPath expression,
but also (at least) type names.

   grouping hhh {
       container test-top {

         typedef MyType { type int8; }

         leaf baz {
             type MyType;
         }
      }
   }

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 07 Aug 2009 17:57:42 -0700

>          typedef MyType { type int8; }

Here is the complete example to show that the proposed
solution does not work:

     grouping hhh {
        container test-top {

          typedef MyType {
             type identityref {
                base MyBase;
             }
             default my-identity;
          }

          leaf baz {
              type MyType;
          }
          leaf copy {
             type leafref {
                path ../baz;
             }
          }
       }
    }

Does the missing prefix refer to the local module or
to the grouping?  Sure looks like both to me.

It would be nice to resolve this important detail.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 10 Aug 2009 12:34:56 +0200 (CEST)

> to the grouping?  Sure looks like both to me.

I think you are mixing two issues here.

The first is when unprefixed identifiers are resolved.  The current
proposal addresses this.

The second issue is how default values can be specified when the
lexicographical representation of the value contains XML prefixes (and
thus depends on the XML context).  This applies to instance-identifier
and identityref.  This issue is not handled in the draft.  We have three
options:  

  1.  do not allow default values for these types
  2.  allow default values but require prefixes
  3.  allow default values and make the prefixes optional

Suggested text for option 3 to go into 9.10.3:

  When an identityref is given a default value using the "default"
  statement, the default value MAY have a prefix.  If a prefix is
  present on the identity name, it refers to an identity defined in
  the module which was imported with that prefix.  Otherwise an
  identity with the matching name MUST be defined in the current
  module or an included submodule.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 10 Aug 2009 15:34:03 -0700

> The first is when unprefixed identifiers are resolved.  The current

IMO, not very well.
It is also really confusing for YANG language users,
which is kind of important.

> and identityref.  This issue is not handled in the draft.  We have three

We seem to disagree on this seemingly unimportant point.

YANG module definition files (i.e., written in YANG)
do not contain any XML prefixes -- only YANG prefixes,
as identified by a [module-name] key or [module-name, revision-date]
key.

Within a NETCONF PDU, XML prefixes are bound to module namespace URI
strings.

There does not have to be a 1:1 correspondence between the
module selected in XML and that selected in YANG.  As Phil
pointed out, unique module names are optional.  The use of
a revision in each module is even optional.

Maybe YANG is too complicated.
Are we building something like an experimental jet to break the
sound barrier or a standard commuter plane any pilot can
fly easily and safely?  Neither?  Both?

I've seen the X-1, and it had so many unlabeled and strange knobs,
I doubt anybody but Chuck Yeager could operate it.
I hope YANG doesn't turn out the same way.  I suspect
it is already too late for that.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 11 Aug 2009 09:55:44 +0200 (CEST)

> We seem to disagree on this seemingly unimportant point.

Hmm.  I have not seen a proposal from you for how to handle
identityref default values.  I apologize if I missed it; if I did
could you point me to it?  In either case, could you comment on my
proposed text?

> fly easily and safely?  Neither?  Both?

FWIW, regarding the first issue, "late binding" in xpath expressions,
one of our customers who is modelling their entire product in YANG ran
into this problem - they need to be able to reference things where the
grouping is used.  It is just one data point of course, but it
indicates that some early users are in fact using also the features we
believe are "advanced".

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 11 Aug 2009 02:47:50 -0700

> could you point me to it?  In either case, could you comment on my

I'm still trying to figure out how the type-name prefix
thing works.

If I have a local type like in my example:

module foo {
  grouping ggg {
    container top-level {
      typedef MyType { ... }
      leaf X { type MyType; }
    }
  }
}

According to your rule, leaf X will have type foo:MyType.
It is bound to the 'foo' namespace because it is a type,
not an XPath expression.

module goo {

   import foo { prefix foo; }
   uses foo:ggg;
}

Now, even though there is a typedef called MyType in
module goo, it is goo:MyType, and not a match.
The YANG compiler is forced to call this an error,
even though the intent was to use the MyType in the
grouping, not the one tied to the foo namespace.

> indicates that some early users are in fact using also the features we

The issue is localized cost -- how to support advanced
features without making everything too complicated.
Maybe it is not possible in this case.  The more we think about
groupings, the more we find there is to know.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 11 Aug 2009 12:13:31 +0200 (CEST)

Yes.

> grouping, not the one tied to the foo namespace.

No, with the module foo above, this is not an error.  Why would it be
an error?  leaf X in the grouping has a type, and it is the same type
in all modules where 'ggg' is used.  This works as before.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 11 Aug 2009 05:03:46 -0700

> in all modules where 'ggg' is used.  This works as before.

it works because we do a late binding on the type name prefix,
not an early binding as your rule specifies.

There is no foo:MyType.
If the prefix 'foo' was present, then foo:MyType should
not resolve in module goo.

Local typedefs are special, because they cannot mask a
global type (a global foo:MyType would be OK, except that the
grouping ggg could be used anywhere but module foo).

I think the rule you are proposing is in the right direction,
but I do not agree that this problem is specific to XPath or
XML prefixes.

It seems to me that all explicit prefixes are
early binding and all missing prefixes are late-binding.
This works whether the definition is in a grouping or not
(same module: early vs. late produces the same result).

If this is the case, then why can't the rule be stated that way?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 11 Aug 2009 14:33:01 +0200 (CEST)

> not an early binding as your rule specifies.

This is getting confusing.  Can we agree that there is no late binding
for types in the current draft 07?  From section 5.4:

   Grouping, type, and identity names are resolved in the context in
   which they are defined, rather than the context in which they are
   used.  Users of groupings, typedefs, and identities are not required
   to import modules or include submodules to satisfy all references
   made by the original definition.  This behaves like static scoping in
   a conventional programming language.

Our proposal is to introduce "late binding" for references in XPath
expressions.  The current rule for type identifiers are the same as
before.

> but I do not agree that this problem is specific to XPath or

Ok, so it seems we agree so far, but in addition to what we propose,
you also want to change the current rule for type/grouping/... to also
use late binding.

> (same module: early vs. late produces the same result).

I prefer to keep the type/grouping rule the way it is, and not do late
binding in this case.  Consider this case:

  module foo { 
    
    typedef my-type { type int32; }

    grouping foo {
      leaf x { type my-type; }
    }
  }

  module goo {

    typedef my-type { type string; }
    
    uses foo:foo;
  }

Now, with your rule, goo:x is of type string.  I think this is
confusing.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 11 Aug 2009 05:50:21 -0700

>     typedef my-type { type int32; }

Add this grouping:

    grouping foobar {
      container fubar {
        typedef my-type { type boolean; }
        leaf y { type my-type; }
      }
   }

So you are saying this is the same as:

    grouping foobar {

   }

This is not illegal (yet) since there needs to be a uses-stmt
to determine if my-type is a conflict.

> Now, with your rule, goo:x is of type string.  I think this is

Either way it is confusing.
Your rule does not work for local typedefs:

  module foo {

    grouping foo {
      container fubar {
        typedef my-type { type boolean; }
        leaf y { type my-type; }
      }
    }
  }

  module goo {

    uses foo:foo;
  }

This is the normal use-case for local typedefs,
not an obscure corner-case.

By your rule, leaf goo:y has type foo:my-type.
There is no foo:my-type.  There is only
a local typedef for my-type within container goo:fubar.

But you say this works by your early-binding rule.
I still don't get it.  Sorry.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 11 Aug 2009 14:59:54 +0200 (CEST)

>         typedef my-type { type boolean; }

This defines a grouping foobar with one container fubar with one leaf
y which is a boolean.

> So you are saying this is the same as:

This defines an empty grouping.  They are not the same.

> not an obscure corner-case.

Agreed.

> By your rule, leaf goo:y has type foo:my-type.

No, by the rules of draft 07 (which I do not want to change), 'y' gets
its type when it is defined; it is a boolean.  No 'uses' can ever
change that.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 11 Aug 2009 06:49:06 -0700

....
> This defines an empty grouping.  They are not the same.

this was a cut instead of a copy.  sorry.

> its type when it is defined; it is a boolean.  No 'uses' can ever

draft-07 is a large document.
Can you point out the exact test that says
leaf y is bound to the local typedef,
even in a grouping?  I agree this is the case,
because I don't want to change my code any more than you do.

The internal pointer to the correct typedef finds
the one inside the grouping.  That never changes,
no matter how many times the group is refined-and-cloned
into a uses-stmt.

   module foo {

      prefix foo;

      typedef my-type { type int8; }

      grouping XXX {
         container xxx {
             typedef my-type { type int16; }
             leaf a { type my-type; }
             leaf b { type foo:my-type; }
         }
     }
  }

yangdump will flag the local 'my-type' typedef as a 'shadow global
type' error, even without any uses-stmt.

But normally (without the global my-type) the xxx grouping would
work fine in another module, so my-type == foo:my-type
in another module.  So I guess you are right.
The local typedef gets bound to module
foo right away, even if it is in a grouping.

It seems weird that the namespace for the element changes
but not the data type of the element, for an external uses-stmt,
especially since local typedefs are not really exported like
real typedefs.

I am still confused whether or not the text in
the draft matches the behavior we supposedly agree is correct,
but I do not have any replacement text, so I will drop
any objection.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 11 Aug 2009 16:01:34 +0200 (CEST)

>From section 5.4:

/martin

---

From: Phil Shafer <[email-masked]>
Date: Tue, 11 Aug 2009 10:41:03 -0400

>expressions.  The current rule for type identifiers are the same as

To be clear, this is not late binding for prefixes, only for
element names.

Thanks,
 Phil

---

From: WashamFan <[email-masked]>
Date: Wed, 12 Aug 2009 10:24:24 +0800

Hi,
  
>  grouping, not the one tied to the foo namespace.

To my understanding, type identifier/XPath exprs/anything else with prefixes in groupings 
should be resolved in where it is defined not it is used.
Once resolved, the references should be kept unchanged
regardless of  "early binding" or "late binding".

In other words, That the references can not be determined until the
grouping is used makes no sense to me.

washam

---

From: "tom.petch" <[email-masked]>
Date: Wed, 12 Aug 2009 21:59:10 +0200

Martin

Top posting as this is a related tangent.

When I struggle to understand yang, I think it is sometimes the choice of words
I struggle with.  Perhaps this I-D fails my boring test.  Technical writing
should bore me; the same concepts should be bound one-to-one to identifiers and
so the same identifiers should recur over and over again.  Boring:-( but
clear:-)

eg, look at the passage you have quoted to Andy:

>    made by the original definition.

coupled with

   o  All derived type names defined within a parent node or at the top-
      level of the module or its submodules share the same type
      identifier namespace.  This namespace is scoped to all descendant
      nodes of the parent node or module.  This means that any
      descendent node may use that typedef, and it MUST NOT define a
      typedef with the same name.

I think that one concept, the typedef name ie the identifier which is the
"typedef" statement's argument, gets referred to as, successively,
 type ..names
 typedefs
 derived type names
 type identifier
 typedef

Life would be more boring, and so for me simpler, if all references were to
typedef
names (or derived type names, although I would find that prolix).  When I see
four identifiers for what seems to be the same concept, I get this nagging
doubt - what am I missing? (please spare me the excitement:-)

Tom Petch