Subject: [netmod] canonical form

From: Martin Bjorklund <[email-masked]>
Date: Wed, 10 Dec 2008 23:42:17 +0100 (CET)

Hi,

In Minneapolis we talked about the canonical form of some data types.
It seems consensus was to try to find a way to formally specify the
canonical form.  In some hallway discussions, we talked about the
following idea.

First, an observation.  The need to specify a canonical form is
typically needed when the type is a string with pattern restrictions.
For example, ipv6 address, domain name.

Another observation is that both XSD and RNG allows multiple
patterns.  In XSD, the patterns are OR:ed and in RNG they are
AND:ed.  It is straightforward to map between these forms.  Suppose
there are two patterns, <a> and <b>.  If they are OR:ed, a single
pattern (<a>|<b>) can be used.  If they are AND:ed, an extra type with
a single pattern <a> can be introduced, and derived from while adding
the pattern <b>.

In YANG we first allowed a single pattern only (since both forms can
be done with a single pattern and type chains), and later we chose to
AND multiple pattern, just like RNG.

The idea we discussed for the canonical form is to change the rules so
that multiple patterns are OR:ed, and introduce a statement 'canonical
true' which can be specificed on one of the patterns.

Example:

  typedef ipv6-address {
    type string {
      pattern
        /* full */
          '((([0-9a-fA-F]{1,4}:){7})([0-9a-fA-F]{1,4})'
         + '(%[\p{N}\p{L}]+)?)' {
        canonical true;
      }
      pattern 
        /* mixed */
        +  '((([0-9a-fA-F]{1,4}:){6})(([0-9]{1,3}\.'
        +      '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}))'
        +    '(%[\p{N}\p{L}]+)?)';
      pattern
        /* shortened */
        +  '((([0-9a-fA-F]{1,4}:)*([0-9a-fA-F]{1,4}))*(::)'
        +    '(([0-9a-fA-F]{1,4}:)*([0-9a-fA-F]{1,4}))*'
        +    '(%[\p{N}\p{L}]+)?)';
      pattern
        /* shortened mixed */
        + '((([0-9a-fA-F]{1,4}:)*([0-9a-fA-F]{1,4}))*(::)'
        +   '(([0-9a-fA-F]{1,4}:)*([0-9a-fA-F]{1,4}))*'
        +   '(([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}))'
        +    '(%[\p{N}\p{L}]+)?)';
    }
  }

  typedef domain-name {
    type string {
      pattern '([a-z0-9][a-z0-9\-]*[a-z0-9]\.)*'
           +  '[a-z0-9][a-z0-9\-]*[a-z0-9]' {
        // lower case
        canonical true;
      }
      pattern '([a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9]\.)*'
           +  '[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9]';
    }
  }

/martin
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Andy Bierman <[email-masked]>
Date: Wed, 10 Dec 2008 15:04:19 -0800

> true' which can be specificed on one of the patterns.

We cannot go back to OR-ing the patterns, as per XSD.
We changed it to AND-ing the patterns because RelaxNG does it
that way.  Although elegant, the solution below will not work
with RelaxNG.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 10 Dec 2008 15:14:15 -0800

> true' which can be specificed on one of the patterns.

How about a new statement called 'canonical-pattern' that is not used
along with all the other AND-ed patterns?  In fact, it really
is not used in NETCONF PDUs at all.  It is available on the
side for applications to check if a value has the canonical form.

   typedef domain-name {
     type string {
       canonical-pattern '([a-z0-9][a-z0-9\-]*[a-z0-9]\.)*'
            +  '[a-z0-9][a-z0-9\-]*[a-z0-9]';
       pattern '([a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9]\.)*'
            +  '[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9]';
     }
   }

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 07:50:38 +0100

> that way.  Although elegant, the solution below will not work

+1. We haven't found a really satisfactory solution to the problem of
canonical form yet, so I think it is not wise to induce changes to other
parts of YANG where consensus has been reached.

I still prefer a more lightweight solution (special description) as an
interim measure for YANG 1.0.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 08:20:03 +0100

> For example, ipv6 address, domain name.
I don't think this is true. We have discussed the problem of "3" versus
"+3" and, in fact, for IP addresses all the different lexical forms come
down to an 32 or 128-bit integer. It will be quite hard to postulate a
single "right" canonical form for an IPv6 address.

I sort of like the approach of DTLL, yes, it still has rough edges, too,
but at least is an indication that there are other possible solutions
out there.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 11 Dec 2008 08:40:49 +0100 (CET)

> > that way.  Although elegant, the solution below will not work

Yes it will.   Just like the solution we have today works with XSD -
it is up to the tools to translate properly.

> interim measure for YANG 1.0.

So do I, but most people at the meeting wanted a formal specification.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 09:46:16 +0100

> it is up to the tools to translate properly.

Maybe, but it will get very complicated in YANG and even more so in
RELAX NG. Think about cases like

P1 or (P2 and P3)

where (P2 and P3) is what represents the canonical form.

> So do I, but most people at the meeting wanted a formal specification.

Is there any pressing need to have it in YANG 1.0? Many people wish to
stop the featuritis and this is nothing else.

I will be happy to work on this after September 09 though.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 09:47:07 +0100

>         +    '(([0-9a-fA-F]{1,4}:)*([0-9a-fA-F]{1,4}))*'

Note that this is incorrect: The shortened form allows the parts to the
left and right of '::' to be arbitrarily long.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Dec 2008 10:41:57 +0100

> true' which can be specificed on one of the patterns.

There were technical reasons to prefer ANDing of pattern and I do not
think the need to define a canonical form does change any of these
reasons. In other words, lets not make arbitrary changes to things we
already sorted out.

My preference at the moment is still to do nothing except adding
clarifying text to the description clauses. If we want to create
syntax, then we should do the following:

typedef foo {
  type string;
  pattern "...";
  pattern "...";
  canonical {
    pattern "...";
    pattern "...";
  }
}

The canonical statement would actually create a derived type only
allowing canonical formats, that is this is roughly equivalent to:

typedef foo {
  type string;
  pattern "...";
  pattern "...";
  description "The canonical format is given by foo-c.";
}

typedef foo-c {
  type foo;
  pattern "...";
  pattern "...";
}

The only benefit of the canonical statement really is that the
types are linked together in a machine readable format. So yet
another version would be:

typedef foo {
  type string;
  pattern "...";
  pattern "...";
  canonical foo-c;
}

typedef foo-c {
  type foo;
  pattern "...";
  pattern "...";
}

The nice aspect of this solution is that it is only one new keyword
and everything else reuses YANG mechanisms. This approach might even
be expanded to handle normalization of groupings or such things in
case this ever becomes necessary.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Phil Shafer <[email-masked]>
Date: Thu, 11 Dec 2008 08:21:30 -0500

>           +  '[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9]';

+1

I like the ORs and the explicit canonical statement.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Thu, 11 Dec 2008 08:26:06 -0500

>that way.  Although elegant, the solution below will not work

I agree the proposed solution is elegant, but disagree about moving
to ORs.  The DSDL generation can translate ORs into whatever RNG
needs (and in a much less painful way than we'll need to translate
ANDs into XSD).

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 14:58:32 +0100

> I agree the proposed solution is elegant, but disagree about moving

Well, it's just a single example which is moreover broken. So let's
first fix the IPv6 patterns and then discuss how the canonical form can
be specified in it.

A better datatyping language is clearly needed not only for YANG, but
it's not a low-hanging fruit, so I think it's better to consider it
really carefully before implementing anything. This is IMO topic for
YANG 2.0.

> needs (and in a much less painful way than we'll need to translate

I don't agree with the change from AND to OR. As Juergen said, there are
technical reasons - one of them was actually fixing the IPv6 patterns.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Dec 2008 15:01:54 +0100

> needs (and in a much less painful way than we'll need to translate

Since pattern OR pattern = pattern|pattern, there is no real value in
having pattern OR pattern. However, translating pattern AND pattern
into a single pattern is really hard so having patterns ANDed has real
value. This is why we decided in favour of ANDing pattern and it does
not matter what XSD or RNG do since translations are possible in all
combinations.

I am wondering on what grounds we open up a previously closed debate.
The question whether multiple pattern are ANDed or ORed is completely
independent of the canonicalization issue.

/js

---

From: Per Hedeland <[email-masked]>
Date: Thu, 11 Dec 2008 15:06:55 +0100

> a single pattern <a> can be introduced, and derived from while adding

But there's a significant difference in the straightforwardness, which
means that AND:ed patterns give more expressive power. With OR:ed
patterns, every 'pattern' after the first is just a useless verbose form
of '|', while AND:ing requires the definition of "dummy" intermediary
types. With AND:ed patterns, you just use '|' or multiple patterns,
within the single type definition where you actually want them.

> true' which can be specificed on one of the patterns.

I think the canonical form as a pattern in general is doubtful - sure it
is formal, but it can't actually be used for anything other than parsing
by a human, since the canonical form is generally something that you
need to *produce* (from an internal or non-canonical form) - a regexp is
no help for that. That's not to say that I can think of an alternative
that *is* useful though...

OR:ing a pattern for the canonical form with pattern(s) covering both
the canonical and non-canonical forms seems useless and wasteful -
what's the point of simultaneously (or maybe even sequentially) matching
against two patterns where one is a superset of the other, when you
don't have an interest in which one of them matches?

--Per Hedeland

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Dec 2008 06:38:06 -0800

Per Hedeland wrote:
>....

+1

I don't see why other toolmakers find this canonical pattern
interesting.  Do you plan some clever way to analyze patterns
and figure out how to change one arbitrary string into
a different string?  Do you plan to brute-force try a billion
times until you randomly generate the canonical form?
Is the point "once you convert by hand, the correct result
will match this pattern"?  For integers, isn't it just
as useful to state "the optional plus sign is removed"
than to put in a cryptic canonical=true pattern?

> don't have an interest in which one of them matches?

exactly

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 11 Dec 2008 16:03:52 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> interesting. 

Neither do I.

In Minneapolis we had three alternatives for how to specify the
canonical rule:

  1) as text, in the description clause
  2) as text, in a new 'canonical' clause (like 'reference')
  3) in some kind of formal way

Personally, I preferred 2 (and still do).  I like 2 b/c it makes the
canonical property stand out for readers to notice it.

However, most people at that time preferred alternative 3.  If we are
going to do that, I like the "simple" solution in my original email.

But maybe this discussion shows that we should not do 3, at least not
now.  If so, we're back to 1 or 2 above.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Dec 2008 07:09:07 -0800

> now.  If so, we're back to 1 or 2 above.

I prefer (1) but the group at the interim meeting did not
think about (or agree to) undoing the AND vs. OR decision
on patterns within the same type-stmt.  A formal solution
would need to work with ANDed patterns we have now.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Dec 2008 16:13:29 +0100

> canonical property stand out for readers to notice it.

I prefer (2) as well.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Dec 2008 19:21:22 +0100

> canonical property stand out for readers to notice it.

I preferred (1) at the meeting, then (3) and me least preferrence
would be (2).  I see no value breaking description clauses in lots of
different clauses so that they all stand out and at the end just
clutter up everything.
 
> now.  If so, we're back to 1 or 2 above.

The good news is that consensus is reached on the mailing list.

/js

---

From: Phil Shafer <[email-masked]>
Date: Thu, 11 Dec 2008 22:45:54 -0500

>different clauses so that they all stand out and at the end just

My concern is that if I have a "valid" value that's _not_ in canonical
form, I can't use it in my client application to match existing
list entries.  If "+1" is valid by the list of patterns, I've no
clue if this is a new entry or if it matches "0x01", "1", "1e0",
etc.  If I at least know there's a canonical form, I can force the
user to use it so I can match existing entries.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 12 Dec 2008 07:30:33 +0100

> user to use it so I can match existing entries.

I fail to see the difference between

   description "... 
                The canoncial form is decimal without a plus sign.";
and

   canoncial "The canoncial form is decimal without a plus sign.";

since in both cases a human is required to read the text and do the
right thing. And yes, for true automation, you really want to have
everything in canonical form at the end (and I think we already agreed
that a server should return canonical form anyways).

Perhaps I don't understand your point.

/js

---

From: Phil Shafer <[email-masked]>
Date: Fri, 12 Dec 2008 02:06:04 -0500

>Perhaps I don't understand your point.

Sorry.  Let me try again.  Suppose I have YANG like:

    typedef positive {
        type string {
            description "The canonical form trims leading pluses";
            pattern "0|[1-9][0-9]*";
            pattern "+[1-9][0-9]*";
        }
    }
    list thing {
        key name;
        leaf name {
            type positive;
        }
        ...
    }

In my client application, I need to prompt the user for a thing
name and find it in my local database.  If I allow the user to enter
"+1", I won't find the data.  I need a mechanism to tell me the
canonical form so I can restrict the user to that pattern when I
have no other choice.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 12 Dec 2008 08:27:59 +0100

> canonical form so I can restrict the user to that pattern when I

OK - got it. But in that case, you really want to force the user
to comply to a type that is canonical by nature:

typedef positive-ca {
    type positive;
    pattern "0|[1-9][0-9]*";
}

I start to like the solution of defining separate types and to link
them together in case we want to do more than just textual
descriptions. This allows us to use all existing and future YANG
mechanisms for type derivations and not just patterns and it allows
data model writers to pick the type they find appropriate.

In general, I believe it is desirable to avoid types that have
non-canonical formats. Exceptions to this rule are those cases where
different formats are widely used (such as IPv6 addresses).

/js

---

From: Phil Shafer <[email-masked]>
Date: Fri, 12 Dec 2008 03:13:15 -0500

>to comply to a type that is canonical by nature:

I want anyone who understands the type to be liberal
and allow anything that matches any of the patterns.
This is normally fine, but for keys, the canonical
form matters more.  If my app doesn't have the type
implemented explicitly, then forcing data into the
canonical form is good enough.

>different formats are widely used (such as IPv6 addresses).

Or integers (0xfe, 0755, 1e4).  Or strings (utf-8).  Or ...

We're transmitting data in text, so canonicalization's part of life.

Thanks,
 Phil

---

From: Per Hedeland <[email-masked]>
Date: Fri, 12 Dec 2008 10:29:59 +0100

> canonical form is good enough.

Could you elaborate on how this would work in practice? E.g. I'm a
netconf client and send you a non-canonical-but-allowed-by-the-pattern
"+1" as a key value, you're the server, now what do you do?

--Per Hedeland

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Dec 2008 13:02:06 +0100

> user to use it so I can match existing entries.

YANG leafs are typed and the standard types such as integer here define
and distinguish lexical space and value space. So in XSLT I can write
even

<xsl:value-of select="foo[key=2*1.5]/value"/>

to address the key "3". A sane and Postel-like approach for YANG would
be to do the same, i.e., compare leafs with numeric types as numbers.

I think is would be better in general to specify value-space values for
types rather than the canonical lexical form. 

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Dec 2008 06:08:14 -0800

> canonical form so I can restrict the user to that pattern when I

I do not think we should worry at all about dumb applications
which do not utilize the YANG files to process the YANG data
models carried in NETCONF PDUs.  You cannot do very much at
all without the YANG file, such as comparing the value-space
instead of the raw XML content.  Since a real YANG app knows
the data type is 'int32', it knows how to deal with the '+' character.

I do not see why the NMS app needs the values by the user
to exactly match the values returned in <get>.  The NMS
has the YANG files, and can convert the data to its canonical
form.

MIB walker programs still work fine.  Real CM applications
are going to need the YANG files to work at all.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 12 Dec 2008 10:57:29 -0500

Andy Bierman writes:
>Real CM applications

Which is why we need to be able to see the canonical
form via the YANG file.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Fri, 12 Dec 2008 11:07:54 -0500

>"+1" as a key value, you're the server, now what do you do?

The server is required to send the content in canonical form.  One
could use the canonical statement to enforce that, but that's not
the use case I was seeing.

My use case is where the type allows a casual form that is translated
into the canonical form.  A client application downloads a list of
entries which includes one named "15".  The user says "edit entry
+15". (I know the "+" is not entirely real world, since everyone
needs to implement integers, but it makes a simple case).

If my application does not have the hardcoded logic to know how to
translate user input into canonical form, it will accidentally make
a new "+15" entry instead of using the existing "15" one.  The user
becomes lost, confused, annoyed, or violent.  Not good.

If there were an indication in the YANG file which pattern is
canonical, my application can see "+15" and know enough to tell the
user that this is not a canonical form and that it's not smart
enough to translate it, so the user will need to re-enter using the
canonical form.  The user may still be annoying, but it's a better
outcome.

Thanks,
 Phil

---

From: Per Hedeland <[email-masked]>
Date: Fri, 12 Dec 2008 17:45:24 +0100

On 12/12/08 17:07, Phil Shafer wrote:
> The server is required to send the content in canonical form.

But the client isn't, right?

> canonical form.  The user may still be annoying, but it's a better

Yes, however my question was not about a user but about a Netconf
client.

--Per Hedeland

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Dec 2008 17:57:02 +0100

> But the client isn't, right?

I think the client must be required in some cases, too, for example,
when specifying a list key.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Dec 2008 09:18:01 -0800

> Which is why we need to be able to see the canonical

Then it needs to be done in a way that doesn't change
the way pattern-stmt works now.  I prefer a separate
canonical-pattern statement instead of a 'canonical'
container around the existing pattern-stmt, because
it is more clear to the human and the parser that
this is not part of the patterns for validating input.

The canonical form for non-string types needs to be
specified in the draft.  So the "+4" or "0x04" into "4" problem
is handled by normative text, not by applying patterns to numbers.

For string types (which already accept the pattern-stmt)
adding a separate canonical-pattern-stmt is useful to
generate warnings or errors, if you want to force the user
to enter the canonical form.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 12 Dec 2008 12:48:34 -0500

>But the client isn't, right?

Right.

>Yes, however my question was not about a user but about a Netconf

Sure, but an the other end of my application is a user
typing in data.  Without the canonical statement, my
app can't use that data effectively.

Thanks,
 Phil

---

From: Per Hedeland <[email-masked]>
Date: Fri, 12 Dec 2008 22:01:35 +0100

> app can't use that data effectively.

Aha, so your application is a Netconf client. OK, this seems to be a
good argument - the server that actually uses the type needs to
"understand" it, and it's reasonable to expect it to "know" how to do
the non-canonical -> canonical transformation, even though it can't
actually use the canonical pattern for that - but expecting a client
working off an arbitrary data model spec to know that is not reasonable,
and it *can* use the canonical pattern.

--Per Hedeland

---

From: David Partain <[email-masked]>
Date: Mon, 15 Dec 2008 13:11:16 +0100

Hi,

I agree with Jürgen.  Let's try to keep the two issues separate if at all 
possible.  I really don't want to re-open issues that are closed unless 
there's a very good reason to do so.

We decided to AND patterns.  Do we really have to change that to solve 
the "canonical form" issue?  I sure hope not.

Cheers,

David

---

From: David Partain <[email-masked]>
Date: Mon, 15 Dec 2008 15:00:00 +0100

Hi,

I'm trying to figure out where this thread is...  Let's see if I have some of 
it right...

As Martin wrote, we seem to have three options on the table.

"In Minneapolis we had three alternatives for how to specify the
canonical rule:

  1) as text, in the description clause
  2) as text, in a new 'canonical' clause (like 'reference')
  3) in some kind of formal way"

The "direction" in Minneapolis was 3, although the result wasn't overwhelming.  
Martin was asked to take this to the list, which he has done.  Thanks!

>From this thread:

1. Andy & Lada have said that they do not want to go back to OR-ing the 
patterns.  As technical contributor, I agree with Jürgen that I haven't seen 
any reason to change this decision.
2. A few voices are saying "wait till after Sep 2009".  I'd personally like to 
settle this now but understand the sentiment...  The "compromise" ground 
could be that we go with "2" above.
3. Andy suggests using a new statement called 'canonical-pattern' that is not 
used along with all the other AND-ed patterns.  This doesn't change
the way pattern-stmt works now.
4. Jürgen has offered a new alternative solution (link to type), but haven't 
seen any discussion on that alternative.
5. Phil's made the argument that this is needed on the "manager application" 
(client) side, which Per seems to agree with.  By having something in the 
YANG model, 

That seems to be where we are.

My interpretation is that this is the "conservative in what you send" field 
whether you're sending from a client or a server.  Seems an important feature 
to me and it seems to capture the problem we're trying to solve.

So we need to reach some kind of rough consensus on this.  There clearly isn't 
one right now.  I guess we'll keep talking.  If we don't reach a seeming 
consensus soon, I'm going to suggest a jabber session to discuss it.

Cheers,

David

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 15 Dec 2008 18:19:37 +0100

Hi David,

for record, I also suggested to consider defining value-space value
instead of canonical lexical value. This is how the built-in types are
supposed to be interpreted and also, for example, RFC 4291 defines the
value-space value of an IPv6 address quite clearly as "128-bit
identifier". In contrast, a canonical lexical form of an IPv6 address
seems rather controversial ("::1" is very useful).

For the time being, the value-space value could be specified simply in a
(special) description clause.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 15 Dec 2008 22:36:39 +0100

On Mon, Dec 15, 2008 at 06:19:37PM +0100, Ladislav Lhotka wrote:
> instead of canonical lexical value. 

I have trouble to follow you. The value space is for most of the types
clear - the canonicalization problem comes from the fact that XML ship
lexical representations and not binary data. The pattern statement
applies to the lexical representation - so how does normalization of
the value space help us?

> This is how the built-in types are supposed to be interpreted and
> also, for example, RFC 4291 defines the value-space value of an IPv6
> address quite clearly as "128-bit identifier". In contrast, a
> canonical lexical form of an IPv6 address seems rather controversial
> ("::1" is very useful).

It is possible to normalize IPv6 addresses such that the 0 compression
is applied to the longest possible sequence and if there is a tie to
the first one. This seems to be what several OSes seem to generate and
this leads to ::1.

> For the time being, the value-space value could be specified simply
> in a (special) description clause.

But the value space is clear - a 128 bit quantity for IPv6. The whole
canonicalization issue are the many different lexical representations
and moving to the value space may not help us with that.

/js

---

From: Per Hedeland <[email-masked]>
Date: Tue, 16 Dec 2008 00:12:08 +0100

> the first one. This seems to be what several OSes seem to generate and

[Not really a comment on value vs lexical space, but rather on the
feasibility of "canonical rule as a pattern (only)".]

True - but can this be expressed with a pattern? I think it just might
be possible to come up with a horribly complex set of regexps that would
reject all forms that didn't follow this rule, but even if it is, I'm
sure that it wouldn't be possible to arrive at the rule by studying
those regexps. I.e. an (informal) textual description is needed in a
case like this.

--Per Hedeland

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 16 Dec 2008 00:31:21 +0100

> those regexps. I.e. an (informal) textual description is needed in a

Yes. And my preference always has been to add appropriate text to the
existing description clause.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 16 Dec 2008 10:03:49 +0100

> applies to the lexical representation - so how does normalization of

Both parties in a NETCONF session will be allowed to use any lexical
form allowed by the datatype (pattern) but also required to use the
value-space value internally for all comparisons, key lookup etc.

> and moving to the value space may not help us with that.

This disscussion started with the problem of comparing list keys that
have different lexical representations but identical value. So it seems
logical to use the value as the discriminating factor. A list item with
key "3" could then be addressed with either "3" or "+3" or "03", if all
three forms are legal lexical representations for the given type. Note
that XPath even allows to use "2*1.5" in this case.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 16 Dec 2008 10:19:28 +0100

> those regexps. I.e. an (informal) textual description is needed in a

In my view the pattern statements will continue to restrict just the
lexical representation, as it is now. Actually, given that the
value-space value must be known for any datatype anyway, the only new
thing would be the requirement on both client and server to use this
value internally. YANG is strongly typed, so it shouldn't lead to any
ambiguities.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 13:16:09 +0100 (CET)

Hi,

Our problem is that there are three different comparisons being done,
and we want them all to do the right thing.  The three are:

  1.  Comparing key values in edit-config.   We want to use the
      value-space value here, not any lexical representation.

      This should not be a problem, but we don't use the term "value
      space" currently.  Maybe we should.

  2.  XPath 1.0 comparison in filtering and must/when expressions.
      This is by definition done on the string representation of the
      value (possibly interpreted as a number by XPath).

      By always using the canonical lexical form, we ensure that
      comparisons can be done consistently. (*)

  3.  Subtree filtering content match comparisons.  Same as (2) above,
      except that it defines its own whitespace trimming rules
      (leading and trailing whitespaces are removed).

(*) This works well as long as all types have a canonical lexical
form.  Unfortunately, we have two types that don't,
'instance-identifier' and 'identityref'.  The lexical representation
of values of these types depend on the XML context in which they
occur.  Specifically they both use namespace prefixes defined in
surrounding XML elements.  For (1) above, where comparisons are done
in the value space, this is not a problem.  But for (2) and (3) these
types cannot be correctly compared (unless the client uses the same
prefixes as the server, which we cannot assume.)

I think we need to introduce the term "value space" in the spec, to
cover (1) above.  (if all types could have a canonical lexical form we
wouldn't need this).

A derived data type by default inherits the value space and canonical
lexical form from its base type (constrained by any restrictions).  A
derived data type might have to specify the value space and, if
applicable, the canonical form.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 17 Dec 2008 13:58:11 +0100

>       space" currently.  Maybe we should.

+1. And for certain new types, it may be necessary to define what the
(value-space) value is.

>       comparisons can be done consistently. (*)

This is however another deviation from XPath spec and practice. XPath
has non-string types, supports numeric operations etc.
 
>       (leading and trailing whitespaces are removed).

But what if xpath capability is used? It will be very confusing if the
comparison/matching semantics is different.

> cover (1) above.  (if all types could have a canonical lexical form we

On the other hand, if all comparisons were based on value space, we
wouldn't need the canonical lexical form.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 14:06:20 +0100 (CET)

> has non-string types, supports numeric operations etc.

I do not try to introduce anything new here, just documenting what
XPath does.  Did I get it wrong?  XPath has strings, numbers (64 bit
float) and  boolean.

> comparison/matching semantics is different.

I agree.  I don't like the whitespace trimming rule, but it is there
in rfc4741.  (it is also incompatible with the three whitespace rules
XSD defines...)

> wouldn't need the canonical lexical form.

Are you (!) suggesting that we change our usage of XPath to do
comparisons based on the value-space value?

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 17 Dec 2008 15:03:22 +0100

> > has non-string types, supports numeric operations etc.

With genuine XPath, an assert like

must "foo:count = 1+1";

would be satisfied whether the foo:count value was set in edit-config as
2 or 2.0. I don't see where the canonical form could come into play.

> comparisons based on the value-space value?

Not me (!), it's IMO the XPath itself that suggests this (see above).

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 15:09:47 +0100 (CET)

> 2 or 2.0. I don't see where the canonical form could come into play.

This just works for the XPath datatypes (i.e. numbers).  But it won't
work for e.g. ipv6 comparisons.  The canonical lexical form is needed for
ipv6 (and others of course).

/martin

---

From: Phil Shafer <[email-masked]>
Date: Wed, 17 Dec 2008 09:22:46 -0500

>      space" currently.  Maybe we should.

Does this mean that an app that doesn't grok lexical space -> value
space mapping for a (new) type can't do key matches?  Will an app
be aware that it can't (and shouldn't) attempt such matches?

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 17 Dec 2008 15:43:15 +0100

> ipv6 (and others of course).

Yes, that's true. But the oft-cited example of "3" versus "+3" is not an
issue, right? I had the impression that these two won't match.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 15:53:19 +0100 (CET)

Phil Shafer <[email-masked]> wrote:
> space mapping for a (new) type can't do key matches? 

Yes.

For example, assume we add a new type 'qname'.  An app that doesn't
understand this type will not know that these two are equivalent:

  <foo xmlns:a="http://example.com/test">a:bar</foo>

and

  <foo xmlns:b="http://example.com/test">b:bar</foo>

> Will an app

If we have a special clause in the typedef, then the app can check if
this new clause is present.  If it is not present, the app can do the
matching, provided it understand the base type.  If the clause is
present, it knows that it cannot do the match.

But the name 'canonical' is not good... is there a term that covers
both "canonical" and "value-space"?

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 16:01:34 +0100 (CET)

> issue, right? I had the impression that these two won't match.

Actually, it seems "+3" is not a Number according to XPath.  If this
really is true, and we did not have the rule about canonical form, an
application might store "+3" in the db, and an xpath comparison with
"3" would then fail.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 17 Dec 2008 16:23:49 +0100

> application might store "+3" in the db, and an xpath comparison with

Indeed, I just checked it, "+3" is not a valid number in XPath.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Wed, 17 Dec 2008 10:34:12 -0500

>Indeed, I just checked it, "+3" is not a valid number in XPath.

FWIW:  If I read their grammar correctly, the unary "-" can be
repeated, making "--3" a valid number.

http://www.w3.org/TR/xpath#NT-UnaryExpr

   [27]   UnaryExpr   ::=   UnionExpr
                            | '-' UnaryExpr

So is "--3" <=> "-3"?

In any case the lex->value issue remains.  I would like to hope
that we can at least clue in apps that lack the specifics for new
types to force them into some sort of not-so-liberal parsing of
user input.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Dec 2008 16:54:37 +0100 (CET)

> types to force them into some sort of not-so-liberal parsing of

Do you have something in mind?

/martin

---

From: Phil Shafer <[email-masked]>
Date: Wed, 17 Dec 2008 11:01:09 -0500

>Do you have something in mind?

I think the canonical statement does the job for the 80% case.

Thanks,
 Phil

---

From: "tom.petch" <[email-masked]>
Date: Sun, 21 Dec 2008 10:53:03 +0100

----- Original Message -----
From: "Martin Bjorklund" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, December 17, 2008 2:06 PM
Subject: Re: [netmod] canonical form

> XPath does.  Did I get it wrong?  XPath has strings, numbers (64 bit

That is much as I see it.  XPath 1.0 has strings, numbers and boolean (pace Node
Sets).  The lexical form for numbers is
 [30]    Number    ::=    Digits ('.' Digits?)?     | '.' Digits
 [31]    Digits    ::=    [0-9]+
ie no octal, hexadecimal or 'float'.  But the character string 'represents a
floating-point number' and must be converted according to IEEE 754-1985 before
being used with function or operator.

(XPath 2.0 is different, perhaps the result of 10 years experience:-)

Tom Petch