Subject: [netmod] types-011: multiple lexical representations

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 26 Oct 2008 10:50:03 +0100

Some types allow multiple representations of the same value. A good
example are IPv6 addresses, e.g. ::1 and 0:0:0:0:0:0:0:1. This raises
several questions:

1) We need to define how comparisions are done. At the interim, there
   was agreement that comparisons are done in the value space so the
   fact that there might be different lexical representations for the
   same value becomes a non-issue. (This needs to be confirmed on the
   mailing list.)

2) At the interim meeting, there was agreement to define normalization
   rules - the basic idea here is the following:

   a) implementations MUST accept normalized values as input
   b) implementations MUST generate normalized values as output
   c) implementations MAY accept non-normalized values as input

   In general, it was preferred that implementations do not have to
   remember the representation that was used to configure a value.  It
   should be sufficient to just store the value.  (This needs to be
   confirmed on the mailing list.)

3) There was no agreement how we write down normalization rules are the
   interim meeting. There are several options:

   (1) we simply add text to description statements

   (2) we introduce a separate statement with a separate description
       clause, e.g.

       normalization {
         description "turn all characters to lower case";
       }

   (3) we introduce a separate statement with a separate description
       clause and allow an additional pattern that can be more
       restrictive for normalized values, e.g.

       normalization {
         pattern "[a-z]";
	 description "normalized representation is in lower case";
       }

   In option (1) and (2), it remains unclear whether the retrictions
   (pattern etc.) associated with a type only cover the normalized
   representation or also other representations that implementations
   MAY allow. One option is to simply just define what MUST be allowed
   and leave everything else to the Postel principle (implementations
   may be lenient in what they accept). This will work for several
   types, but might not work for all of them.

4) I got the task to go over all definitions in the -type document to
   identify types that may need normalization rules.  I have done that
   and for each type we need to decide between the following options:

   (i)  we further restrict the lexical space so that we have a 1:1
        mapping of the lexical representation to the value space

   (ii) we define what the normalization rules are

   * date-and-time: 2008-10-26T10:35:00Z
                    2008-10-26T10:35:00+00:00

   * ipv4-address:  numeric versus textual zone index

   * ipv6-address:  shortened and mixed forms, zone index

   * ipv4-prefix:   unused bits (127.0.0.0/8 vs. 127.1.2.3/8)

   * ipv6-prefix:   shortened and mixed forms, unused bits

   * domain-name:   lower-case normalization, make this mandatory
                    like in the uri description statement?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 29 Oct 2008 15:37:39 +0100 (CET)

Hi,

>    same value becomes a non-issue. (This needs to be confirmed on the

I agree with this.

>    c) implementations MAY accept non-normalized values as input

I do not agree with c.  Which I guess means that I don't agree with b
either.

I see a value in having both a normalized value and a less constrained
lexical value if the rules were:

    a') implementations MUST accept normalized values as input
    b') implementations SHOULD generate normalized values as output
    c') implementations MUST accept non-normalized values as input

If we do a + b in your list, then I don't think we should bother with
specifying anything but the normalized value.

>    confirmed on the mailing list.)

I agree with this.

>          description "turn all characters to lower case";

Or maybe 'normalization "turn on all characters to lower case";'

> 	 description "normalized representation is in lower case";

I wouldn't mind (3), but I don't think it is necessary.  I like (2)
better than (1) because it tells a module designer that YANG
recognizes the problem of normalization, and provides a solution for
it.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 29 Oct 2008 15:48:51 +0100

On Wed, Oct 29, 2008 at 03:37:39PM +0100, Martin Bjorklund wrote:

[...]
 
> recognizes the problem of normalization, and provides a solution for

A module designer either puts in the right text or not because he
either understands the issue or not. I do not think that the existance
of an additional keyword has much impact on whether the designer
understands the issue or not. So I personally can live with (3)
although (1) might be good enough - depends a little bit on the rules
though and what the regex really means. In other words, I think (2)
is a bad choice.

/js

PS: The URI type in the collection of YANG types actually has
    normalization text that was carried over from the corresponding
    SMIv2 TC definition. This tells me that we dealt with some of this
    before by using description clauses.

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 29 Oct 2008 16:01:12 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> either understands the issue or not.

Sure, but with an explicit keyword the module designer that *does*
think about the issue knows that the concept is supported in the
language.  And maybe the explicit keyword will help the designer
understand the issue and remember to think about it.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 29 Oct 2008 17:26:46 +0100

> >    same value becomes a non-issue. (This needs to be confirmed on the

+1

>     c') implementations MUST accept non-normalized values as input

+1

> >    confirmed on the mailing list.)

+1

> recognizes the problem of normalization, and provides a solution for

I prefer (2), and think about (3) for YANG 2.0.

Actually, there is another option which DTLL tries to implement: provide
expressions for obtaining the value-space value from various lexical
representations. For example, IPv4 addresses would be mapped to uint32
in the value space.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 29 Oct 2008 17:32:02 +0100

> understand the issue and remember to think about it.

Right, and descriptions sometimes deal with other things and may be
quite long, in which case it would be easy to miss the part about
normalization.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 29 Oct 2008 10:01:29 -0700

Ladislav Lhotka wrote:
.....
> I prefer (2), and think about (3) for YANG 2.0.

I prefer (1).
I do not see why normalization implementation issues
deserve their own description clause.  It isn't more
important than what the knob does, or security concerns,
or network impact, etc.

Multiple description clauses will just lead to lots of CLRs
about what text goes where.

(3) sounds nice for YANG 3.0.
If we try to automate everything now, YANG will collapse
under its own weight before taking its first baby steps.

> representations. For example, IPv4 addresses would be mapped to uint32

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 29 Oct 2008 10:19:38 -0700

Hi -

While I certainly don't want to derail what appears to be a
WG consensus, I think it's important to keep in mind that
this direction (comparisons in "the value space",
accepting non-normalized input from clients, and more-or-
less ad hoc normalization rules associated with new bits
of data at the time they are defined) has serious implications
if there is ever a need to support delegation / subagent
architectures or generic repositories.  It requires the parts
of a system analogous to a master agent to have the ability
to dynamically acquire new rules corresponding to the normalized
comparison functions for new subagents, or for the master
agent to "farm out" the comparisons to the subagent.  While
this is reasonable for filter-like applications, for handling instance
naming it scales badly.  I know this from first-hand experience,
as the path being taken by netmod here is the one that was taken
by CMIP, and this is one of the things that most CMIP people
will admit was a mistake that could have easily been avoided.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 29 Oct 2008 10:31:42 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Ladislav Lhotka" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, October 29, 2008 10:01 AM
> Subject: Re: [netmod] types-011: multiple lexical representations

....
> important than what the knob does, or security concerns,

Think RFC 3454 as the starting point for human-readable text
if one does ad hoc normalization rules.
 
Randy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 29 Oct 2008 16:23:30 -0400

"Randy Presuhn" writes:
>It requires the parts
>agent to "farm out" the comparisons to the subagent.

Yes, this is definitely a weakness, but I see the issue as lesser
for agents and greater for managers.  If an applications loads
"mine.yang" and it defines a new type, the app won't automatically
know how to canonicalize values.  So if I make a
"junos:ipprefix-or-interface" type, it won't know that "10/8" gets
canonicalized to "10.0.0.0/8".  This isn't good, but having a
canonical form beats treating "10/8" and "10.0.0.0/8" as distinct
prefixes.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 29 Oct 2008 22:29:19 +0100 (CET)

> canonical form beats treating "10/8" and "10.0.0.0/8" as distinct

I think that if we take Randy's comment seriously, there are only two
alternatives:  don't allow multiple representations of the same value,
or have a formal syntax to specify how the canonical form is derived
from the multiple representations (i.e. do something like DTLL).  IMO
the latter is not feasible, so that leaves us with having a one-to-one
mapping between the lexical space and the value space.

Did I get this right Randy?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 29 Oct 2008 15:08:31 -0700

> mapping between the lexical space and the value space.

What happened to the Postel Principle?
A NETCONF agent should not break because it get +4 instead or 4,
or leading whitespace, etc.  An SNMP agent accepts more
than one ASN.1/BER encoding, accepts varbinds in any order, etc.
to make it easier on applications.  Let's not do something
intended to help applications which actually makes things worse.

> ___

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 29 Oct 2008 18:01:46 -0700

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Cc: <[email-masked]>; <[email-masked]>
> Sent: Wednesday, October 29, 2008 2:29 PM

Close enough.  :-)  The distinction between scoping and filtering
in the CMIP world permits less draconian solutions there.  The cases
where only matching for equality would be needed can be handled
by canonicalization alone.  More elaborate matching rules (e.g., ordering,
or substring (substring of IPv6 address, anyone?)) get much more
complicated.  I don't know what the right answer wil be here, just that
it's something that requires very careful thought if you think there
will be a need to support multiple, independently-developed software
components that implement the various models supported in
a system.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 29 Oct 2008 18:07:38 -0700

Hi -

> To: "Martin Bjorklund" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, October 29, 2008 3:08 PM

...
> intended to help applications which actually makes things worse.
...

If we're bringing in SNMP, let's use the correct analogy.
The rules for the encoding of INDEX values into an Object
Identifier are a clear example of the kind of canonicalization
I'm concerned about.  There is exactly one correct encoding
here, with well-defined (albeit funky) rules for ordering and
comparison for equality.  This is one of the things SNMP got
right.

Randy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 30 Oct 2008 07:37:07 +0100

> mapping between the lexical space and the value space.
The former option is IMO not realistic. Multiple lexical representations
are allowed for numeric types, so why shoudn't they be for other types?
My understanding is that the mapping from lexical to value space has to
be "hardwired" in the agent or manager code anyway so that its verbal
specification in description or whatever strings should in principle
suffice. Loading new types dynamically and being able to handle them
correctly is another league.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 30 Oct 2008 12:57:07 +0100

> intended to help applications which actually makes things worse.

One option is indeed to define a single lexical representation for all
YANG types, that is you can't formally ship ::1 as an IPv6 address and
you have to use 0:0:0:0:0:0:0:1 instead. Then we can leave it to
implementations to apply the postel principle and to accept ::1 where
this is feasible.  This way, the standard is clean and we have moved
the problem into implementation space. We also accept potential
surprises of users who might run into unexpected interoperability
problems if they start to rely on the "lenient" behaviour of NETCONF
implementations.

Is this the way to go? This surely makes many pattern much simpler but
still requires somewhat complex text to write down the specifics of
the lexical representation so that we actually achieve a 1:1 mapping.

/js