Subject: [netmod] 6021 ipv4-prefix

From: <[email-masked]>
Date: Tue, 16 Apr 2019 17:06:49 -0400

Y'all --

It looks like the text in 6021 around the ipv4-prefix is being... mis-read? 

https://github.com/CESNET/libyang/issues/755

The libyang folk seem to be taking the prefix as the mask, rather than the prefix here? Canonizing should mean replacing anything in the host bits with 0's, so only the prefix is left, rather than converting all the prefix bits to 1's.

We might need to clarify this with the libyang folk.

😊 /r

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Apr 2019 09:16:15 +0200 (CEST)

> We might need to clarify this with the libyang folk.

I see that Michal fixed the bug in libyang. Good.

There is another thing I am unsure about.

What is the netconf server supposed to do if a client tries to store 
192.168.1.1/24 in ipv4-prefix ? Or 2001:db8::1/64 in ipv6-prefix?

Reading the canonical format description in 6021 one might intepret that 
the netconf server should just truncate the host bits and store these as 
192.168.1.0/24 and 2001:db8::/64 ? This means the netconf server actually 
stored something else than the client tried to commit (the resulting 
uint32 and uint128 will have different information than was commited by 
the netconf client).

Or should the netconf server throw an error if the client tries to commit 
data that is not according to the bit pattern described in the canonical 
format?

I guess I am getting confused by the "canonical format" term being used in 
IPv6 for describing the ascii representation of the value, but both in 
IPv4 and IPv6 it's also used to describe how the bits should be set (and 
not be set) depending on prefix/mask.

Also, the IPv4 canoncical format representation doesn't describe at all 
the ascii representation, so for instance 192.168.001.001 would be valid 
according to 6021. I haven't seen this to be a problem in reality though, 
because IPv4 addresses are typically "compressed" the same way, all the 
time. If we're revving 6021, then perhaps some text about ascii 
representation format should be to use the format used by the posix 
function inet_ntoa() ?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 18 Apr 2019 10:06:43 +0200

On Thu, Apr 18, 2019 at 09:16:15AM +0200, Mikael Abrahamsson wrote:
> Reading the canonical format description in 6021 one might intepret that the
> netconf server should just truncate the host bits and store these as
> stored something else than the client tried to commit (the resulting uint32
> and uint128 will have different information than was commited by the netconf
> client).

The values 2001:db8::1/64 and 2001:db8::/64 are both legal inputs but
they result in the same value. In situations where multiple inputs
resolve to the same value, we define a canonical representation. And
in the case of ipv6-prefix, the canonical representation is
2001:db8::/64. Hence, if you configure 2001:db8::1/64, then the server
will accept that input report the value back as 2001:db8::/64. The
main reason for having canonical formats is to make comparisons easy
and predictable. Think about xpath expressions - without a predictable
canonical representations, they would become rather complex since they
would have to deal with several different representations.
 
> data that is not according to the bit pattern described in the canonical

No, as explained above, the server should accept the input and convert
it into canonical format.

> IPv6 for describing the ascii representation of the value, but both in IPv4
> and IPv6 it's also used to describe how the bits should be set (and not be
> set) depending on prefix/mask.

I am not sure what the problem is here. If your implementation stores
the address in binary format, then the canonical representation would
clear the unused bits - or you leave them in and you ignore them when
the server produces the canonical textual representation. This is
implementation detail (I would implement this by clearing the bits
also in the binary format but others may choose to do differently).

> Also, the IPv4 canoncical format representation doesn't describe at all the
> ascii representation, so for instance 192.168.001.001 would be valid
> representation format should be to use the format used by the posix function
> inet_ntoa() ?

In both RFCs (6021 and 6991), I think that the pattern takes care of
leading zeros, i.e., 192.168.001.001 will be rejected. I do not know
the POSIX specification of inet_ntoa(), the man pages I have (usually
close to the POSIX specification) do not detail the exact format. The
version agnostic functions are actually inet_ntop() and inet_pton()
and they may not even be a POSIX standard. And inet_ntop() and
inet_pton() likely do the right thing on most systems that also
implement RFC 5952. (The definition of inet_ntop() in the open group
specification [1] is not very tight about the textual format.)

/js

[1] http://pubs.opengroup.org/onlinepubs/9699919799/

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 18 Apr 2019 08:09:59 +0000

Hi Mikael,

On 4/18/19, 3:17 AM, "netmod on behalf of Mikael Abrahamsson" <[email-masked] on behalf of [email-masked]> wrote:

    

Since the constraint on the non-masked portion of the prefix is solely in the description, there is nothing to prevent this and I'm sure the ipv4-prefix and ipv6-prefix types are being used incorrectly.

Thanks,
Acee

    

    
    _______________________________________________
    netmod mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/netmod

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Apr 2019 11:43:05 +0200 (CEST)

> would have to deal with several different representations.

Yes, I perfectly understand the IPv6 compression canonical format, since 
it results in two bit fields expressed in ASCII can be string compared.

However, from a networking point of view 2001:db8::/64 and 2001:db8::1/64 
are NOT the same, and I believe all systems I have interacted with would 
throw an error if I tried to commect 2001:db8::1/64 when the system 
expected the last 64 bits to be zeroed.

2001:db8::/64 2001:db8:0::/64 is the same thing when you use ut fir 
configuration, after the configuration has been applied in the system. 
They're two different (correct) representations of the same configuration 
item. I fully understand why the canonical format is needed here and how 
it's supposed to be used.

2001:db8::/64 and 2001:db8::1/64 are NOT the same if you use them.

https://tools.ietf.org/html/rfc6020#section-9.1

"   For most types, there is a single canonical representation of the
    type's values.  Some types allow multiple lexical representations of
    the same value, for example, the positive integer "17" can be
    represented as "+17" or "17".  Implementations MUST support all
    lexical representations specified in this document."

I don't know what the word "lexical" representation means here. Let's take 
another example.

If I commit "+17.4" into an integer, should I expect the netconf server to 
round this down to 17 and commit that? This is effectively the same thing 
as the above example. When I tried this just now, I got that 17.4 is not a 
valid integer from the software I am using. Is this software doing the 
wrong thing?

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 18 Apr 2019 12:26:04 +0200

> 2001:db8::/64 and 2001:db8::1/64 are NOT the same if you use them.

Why are they not the same if you define a prefix?
 
> round this down to 17 and commit that? This is effectively the same thing as
> the above example. When I tried this just now, I got that 17.4 is not a
> valid integer from the software I am using. Is this software doing the wrong
> thing?

+17.4 is not an integer, so this is an error (not because of the + but
because of the . followed by additional digits). +17 is I think a
valid integer value but the + will be dropped in the canonical
representation.

/js

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Apr 2019 12:53:22 +0200 (CEST)

> Why are they not the same if you define a prefix?

Because they're not. One of them is a valid prefix, the other one isn't.

> because of the . followed by additional digits). +17 is I think a valid 
> integer value but the + will be dropped in the canonical representation.

Yes, but 2001:db8::1/64 isn't valid prefix (because the host portion of 
the prefix isn't 0) so why should it be "rounded" when 17.4 shouldn't be 
rounded if an integer input is expected?

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 18 Apr 2019 13:12:41 +0200

> Because they're not. One of them is a valid prefix, the other one isn't.

Well, this has gone twice through WG last call and twice through IESG
review plus several directorate reviews. In the canonical format, the
non-prefix bits are all zero. I fail to see why this is now a problem.

> Yes, but 2001:db8::1/64 isn't valid prefix (because the host portion of the
> prefix isn't 0) so why should it be "rounded" when 17.4 shouldn't be rounded
> if an integer input is expected?

The non-prefix bits are irrelevant for the prefix and the canonical
format has the non-prefix bits all set to zero. I understand that you
prefer 2001:db8::1/64 to be an error but RFC 6021 and RFC 6991
consider this as valid input that can be safely interpreted to mean
2001:db8::0/64.

/js

---

From: Lou Berger <[email-masked]>
Date: Thu, 18 Apr 2019 09:12:37 -0400

Having worked with UIs that have the behavior of accepting an 
address/prefix-len and mapping it to a prefix, (i.e., network/prefix-len 
and zeroing out the non-significant bits)  - some users really like it 
as they don't have to do the transformation from address to network, 
notably for odd length prefixes, while other users hate it as the system 
shows/does something different than what they entered.

In the end the current definition is what it is.  If we want something 
different we can define it. I personally think an address/prefix-len 
would be useful, and would leave ip-prefix as is.  (again just an 
individual's opinion.)

Lou

---

From: William Lupton <[email-masked]>
Date: Thu, 18 Apr 2019 14:25:16 +0100

On Thu, 18 Apr 2019 at 09:10, Acee Lindem (acee) <[email-masked]> wrote:

> Since the constraint on the non-masked portion of the prefix is solely in
> the description, there is nothing to prevent this and I'm sure the
> ipv4-prefix and ipv6-prefix types are being used incorrectly.
>

Is a requirement in a description any less normative than a requirement
expressed directly using YANG syntax?

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 18 Apr 2019 13:34:56 +0000

+1

It also would be nice if we could loosen the YANG migration rules so we could change leaves from ipv4-prefix to ipv4-address-and-length (or whatever we decide to call it) to handle the cases where the former has been used incorrectly. However, this is a separate issue. 

Acee

On 4/18/19, 9:26 AM, "netmod on behalf of Lou Berger" <[email-masked] on behalf of [email-masked]> wrote:

    
    Lou

---

From: Kristian Larsson <[email-masked]>
Date: Thu, 25 Apr 2019 23:20:57 +0200

> consider this as valid input that can be safely interpreted to mean

Vice versa, if an implementation does treat 2001:db8::1/64 as a syntax 
error, is that implementation incorrect?

    kll

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 25 Apr 2019 23:51:34 +0200

> error, is that implementation incorrect?

I think so. The types do not require that non-prefix bits are zero
when a value is received. However, a server must report the canonical
value, in this case 2001:db8::/64.

/js

---

From: Kristian Larsson <[email-masked]>
Date: Fri, 26 Apr 2019 00:07:31 +0200

> value, in this case 2001:db8::/64.

Cisco NSO treats 2001:db8::1/64 as a syntax error for a leaf of type 
ip-prefix (or ip6-prefix).

It would be interesting to hear Martins opinion on this.

Kind regards,
    Kristian.

---

From: Jeff Tantsura <[email-masked]>
Date: Thu, 25 Apr 2019 22:51:03 -0700

+1

Cheers,
Jeff

> On Apr 18, 2019, at 6:12 AM, Lou Berger <[email-masked]> wrote:
> 
> Having worked with UIs that have the behavior of accepting an address/prefix-len and mapping it to a prefix, (i.e., network/prefix-len and zeroing out the non-significant bits)  - some users really like it as they don't have to do the transformation from address to network, notably for odd length prefixes, while other users hate it as the system shows/does something different than what they entered.
> 
> In the end the current definition is what it is.  If we want something different we can define it. I personally think an address/prefix-len would be useful, and would leave ip-prefix as is.  (again just an individual's opinion.)
> 
> Lou
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 26 Apr 2019 09:39:45 +0200

On Thu, 2019-04-25 at 23:51 +0200, Juergen Schoenwaelder wrote:
> > > > > valid
> > > > the

Agreed. The description only says (and only for ipv6-prefix) that the host bits
should be zero, i.e. no strict requirement.

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 26 Apr 2019 09:44:43 +0200

> ip-prefix (or ip6-prefix).

It is a server backend job to take care about this, the YANG type can stay more
liberal. Similar differences in implementations are rather typical.

Lada

---

From: Kristian Larsson <[email-masked]>
Date: Fri, 26 Apr 2019 12:56:57 +0200

On 2019-04-26 09:39, Ladislav Lhotka wrote:
> Agreed. The description only says (and only for ipv6-prefix

I think it says it for ipv4-prefix too:

   ...
   The canonical format of an IPv4 prefix has all bits of
   the IPv4 address set to zero that are not part of the
   IPv4 prefix.";

> ) that the host bits
There is no strict requirement of what? Accepting the data? Throwing an 
error? Ambiguity of what you are referencing makes it impossible for me 
to parse your sentence. Please elaborate.

I'm having trouble unifying the following:
- Juergen says RFC6021 and 6991 consider 2001:db8::1/64 to be valid 
input that can safely be interpreted to mean 2001:db8::/64
- NSO instead treats 2001:db8::1/64 as a syntax error

If 6021+6991 says it is valid input, then NSO must accept it, no?

Or does 6021+6991 say such a value MAY be treated as valid input? And if 
it does accept it, it must then store or at least always return it in 
the canonical format?

    kll

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Fri, 26 Apr 2019 11:16:41 +0000

Agreed. 

Thanks,
Acee

On 4/26/19, 1:52 AM, "netmod on behalf of Jeff Tantsura" <[email-masked] on behalf of [email-masked]> wrote:

    +1
    
    Cheers,
    Jeff

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 26 Apr 2019 13:18:29 +0200

On Fri, Apr 26, 2019 at 12:56:57PM +0200, Kristian Larsson wrote:
> - Juergen says RFC6021 and 6991 consider 2001:db8::1/64 to be valid input
> that can safely be interpreted to mean 2001:db8::/64
> Or does 6021+6991 say such a value MAY be treated as valid input? And if it
> does accept it, it must then store or at least always return it in the
> canonical format?

I do not find anything in 6021+6991 that says 2001:db8::1/64 is
illegal input. If it were illegal, we would not need the definition of
the canonical format that is in 6021+6991. Apparently text could have
been more explicit but if you connect the bits and pieces, I think the
conclusion must be that 2001:db8::1/64 is allowed input, i.e., you do
not have to clear the bits that are irrelevant but the server will do
this since it has to return the value in canonical format.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 26 Apr 2019 13:28:10 +0200

>    the IPv4 address set to zero that are not part of the

This defines the canonical format, and Juergen explained its role earlier in
this thread.

> There is no strict requirement of what? Accepting the data? Throwing an

That the value of the ipv[46]-prefix type must have the host bits set to zero.
In other words, values that do not have this property are still valid.

> If 6021+6991 says it is valid input, then NSO must accept it, no?

I am not familiar with NSO. If it uses the the types from ietf-inet-types and
refuses to accept such input values from a NETCONF/RESTCONF client, then it
looks like a bug to me.

> Or does 6021+6991 say such a value MAY be treated as valid input? And if

It IS a valid value: it matches the regex pattern and nothing in the description
says otherwise.

> it does accept it, it must then store or at least always return it in 

RFC 7950 says in sec. 9.1 that the server MUST return values in the canonical
form and that the values are conceptually stored in the canonical form (in a
datastore).

Lada

---

From: Kristian Larsson <[email-masked]>
Date: Fri, 26 Apr 2019 17:07:52 +0200

> this since it has to return the value in canonical format.

Thanks for your time in answering this Juergen, much appreciated.

I think the canonical representation is quite clear as is the part that 
the server must return data (and conceptually store) in canonical 
format. What is much less clear is the allowed input formats which then 
includes 2001:db8::1/64. I think it could be worthwhile to explicitly 
state this as it is a bit surprising. Unlike say the uintX types, there 
is no "lexical representation" section for ip-prefix (I presume because 
they are not basetypes and so the lexical presentation follows the base, 
string in this case + the pattern) that explains things in any detail. 
Do you think we could add some clarifications, perhaps to the 
description of the type about this? Or could we even add a lexical 
representation section with examples? Or just an examples section?

Kind regards,
    Kristian.

---

From: Kristian Larsson <[email-masked]>
Date: Fri, 26 Apr 2019 17:15:49 +0200

> This defines the canonical format, and Juergen explained its role earlier in

Ah yes, I see now, you're right.

> refuses to accept such input values from a NETCONF/RESTCONF client, then it

It does use these types so filing bug report I guess.

> form and that the values are conceptually stored in the canonical form (in a

Thanks for the reference, it helps! :)

Kind regards,
    Kristian.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 26 Apr 2019 17:36:23 +0200

On Fri, Apr 26, 2019 at 05:07:52PM +0200, Kristian Larsson wrote:
> 
> I think the canonical representation is quite clear as is the part that the
> server must return data (and conceptually store) in canonical format. What
> is much less clear is the allowed input formats which then includes
> 2001:db8::1/64. I think it could be worthwhile to explicitly state this as
> it is a bit surprising. Unlike say the uintX types, there is no "lexical
> representation" section for ip-prefix (I presume because they are not
> basetypes and so the lexical presentation follows the base, string in this
> case + the pattern) that explains things in any detail. Do you think we
> could add some clarifications, perhaps to the description of the type about
> this? Or could we even add a lexical representation section with examples?
> Or just an examples section?
>

I have added text along these lines in my sources (goes behind the
definition of the canonical format):

      The definition of ipv6-prefix does not require that bits,
      which are not part of the prefix, are set to zero. However,
      implementations have to return values in canonical format,
      which requires non-prefix bits to be set to zero. This means
      that 2001:db8::1/64 must be accepted as a valid value but it
      will be converted into the canonical format 2001:db8::/64.

I have added similar text to the definition of ipv4-prefix. I hope
that text like this clarifies the situation inline in the type
definition.

/js

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Fri, 26 Apr 2019 16:55:02 +0000

Hi Juergen, 

On 4/26/19, 11:36 AM, "netmod on behalf of Juergen Schoenwaelder" <[email-masked] on behalf of [email-masked]> wrote:

    

I must admit that I think this is the worst possible outcome. Independent of the original intent, at a high level it is just not a good idea to accept the non-canonical prefix format and return the canonical format. 

Thanks,
Acee

    /js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 26 Apr 2019 19:30:14 +0200

> I must admit that I think this is the worst possible outcome. Independent of the original intent, at a high level it is just not a good idea to accept the non-canonical prefix format and return the…

So you propose to deprecate the definitions and create new ones?
Otherwise, I can't follow why a clarification can be the worst
possible outcome.

Note that we do have different lexical representations this in several
other places. We accept +17 to mean 17 (Section 9.1 of RFC 7950.)

/js

---

From: tom petch <[email-masked]>
Date: Sat, 27 Apr 2019 12:23:05 +0000

----- Original Message -----
From: "Kristian Larsson" <[email-masked]>
Sent: Thursday, April 25, 2019 11:07 PM

> It would be interesting to hear Martins opinion on this.

Kristian

No way can I approach being Martin but ..

this topic has been active on the 6man list and to some extent on the
main IETF list with one very active participant arguing that because
widely-used implementations allow something that the RFC does not
(making use of the 64 zero bits of fe80::/10) then the RFC is wrong and
should be changed.  Happily this seems to be a minority view but, like a
hydra, the thread keeps coming back to life in another form see e.g.

Re: encoding link ID in link-local addrs
Re: about violation of standards
Globally Unique Link Local Addresses
mailing list activity
who should try to explain
who shouuld update

even

Re: [Technical Errata Reported] RFC7421 (5699)

In fact almost any post to 6man in the past 10 days (of which there are
many)

Tom Petch

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 27 Apr 2019 14:29:33 +0200

> hydra, the thread keeps coming back to life in another form see e.g.

This is NOT this topic. Lets please keep this out of here.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Apr 2019 10:25:53 +0200

>           will be converted into the canonical format 2001:db8::/64.

This just says how the canonical format works, and this is already
defined in RFC 7950. I know this particular type is a hot topic right
now, but there are other places where the canonical format may come into
play. I think that YANG readers should just understand what it is about.

Perhaps it would suffice to add a reference to the corresponding RFC
7950 section:

OLD
    The canonical format of an IPv[46] prefix has all bits of ...

NEW
    The canonical format (see sec. 9.1 of RFC 7950) of an IPv[46] prefix
    has all bits of ...

> I must admit that I think this is the worst possible
> outcome. Independent of the original intent, at a high level it is
> just not a good idea to accept the non-canonical prefix format and
> return the canonical format.

But this is how the canonical format is supposed to work! There are
other types with multiple lexical formats, and the canonical format is
basically just the Postel principle applied to the server: Be liberal in
what you accept, and conservative in what you send.

The expected server behaviour in this case can be a problem only if the
ipv[46]-prefix is (mis)used as IP address & network prefix length combo.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 29 Apr 2019 10:48:25 +0200 (CEST)

> It would be interesting to hear Martins opinion on this.

I did some digging, and it turns out that we had this type internally
before it was part if ietf-inet-types, where we did not require that
all non-prefix bits were zero, but at one point (after
draft-ietf-netmod-yang-types-00 back in 2008) checked in a fix:

  The confd:ipv4Prefix and confd:ipv6Prefix types now require that all
  bits that do not belong to the prefix are set to zero. This is for
  compatibility with the corresponding YANG types defined by the IETF
  NETMOD working group.

You may want to see the threads:

https://mailarchive.ietf.org/arch/msg/netmod/bXL0Mec_ZVVyalmK3pNHkczm6ZI

https://mailarchive.ietf.org/arch/msg/netmod/3Wz5BPgxZajCZloAOjU-ycfr9Lg

Specifically Juergen's proposal:

      Require that all bits that are not part of the prefix are set to
      zero (192.0.2.8/24 becomes an invalid representation of an IPv4
      prefix)

I can't find any discussion in the archive about allowing non-zero non-prefix
bits.  So I think that the original intention was to be strict in
these types.  I agree that the current description text needs
clarification in either case.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Apr 2019 11:30:47 +0200

>       zero (192.0.2.8/24 becomes an invalid representation of an IPv4

Interestingly, the revisions of draft-ietf-netmod-yang-types preceding this
thread also had this sentence for ipv4-prefix:

       The IPv4 address represented in dotted quad notation
       should have all bits that do not belong to the prefix
       set to zero.

In the immediately following revision (draft-ietf-netmod-yang-types-02), this
sentence was removed, though not for ipv6-prefix.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 11:47:31 +0200

> clarification in either case.

Looking at the archive again, the second message seems to indicate
that the idea was to require non-prefix bits to be zero for IPv4 but
not for IPv6, which could explain why we have the SHOULD text for
ip6-prefix but not for ipv4-prefix. In retrospect, having different
requirements for non-prefix bits in ipv4-prefix and ip6-prefix sounds
somewhat weird. The first message you refer to only indicates that we
need to think about canonical formats for the ipvX-prefix types.

Since the text does not say that non-zero non-prefix bits are
disallowed, I think the clarification I have proposed is a path
forward to resolve this.

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 09:51:41 +0000

Hi Juergen,

> -----Original Message-----
> From: netmod <[email-masked]> On Behalf Of Juergen
> Schoenwaelder
> Sent: 26 April 2019 18:30
> To: Acee Lindem (acee) <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [netmod] 6021 ipv4-prefix
> 
> > I must admit that I think this is the worst possible outcome. Independent of the
> original intent, at a high level it is just not a good idea to accept the non-
> canonical prefix format and return the canonical format.
> >
> 
> Otherwise, I can't follow why a clarification can be the worst possible outcome.
> 
> Note that we do have different lexical representations this in several other
> places. We accept +17 to mean 17 (Section 9.1 of RFC 7950.)

This feels somewhat different.  I think that it well understood that these are just the same thing.  E.g. anything that parses these into a integer type will internally end up with the same value in both cases.

I have a related question on the fraction-digits type:

     typedef my-decimal {
       type decimal64 {
         fraction-digits 2;
         range "1 .. 3.14 | 10 | 20..max";
       }
     } 

Should a server accept a value of "3.140" for my-decimal?

What about "3.141"?  I presume that servers would generally not accept (and then round) this value, and except clients to round appropriately before passing the value in.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 12:02:13 +0200

> This feels somewhat different.  I think that it well understood that these are just the same thing.  E.g. anything that parses these into a integer type will internally end up with the same value i…

For me, 10.0.0.0/8 and 10.0.0.1/8 both denote the same IPv4 prefix.

> What about "3.141"?  I presume that servers would generally not accept (and then round) this value, and except clients to round appropriately before passing the value in.

Please start a separate thread if you want to discuss this.

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 10:19:01 +0000

> From: Juergen Schoenwaelder <[email-masked]>
> Sent: 29 April 2019 11:02
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Acee Lindem (acee) <[email-masked]>; [email-masked]
> > > original intent, at a high level it is just not a good idea to
> > > accept the non- canonical prefix format and return the canonical format.
> > > >
> > >
> > > Note that we do have different lexical representations this in
> > > several other places. We accept +17 to mean 17 (Section 9.1 of RFC
> > > 7950.)
> >
> > This feels somewhat different.  I think that it well understood that these are
> just the same thing.  E.g. anything that parses these into a integer type will
> internally end up with the same value in both cases.
> >
> 

It is obvious to me that internally the router should treat these the same, i.e. in the canonical format.
It is also obvious to me that the operational value reported for this should be "10.0.0.0/8".

But it isn't obvious to me that if the input configuration contains "10.0.0.1/8" then when the client requests that configuration back again it should get "10.0.0.0/8" back rather than the value that they provided in the input configuration.

To me, that probably means that a sensible client should just use the canonical format.  Does it improve interop for the type to allow the non-canonical format on input?  That isn't obvious to me either.

> > What about "3.141"?  I presume that servers would generally not accept (and
> then round) this value, and except clients to round appropriately before passing
> the value in.
> 

I was attempting to use it as a similar example for consistency.  I.e. one where extra data is provided and whether the input is validated strictly or loosely.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 12:34:51 +0200

> To me, that probably means that a sensible client should just use the canonical format.  Does it improve interop for the type to allow the non-canonical format on input?  That isn't obvious to me e…

We have the same with +7 and 7 - if you configure an integer to be +7
you get the value 7 back. The alternative would be to generally
disallow any types that accept multiple representations in YANG. This
would then be a YANG next issue to bring up. In YANG 1 and 1.1, we do
support "liberal inputs". (And yes, I know that some of this is also
encoding specific, the hidden can or worms is indeed bigger. But I
like to have this discussion scoped to the RFC 6991bis effort.)

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 13:33:22 +0000

> Sent: 29 April 2019 11:35
> > But it isn't obvious to me that if the input configuration contains "10.0.0.1/8"
> then when the client requests that configuration back again it should get
> "10.0.0.0/8" back rather than the value that they provided in the input
> configuration.
> >
> > To me, that probably means that a sensible client should just use the canonical
> format.  Does it improve interop for the type to allow the non-canonical format
> on input?  That isn't obvious to me either.
> >
> 
> We have the same with +7 and 7 - if you configure an integer to be +7 you get
> the value 7 back. The alternative would be to generally disallow any types that
> accept multiple representations in YANG. This would then be a YANG next issue
> to bring up. In YANG 1 and 1.1, we do support "liberal inputs". (And yes, I know
> that some of this is also encoding specific, the hidden can or worms is indeed
> bigger. But I like to have this discussion scoped to the RFC 6991bis effort.)

I think that there is a difference between a canonical representation for base types known to YANG, vs a defined canonical representation in a typedef's description that requires additional typedef specific code to behave correctly under various scenarios (e.g. server input, when comparing instance values).

I do agree that a clarification is better than the ambiguity that we have now.

But I'm not convinced that allowing ipv4-prefix values in the non-canonical format is necessarily the right thing to do.  If we were defining these as a new type today then would we make the same choice of typedef definition?

Or is a significant part of your proposal/reasoning to ensure backwards compatibility with what we have today?

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 15:46:15 +0200

> Or is a significant part of your proposal/reasoning to ensure backwards compatibility with what we have today?

I am trying to clarify what the existing definition says since there
apparently have been different interpretations.

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 14:02:31 +0000

> Sent: 29 April 2019 14:46
> > But I'm not convinced that allowing ipv4-prefix values in the non-canonical
> format is necessarily the right thing to do.  If we were defining these as a new
> type today then would we make the same choice of typedef definition?
> >
> >
> I am trying to clarify what the existing definition says since there apparently
> have been different interpretations.

Given the definition of ipv6-prefix already contains:

      " The IPv6 address should have all bits that do not belong
       to the prefix set to zero."

I think that a better solution might be to add the equivalent text to the ipv4-prefix definition:

      " The IPv4 address should have all bits that do not belong
       to the prefix set to zero."

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Apr 2019 16:24:26 +0200

>        to the prefix set to zero."

But this still essentially permits the client to send a value with those bits
set, and the server has to be prepared to handle it.

If the goal is to get rid of the difference between ipv4- and ipv6-prefix, which
makes sense, then I prefer to remove this sentence from ipv6-prefix.

Lada

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 14:53:40 +0000

Hi Lada,

> From: netmod <[email-masked]> On Behalf Of Ladislav Lhotka
> Sent: 29 April 2019 15:24
> To: [email-masked]
> > > format is necessarily the right thing to do.  If we were defining
> > > these as a new type today then would we make the same choice of
> > > typedef definition?
> > > >
> > > > Or is a significant part of your proposal/reasoning to ensure
> > > > backwards
> > > compatibility with what we have today?
> > >
> But this still essentially permits the client to send a value with those bits set, and
> the server has to be prepared to handle it.

My aim with this text is to:
 - encourage clients to use canonical format, since that seems to cause the least problems.
 - align the v4 and v6 definitions.
 - retain the existing flexibility for servers to choose what they support, noting that any change in behaviour here will be non-backwards compatible.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 17:36:43 +0200

>  - retain the existing flexibility for servers to choose what they support, noting that any change in behaviour here will be non-backwards compatible.

The 'should' text is of limited value since the 'should' text does not
change what is allowed. I hope the newly proposed text spells out more
clearly that a server turns values into canonical format that are not
in canonical format and implementors can then conclude that using
canonical format in the first place is perhaps a good idea - or it is
convenient to have the conversion done by the server.

Note that the canonical format for IPv6 prefixes has much more to it
than setting the non-prefix bits to zero, see RFC 5952 for all the
details that affect the text representation of the address part.

I believe we are not in the position to tell clients that they should
or should not do. If I push the config value 2001:DB8::/64 (since my
database has values stored using uppercase characters) and it comes
back as 2001:db8::/64, then so be it; it might be convenient for me to
be allowed to push 2001:0DB8:0:0:0:0:0:0/64 and not having to worry
about how to produce the RFC 5952 representation in my glue code that
ties into my database backend. Why would we say clients should not
send values in non-canonical format? Why would we say clients have to
take the pain to ensure everything is turned into 2001:db8::/64 (to
continue the example) before sending values to the server?

Note that the ipv6-address definition has the same canonicalization
properties (if we consider the address portion of ipv6-prefix) and
there is no text saying you should send 2001:db8::1 instead of
2001:0DB8::1.

I think our mission instead is to make it clear what the canonical
format is and that servers will turn lexical representations they
accept into canonical lexical representation. This way people can take
informed decisions about what is appropriate for their specific
clients.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Apr 2019 17:50:56 +0200

On Mon, 2019-04-29 at 15:32 +0000, Rob Wilton (rwilton) wrote:
> BTW, did you mean to drop the alias?

No, my frequent mistake. :-( Putting it back, please see below.

> > From: Ladislav Lhotka <[email-masked]>
> > Sent: 29 April 2019 16:15
> > On Mon, 2019-04-29 at 14:53 +0000, Rob Wilton (rwilton) wrote:
> > > Hi Lada,
> > > 
> > > > > > > But I'm not convinced that allowing ipv4-prefix values in the
> > > > > > > non- canonical
> > > > > > format is necessarily the right thing to do.  If we were
> > > > > > defining these as a new type today then would we make the same
> > > > > > choice of typedef definition?
> > > > > > I am trying to clarify what the existing definition says since
> > > > > > there apparently have been different interpretations.
> > > > > 
> > > > But this still essentially permits the client to send a value with
> > > > those bits set, and the server has to be prepared to handle it.
> > > 
> > It would be good to clarify the implications of sending non-canonical
> > values, and
> > perhaps give recommendations, but not in the description of a particular
> > derived
> > type. I guess it also depends on the character of the client - a curl script
> > cannot
> > be expected to perform extensive normalization.
> 
> I think, for ipv4-prefix, that an existing server could reasonably do any of
> the following:
>  - accept non canonical values and convert them to the canonical format
> internally, or just reject non canonical values
>  - return the canonical value or return the exact value that was configured by
> the client.
>  
> RFC 7950 states that the canonical value is returned for native YANG
> types.  It doesn't say anything about returning canonical values defined in a 

Hmm, my understanding so far has been that the rules for lexical/canonical
values work the same for both built-in an derived types. It is true that these
concepts are defined inside Section 9 (Built-In Types) but I am not sure if this
was really intended to be limited to built-in types. For one, using IP addresses
as list keys would then be impossible.

Lada

> typedef description statements.
> 
> > >  - retain the existing flexibility for servers to choose what they
> > > support, noting that any change in behaviour here will be non-backwards
> > compatible.
> > 
> > I am not sure that I understand what you mean. Are you saying that a server
> > can
> > choose to reject a uint8 value like "+7" as invalid? Such a server is IMO
> > non-
> > compliant.
> 
> A server should accept "+7" and return "7", as per RFC 7950.
> 
> I think that this question is about derived type definitions, and the
> ipv4/ipv6-prefix type definition in particular.
>  
> Thanks,
> Rob
> 
> 
> > Lada
> > 
> > > Thanks,
> > > Rob
> > > 
> > > 
> > > > If the goal is to get rid of the difference between ipv4- and
> > > > ipv6-prefix, which makes sense, then I prefer to remove this
> > > > sentence from ipv6-prefix.
> > > > Lada
> > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > > > > /js
> > > > > > 
> > > > > > --

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 15:58:03 +0000

> Sent: 29 April 2019 16:51
> Cc: NETMOD WG <[email-masked]>
> > > > > > > On Mon, Apr 29, 2019 at 01:33:22PM +0000, Rob Wilton
> > > > > > > (rwilton)
> > > > > > > wrote:
> > > > > > > defining these as a new type today then would we make the
> > > > > > > same choice of typedef definition?
> > > > > > > > Or is a significant part of your proposal/reasoning to
> > > > > > > > ensure backwards
> > > > > > > I am trying to clarify what the existing definition says
> > > > > > > since there apparently have been different interpretations.
> > > > > >
> > > > > But this still essentially permits the client to send a value
> > > > > with those bits set, and the server has to be prepared to handle it.
> > > >
> > > It would be good to clarify the implications of sending
> > > non-canonical values, and perhaps give recommendations, but not in
> > > the description of a particular derived type. I guess it also
> > > depends on the character of the client - a curl script cannot be
> > > expected to perform extensive normalization.
> >
> > I think, for ipv4-prefix, that an existing server could reasonably do
> > any of the following:
> >  - accept non canonical values and convert them to the canonical
> > format internally, or just reject non canonical values
> >  - return the canonical value or return the exact value that was
> > configured by the client.
> >
> concepts are defined inside Section 9 (Built-In Types) but I am not sure if this was
> really intended to be limited to built-in types. For one, using IP addresses as list
> keys would then be impossible.

But according to RFC-7950, from a language POV, I think that it is reasonable to interpret the canonical format of ipv4-prefix to match that of its base YANG type, i.e. string.

9.4.2.  Canonical Form

   The canonical form is the same as the lexical representation.  No
   Unicode normalization of string values is performed.

Section "9.1.  Canonical Representation" does not state that the canonical format of a type may be overridden by a description statement.

Thanks,
Rob

> > > I am not sure that I understand what you mean. Are you saying that a
> > > server can choose to reject a uint8 value like "+7" as invalid? Such
> > > a server is IMO
> > > non-
> > > compliant.
> >

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 18:12:24 +0200

> Section "9.1.  Canonical Representation" does not state that the canonical format of a type may be overridden by a description statement.

Section 9.1 talks about 'types' - the text does not indicate that this
is restricted to built-in types.

We are defining canonical formats of typedefs since RFC 6021, which
was published together with the first version of YANG (RFC 6020). Are
you telling us that we got this all wrong? How do you test for say an
IPv6 address in an xpath expression if you cannot predict how it is
lexically represented?

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 29 Apr 2019 17:08:41 +0000

> Sent: 29 April 2019 17:12
> Cc: Ladislav Lhotka <[email-masked]>; NETMOD WG <[email-masked]>
> > But according to RFC-7950, from a language POV, I think that it is reasonable
> to interpret the canonical format of ipv4-prefix to match that of its base YANG
> type, i.e. string.
> >
> > 9.4.2.  Canonical Form
> >
> > Section "9.1.  Canonical Representation" does not state that the canonical
> format of a type may be overridden by a description statement.
> >
> 
> Section 9.1 talks about 'types' - the text does not indicate that this is restricted
> to built-in types.

The whole of section 9 is for "Built-In-Types" and doesn't cover the typedef statement.  In fact, it states "When a server sends XML-encoded data, it MUST use the canonical form defined in this section."

If YANG allows a typedef to refine the canonical definition of a base type, then I think that the YANG RFC should be explicit on this (e.g. in YANG Next).  Particularly, because this requires a server implementation to read/understand the description associated with a leaf/typedef in case they have to add specific canonicalization code to implement the leaf/typedef.

> 
> We are defining canonical formats of typedefs since RFC 6021, which was
> published together with the first version of YANG (RFC 6020). Are you telling us
> that we got this all wrong?

I'm not sure that we have really got a simple solution for either clients or servers:
 1) Clients may use non canonical format on configuration input
 2) But clients must still use canonical format for xpath expressions
 3) Clients must also handle canonical format being returned on any get requests.
 4) Servers must perform normalization of any type to a canonical format, as defined in the type/typedef/leaf description.

Possibly having a keyword in YANG to mark leaves/typedefs where the canonical definition refines the canonical type associated with the base type might have been helpful.

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 29 Apr 2019 19:12:24 +0200

On Mon, 2019-04-29 at 15:58 +0000, Rob Wilton (rwilton) wrote:
> to interpret the canonical format of ipv4-prefix to match that of its base
> YANG type, i.e. string.
> 
> 9.4.2.  Canonical Form
> 

As I wrote, accepting this would mean that different strings representing the
same IPv6 address would be different as list keys, because keys are compared
based on their canonical value. The same problem would be in XPath expressions.
I guess this is not what we want.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Apr 2019 19:25:45 +0200

On Mon, Apr 29, 2019 at 05:08:41PM +0000, Rob Wilton (rwilton) wrote:
> 
> If YANG allows a typedef to refine the canonical definition of a
> base type, then I think that the YANG RFC should be explicit on this
> (e.g. in YANG Next).  Particularly, because this requires a server
> implementation to read/understand the description associated with a
> leaf/typedef in case they have to add specific canonicalization code
> to implement the leaf/typedef.

Description statements in general are expected to be read and
understood and implemented where necessary. But I now see that the
fact that this section 9.1 is under section 9 which is titled built-in
types is causing the confusion. This is, indeed, unfortunate.
 
>  4) Servers must perform normalization of any type to a canonical format, as defined in the type/typedef/leaf description.

Exactly. Note that clients only send xpath for filtering (if they want
to filter via xpath). What is more important is that module authors
can predict the format of values when they write when or must
expressions. And as Lada points out, having predictable key values
also is kind of desirable.

The reality is that there are many different ways to write an IPv6
address and the idea was to accept them on input but to subsequently
work with the normalized canonical representation. And this is in
ietf-yang-types and ietf-inet-types since day one. But yes, the
text in section 9.1 seems to be misplaced.

/js

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 30 Apr 2019 07:24:17 +0200 (CEST)

On Mon, 29 Apr 2019, Juergen Schoenwaelder wrote:

> I believe we are not in the position to tell clients that they should or 
> should not do. If I push the config value 2001:DB8::/64 (since my 
> database has values stored using uppercase characters) and it comes back 
> as 2001:db8::/64, then so be it; it might be convenient for me to be 
> allowed to push 2001:0DB8:0:0:0:0:0:0/64 and not having to worry about 
> how to produce the RFC 5952 representation in my glue code that ties 
> into my database backend. Why would we say clients should not send 
> values in non-canonical format? Why would we say clients have to take 
> the pain to ensure everything is turned into 2001:db8::/64 (to continue 
> the example) before sending values to the server?
>

2001:0DB8::1 and 2001:db8::1 represent the same thing in the end. 
2001:db8::/64 and 2001:db8::1/64 is not the same thing in the end.

The fact that the server accepts 2001:db8::1/64 and turns it into 
2001:db8::/64 is something I believe should be explicitly stated. It's not 
obvious from neither the description of what a "canonical format" is in 
the YANG spec whether this also applies to dropping actual representation.

A contrived example is if I specify a canonical format for a string that 
is [a-f] and I sent it "abcdefghij" then the server just accepts this and 
turns it into "abcdef"? That's basically what's happening above. A lot of 
people would be astonished by this behaviour, and I think this should be 
more explicit when this is happening. From what we're seeing now from 
different implementations doing differently, this is not obvious.

> format is and that servers will turn lexical representations they accept 
> into canonical lexical representation. This way people can take informed 
> decisions about what is appropriate for their specific clients.

The ipv6-prefix type doesn't have a lexical definition. An integer has 
lexical definition. Does this mean everything should have both just to 
make sure everything is clear? And should a server disallow input that 
doesn't adhere to the lexical format?

I just want it to be 100% clear from reading about what lexical and 
canonical format is what the server should and shouldn't do in each case.

I still think there is a difference between representation of the 
underlying information being changed or not. For instance, DNS domain 
names are case insensitive. Turning input into lower case (canonical 
format) is fine, this changes nothing operationally. Accepting client 
edit-config and just removing ASCII characters can cause unexpected and 
baffling behavior.

I still believe that if a client does edit-config on a DNS domain and 
include for instance a "%" in the name then the server should throw an 
error. So same thing with the ipv6-prefix, it needs to be defined whether 
the server should accept (and zero) host bits or not.

Do we have other types with this kind of ambiguity?

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 30 Apr 2019 08:17:37 +0200

I think we go in circles in this thread and I will stop explaining
things again and again. I suggest people look at the next revision
and if anything remains unclear, people can send concrete edit
proposals.

/js

On Tue, Apr 30, 2019 at 07:24:17AM +0200, Mikael Abrahamsson wrote:
> obvious from neither the description of what a "canonical format" is in the
> YANG spec whether this also applies to dropping actual representation.
> 
> A contrived example is if I specify a canonical format for a string that is
> [a-f] and I sent it "abcdefghij" then the server just accepts this and turns
> it into "abcdef"? That's basically what's happening above. A lot of people
> would be astonished by this behaviour, and I think this should be more
> explicit when this is happening. From what we're seeing now from different
> implementations doing differently, this is not obvious.
> 
> lexical definition. Does this mean everything should have both just to make
> sure everything is clear? And should a server disallow input that doesn't
> adhere to the lexical format?
> 
> I still think there is a difference between representation of the underlying
> information being changed or not. For instance, DNS domain names are case
> insensitive. Turning input into lower case (canonical format) is fine, this
> changes nothing operationally. Accepting client edit-config and just
> removing ASCII characters can cause unexpected and baffling behavior.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 30 Apr 2019 10:46:34 +0200 (CEST)

> and if anything remains unclear, people can send concrete edit

You don't have to explain it. Let me try in a different way.

https://tools.ietf.org/html/rfc7950#section-9.1

"For most types, there is a single canonical representation of the
    type's values."

Is it generally ok that the canonical value potentially represents a 
different bit field/value than what the client sent?

If it is (and that's fine by me), I think this should be made more clear 
in the next rev of the YANG specification. I feel the whole 
"canonical/lexical format" concept is underspecified, for instance in the 
case of ipv6-prefix. In the text you suggested before that fixes 
ipv6-prefix. Do we have more types where this needs to be fixed?

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 30 Apr 2019 11:09:05 +0200

> different bit field/value than what the client sent?

Yes. I explained that the canonicalization of IPv6 addresses is much
more involved than clearing some unused bits in an IPv6 prefix.

> If it is (and that's fine by me), I think this should be made more clear in
> the next rev of the YANG specification. I feel the whole "canonical/lexical
> format" concept is underspecified, for instance in the case of ipv6-prefix.

There is text defining the canonical format since day one. I proposed
an addition that hopefully makes this even clearer.

> In the text you suggested before that fixes ipv6-prefix. Do we have more
> types where this needs to be fixed?

We are not 'fixing' anything. The canonical format is nothing new. The
text aims at explaining things better. Yes, there are many more types
that have a canonical representation. Read the other email messages in
this thread or simply search for 'canonical' in the type definitions.

I think the descriptions are actually all quite clear (but then I am
biased of course).

/js

---

From: Randy Presuhn <[email-masked]>
Date: Tue, 30 Apr 2019 15:30:50 -0700

Hi -

On 4/30/2019 1:46 AM, Mikael Abrahamsson wrote:
...
...

The *value* represented is the same.  The sequences of bytes used
to represent that value may be different.  Some information sent
by the client may be extraneous to the *value*.  Consider the case
of currency values entered into some application.  A robust application
won't care whether I enter $1,234,567.89 or 1234567.89 and if it
subsequently chooses to display it as "1.234.567,89 USD" I can't
complain that the value is different, even though several bytes of
my input have clearly been discarded.

This situation is hardly unique to netconf. I recall coding
for such situations forty years ago.  It has been a fact of life
throughout the history of ASN.1 and especially BER. It will continue
to be a consideration at least as long as folks feel the need to
support "human readable" representations on input.  Frankly,
I was surprised that anyone was surprised by this.

Randy