Subject: Re: [netmod] 6021 ipv4-prefix

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 1 May 2019 10:55:38 +0200 (CEST)

On Tue, 30 Apr 2019, Juergen Schoenwaelder wrote:

> On Tue, Apr 30, 2019 at 10:46:34AM +0200, Mikael Abrahamsson wrote:
>> On Tue, 30 Apr 2019, Juergen Schoenwaelder wrote:
>>
>>> I think we go in circles in this thread and I will stop explaining
>>> things again and again. I suggest people look at the next revision
>>> and if anything remains unclear, people can send concrete edit
>>> proposals.
>>
>> You don't have to explain it. Let me try in a different way.
>>
>> https://tools.ietf.org/html/rfc7950#section-9.1
>>
>> "For most types, there is a single canonical representation of the
>>    type's values."
>>
>> Is it generally ok that the canonical value potentially represents a
>> different bit field/value than what the client sent?
>
> Yes. I explained that the canonicalization of IPv6 addresses is much
> more involved than clearing some unused bits in an IPv6 prefix.

The canonicalization of IPv6 addresses doesn't change the resulting 128 
bit pattern. Canonicalization of IPv6-prefix *does* change the bit 
pattern. Also, it doesn't say in 
https://tools.ietf.org/html/rfc7950#section-9.1 whether the server should 
accept bit-fields that do not adhere to the canonical representation or 
not.

So while you seem to think I am not reading your text, it seems to me 
you're not reading what I am saying either. You're not responding to the 
points I am trying to make anyway.

https://tools.ietf.org/html/rfc7950#section-9.1

This talks about *values*. If you drop bits in IPv6-prefix, then it's not 
the same *value* anymore.

So https://tools.ietf.org/html/rfc7950#section-9.1 should be changed 
in future revisions to avoid confusion.

> We are not 'fixing' anything. The canonical format is nothing new. The
> text aims at explaining things better. Yes, there are many more types
> that have a canonical representation. Read the other email messages in
> this thread or simply search for 'canonical' in the type definitions.
>
> I think the descriptions are actually all quite clear (but then I am
> biased of course).

There are lots of implications that are *not* clear in 
https://tools.ietf.org/html/rfc7950#section-9.1.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 1 May 2019 10:58:46 +0200 (CEST)

On Tue, 30 Apr 2019, Randy Presuhn wrote:

> Hi -
>
> On 4/30/2019 1:46 AM, Mikael Abrahamsson wrote:
> ....
> ....
> The *value* represented is the same.  The sequences of bytes used
> to represent that value may be different.  Some information sent

No. 2001:db8::/64 and 2001:db8::1/64 isn't the same *value*. These are not 
the same bits set.

> by the client may be extraneous to the *value*.  Consider the case
> of currency values entered into some application.  A robust application
> won't care whether I enter $1,234,567.89 or 1234567.89 and if it
> subsequently chooses to display it as "1.234.567,89 USD" I can't
> complain that the value is different, even though several bytes of
> my input have clearly been discarded.

That's a completely different example. In your example the value is the 
same, in mine (ipv6-prefix) it isn't.

> This situation is hardly unique to netconf. I recall coding for such 
> situations forty years ago.  It has been a fact of life throughout the 
> history of ASN.1 and especially BER. It will continue to be a 
> consideration at least as long as folks feel the need to support "human 
> readable" representations on input.  Frankly, I was surprised that 
> anyone was surprised by this.

I have no problem understanding the canonicalization of IPv6 address, 
because the underlying value doesn't change by canonicalizating it. This 
is not the case with canonicalizating IPv6-prefix, where the resulting 128 
bit field *changes* when you canonicalize it.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 1 May 2019 13:17:12 +0200

On Wed, May 01, 2019 at 10:55:38AM +0200, Mikael Abrahamsson wrote:
> 
> So while you seem to think I am not reading your text, it seems to me you're
> not reading what I am saying either. You're not responding to the points I
> am trying to make anyway.
> 

I personally do take the standpoint that irrelevant bits do not matter
for the value of a prefix, i.e., 192.168.0.1/24 and 192.168.0.0/24 are
two different representations for the same prefix. You seem to take
the standpoint that 192.168.0.1/24 and 192.168.0.0/24 are different
prefixes since bits that are irrelevant do differ.

Apparently there are different views that people have concerning
prefixes. I think I have seen the following three alternatives:

a) non-prefix bits that are set to one are illegal in a prefix
b) non-prefix bits are irrelevant but they need to be preserved
c) non-prefix bits are irrelevant, ignore them and the canonical
   representation has non-prefix bits set to zero

The RFC 6991 definitions do c). If there is consensus that c) is
wrong, we need to deprecate the definitions and create new ones after
finding consensus on either b) or a).

System/kernel interfaces seem to show different behaviours. My Linux
box seems to do a), my MacOS box seems to do c). The system/kernel
interfaces likely all do the same if all non-prefix bits are zero,
i.e., when the system receives data in the canonical format.

Option b) seems to be the most expensive to implement (your server may
have to clear non-prefix bits before pushing the prefix into the
kernel and it needs to map data received from the kernel back to a
prefix that has unused bits preserved in the datastore). Hence, for me
the choice is between a) and c) and given that we have c) already
defined for years, my first preference is to just keep things as they
are.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 1 May 2019 15:12:47 +0200 (CEST)

> prefixes since bits that are irrelevant do differ.

No, I am saying this is underspecified or actually wrongly specified in 
the current documents + proposed text regarding what canonical format is 
and isn't, and how the server and clients handle this.

I am fine with the current proposed text to specify this for ipv6-prefix, 
but I am also pointing out that I think when YANG 1.2 is specced, the 
definition for "canonical format" needs more/changed text.

> finding consensus on either b) or a).

I am fine with c), but I am also saying I disagree with your view that 
this this behaviour has been specified "since the beginning". This might 
have been obvious to you from the beginning, but it's not wrotten down 
properly (at least I haven't seen text that makes me clearly understand 
the expected behaviour). I think the text specifying what "canonical 
format is" referring to "same *value*" is wrong. +17 and 17 is the same 
integer, 192.168.0.1/24 and 192.168.0.0/24 are not the same *value*. It's 
misleading to refer to the canonical form having the *same* *value* when 
we're throwing away information.

If you write 2001:db8:0:1 as 2001:db8::1 then you're compressing the text 
form without throwing away actual information. It's the *same value* buth 
*different text representation*. 2001:db8::/64 and 2001:db8::1/64 is *not 
the same value*.

I am fine with us continuing with c) above, I have long stopped arguing 
for anything else. What I am though saying is that I want 
https://tools.ietf.org/html/rfc7950#section-9.1 (or elsewhere) to have 
better text on this matter when the documents are revved next time.

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 1 May 2019 13:18:23 +0000

Hi Mikael,

> -----Original Message-----
> From: Mikael Abrahamsson <[email-masked]>
> Sent: 01 May 2019 14:13
> To: Juergen Schoenwaelder <[email-masked]>
> Cc: Rob Wilton (rwilton) <[email-masked]>; [email-masked]
> Subject: Re: [netmod] 6021 ipv4-prefix
> 
> No, I am saying this is underspecified or actually wrongly specified in the current
> documents + proposed text regarding what canonical format is and isn't, and
> how the server and clients handle this.
> 
> I am fine with the current proposed text to specify this for ipv6-prefix, but I am
> also pointing out that I think when YANG 1.2 is specced, the definition for
> "canonical format" needs more/changed text.

I think that it is quite likely that this will get fixed in YANG 1.2.

It is being tracked as a potential issue for YANG 1.2 here, https://github.com/netmod-wg/yang-next/issues/83, which means that this issue should at least be discussed/considered for the next version of YANG.

If there are particular points of clarification that you think are important/required then adding them as comments to that github issue would be helpful.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 1 May 2019 17:53:21 +0200

On Wed, May 01, 2019 at 03:12:47PM +0200, Mikael Abrahamsson wrote:
> 
> I am fine with c), but I am also saying I disagree with your view that this
> this behaviour has been specified "since the beginning". This might have
> been obvious to you from the beginning, but it's not wrotten down properly
> (at least I haven't seen text that makes me clearly understand the expected
> behaviour). I think the text specifying what "canonical format is" referring
> to "same *value*" is wrong. +17 and 17 is the same integer, 192.168.0.1/24
> and 192.168.0.0/24 are not the same *value*. It's misleading to refer to the
> canonical form having the *same* *value* when we're throwing away
> information.
>

The basic disconnect here may be that for me the prefix is the value
while for you the value is the prefix plus the unused bits.

> *different text representation*. 2001:db8::/64 and 2001:db8::1/64 is *not

With your definition of 'value' it is not the same, with my definition
of 'value' it is the same.

For me, the value space of the ipv6-prefix type is the set of all
possible ipv6-prefixes. And with this, 2001:db8::/64 and
2001:db8::1/64 are two different textual representations that resolve
to the same prefix, i.e., the same value in the value space. I would
go even further and make the following distinction between

- textual representations of values of a type
- the value space of a type
- internal representations of values of a type

I think we have this discussion because you likely map the textual
representations of a prefix into an internal representation that can
capture details that are (in my model) not relevant for the value
space itself.

(Even for the case of simple signed integers, it depends on my internal
number representation whether normalizing +7 to 7 causes a change of
the internal representation or not.)

Once we add support to YANG for binary encodings, we will get
additional complexity since we will map the value space to multiple
'external' representations and for the same type (=value space), there
may be differences how 'external' representations map to the value
space. A binary representation of an IPv6 address may have a 1:1
mapping to the value space while our textual representation already
has a n:1 relationship. Or we go into the direction to require that
all 'external' representations must only have 1:1 relationships, i.e.,
only textual representations in canonical format will be legal. We
will see...

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 1 May 2019 16:21:37 +0000

> From: Juergen Schoenwaelder <[email-masked]>
> Sent: 01 May 2019 16:53
> To: Mikael Abrahamsson <[email-masked]>
> > this this behaviour has been specified "since the beginning". This
> > might have been obvious to you from the beginning, but it's not
> > wrotten down properly (at least I haven't seen text that makes me
> > clearly understand the expected behaviour). I think the text
> > specifying what "canonical format is" referring to "same *value*" is
> > wrong. +17 and 17 is the same integer, 192.168.0.1/24 and
> > 192.168.0.0/24 are not the same *value*. It's misleading to refer to
> > the canonical form having the *same* *value* when we're throwing away
> information.
> >
> 
> The basic disconnect here may be that for me the prefix is the value while for
> you the value is the prefix plus the unused bits.

[To frame this comment, I don't object to your currently proposed solution]

I consider the extra bits after the valid part of the prefix to be unwanted noise.

For me, one hypothetical question I have is whether the potential convenience of allowing 192.168.0.1/24 (for some use cases) outweighs the risk that the wrong value has been provided but cannot be checked (e.g. 192.168.0.1/2).

Similarly, if you consider strings with a max length, I would expect a server to reject a configured string that was too long rather that treat the excessive characters as noise and silently truncate the input.

If it was down to me, and I was defining this now, then I would choose the stricter input (as per the should currently in the ipv6-prefix definition).  But no longer care if the consensus is in the other direction.

Whichever way we change the definitions, I believe that they are being changed in a non-backwards-compatible way that will impact some clients or server implementations.

Thanks,
Rob

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 1 May 2019 21:46:44 +0200 (CEST)

> while for you the value is the prefix plus the unused bits.

My disconnect is what the server should do when it encounters a value 
where the bits are non-zero.

Where is the text that tells the server implementor whether to throw an 
error when client commits non-zero bits, or to just throw the bits away 
and store the value in the canonical format?

> the internal representation or not.)

I think this example is not relevant to this discussion. +7 and 7 doesn't 
change any integer backend representation. It's the same value.

Again, I have no problem with the server throwing away the bits at commit 
time, I just want it to be clear from the specs that this is the correct 
behaviour and what the server should do when the above text is not true:

"The IPv6 address should have all bits that do not belong
       to the prefix set to zero."

Throw an error or "fix it"? It seems it should "fix it", so we should have 
text that reflects this. I have no idea where this text should go, though.

---

From: Randy Presuhn <[email-masked]>
Date: Wed, 1 May 2019 19:41:30 -0700

Hi -

On 5/1/2019 12:46 PM, Mikael Abrahamsson wrote:
...
> and store the value in the canonical format?

Such text would be an inappropriate constraint the server's
internal representation.  We should only specify
the externally-visible behaviour: that the reported value
will be in the canonical format.  Whether an implementation
preserves extraneous cruft in its internal representation is
purely an implementation decision, and not subject to standardization.

> I think this example is not relevant to this discussion. +7 and 7
> doesn't change any integer backend representation. It's the same value.

You don't know whether it does or does not affect the internal
representation used by the backend.  If the backend is a textual
configuration file virtualized through netconf, the "+" might
very well be preserved in the file, even though it would disappear
in response to a query.  We only know that for purposes
of the protocol's operation, the server needs to behave as though
it is in the canonical form.

> Again, I have no problem with the server throwing away the bits at
> commit time, I just want it to be clear from the specs that this is the
> correct behaviour and what the server should do when the above text is
> not true:
>
> Throw an error or "fix it"?

Since the language is "should" an error seems inappropriate.

> It seems it should "fix it", so we should
> have text that reflects this.

False dichotomy.  An implementation might actually preserve
those bits, though of course they'd never be seen again (at
least not on a netconf interface) since the netconf server
will always behave as though the value were in its canonical
form, regardless of the internal representation.

> I have no idea where this text should go,
> though.

I agree with the earlier sentiment that anything addressing
this really belongs in whatever further clarification of
canonicalization goes into the update.

Randy

---

From: tom petch <[email-masked]>
Date: Thu, 2 May 2019 09:06:49 +0000

----- Original Message -----
From: "Juergen Schoenwaelder" <[email-masked]>
Sent: Wednesday, May 01, 2019 12:17 PM

> finding consensus on either b) or a).

And I think that c) is the correct answer (and would be for those not
involved in netmod but who have experience of the IETF - did I hear 'be
liberal in what you accept'?)

I have worked with systems that made every effort to find a reason not
to accept something that had been input, and I have worked with systems
that bend over backwards to accept something a bit flaky; different
developers have different cultures but I have always found the culture
of the IETF to be very clear on this point.  Mac, Linux and Windows have
different cultures.

There is a caveat and it is that the context must be clear.  Randy cited
' ... 1234567.89 and if it
subsequently chooses to display it as "1.234.567,89 USD" '
Well, where he is, may be, but not here where it could be

1.234.567,89 Euro
1,234,567.89 Euro
123456789 Yen
or ...
depending on the locale.

Going back to
192.168.0.1/24
then the meaning is clear as long as it is unambiguous that it is a
prefix and not more; as has been commented on (a lot of messages ago),
this could be a compound object, specifying
address+ mask
address + prefix + prefix length
etc
so it matters that the context is clear, both for the object instance
and for the organisation doing the specification.

Tom Petch

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 2 May 2019 13:35:43 +0200 (CEST)

> purely an implementation decision, and not subject to standardization.

I am talking about what goes on the wire. If the client does an 
edit-config with ipv6-prefix 2001:db8::1/64, should the server convert 
this into 2001:db8::/64 or throw an error on the edit-config operation.

Jurgen seems to say it should convert it and not throw an error, and I'd 
like text to say that indeed, this is proper behaviour. Nobody has so far 
been able to tell me where this text currently is, so that's why I'm 
asking for it to be added. Either this should go into an update to 
https://tools.ietf.org/html/rfc7950#section-9.1 or it should go into each 
and every definition of types (or both of them).

> form, regardless of the internal representation.

Again, I am talking about what goes on the wire, what is seen when issuing 
"get" or "edit-config" etc.

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 02 May 2019 14:38:41 +0200

> been able to tell me where this text currently is, so that's why I'm

If we agree that a type defines the set of legal on-the-wire values
(possibly modulo representation - JSON/XML/...), then section 4.2.2.1 in
RFC 7950 says:

[A leaf instance] has exactly one value of a particular type ...

So why should a server throw an error if this is satisfied?

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Thu, 2 May 2019 16:32:40 +0000

> From: netmod <[email-masked]> On Behalf Of Ladislav Lhotka
> Sent: 02 May 2019 13:39
> To: Mikael Abrahamsson <[email-masked]>; Randy Presuhn
> <[email-masked]>
> Cc: [email-masked]
> >>> Where is the text that tells the server implementor whether to throw
> >>> an error when client commits non-zero bits, or to just throw the
> >>> bits away and store the value in the canonical format?
> >>
> >> Such text would be an inappropriate constraint the server's internal
> >> representation.  We should only specify the externally-visible
> >> behaviour: that the reported value will be in the canonical format.
> >> Whether an implementation preserves extraneous cruft in its internal
> >> representation is purely an implementation decision, and not subject
> >> to standardization.
> >
> > Jurgen seems to say it should convert it and not throw an error, and
> > I'd like text to say that indeed, this is proper behaviour. Nobody has
> > so far been able to tell me where this text currently is, so that's
> > why I'm
> 
> If we agree that a type defines the set of legal on-the-wire values (possibly
> modulo representation - JSON/XML/...), then section 4.2.2.1 in RFC 7950 says:
> 

I don't think that there is text for a derived type that defines a canonical representation that:
(i) The server must internally treat the data as if it were in the canonical form
(ii) If the data is returned then it must be in the canonical form.

We should fix this in YANG 1.2 so this is clearer, but YANG 1.2 isn't going to be here for a while.  I don't think that we can do this as an erratum, but perhaps the 6991bis could, for each type state that the it is handled equivalently to RFC 7950 section 9.1.  Or would that just end up being noise?

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 May 2019 10:23:01 +0200

> (ii) If the data is returned then it must be in the canonical form.

The text is in subsection 9.1, which really should have been outside
sec. 9 (Built-In Types), and applicable to derived types as well. 

But anyway, Mikael wrote about the case when the client sends a value
(specifically, belonging to the ipv6-prefix type) that is NOT in the
canonical format.

Lada

---

From: tom petch <[email-masked]>
Date: Fri, 3 May 2019 12:08:10 +0000

----- Original Message -----
From: "Mikael Abrahamsson" <[email-masked]>
To: "Randy Presuhn" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, May 02, 2019 12:35 PM

> Jurgen seems to say it should convert it and not throw an error, and
I'd
> and every definition of types (or both of them).

Mikael

How about RFC791, still much quoted in all aspects of the work of the
IETF?

" In general, an implementation must be conservative
  in its sending behavior, and liberal in its receiving behavior.  That
  is, it must be careful to send well-formed datagrams, but must accept
  any datagram that it can interpret (e.g., not object to technical
  errors where the meaning is still clear)."

We did not have MUST in those days, but had we, this would have been one
IMHO.

Tom Petch

---

From: Christian Hopps <[email-masked]>
Date: Fri, 3 May 2019 15:24:51 -0400

> On May 3, 2019, at 8:08 AM, tom petch <[email-masked]> wrote:
> 

So, this is a good opportunity to mention what has bothered me during this discussion.

Let's for a moment leave aside the "standards language" etc part, and instead consider "What's actually useful for people who try and run networks."

NETCONF and YANG have the concept of validating configuration, this is very useful for users. In a previous job I incorrectly started out to use ipv4-prefix in a model where I really wanted an ipv4-address-and-prefix (i.e., an interface context). Now consider the reverse case then where the model really expects a prefix only, and the user for some reason thinks it will accept and can make use of an "address-and-prefix". The most obvious indication that the user has got this wrong is that they have host bits set. So if the server accepts the value but silently discards the host bits the error is not caught and the user and server probably have different ideas about what's going to happen.

IOW, I don't think "where the meaning is still clear" applies to stripping host bits from a value, in fact I think it's more clear that stripping the host bits is actually ignoring (getting wrong) the user intent.

Thanks,
Chris.