Subject: [Netconf] Unable to filter strings with leading whitespace - errata on Netconf RFC?

From: Balazs Lengyel <[email-masked]>
Date: Fri, 20 Feb 2015 14:55:41 +0100

Hello,
I assume that " hello" with the leading space is a perfectly OK value in 
Netconf/YANG for a string leaf. I also think that the " hello" string is 
not the same as the "hello" string without the space.
However https://tools.ietf.org/html/rfc6241#section-6.2.5 part of the 
Netconf RFC says about filtering

"Leading and trailing whitespace characters are ignored, but any
       whitespace characters within a block of text characters are not
       ignored or modified."

IMHO this is an error for YANG string leafs. It is perfectly OK to 
convert an integer "  1" into "1" but converting the string " hello" 
into "hello" is changing its content. So there should be an errata 
stating that for string type content all white space content is 
considered during filtering.

Agree? If you do not agree, how am I supposed to filter for " hello" 
instead of "hello"?
regards Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
Senior Specialist
ECN: 831 7320                        Tel: +36-1-437-7320
Mobile: +36-70-330-7909              email: [email-masked]

---

From: t.petch <[email-masked]>
Date: Sun, 22 Feb 2015 10:59:00 +0000

----- Original Message -----
From: "Balazs Lengyel" <[email-masked]>
To: <[email-masked]>
Cc: "Peter Labraaten" <[email-masked]>
Sent: Friday, February 20, 2015 1:55 PM

> Agree? If you do not agree, how am I supposed to filter for " hello"

You can't!  I think that the Netconf RFC is very clear.  YANG does allow
spaces to be significant or not, depending on the quoting (RFC6020
s6.1.3) but Netconf, as you point out, does not.

Nor is this an erratum, which would imply that the RFC wrongly (or
unclearly) states the consensus of the WG.

The text you cite first appears in
  draft-ietf-netconf-prot-06
in April 2005, following an e-mail from Andy  at
http://www.ietf.org/mail-archive/web/netconf/current/msg00414.html

The earlier text also came from Andy
http://www.ietf.org/mail-archive/web/netconf/current/msg00004.html

There was a lengthy discourse on subpath filtering between the two but
that was about whether to have home-grown filtering, a subset of XPath
or full XPath or some combination thereof.

Tom Petch

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Andy Bierman <[email-masked]>
Date: Sun, 22 Feb 2015 08:10:11 -0800

On Fri, Feb 20, 2015 at 5:55 AM, Balazs Lengyel
<[email-masked]> wrote:
> Netconf/YANG for a string leaf. I also think that the " hello" string is not
> the same as the "hello" string without the space.
> IMHO this is an error for YANG string leafs. It is perfectly OK to convert
> an integer "  1" into "1" but converting the string " hello" into "hello" is
> changing its content. So there should be an errata stating that for string
> type content all white space content is considered during filtering.
>
> Agree? If you do not agree, how am I supposed to filter for " hello" instead
> of "hello"?

I don't think our server actually trims whitespace.
(Any server using content_match_test from yuma does the same thing).
It compares the entire value.

This text is probably left over from when the WG thought it was OK
for the XML parser to trim whitespace.

There was an incorrect assumption the following are equivalent

      <foo>
         some string
      </foo>

      <foo>some string</foo>

> regards Balazs

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 Feb 2015 14:21:17 +0100

> Agree? If you do not agree, how am I supposed to filter for " hello" 

I agree. It is perfectly legal (albeit hardly recommendable) to define
the following enums in YANG, and they have to be regarded as distinct by
a NETCONF server:

enum "foo";
enum " foo";

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Feb 2015 08:29:21 -0800

> the following enums in YANG, and they have to be regarded as distinct by

This is not allowed,  From 9.6.4:

   It takes as an argument a string which is the assigned name.  The
   string MUST NOT be empty and MUST NOT have any leading or trailing
   whitespace characters.  The use of Unicode control codes SHOULD be
   avoided.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 Feb 2015 19:55:19 +0100

> On 23 Feb 2015, at 17:29, Andy Bierman <[email-masked]> wrote:
> 

OK, sorry, my fault. But still, e.g. list keys “foo” and “ foo” must be considered different. Trimming whitespace in XML *elements* breaks all XML tools.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 23 Feb 2015 20:42:54 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> OK, sorry, my fault. But still, e.g. list keys “foo” and “ foo” must
> be considered different.

Yes, they are absolutely different.  But you cannot use subtree
filtering to get just one of them.  (You can get both, and then apply
client-side logic to single out the one you actually were interested
in).

> Trimming whitespace in XML *elements* breaks
> all XML tools.

Not true.  Whitespaces are preserved in XML, yes, but that just means
that generic tools will not do any trimming.  Any compliant subtree
filtering implementation will thus have to do the trimming after XML
parsing.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 Feb 2015 21:16:08 +0100

> On 23 Feb 2015, at 20:42, Martin Bjorklund <[email-masked]> wrote:
> 

This is not how I read the text that Balazs cited: it says that leading and trailing whitespace is ignored, and I understand it is ignored in content match nodes. Therefore, the list entry with “ foo” key must be inaccessible, which is exactly what Balazs concluded.

> filtering implementation will thus have to do the trimming after XML

Yes, but it means a complication, e.g. for converting subtree filters to XPath, also because this whitespace trimming is not the same as the effect of the normalize-space function.

Lada

---

From: t.petch <[email-masked]>
Date: Mon, 23 Feb 2015 21:07:18 +0000

----- Original Message -----
From: "Ladislav Lhotka" <[email-masked]>
To: "Martin Björklund" <[email-masked]>
Cc: "Netconf" <[email-masked]>; "Peter Labraaten"
<[email-masked]>
Sent: Monday, February 23, 2015 8:16 PM
> >>>>> I assume that " hello" with the leading space is a perfectly OK
value
> >>>>> in
> >>>>> Netconf/YANG for a string leaf. I also think that the " hello"
string
> >>>>> is
> >>>> the following enums in YANG, and they have to be regarded as
distinct
> >>>> by

> This is not how I read the text that Balazs cited: it says that
leading and trailing whitespace is ignored, and I understand it is
ignored in content match nodes. Therefore, the list entry with “ foo”
key must be inaccessible, which is exactly what Balazs concluded.

Lada

I agree.  My reading of the text is that stripping the spaces applies to
the content match node and that no such normalisation is performed on
the data base element ("the leaf  node element content").

Tom Petch

> Yes, but it means a complication, e.g. for converting subtree filters
to XPath, also because this whitespace trimming is not the same as the
effect of the normalize-space function.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 23 Feb 2015 13:51:27 -0800

> the data base element ("the leaf  node element content").

I think the issue Balazs raised is questioning why the server would alter
the filter match expression.  Why is this a feature?  The client can create
string leafs with leading and trailing whitespace, but not filter them
with a content match node.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 24 Feb 2015 10:39:45 +0100

Hello,

Although this might not be 100% standard compliant, I wrote the
following internal guideline in Ericsson:

- When Netconf was designed they tried to make it data model
agnostic as there was no accepted data modeling language at the
time. Thus it would handle string attributes the same way as integer
or binary attributes. While separating layers (protocol versus data
model) is generally a good idea, this level of "data model agnostic"
treatment is taking it too far, it is a mistake.

- Comparing values does not and should not mean the same thing for
strings and integers,

while the string "    aa  " and "aa" are different

the numbers  "   +1" and "1" are not.

- RFC6241 clearly states that whitespace has to be considered part
of a string

Our responsibility is not just to follow the standard, but to be
intuitive.  Trimming whitespace from a string is not intuitive, I
would rather call it surprising.

Also if we would blindly follow RFC6241:

"Leading
and trailing whitespace characters are ignored, but any
whitespace characters within a block of text characters are
not ignored or modified."

would mean that you would never ever get a content match for  the
string "  aa" as ignoring  the whitespace in the content match input
means, the content match input is first converted to "aa" and that
is what we try to match, which will never be equal to "  aa".

regards Balazs

On 2015-02-23 22:51, Andy Bierman
wrote:

----- Original Message -----
From: "Ladislav Lhotka" <[email-masked]>
To: "Martin Björklund" <[email-masked]>
Cc: "Netconf" <[email-masked]>; "Peter Labraaten"

On Mon, Feb 23, 2015 at 5:21 AM, Ladislav Lhotka <[email-masked]>
wrote:

Hello,

value

in

string

is
not the same as the "hello" string without the space.
However https://tools.ietf.org/html/rfc6241#section-6.2.5 part

of the

"Leading and trailing whitespace characters are ignored, but any
whitespace characters within a block of text characters are

not

ignored or modified."

IMHO this is an error for YANG string leafs. It is perfectly OK

to

convert an integer " 1" into "1" but converting the string "

hello"

into "hello" is changing its content. So there should be an

errata

Agree? If you do not agree, how am I supposed to filter for "

hello"

instead of "hello"?

I agree. It is perfectly legal (albeit hardly recommendable) to

define

distinct

by
a NETCONF server:

enum "foo";
enum " foo";

string MUST NOT be empty and MUST NOT have any leading or

trailing

whitespace characters. The use of Unicode control codes SHOULD

be

avoided.

OK, sorry, my fault. But still, e.g. list keys “foo” and “ foo”

must

be considered different.

Yes, they are absolutely different. But you cannot use subtree
filtering to get just one of them. (You can get both, and then

apply

Lada

Tom Petch

Andy

Not true. Whitespaces are preserved in XML, yes, but that just

means

Lada

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 Feb 2015 10:55:06 +0100

Hi,

please note that RFC 6020 takes care of defining canonical
representations for most data types. RFC 6991 follows this practice
and does the same for the common typedefs. Hence, the representation
of a positive integer you can rely on is 1 and not +1.

/js

On Tue, Feb 24, 2015 at 10:39:45AM +0100, Balazs Lengyel wrote:
>    Hello,
>    Although this might not be 100% standard compliant, I wrote the following
>    internal guideline in Ericsson:
>    - When Netconf was designed they tried to make it data model agnostic as
>    there was no accepted data modeling language at the time. Thus it would
>    handle string attributes the same way as integer or binary attributes.
>    While separating layers (protocol versus data model) is generally a good
>    idea, this level of "data model agnostic" treatment is taking it too far,
>    it is a mistake.
>    - Comparing values does not and should not mean the same thing for strings
>    and integers,
>    - RFC6241 clearly states that whitespace has to be considered part of a
>    string
> 
>    intuitive.  Trimming whitespace from a string is not intuitive, I would
>    rather call it surprising.
> 
>    whitespace characters within a block of text characters are not ignored or
>    modified."
>    would mean that you would never ever get a content match for  the string
>    "  aa" as ignoring  the whitespace in the content match input means, the
>    content match input is first converted to "aa" and that is what we try to
>    match, which will never be equal to "  aa".
> 
>    regards Balazs
> 
>    On 2015-02-23 22:51, Andy Bierman wrote:
> 
>  On Mon, Feb 23, 2015 at 1:07 PM, t.petch [1]<[email-masked]> wrote:
> 
>  From: "Ladislav Lhotka" [2]<[email-masked]>
>  To: "Martin Bj�rklund" [3]<[email-masked]>
>  Cc: "Netconf" [4]<[email-masked]>; "Peter Labraaten"
>  [5]<[email-masked]>
>  On 23 Feb 2015, at 20:42, Martin Bjorklund [6]<[email-masked]> wrote:
> 
>  Ladislav Lhotka [7]<[email-masked]> wrote:
> 
> 
>  On 23 Feb 2015, at 17:29, Andy Bierman [8]<[email-masked]> wrote:
> 
>  On Mon, Feb 23, 2015 at 5:21 AM, Ladislav Lhotka [9]<[email-masked]>
>  wrote:
> 
>  Balazs Lengyel [10]<[email-masked]> writes:
> 
> 
>  Hello,
>  However [11]https://tools.ietf.org/html/rfc6241#section-6.2.5 part
> 
>  of the
> 
>  OK, sorry, my fault. But still, e.g. list keys "foo" and " foo"
> 
>  must
> 
>  be considered different.
> 
>  ignored in content match nodes. Therefore, the list entry with " foo"
>  [12][email-masked]
>  [13]https://www.ietf.org/mailman/listinfo/netconf
> 
> 
>  [14][email-masked]
>  [15]https://www.ietf.org/mailman/listinfo/netconf
> 
>  [16][email-masked]
>  [17]https://www.ietf.org/mailman/listinfo/netconf
> 
>  --
>  Mobile: +36-70-330-7909              email: [18][email-masked]
> 
> References
> 
>    Visible links
>    1. mailto:[email-masked]
>    2. mailto:[email-masked]
>    3. mailto:[email-masked]
>    4. mailto:[email-masked]
>    5. mailto:[email-masked]
>    6. mailto:[email-masked]
>    7. mailto:[email-masked]
>    8. mailto:[email-masked]
>    9. mailto:[email-masked]
>   10. mailto:[email-masked]
>   11. https://tools.ietf.org/html/rfc6241#section-6.2.5
>   12. mailto:[email-masked]
>   13. https://www.ietf.org/mailman/listinfo/netconf
>   14. mailto:[email-masked]
>   15. https://www.ietf.org/mailman/listinfo/netconf
>   16. mailto:[email-masked]
>   17. https://www.ietf.org/mailman/listinfo/netconf
>   18. mailto:[email-masked]

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 24 Feb 2015 11:04:40 +0100

> On 24 Feb 2015, at 10:39, Balazs Lengyel <[email-masked]> wrote:
> 
> Hello,
> Although this might not be 100% standard compliant, I wrote the following internal guideline in Ericsson:
> - When Netconf was designed they tried to make it data model agnostic as there was no accepted data modeling language at the time. Thus it would handle string attributes the same way as integer or binary attributes. While separating layers (protocol versus data model) is generally a good idea, this level of "data model agnostic" treatment is taking it too far, it is a mistake.  
> - Comparing values does not and should not mean the same thing for strings and integers, 

Actually, RFC 6020 says nothing about the handling of whitespace in numeric instances. I think it should.

> - RFC6241 clearly states that whitespace has to be considered part of a string
> 
> Our responsibility is not just to follow the standard, but to be intuitive.  Trimming whitespace from a string is not intuitive, I would rather call it surprising.

+1

Lada

> "Leading and trailing whitespace characters are ignored, but any whitespace characters within a block of text characters are not ignored or modified."
> would mean that you would never ever get a content match for  the string "  aa" as ignoring  the whitespace in the content match input means, the content match input is first converted to "aa" and that is what we try to match, which will never be equal to "  aa".  
> 
> regards Balazs
> 
>>> Cc: "Netconf" 
>>> <[email-masked]>
>>> ; "Peter Labraaten"
>>>>> Ladislav Lhotka 
>>>>> <[email-masked]>
>>>>>>>>> However 
>>>>>>>>> https://tools.ietf.org/html/rfc6241#section-6.2.5
>>>>>>>>>  part
>>>>>>>>> 
>>> of the
>>>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 Feb 2015 11:12:38 +0100

> Actually, RFC 6020 says nothing about the handling of whitespace in numeric instances. I think it should.

RFC 6020 section 9.2:

   An integer value is lexically represented as an optional sign ("+" or
   "-"), followed by a sequence of decimal digits.  If no sign is
   specified, "+" is assumed.

[...]

   The canonical form of a positive integer does not include the sign
   "+".  Leading zeros are prohibited.  The value zero is represented as
   "0".

I find this pretty clear.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 24 Feb 2015 11:18:53 +0100

> On 24 Feb 2015, at 11:12, Juergen Schoenwaelder <[email-masked]> wrote:
> 

OK, but then Balazs’ advice regarding numbers “  +1” and “1” is incorrect.

Lada

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 24 Feb 2015 11:48:35 +0100

Hello Lada,

I wrote:

--------------------------

- Comparing values
does not and should not mean the same thing for strings and
integers,

--------------------------

So as I see it (in agreement with RFC6020)  the following are all
equivalent for an integer type:

"1" "+1" "  1"

Balazs

On 2015-02-24 11:18, Ladislav Lhotka
wrote:

RFC 6020 section 9.2:

Lada

[...]

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 24 Feb 2015 11:50:52 +0100

> On 24 Feb 2015, at 11:48, Balazs Lengyel <[email-masked]> wrote:
> 
> Hello Lada, 
> I wrote:
> So as I see it (in agreement with RFC6020)  the following are all equivalent for an integer type:
> "1" "+1" "  1”

The third form is an invalid lexical representation.

Lada

> Balazs
> 
> On 2015-02-24 11:18, Ladislav Lhotka wrote:

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 Feb 2015 12:09:21 +0100

> The third form is an invalid lexical representation.

And the server's canonical representation is "1".

/js

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 24 Feb 2015 12:33:55 +0100

> The third form is an invalid lexical representation.
[BALAZS]: yes true,
but following the Postel principle (be liberal in what you accept) and 
seeing that whitespace is sometimes
added just for formating, and that this is not a source of possible 
misunderstandings for an integer, I see it
OK to accept it by trimming leading and trailing whitespace.

---

From: t.petch <[email-masked]>
Date: Tue, 24 Feb 2015 11:37:34 +0000

----- Original Message -----
From: "Balazs Lengyel" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, February 24, 2015 9:39 AM
following internal guideline in Ericsson:
> - When Netconf was designed they tried to make it data model agnostic
as there was no accepted data modeling language at the time. Thus it
would handle string attributes the same way as integer or binary
attributes. While separating layers (protocol versus data model) is
generally a good idea, this level of "data model agnostic" treatment is
taking it too far, it is a mistake.
strings and integers,
> "Leading and trailing whitespace characters are ignored, but any
whitespace characters within a block of text characters are not ignored
or modified."
string "  aa" as ignoring  the whitespace in the content match input
means, the content match input is first converted to "aa" and that is
what we try to match, which will never be equal to "  aa".

I have a certain sympathy with the author of the text in RFC6241.  You
would say that

Balazs Lengyel
and
Balazs Lengyel
are different because the second one is actually
"Balazs Lengyel  "
to which I would say what an opportunity for a phishing attack!  I
cannot remember if attacks based on something like this were happening
10 years ago, but I have some distant memory thereof.

It is unfortunate that the canonical form in Netconf is not the same as
in YANG but I would find the YANG form the more surprising.  Some
databases I deal with even turn some punctuation characters into white
space before performing a comparison.  Again, some databases have stem
searches with a range of regular expressions in which case the search
you are looking for would be no problem.

Tom Petch

> To: "Martin BjÃ¶rklund" <[email-masked]>
> OK, sorry, my fault. But still, e.g. list keys â€œfooâ€ and â€œ
fooâ€
> ignored in content match nodes. Therefore, the list entry with â€œ
fooâ€
> effect of the normalize-space function.

------------------------------------------------------------------------
--------

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 24 Feb 2015 19:17:19 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> ignored in content match nodes.

Yes, you're right.  I got it wrong.

/martin