Subject: [netmod] Disallow leading/trailing whitespaces and non-printable chars in strings

From: Gerhard Muenz <[email-masked]>
Date: Fri, 02 Jul 2010 16:32:54 +0200

Dear all,

The YANG guidelines say:

4.10. Data Types

   ...

   For string data types, data definition semantics SHOULD NOT rely on
   preservation of leading and trailing whitespace characters.

So, I assume that leading and trailing whitespaces should be disallowed
by the pattern statement if different string node values in a YANG
module need to match.

What would be the right pattern statement to disallow leading and
trailing whitespaces as well as all non-printable characters?

Currently, I have:

      length "1..max";
      pattern "\S([#20\c]*\S)?";

Thanks for your help.

Gerhard

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 07:40:37 -0700

>       pattern "\S([#20\c]*\S)?";

The YANG Guildelines are much more conservative than the YANG spec.
That is why they exist -- to prevent YANG from needing these CLRs.
For example, YANG allows billion char identifiers, but a standard YANG
module can only use 64 character identifiers.

The intent of this separation seems to be to allow vendors
to be more liberal in their own proprietary YANG usage
than standard YANG modules, which need to stick to a
common subset of all YANG behavior.

> Gerhard

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 18:16:45 +0200

>    preservation of leading and trailing whitespace characters.

Now that I read this in isolation, I am not sure why we have this
rule. I believe things are simpler if a value simply does not change.
If we allow white space changes, we make things more complex rather
than simpler...

>       pattern "\S([#20\c]*\S)?";

And this kind of proves my point. ;-)

/js 

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 09:43:44 -0700

> If we allow white space changes, we make things more complex rather

Except this does not deal with the reality that different XML parser
implementations are likely to deal with this issue the same way.

We can remove the guideline, and just let users get what they get
from implementors.  The XML spec says that whitespace is
significant, so we can defer to their ownership of the problem.
XML says the client can rely on preservation of whitespace,
so maybe there is no need to worry about it, even if it is common
practice for server code to strip strings before using them. 

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 18:55:45 +0200

> practice for server code to strip strings before using them. 

I believe we are heading in the wrong direction if we encourage data
model writers to create rather complex regular expressions just
because something down in the NETCONF encoding layer might not follow
a particular standard. We better have test cases / test suites that
make it clear what a correct implementation should do and fix the
problem where it is, perhaps even adding clear language to 4741bis.

/js

---

From: Phil Shafer <[email-masked]>
Date: Fri, 2 Jul 2010 13:11:20 -0400

>If we allow white space changes, we make things more complex rather

This rule is required to deal with the variety of XML
processing tools that treat whitespace as something
that can be freely introduced, especially indentation
and newlines.  XSLT is notorious in this regard.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 19:44:11 +0200

> and newlines.  XSLT is notorious in this regard.

How good that we use something as advanced as XML then...

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 11:00:04 -0700

> How good that we use something as advanced as XML then...

So we put up an 'orange cone' and tell the drivers
not to hit it.  Relying on preservation of leading and
trailing whitespace is not what tools do today at all.
Why start relying on it now?  Better to advice people
to follow the industry practice.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 20:10:42 +0200

> to follow the industry practice.

I love "industry practice". So do we have an example where it makes
sense to allow white space even though we know it won't be preserved?
In other words, why allow leading/trailing white space at all if it
can be arbitrarily changed? Are there "industry practice" tools add
leading/trailing white space or just "industry practice" tools
removing leading/trailing white space?

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 11:58:23 -0700

> removing leading/trailing white space?

So you want to change SHOULD NOT to MUST NOT?
OK -- I do not object to that.

But this means MUST NOT rely on whitespace preservation.
Is that the same thing as MUST NOT use leading or trailing whitespace
in a pattern?  Is there a need for both guidelines?

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 21:30:20 +0200

> in a pattern?  Is there a need for both guidelines?

As usual, we do not communicate well. Give me an example where it
makes sense to allow for leading/trailing white space in a pattern
that can be arbitrarily removed. My point is that in such a case, the
pattern should likely not allow leading/trailing white space in the
first place since the leading/trailing white space obviously is not
meaningful.

I have a problem with encouraging people to write pattern like the one
that started this thread (unfortunately we do not know what this type
is going to model).

/js

---

From: Phil Shafer <[email-masked]>
Date: Fri, 2 Jul 2010 15:12:33 -0400

>How good that we use something as advanced as XML then...

Yup, like most things, it's simple only went viewed from a distance.

On a related XML encoding note we finally gave up and started
encoding control characters (which XML does not support (shockingly))
using a "private use" unicode range starting at 0xE000.  I could
not find a better way to do it, and it needed to be done.  Can/should
we standardize on this encoding?  Is there a means of documenting it?

My current encoding issue is that because XML lacks a real escaping
mechanism, strings in XSLT XPath expression cannot have both single
and double quotes, even when resorting to the "&quot;" and "&pos;"
encodings.  This means there's no simple, safe way of lifting a
string from, say, a YANG file and stuffing it into an XPath string.
I'm having to break the string up, and turn it into a concat() of
two or more strings, which is just plain sad.

Had no one heard of the backslash character?  :-(  (aka &sad;)

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 12:36:58 -0700

>> in a pattern?  Is there a need for both guidelines?

I think we are communicating.
Provide some OLD: and NEW: text proposals
in RFC Editor format, and I will have a better
idea what you want to change in the document.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Jul 2010 21:53:57 +0200

> idea what you want to change in the document.

OLD:

NEW:

   Since XML tools often do not preserve leading and trailing white
   space characters, string data types SHOULD where possible avoid
   leading and trailing white space characters.

Rationale:

- Provide an explanation why the rule exists.  Encourage people not to
- allow leading/trailing white space, the previous text did encourage
  people to allow such characters by making them optional.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 02 Jul 2010 13:09:33 -0700

>    leading and trailing white space characters.

s/avoid leading/avoid using leading/

I thought you wanted MUST instead of SHOULD?

I don't careeither way  --your edit is fine with me.
(See how easy that was? :-)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 02 Jul 2010 23:24:57 +0200 (CEST)

>    leading and trailing white space characters.

What sort of XML tool does not preserve leading and trailing
whitespace?  In my experience XML tools do preserve this whitespace.

Phil had a different explaination:

phil> This rule is required to deal with the variety of XML
phil> processing tools that treat whitespace as something
phil> that can be freely introduced, especially indentation
phil> and newlines.  XSLT is notorious in this regard.

But if this is the case, the rule above is not sufficient, and if we
want to guard against tools that introduce whitespace we need other
mechanisms to deal with this in the YANG language (which I do not want
to do).

XML itself has no special rules for this whitespace in element data,
and that's just how it works.  We should simply play along.

Also, I wonder what the practical impact of this rule is.  Are we
saying that the 'string' data type should not be used?  Do we have an
alternative, or should every data model invent their own
my-better-string data type?

/martin

---

From: Gerhard Muenz <[email-masked]>
Date: Sat, 03 Jul 2010 13:47:30 +0200

> that started this thread (unfortunately we do not know what this type

In IPFIX-CONFIG, we use string nodes <name> as identifiers.
They are used as key in config lists etc., and they are being referenced 
by leafref. Hence, we have to make sure that if the user configures the 
same string at two different places (nodes), they must be recognized as 
identical.

Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Sat, 03 Jul 2010 13:52:18 +0200

On 03.07.2010 13:47, Gerhard Muenz wrote:
> In IPFIX-CONFIG, we use string nodes<name>  as identifiers.

And the other way round, of course: if the strings differ, they must be 
recognized as different.

I assume that other people might have this problem. So, I suggest to 
either remove the paragraph from the guidelines or to provide a solution 
for this problem, e.g. an example pattern or a common type.

Thanks,
Gerhard

---

From: Andy Bierman <[email-masked]>
Date: Sat, 03 Jul 2010 07:10:34 -0700

On 07/03/2010 04:52 AM, Gerhard Muenz wrote:
> And the other way round, of course: if the strings differ, they must
> be recognized as different.
>
> either remove the paragraph from the guidelines or to provide a
> solution for this problem, e.g. an example pattern or a common type.
>

Removing the guideline is fine with me.
I do not think using leading whitespace in a name string
is a good idea.  I cannot think of any use-case at all
where using leading and trailing whitespace in a
YANG string value would be helpful.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 3 Jul 2010 23:08:18 +0200

> same string at two different places (nodes), they must be recognized as 

In this case, it seems you like to restrict the string such that 
it does not contain any leading/trailing whitespace (perhaps no
whitespace at all).

/js

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 05 Jul 2010 17:15:51 +0200

> it does not contain any leading/trailing whitespace (perhaps no

We want to allow whitespaces in the middle of the string if possible.

So, is the following pattern appropriate?

Are there better alternatives?

Or shall I just ignore the issue?

Thanks,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 06 Jul 2010 09:41:21 +0200

Hi Jürgen,

Juergen Schoenwaelder wrote:
> On Mon, Jul 05, 2010 at 05:15:51PM +0200, Gerhard Muenz wrote:
> 
> Should that not be #x20 instead of #20?

Indeed.

I changed it to

      pattern "\S([\s\c]*\S)?";

yet this does not allow "/".

For simplicity, I use

      pattern "\S(.*\S)?";

for the moment.

Nevertheless, I think that it is strange that the guidelines indirectly
disadvice the usage of string type nodes as keys and targets of leafrefs.

Either the string type is robust enough to be used in these cases
without any patterns or it should be stated explicitly that unrestricted
strings should not be used. In the later case, it would be nice to have
a second string type which can be used instead.

Sorry for insisting on this, but from the point of view of a data model
writer, the current guidelines are a bit confusing in this regard.

Thanks,
Gerhard

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 6 Jul 2010 10:07:28 +0200

> writer, the current guidelines are a bit confusing in this regard.

I agree and I hope that NETCONF implementations do not mess around
with any values in XML elements that are part of the (config) payload.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 06 Jul 2010 22:05:58 +0200 (CEST)

> with any values in XML elements that are part of the (config) payload.

+1

I think this guideline should be removed, and it should be fine to use
'string' as is.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 07 Jul 2010 11:34:35 +0200

> YANG string value would be helpful.

Perhaps YANG could define the canonical value for strings to be
normalized in some way, either as XML attribute values or just by
stripping leading and trailing whitespace.

Lada

> 
> > Thanks,
> > Gerhard
> >
> 
> Andy
> 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 7 Jul 2010 11:48:45 +0200

> stripping leading and trailing whitespace.

I believe NETCONF implementations should treat a plain string value as
an opaque value and not mess around with it. There might be "industry
practice" tools that do this kind of bad thing but that is not a good
reason for us to introduce the same misbehaviour into NETCONF.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 07 Jul 2010 12:02:08 +0200

> reason for us to introduce the same misbehaviour into NETCONF.

I agree, except that using strings as list keys may lead to tricky
problems, especially when the values are entered manually. That's why
key-like things are invariably implemented as attributes in XML, e.g.

<key value="foo"/>

rather than

<key>foo</key>

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 7 Jul 2010 12:20:12 +0200

On Wed, Jul 07, 2010 at 12:02:08PM +0200, Ladislav Lhotka wrote:
> Juergen Schoenwaelder p????e v St 07. 07. 2010 v 11:48 +0200:

As a data model writer, I would not use unconstrained strings as
identifiers or keys. But this is a issue unrelated to the question
whether NETCONF messes around with string values.

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 07 Jul 2010 09:05:53 -0700

> whether NETCONF messes around with string values.

Let's review...

1) Phil pointed out that some tools inject whitespace
     into XML
2) Nobody is claiming that using leading or trailing
    whitespace is a good idea.  (There could be a corner-case
    like Lada's HTML 'banner' leaf, although unlikely.)
3) If the server encounters L or T WSP, it has to decide
    what to do with it.  Should the server assume the 1 in 10,000
    chance that the client really wants this WSP, or assume the
    9,999 in 10,000 chance that it is injected WSP and remove it?

Forcing every server to implement the 1 in 10,000 chance
is a typical IETF move, but it goes against the Postel Principle
and is IMO the wrong thing to do.

There are claims in various NETCONF documents that NETCONF
and even YIN are intended to work well with XSLT.  Is this marketing
hype or the true intent of the NETCONF and NETMOD WGs?

I don't mind removing the guideline completely, but I object to
forcing the server to do anything wrt/ whitespace.
XML already says what to do, so the server is breaking an XML
rule already, and there is no need for redundant guidelines like this one.

Andy