Subject: Re: [netmod] Alexey Melnikov's Discuss on draft-ietf-netmod-module-tags-07: (with DISCUSS)

From: Christian Hopps <[email-masked]>
Date: Wed, 12 Feb 2020 19:30:23 -0500

The intent in the document is to place as few restrictions on tags as possible to allow for future-proofing and organic growth of use both within and outside of SDOs. For standard tags we trust IANA (and the human behind the process) to make the call on whether a tag is already present. :)

Having worked for a company where a lot of XML string data was non-ascii I find limiting to ascii to be rather restrictive.

Thanks,
Chris.

> On Apr 11, 2019, at 9:41 AM, Alexey Melnikov via Datatracker <[email-masked]> wrote:
> 
> Alexey Melnikov has entered the following ballot position for
> draft-ietf-netmod-module-tags-07: Discuss
> 
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-netmod-module-tags/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> This is generally a fine document, but after checking RFC 7950 syntax for
> strings I question why you think you need non ASCII tags. There are so many
> problems that can arise from that. For example, how would IANA be able to
> enforce uniqueness of Unicode tags written in different Unicode
> canonicalisation forms?
> 
> 
> 
>

---

From: "Alexey Melnikov" <[email-masked]>
Date: Thu, 13 Feb 2020 13:10:02 +0000

Hi Christian,

On Thu, Feb 13, 2020, at 12:30 AM, Christian Hopps wrote:
> The intent in the document is to place as few restrictions on tags as 
> possible to allow for future-proofing and organic growth of use both 
> within and outside of SDOs. For standard tags we trust IANA (and the 
> human behind the process) to make the call on whether a tag is already 
> present. :)

And the problem with that is that because there might be multiple ways to encode in Unicode visually indistinguishable tags IANA would end up asking IESG for help.

So you need to at minimum specify a Unicode normalization form to use. I suggest you normatively reference RFC 5198 here.

> Having worked for a company where a lot of XML string data was 
> non-ascii I find limiting to ascii to be rather restrictive.

Best Regards,
Alexey

---

From: Schönwälder, Jürgen <[email-masked]>
Date: Thu, 13 Feb 2020 18:38:58 +0000

And a longer term solution might be to define a YANG Net-Unicode
string datatype that can be used in all situations where
non-normalized strings may cause problems. The problem (if one
agrees it is one) is likely much bigger than just YANG tags,
there likely are many uses of YANG strings where normalization
would be desirable.

/js

On Thu, Feb 13, 2020 at 01:10:02PM +0000, Alexey Melnikov wrote:
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Christian Hopps <[email-masked]>
Date: Thu, 13 Feb 2020 14:51:51 -0500

> On Feb 13, 2020, at 8:10 AM, Alexey Melnikov <[email-masked]> wrote:
> 
> Hi Christian,
> 

I don't have a problem including some text in the IANA guidance if we need it; however, the guidance already says:

  "New values should be well considered and not achievable through a
   combination of already existing IETF tags."

How could we arrive at a place where IANA is confused about allowing 2 visually indistinguishable strings given the above statement? The registry already requires IETF review, and it seems counter to the existing IANA guidance to even be talking about distinguishing visually indistinguishable tag values. IOW simply by talking about the need to compare 2 strings that are visually indistinguishable, we diminish the much more important intent of the initial guidance.

Thanks,
Chris.

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 14 Feb 2020 10:30:50 +0000

Hi Juergen,

This sounds potentially useful to me, although should this be for general unicode strings (e.g. ones that might include spaces), or just identifiers (without any spaces)?.  Is this something that could/should go into rfc6991-bis, or at least be discussed in that context?

I would have thought that normalization would be required wherever a configurable unicode string is used as a list key, or leaf-list.

Thanks,
Rob

> -----Original Message-----
> From: iesg <[email-masked]> On Behalf Of Schönwälder, Jürgen
> Sent: 13 February 2020 18:39
> To: Alexey Melnikov <[email-masked]>
> Cc: [email-masked]; [email-masked];
> [email-masked]; Joel Jaeggli <[email-masked]>; Christian Hopps
> <[email-masked]>; The IESG <[email-masked]>
> Subject: Re: [netmod] Alexey Melnikov's Discuss on draft-ietf-netmod-
> module-tags-07: (with DISCUSS)
> 
> And a longer term solution might be to define a YANG Net-Unicode string
> datatype that can be used in all situations where non-normalized strings
> may cause problems. The problem (if one agrees it is one) is likely much
> bigger than just YANG tags, there likely are many uses of YANG strings
> where normalization would be desirable.
> 
> /js
> 
> > > The intent in the document is to place as few restrictions on tags
> > > as possible to allow for future-proofing and organic growth of use
> > > both within and outside of SDOs. For standard tags we trust IANA
> > > (and the human behind the process) to make the call on whether a tag
> > > is already present. :)
> >
> > And the problem with that is that because there might be multiple ways
> to encode in Unicode visually indistinguishable tags IANA would end up
> asking IESG for help.
> >
> > So you need to at minimum specify a Unicode normalization form to use. I
> suggest you normatively reference RFC 5198 here.
> >
> > > > When responding, please keep the subject line intact and reply to
> > > > all email addresses included in the To and CC lines. (Feel free to
> > > > cut this introductory paragraph, however.)
> > > >
> > > >
> > > > Please refer to
> > > > https://www.ietf.org/iesg/statement/discuss-criteria.html
> > > > This is generally a fine document, but after checking RFC 7950
> > > > syntax for strings I question why you think you need non ASCII
> > > > tags. There are so many problems that can arise from that. For
> > > > example, how would IANA be able to enforce uniqueness of Unicode
> > > > tags written in different Unicode canonicalisation forms?
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 14 Feb 2020 10:43:09 +0000

Hi Alexey, Christian, 

Allowing Unicode but requiring normalization as per RFC 5198 for IANA managed tags makes sense to me.

But does the server also need to normalize any configured tags?  I.e. should the description for the tag typedef also specify that tags SHOULD be normalized, and specify a normalization method that SHOULD be used?  Or is the onus on the client to use sensible (i.e. already normalized) values, and if so, does that need to be stated?

Thanks,
Rob

> From: iesg <[email-masked]> On Behalf Of Alexey Melnikov
> Sent: 13 February 2020 13:10
> To: Christian Hopps <[email-masked]>
> Cc: [email-masked]; Joel Jaeggli <[email-masked]>; The IESG
> <[email-masked]>; [email-masked]; [email-masked]
> Subject: Re: Alexey Melnikov's Discuss on draft-ietf-netmod-module-tags-
> 07: (with DISCUSS)
> 
> Hi Christian,
> 
> And the problem with that is that because there might be multiple ways to
> encode in Unicode visually indistinguishable tags IANA would end up asking
> IESG for help.
> 
> > > syntax for strings I question why you think you need non ASCII tags.
> > > There are so many problems that can arise from that. For example,
> > > how would IANA be able to enforce uniqueness of Unicode tags written
> > > in different Unicode canonicalisation forms?
> > >
> > >
> > >
> > >
> >
> >

---

From: Christian Hopps <[email-masked]>
Date: Fri, 14 Feb 2020 06:01:24 -0500

I was not approaching this discuss with this level of change in mind. How many years does it take to get a YANG model even one as simple as this completed?

Thanks,
Chris.

> On Feb 14, 2020, at 5:43 AM, Rob Wilton (rwilton) <[email-masked]> wrote:
> 
> Hi Alexey, Christian, 
>

---

From: Christian Hopps <[email-masked]>
Date: Fri, 14 Feb 2020 06:06:21 -0500

For the record this one is 3 years and counting. For a list of tags.

> On Feb 14, 2020, at 6:01 AM, Christian Hopps <[email-masked]> wrote:
>

---

From: Christian Hopps <[email-masked]>
Date: Fri, 14 Feb 2020 06:21:24 -0500

How about I add this to the description of "typedef tag" in the module:

       description
         "A tag is a type 'string' value that does not include carriage
          return, newline or tab characters. It SHOULD begin with a
          registered prefix; however, tags without a registered prefix
-         SHOULD NOT be treated as invalid.";
+         SHOULD NOT be treated as invalid. For the purposes of comparison
+         non-ascii strings should use 'NFC' (RFC5198) normalization";
     }

Thanks,
Chris.

> On Feb 14, 2020, at 6:06 AM, Christian Hopps <[email-masked]> wrote:
>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 14 Feb 2020 12:35:02 +0000

Hi Chris,

I think that would be okay, although perhaps using SHOULD rather than "should" might be better, given that SHOULD NOT is used elsewhere in the description.

Thanks,
Rob

> From: Christian Hopps <[email-masked]>
> Sent: 14 February 2020 11:21
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Christian Hopps <[email-masked]>; Alexey Melnikov
> <[email-masked]>; [email-masked]; Joel Jaeggli
> <[email-masked]>; The IESG <[email-masked]>; [email-masked]; draft-ietf-
> [email-masked]
> >> I was not approaching this discuss with this level of change in mind.
> How many years does it take to get a YANG model even one as simple as this
> completed?
> >>
> >> Thanks,
> >> Chris.
> >>
> >>> But does the server also need to normalize any configured tags?  I.e.
> should the description for the tag typedef also specify that tags SHOULD
> be normalized, and specify a normalization method that SHOULD be used?  Or
> is the onus on the client to use sensible (i.e. already normalized)
> values, and if so, does that need to be stated?
> >>>
> >>> Thanks,
> >>> Rob
> >>>
> >>>
> >>>> Cc: [email-masked]; Joel Jaeggli <[email-masked]>; The
> >>>> IESG <[email-masked]>; [email-masked];
> >>>> [email-masked]
> >>>>> (and the human behind the process) to make the call on whether a
> >>>>> tag is already present. :)
> >>>>
> >>>> And the problem with that is that because there might be multiple
> >>>> ways to encode in Unicode visually indistinguishable tags IANA
> >>>> would end up asking IESG for help.
> >>>>
> >>>> So you need to at minimum specify a Unicode normalization form to
> >>>> use. I suggest you normatively reference RFC 5198 here.
> >>>>
> >>>>>> all email addresses included in the To and CC lines. (Feel free
> >>>>>> to cut this introductory paragraph, however.)
> >>>>>>
> >>>>>>
> >>>>>> Please refer to
> >>>>>> -----------------------------------------------------------------
> >>>>>> ---
> >>>>>> --
> >>>>>> DISCUSS:
> >>>>>> -----------------------------------------------------------------
> >>>>>> ---
> >>>>>> --
> >>>>>>
> tags.
> >>>>>> how would IANA be able to enforce uniqueness of Unicode tags
> >>>>>> written in different Unicode canonicalisation forms?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>
> >

---

From: Schönwälder, Jürgen <[email-masked]>
Date: Fri, 14 Feb 2020 14:30:31 +0000

Rob,

I think there are two related issues here:

a) If we need normalized strings (to avoid comparison suprises), we
   should have a common type for them; rfc6991-bis would be a proper
   home. I am _not_ saying we should delay the tags document for this,
   but we should think about providing a solution that can be easily
   reused. Right now, we often use strings as part of keys, which can
   lead to comparison issues.

b) It seems that normalized strings only solve part of the problem. If
   an organization creates names for 'things', the organization likely
   wants to further restrict the format of these names to something
   sensible to avoid fun with different kinds of hyphens or emojis or
   ... So while creative unicode characters may technically work,
   there will likely be good reasons to avoid some of them. (There are
   reasons why we have coding styles for most programming languages.)
   These rules may, however, differ between organizations.

We should not confuse a) and b). If IANA needs additional guidelines
for tags (their coding style for tags), then we should provide these
guidelines, i.e., this is a type b) action. The type a) action is
needed to technically ensure that comparisons do not lead to
surprises. But a) won't be an answer for all type b) issues. Of
course, we could give IANA a 'coding style' that avoids any
normalization issues. This would make IANA assigned tags safe but
would not avoid comparison surprises for other sources of tags.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 14 Feb 2020 09:26:08 -0800

On Fri, Feb 14, 2020 at 3:01 AM Christian Hopps <[email-masked]> wrote:

> I was not approaching this discuss with this level of change in mind. How
> many years does it take to get a YANG model even one as simple as this
> completed?
>
>

I strongly agree.
Just because there are refinements to the YANG "string" data type that are
possible
does not mean the Module Tags RFC has to solve that problem, or wait until
it is solved.
Every YANG module using the "string" data type (all of them?) could have
the same problem.

> Chris.

Andy

> should the description for the tag typedef also specify that tags SHOULD be
> normalized, and specify a normalization method that SHOULD be used?  Or is
> the onus on the client to use sensible (i.e. already normalized) values,
> and if so, does that need to be stated?
> >
> > Thanks,
> > Rob
> >
> >
> to

---

From: Randy Presuhn <[email-masked]>
Date: Fri, 14 Feb 2020 10:05:09 -0800

Hi -

> +         non-ascii strings should use 'NFC' (RFC5198) normalization";

There are other considerations beyond normalization form.
For the tip of the iceberg, see the definition of SnmpAdminString
in RFC 3411, or the "SHOULD be avoided" stuff in RFC 5198.

For things like tags where one would like to minimize accidental
visual punning, I'd suggest NFKC should probably be given some
consideration.   Excellent presentation of the issues is
available at https://unicode.org/reports/tr15/

That said, these are issues that were raised in the earliest days of 
Netmod /
Netconf, and no one should be surprised that they haven't gone away by
themselves.  So I find my self in ironic agreement with Andy that it doesn't
make sense to wait for / impose a global solution, because there's plenty
of evidence that having all sorts of ill-defined cases of questionable
interoperability in theory hasn't been sufficient to prevent adoption of
the technology.

Randy

---

From: Christian Hopps <[email-masked]>
Date: Mon, 17 Feb 2020 06:15:54 -0500

> On Feb 14, 2020, at 1:05 PM, Randy Presuhn <[email-masked]> wrote:
> 
> Hi -
> 

Do you think that the above is appropriate to include, or should be left out as doing more harm than helping?

BTW, I did look at the "SHOULD be avoided" (occurs twice that I saw) once dealing with LFs and CRs which lucky for us is not part of a tags allowable characters.

"
     typedef tag {
       type string {
         length "1..max";
         pattern '[\S ]+';
       }
"

The other is about private use unicode, and I don't know anything about that.

I think we either include the diff quoted above, or do nothing, as this isn't the document, as we all seem to agree, to fix the larger unicode issues with YANG string type.

Thanks,
Chris.

> That said, these are issues that were raised in the earliest days of Netmod /

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 17 Feb 2020 14:07:30 +0000

Hi Juergen,

Please see inline ...

> From: Schönwälder, Jürgen <[email-masked]>
> Sent: 14 February 2020 14:31
> Cc: Alexey Melnikov <[email-masked]>; [email-masked]; Joel
> Jaeggli <[email-masked]>; Christian Hopps <[email-masked]>; The IESG
> <[email-masked]>
[RW] 

I agree.  Note, I am also not proposing that we delay module-tags for rfc6991-bis.

RFC 7950 states that strings are not normalized by default (section 9.4.2).  Thinking about this some more, I think that it is reasonable to make it the client's responsibility to normalize strings, if required.

Chris, this would mean that no change to the typedef description is required.

> We should not confuse a) and b). If IANA needs additional guidelines for
> tags (their coding style for tags), then we should provide these
> guidelines, i.e., this is a type b) action. The type a) action is needed
> to technically ensure that comparisons do not lead to surprises. But a)
> won't be an answer for all type b) issues. Of course, we could give IANA a
> 'coding style' that avoids any normalization issues. This would make IANA
> assigned tags safe but would not avoid comparison surprises for other
> sources of tags.
[RW] 

So, solving B seems reasonable for the IANA defined module tags, following Alexey's suggestion of referencing RFC 5198 for normalization.

Thanks,
Rob

> > This sounds potentially useful to me, although should this be for
> general unicode strings (e.g. ones that might include spaces), or just
> identifiers (without any spaces)?.  Is this something that could/should go
> into rfc6991-bis, or at least be discussed in that context?
> >
> > I would have thought that normalization would be required wherever a
> configurable unicode string is used as a list key, or leaf-list.
> >
> > Thanks,
> > Rob
> >
> >
> > > agrees it is one) is likely much bigger than just YANG tags, there
> > > likely are many uses of YANG strings where normalization would be
> desirable.
> > >
> > > /js
> > >
> > > > > The intent in the document is to place as few restrictions on
> > > > > tags as possible to allow for future-proofing and organic growth
> > > > > of use both within and outside of SDOs. For standard tags we
> > > > > trust IANA (and the human behind the process) to make the call
> > > > > on whether a tag is already present. :)
> > > >
> > > > ways
> > > to encode in Unicode visually indistinguishable tags IANA would end
> > > up asking IESG for help.
> > > >
> > > > use. I
> > > > > > When responding, please keep the subject line intact and reply
> > > > > > to all email addresses included in the To and CC lines. (Feel
> > > > > > free to cut this introductory paragraph, however.)
> > > > > >
> > > > > >
> > > > > > Please refer to
> > > > > > --------------------------------------------------------------
> > > > > > ----
> > > > > > ----
> > > > > > DISCUSS:
> > > > > > --------------------------------------------------------------
> > > > > > ----
> > > > > > ----
> > > > > >
> > > > > > example, how would IANA be able to enforce uniqueness of
> > > > > > Unicode tags written in different Unicode canonicalisation
> forms?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >

---

From: Christian Hopps <[email-masked]>
Date: Mon, 17 Feb 2020 10:40:30 -0500

> On Feb 17, 2020, at 9:07 AM, Rob Wilton (rwilton) <[email-masked]> wrote:
> 
> Hi Juergen,
> 
> Please see inline ...
> 

I will not put the additional text in the typedef and instead put it in the guidance for the IANA registry then:

 This registry allocates tags that have the registered prefix
 "ietf:". New values should be well considered and not achievable
-through a combination of already existing IETF tags.
+through a combination of already existing IETF tags. For comparing
+non-ascii strings, 'NFC' [[RFC5198]] normalization SHOULD be used.

Unless there are further objections, I believe this, and a small change to the security section suggested by Benjamin K, will clear the remaining DISCUSS so I will republish soon.

Thanks,
Chris.

---

From: Randy Presuhn <[email-masked]>
Date: Mon, 17 Feb 2020 08:51:11 -0800

Hi -

On 2/17/2020 3:15 AM, Christian Hopps wrote:
...
 > BTW, I did look at the "SHOULD be avoided" (occurs twice that I saw) 
once dealing with LFs and CRs which lucky for us is not part of a tags 
allowable characters.

There are lots of other things that complicate life.  The Yang string 
definition
circumscribes some of them, but not all.

>           pattern '[\S ]+';

This pattern doesn't make sense to me when I try to understand it using
https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-classes
It excludes "symbols", but permits, for example, paragraph separators and
formatting characters and such delights as zero-width non-joiner. Also, in
complementing the "all symbols" category, it seems to me it already permits
space, so I don't see why it calls out space again.

Randy

---

From: Schönwälder, Jürgen <[email-masked]>
Date: Mon, 17 Feb 2020 17:09:53 +0000

> Unless there are further objections, I believe this, and a small change to the security section suggested by Benjamin K, will clear the remaining DISCUSS so I will republish soon.

Perhaps this wording is clearer about who is responsible for normalization:

-through a combination of already existing IETF tags.
+through a combination of already existing IETF tags. IANA assigned
+tags must conform to Net-Unicode as defined in RFC 5198 and they shall
+not need normalization.

And then lets hope that others assigning tags follow this advice. ;-)

/js

---

From: Christian Hopps <[email-masked]>
Date: Mon, 17 Feb 2020 14:47:37 -0500

> On Feb 17, 2020, at 11:51 AM, Randy Presuhn <[email-masked]> wrote:
> 
> Hi -
> 
> There are lots of other things that complicate life.  The Yang string definition

The intent was to have the pattern match the description immediately below it:

"A tag value is composed of a standard prefix followed by any type 'string' value that does not include carriage return, newline or tab characters."

Does this pattern fail in doing that?

If this requires anymore work to get right then I think we should drop the pattern, as this isn't the document to come up with a better way to deal with the apparent ugliness of UTF strings in YANG. The overriding intent is to leave it to the users to decide what they want to put in there.

Thanks,
Chris.

---

From: Christian Hopps <[email-masked]>
Date: Mon, 17 Feb 2020 14:49:50 -0500

> On Feb 17, 2020, at 12:09 PM, Schönwälder, Jürgen <[email-masked]> wrote:
> 
> IANA assigned

Seems good to me. Changed locally.

Thanks,
Chris.

---

From: Per Hedeland <[email-masked]>
Date: Mon, 17 Feb 2020 20:57:21 +0100

>> space, so I don't see why it calls out space again.

\S is not the *complement* of the "all symbols" category, it *is* the
"all symbols" category - a.k.a. the complement of \s ([#x20\t\n\r]).

 > Does this pattern fail in doing that?

No, as far as I can see that is exactly what it does.

--Per

 > If this requires anymore work to get right then I think we should drop the pattern, as this isn't the document to come up with a better way to deal with the apparent ugliness of UTF strings in 
YANG. The overriding intent is to leave it to the users to decide what they want to put in there.

---

From: Randy Presuhn <[email-masked]>
Date: Mon, 17 Feb 2020 13:42:35 -0800

Hi -

On 2/17/2020 11:47 AM, Christian Hopps wrote:
>> On Feb 17, 2020, at 11:51 AM, Randy Presuhn 
>> <[email-masked]> wrote: Hi - On 2/17/2020 3:15 AM, 
>> Christian Hopps wrote: ...
>>> once dealing with LFs and CRs which lucky for us is not part of a 
>>> tags allowable characters. 
>> definition circumscribes some of them, but not all.
>>> " typedef tag { type string { length "1..max"; pattern '[\S ]+'; } " 
>> It excludes "symbols", but permits, for example, paragraph separators 
>> and formatting characters and such delights as zero-width non-joiner. 
>> Also, in complementing the "all symbols" category, it seems to me it 
>> already permits space, so I don't see why it calls out space again. 
> The intent was to have the pattern match the description immediately 
> below it: "A tag value is composed of a standard prefix followed by 
> any type 'string' value that does not include carriage return, newline 
> or tab characters." Does this pattern fail in doing that?

Yes, what it accomplishes does not match the stated intent.
I suspect you may have intended something like '[\Z ]+'
See https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-classes

Randy

---

From: Christian Hopps <[email-masked]>
Date: Mon, 17 Feb 2020 17:14:29 -0500

> On Feb 17, 2020, at 4:42 PM, Randy Presuhn <[email-masked]> wrote:
> 
> Hi -
> 
>>> On Feb 17, 2020, at 11:51 AM, Randy Presuhn <[email-masked]> wrote: Hi - On 2/17/2020 3:15 AM, Christian Hopps wrote: ...
>>> There are lots of other things that complicate life. The Yang string definition circumscribes some of them, but not all.
>>> This pattern doesn't make sense to me when I try to understand it using https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-classes It excludes "symbols", but permits, for example, paragraph separators and formatting characters and such delights as zero-width non-joiner. Also, in complementing the "all symbols" category, it seems to me it already permits space, so I don't see why it calls out space again. 
>> The intent was to have the pattern match the description immediately below it: "A tag value is composed of a standard prefix followed by any type 'string' value that does not include carriage return, newline or tab characters." Does this pattern fail in doing that?
> 

I'm finding this hard to believe looking at the definition of "\S" which is "everything but space, tab, newline and carriage return" and then adding "space". Seems to match the definition unless we quibble over the prefix (which I don't think we are).

> See https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-classes

I don't think that's a valid pattern.

If you are talking about the property categories (where I see 'Z' mentioned as "All separators") then there doesn't appear to be a "lower means include, upper means exclude" relationship. Also it appears that to refer to one of these things the syntax is actually "\P{Z}" or "\p{Z}" not just "Z". So translating maybe that's "[\P{Z} ]"? I see nothing that defines how "catEsc" (\p{}) vs "compEsc" (\P{}) are different, but maybe the upper here means exclude.

I'm more inclined to just ditch any pattern or restriction the more this gets discussed. Let the user do what they want. If they want to include crazy unicode stuff (almost certainly they dont) then I guess that's what they want.

Thanks,
Chris.

---

From: Per Hedeland <[email-masked]>
Date: Mon, 17 Feb 2020 23:31:07 +0100

On 2020-02-17 23:14, Christian Hopps wrote:
 >>>> This pattern doesn't make sense to me when I try to understand it using https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-classes It excludes "symbols", but permits, for example, 
paragraph separators and formatting characters and such delights as zero-width non-joiner. Also, in complementing the "all symbols" category, it seems to me it already permits space, so I don't see 
why it calls out space again.
 >>> The intent was to have the pattern match the description immediately below it: "A tag value is composed of a standard prefix followed by any type 'string' value that does not include carriage 
return, newline or tab characters." Does this pattern fail in doing that?
 > I'm finding this hard to believe looking at the definition of "\S" which is "everything but space, tab, newline and carriage return" and then adding "space". Seems to match the definition unless we 
quibble over the prefix (which I don't think we are).

+1

 > I don't think that's a valid pattern.

+1

 > If you are talking about the property categories (where I see 'Z' mentioned as "All separators") then there doesn't appear to be a "lower means include, upper means exclude" relationship. Also it 
appears that to refer to one of these things the syntax is actually "\P{Z}" or "\p{Z}" not just "Z". So translating maybe that's "[\P{Z} ]"? I see nothing that defines how "catEsc" (\p{}) vs "compEsc" 
(\P{}) are different, but maybe the upper here means exclude.

The description is right above the definitions:

   The set containing all characters that have property X, can be
   identified with a category escape \p{X}. The complement of this set
   is specified with the category escape \P{X}. ([\P{X}] = [^\p{X}])

So yes, \P{Z} would be the complement of "All Separators", while your
original \S is the complement of \s ([#x20\t\n\r]). I.e. \P{Z} would
exclude "more separators", but is hardly worth the trouble I think -
and it is *not* the "stated intent".

 > I'm more inclined to just ditch any pattern or restriction the more this gets discussed. Let the user do what they want. If they want to include crazy unicode stuff (almost certainly they dont) 
then I guess that's what they want.

FWIW, as I already wrote, I think your original pattern is fine (and I
think Randy needs to have a closer look at the section he references).

--Per

---

From: Kent Watsen <[email-masked]>
Date: Mon, 17 Feb 2020 22:36:30 +0000

FWIUW, I created a YANG-Next issue to track the needed long-term update to RFC 7950:

https://github.com/netmod-wg/yang-next/issues/102 <https://github.com/netmod-wg/yang-next/issues/102>

Kent

> On Feb 17, 2020, at 5:31 PM, Per Hedeland <[email-masked]> wrote:
>

---

From: Randy Presuhn <[email-masked]>
Date: Mon, 17 Feb 2020 16:31:33 -0800

Hi -

On 2/17/2020 2:14 PM, Christian Hopps wrote:
...
> If you are talking about the property categories (where I see 'Z' 
> mentioned as "All separators") then there doesn't appear to be a 
> "lower means include, upper means exclude" relationship. Also it 
> appears that to refer to one of these things the syntax is actually 
> "\P{Z}" or "\p{Z}" not just "Z". So translating maybe that's "[\P{Z} 
> ]"? I see nothing that defines how "catEsc" (\p{}) vs "compEsc" (\P{}) 
> are different, but maybe the upper here means exclude.

My error.  You're right that "[\P{Z} ]" is what the grammar would require
to eliminate all the space-like things while still allowing #x20 space.
The problem with "\S" is that the only kind of space it eliminates is #x20,
(which you then add back in in the regular expression) while Unicode has
a whole menagerie of other space-like creatures one might well want to
exclude from tags while still permitting #x20, as well as lots of other 
stuff
that is probably a bad idea to permit in tags.  That said...

> I'm more inclined to just ditch any pattern or restriction the more 
> this gets discussed. Let the user do what they want. If they want to 
> include crazy unicode stuff (almost certainly they dont) then I guess 
> that's what they want.

Yeah, doing so is almost a design pattern in Yang:  try to constrain 
things with
regular expressions, find out the expression is either too limiting or 
allows too
many invalid strings, then give up.  :-)  No judgement here - it is what 
it is, and
"done" has to take precedence over "perfect."

Randy

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Tue, 18 Feb 2020 11:12:47 +0000

I agree with Per.

I think that the original pattern (with the \S) is fine and captures the intent clearly and simply.

I'm also not convinced of how many implementations would properly honour the XML regex property categories, e.g., if the pattern statement has been translated to another regex engine.

Thanks,
Rob

> From: netmod <[email-masked]> On Behalf Of Per Hedeland
> Sent: 17 February 2020 22:31
> To: Christian Hopps <[email-masked]>; Randy Presuhn
> <[email-masked]>
> Cc: [email-masked]
>  >> On Feb 17, 2020, at 4:42 PM, Randy Presuhn
> <[email-masked]> wrote:
>  >>
>  >> Hi -
>  >>
>  >>>>> BTW, I did look at the "SHOULD be avoided" (occurs twice that I
> saw) once dealing with LFs and CRs which lucky for us is not part of a
>  >>>> This pattern doesn't make sense to me when I try to understand it
> using https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#charcter-
> classes It excludes "symbols", but permits, for example, paragraph
> separators and formatting characters and such delights as zero-width non-
> joiner. Also, in complementing the "all symbols" category, it seems to me
> it already permits space, so I don't see why it calls out space again.
> below it: "A tag value is composed of a standard prefix followed by any
> type 'string' value that does not include carriage return, newline or tab
> characters." Does this pattern fail in doing that?
>  >>
>  > I'm finding this hard to believe looking at the definition of "\S"
> which is "everything but space, tab, newline and carriage return" and then
> adding "space". Seems to match the definition unless we quibble over the
> prefix (which I don't think we are).
> 
> +1
> 
> mentioned as "All separators") then there doesn't appear to be a "lower
> means include, upper means exclude" relationship. Also it appears that to
> refer to one of these things the syntax is actually "\P{Z}" or "\p{Z}" not
> just "Z". So translating maybe that's "[\P{Z} ]"? I see nothing that
> defines how "catEsc" (\p{}) vs "compEsc"
> exclude "more separators", but is hardly worth the trouble I think - and
> it is *not* the "stated intent".
>

---

From: Schönwälder, Jürgen <[email-masked]>
Date: Tue, 18 Feb 2020 11:19:57 +0000

> I'm also not convinced of how many implementations would properly honour the XML regex property categories, e.g., if the pattern statement has been translated to another regex engine.

That's a weird argument. If there is agreement on a restriction and we
can express that restriction in a pattern, we should IMHO do so.

Note the order: It is backwards if there is first a pattern and as a
consequence we agree on that specific restriction.

Ideally, the agreed upon restriction is stated in the description and
if possible also expressed as a pattern. This way it does not matter
how implementations enforce the restriction and it is clear to
everyone what the pattern is trying to achieve.

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Tue, 18 Feb 2020 18:32:25 +0000

Hi Juergen,

> Sent: 18 February 2020 11:20
> > I'm also not convinced of how many implementations would properly honour
> the XML regex property categories, e.g., if the pattern statement has been
> translated to another regex engine.
> >
> 
> That's a weird argument. If there is agreement on a restriction and we can
> express that restriction in a pattern, we should IMHO do so.

Not really.

I believe that ultimately the configuration is the responsibility of the client.  If there are reasonable steps that a server can take to help police that configuration and reject obvious errors then that is obviously helpful, but it most cases a server cannot check that the value provided by the client is correct, which I suspect is the far more likely error (e.g. a client configures the wrong IP address rather than a malformed IP address).

So, in the case of regexes, I still prefer a shorter, easier to read/understand, and likely more performant pattern statement instead of a longer, complex, and harder to verify pattern statement.

In particular, I note that unicode properties do not seem to be widely used in YANG pattern statements (e.g. perhaps only 10 unique pattern instances in all YANG modules on github YANG repo, some of which look plausibly wrong to me (unless they intended to include ASCII control characters), and hence I question whether it is worth using them, but maybe I am being biased towards languages using the Latin alphabet.

Further, I was concerned that there didn't seem to be widespread support for unicode properties within standard regex engines, but it looks like this may have changed, and perhaps support for them is more widespread now, even if the exact specification of what character properties are defined and what characters are included in those properties seem to able to vary over time depending on the unicode version, and presumably between implementations.

In the case of module tags, I'm not sure what would go wrong if a space was included in the tag name.  I suspect nothing, just that it could be confusing to a user when displayed.

So, I'm still think that pragmatically the existing regex is fine.  But if the consensus is to use \P{Z} or better \P{Separator} then I would also be okay with that.

> Ideally, the agreed upon restriction is stated in the description and if
> possible also expressed as a pattern. This way it does not matter how
> implementations enforce the restriction and it is clear to everyone what
> the pattern is trying to achieve.

Yes. I agree to both these points.

Thanks,
Rob