Subject: [netmod] regular expression flavours (again)

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 12 Jun 2019 10:01:30 +0200

Hi,

I think it is very unfortunate that OpenConfig folks use POSIX regular
expressions instead of XSD. This of course causes confusion, see e.g. this PR:

https://github.com/CZ-NIC/yangson/pull/22

Given the popularity of OpenConfig models, it seems to me that it would be
better if YANG somehow accomodates such (largely cosmetic) differences.

One option would be that each YANG module declares the regex flavour used in its
patterns. Yes, it is a complication, but having separate forks of YANG is much
worse.

ISO Schematron adopted a similar approach with "query language binding" that can
be declared with the schema element, for example

<schema xmlns="http://purl.oclc.org/dsdl/schematron" queryBinding="exslt">

Any thoughts on this?

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 12 Jun 2019 09:25:38 +0000

Hi Lada,

OpenConfig have already introduced a module level extension statement to indicate that their pattern statements use POSIX regular expressions.

  extension regexp-posix {
     description
      "This extension indicates that the regular expressions included
      within the YANG module specified are conformant with the POSIX
      regular expression format rather than the W3C standard that is
      specified by RFC6020 and RFC7950.";
  }

Defined in https://github.com/openconfig/public/blob/master/release/models/openconfig-extensions.yang

I don't think that this should become part of the YANG standard.

Thanks,
Rob

> -----Original Message-----
> From: netmod <[email-masked]> On Behalf Of Ladislav Lhotka
> Sent: 12 June 2019 09:02
> To: NETMOD WG <[email-masked]>
> Subject: [netmod] regular expression flavours (again)
> 
> Hi,
> 
> One option would be that each YANG module declares the regex flavour used
> in its patterns. Yes, it is a complication, but having separate forks of
> YANG is much worse.
> 
> ISO Schematron adopted a similar approach with "query language binding"
> that can be declared with the schema element, for example
> 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 12 Jun 2019 11:25:56 +0200

> better if YANG somehow accomodates such (largely cosmetic) differences.

The differences are not cosmetic. See previous discussions on this topic.

> patterns. Yes, it is a complication, but having separate forks of YANG is much

It is sad that OpenConfig does not follow the standard. If OpenConfig
can't live without POSIX regular expressions, then OpenConfig should
have used proper extension statements instead of simply changing the
semantics of the YANG pattern statement to measn something
different. That said, they do seem to declare something like
oc-ext:regexp-posix; but it would have been much smarter to use for
example oc-posix:regex instead of changing the semantics of the
pattern statement.

I would rather see OpenConfig adopting the standard or fixing their
POSIX regular expression solution so that it avoids changing the
semantics of YANG statements. Having statements mean different things
depending on some context is pretty bad design.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 12 Jun 2019 11:39:37 +0200

Hi Rob,

On Wed, 2019-06-12 at 09:25 +0000, Rob Wilton (rwilton) wrote:
> Hi Lada,
> 
> OpenConfig have already introduced a module level extension statement to
> indicate that their pattern statements use POSIX regular expressions.

What I meant was rather something like

    pattern-language "POSIX";

with the default being "XSD". It could possibly also include your proposal of
XSD regex subset (issue #21).

This could in fact be a good candidate for a critical extension.  

> Defined in 
> https://github.com/openconfig/public/blob/master/release/models/openconfig-extensions.yang
> 

I suspect that developers of YANG tools are being pushed by users to include
support for this anyway, so I don't see much benefit from blocking it.

Lada

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 12 Jun 2019 09:53:01 +0000

Hi Lada,

> From: Ladislav Lhotka <[email-masked]>
> Sent: 12 June 2019 10:40
> To: Rob Wilton (rwilton) <[email-masked]>; NETMOD WG <[email-masked]>
> Subject: Re: [netmod] regular expression flavours (again)
> 
> Hi Rob,
> 
> > OpenConfig have already introduced a module level extension statement
> > to indicate that their pattern statements use POSIX regular expressions.
> 

But depending on how many pattern languages are supported, wouldn’t that make implementing YANG a lot harder, since implementations would need to support all the allowed pattern languages?

Also, IIRC, POSIX regex doesn't support unicode, which means that it isn't necessarily a good choice for a language that does support unicode strings by default.

> with the default being "XSD". It could possibly also include your proposal
> of XSD regex subset (issue #21).

Really my aim was the other way around.  I.e. have all YANG pattern statements in just one simple regex language that could trivially be converted to run on any common regex engine.

> I suspect that developers of YANG tools are being pushed by users to
> include support for this anyway, so I don't see much benefit from blocking
> it.

I don't see this as blocking it at all, but rather that a pragmatic common solution already exists without requiring any standardization work.

Thanks,
Rob

> > > I think it is very unfortunate that OpenConfig folks use POSIX
> > > regular expressions instead of XSD. This of course causes confusion,
> > > see e.g. this
> > > PR:
> > >
> > > Given the popularity of OpenConfig models, it seems to me that it
> > > would be better if YANG somehow accomodates such (largely cosmetic)
> differences.
> > >
> > > One option would be that each YANG module declares the regex flavour
> > > used in its patterns. Yes, it is a complication, but having separate
> > > forks of YANG is much worse.
> > >

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 12 Jun 2019 13:01:35 +0200

On Wed, 2019-06-12 at 09:53 +0000, Rob Wilton (rwilton) wrote:
> But depending on how many pattern languages are supported, wouldn’t that make
> implementing YANG a lot harder, since implementations would need to support
> all the allowed pattern languages?

Not necessarily, implementors should be able decide which flavours to support.
So OpenConfig would be a niche that uses POSIX, and they have to use tools that
support it.

The real problem here is that they use an extension for changing the semantics
of a built-in YANG statement, which is IMO not permitted by RFC 7950 and causes
implementations to break without warning, or even opens security holes. If it
was a critical extension, and I didn't want to implement the POSIX flavour, my
implementation should be able to notice it and say "Sorry, I can't work with
this module".

> Also, IIRC, POSIX regex doesn't support unicode, which means that it isn't
> necessarily a good choice for a language that does support unicode strings by
> default.

Yes, this has also been discussed previously. I am not advocating POSIX in
favour of XSD, I just want to guarantee that YANG tools can be made so as to
behave predictably and safely. We cannot expect YANG BFUs to understand the dog-
fighting between NETMOD and OpenConfig.

> Really my aim was the other way around.  I.e. have all YANG pattern statements
> in just one simple regex language that could trivially be converted to run on
> any common regex engine.

I understand this, but I think the full XSD pattern language has to remain an
option.

Lada

> I don't see this as blocking it at all, but rather that a pragmatic common
> solution already exists without requiring any standardization work.
> 
> Thanks,
> Rob
> 
> 
> > Lada
> > 
> > > Thanks,
> > > Rob
> > > 
> > >

---

From: Robert Varga <[email-masked]>
Date: Thu, 13 Jun 2019 15:22:13 +0200

>>>     pattern-language "POSIX";

Yeah, that is what I proposed here:
https://github.com/openconfig/public/issues/44#issuecomment-416320334

> So OpenConfig would be a niche that uses POSIX, and they have to use tools that

Well, some of the languages need to be mandatory-to-implement otherwise
things like unions stop working consistently.

> implementation should be able to notice it and say "Sorry, I can't work with

I could argue that tailf:action introduced such an extension, as users
expected augment to work with it -- except a parser cannot assume an
extension it knows nothing about actually defines a thing in the schema
tree. In my mind that constitutes prior art of getting things work in
the field before standardizing.

It makes little difference to me as an implementer whether the users
scream at me for the patterns not working as expected (we've had
equivalents of the issue you referenced pop up for years now) or they
scream because the tooling rejects to work with the models.

In either case the source is whoever wrote the models and it is the
expectation that tools work Just Work (tm).

Regards,
Robert

---

From: Robert Varga <[email-masked]>
Date: Thu, 13 Jun 2019 15:31:49 +0200

On 12/06/2019 11:25, Juergen Schoenwaelder wrote:
> That said, they do seem to declare something like

Yes, this would be preferable, but it has a downside: now you need to
carry the patterns twice and must make sure they are in sync. If they
are not, all sorts of implementation-dependent breakage would ensue.

The differences are far from minor where specification is concerned, but
they certainly minor in the vast majoritiy of the use cases and most
users do not even understand REs enough to know they are dealing with a
maze of twisty flavors, all alike.

Do we have an easy way of understanding the corpus of pattern arguments
being 'out there' in real use?

Can we engineer a workable solution for the general case without getting
everyone know the differences in RE engines? Something along the lines
of this:
https://github.com/openconfig/public/issues/44#issuecomment-501629497
perhaps?

Thanks,
Robert

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 13 Jun 2019 15:44:19 +0200 (CEST)

Robert Varga <[email-masked]> wrote:
> carry the patterns twice and must make sure they are in sync.

In the OC models they would/could just use the oc-posix:regex statement, and
not even use the pattern statement.

The current solution doesn't really work, since it requires tools to
understand the "regexp-posix" extension.

And yes, I'm the first to admit that tailf:action had the same kind of
problem.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 13 Jun 2019 16:06:55 +0200

> are not, all sorts of implementation-dependent breakage would ensue.

Authors could simply use the YANG pattern statement as it is defined
and there is zero extra cost.

> https://github.com/openconfig/public/issues/44#issuecomment-501629497

Please lets not (even with the best intentions) create yet another
regular expression flavour.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 14 Jun 2019 09:58:30 +0200

> The differences are not cosmetic. See previous discussions on this topic.

What I meant is that such differences are still relatively easy to handle,
unlike e.g. the schema structure that is incompatible with the current
conventions and NMDA.

> depending on some context is pretty bad design.

I agree with all this but, apparently, OpenConfig people don't care too much.
And since they are much more agile than the IETF, our views may soon become
irrelevant.

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 14 Jun 2019 11:12:25 +0200

> And since they are much more agile than the IETF, our views may soon become

If this becomes our attitude, we already are irrelevant.

I have no idea why someone would go for Posix REs of all things, but if OpenConfig wants to do that, more power to them.  However, YANG has well-defined extension points, and arbitrarily changing the semantics of the pattern statement is not one of them.  So we have to make sure they know that this is not tolerable, and probably show them the way on a good way to make use of the existing extension points.

For CDDL, we essentially have the same problem (nobody likes W3C XSD, not even their REs, but there were good reasons to choose them, not the least that YANG also chose them).  CDDL also has an extension point that is useful here, and RFC 8610 Section 3.8.3.2 was added specifically to guide CDDL users to use the right extension point once the need comes up:

https://tools.ietf.org/html/rfc8610#section-3.8.3.2

I would expect the netmod community to write a similar section, in a separate I-D of its own if need be, and obtain IETF consensus on it.  Make the OpenConfig people aware of it (now, and when it’s done).  Github issues are a good way to do the latter.

Grüße, Carsten

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 14 Jun 2019 09:29:39 +0000

> From: netmod <[email-masked]> On Behalf Of Juergen Schoenwaelder
> Sent: 13 June 2019 15:07
> To: Robert Varga <[email-masked]>
> Cc: NETMOD WG <[email-masked]>
> > Can we engineer a workable solution for the general case without
> > getting everyone know the differences in RE engines? Something along
> > the lines of this:
> Please lets not (even with the best intentions) create yet another regular
> expression flavour.

Sorry, but I still think that is exactly what we should do. 😊

What we have today is seemingly not working for the industry:

Some implementations might link against the libxml2 regex engine.  Great, they are compliant with IETF YANG models, but not OpenConfig.

Perhaps there are some other implementations that have written their own XML regex engine for the language of their choice.  I suspect that there are few in this camp!

For everyone else, I think that they do what OpenConfig is trying to do, and they just throw the pattern statements into their chosen language's default regex engine.  And the thing is, assuming the beginning/end anchors are handled appropriately, I suspect that this works for 99% of the pattern statements in YANG models.

OpenConfig state that they are using POSIX regex, with the expectation that these regular expressions can just be interpreted by most languages.  Unfortunately, that isn't quite true either, seemingly most languages are derived from PCRE, and there is a minor difference on how '\' is handled within a character group.  There also seems to be a difference of first vs longest match for alternation, but I'm not convinced that matters for pattern statements.  Finally, POSIX ERE regex is only defined by ASCII, and doesn't cover Unicode, so isn't a great choice for YANG.

Personally, I think that standardizing a basic regex language that is a common subset of what the common languages implement would probably be beneficial in the industry in general.

I'm sure that someone can post an XKCD of why this is a bad idea 😉

Thanks,
Rob

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 14 Jun 2019 11:56:35 +0200

> I'm sure that someone can post an XKCD of why this is a bad idea 😉

Yeah, going ahead and standardizing another regex dialect that is subtly incompatible with everything else is exactly what we need.

We could even make sure that dialect is actually useful in a pattern statement (e.g., by making it self-anchoring).

But wait, somebody has already done that work for us!

W3C did, when they designed their XSD-types…
So we are done already!

Now the main deployability problem with W3C XSD regexes is that they added some functionality that is sorely missing in other dialects, such as character class subtraction, so it is more than an hour of work to write a converter from XSD regexes to you favorite flavor.  Maybe we should encourage some open source software in this space…

Grüße, Carsten

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 14 Jun 2019 10:08:36 +0000

> From: Carsten Bormann <[email-masked]>
> Sent: 14 June 2019 10:57
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Juergen Schoenwaelder <[email-masked]>; Robert
> Varga <[email-masked]>; NETMOD WG <[email-masked]>
> Yeah, going ahead and standardizing another regex dialect that is subtly
> incompatible with everything else is exactly what we need.

This is not what I am suggesting.

We don't need a standard regex dialect that is subtly incompatible with all the normal regex implementations because that is exactly what W3C produced.

What I am suggesting is standardizing a regex dialect that is well defined and widely compatible with normal regex engines.  I.e. a standardized subset of PCRE.

> 
> We could even make sure that dialect is actually useful in a pattern
> statement (e.g., by making it self-anchoring).

I don't think that adding/removing the anchor characters is really the issue, since they are trivial to add/remove as required.

> W3C did, when they designed their XSD-types… So we are done already!

I disagree, please see above.

> 
> Now the main deployability problem with W3C XSD regexes is that they added
> some functionality that is sorely missing in other dialects, such as
> character class subtraction, so it is more than an hour of work to write a
> converter from XSD regexes to you favorite flavor.  Maybe we should
> encourage some open source software in this space…

Or perhaps we could define a regex language that worked with normal implementations without requiring any conversion.

Thanks,
Rob

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 14 Jun 2019 12:45:28 +0200

> Or perhaps we could define a regex language that worked with normal implementations without requiring any conversion.

Unlikely, as it is easy to write an regex that inadvertently triggers some “feature" in a specific flavor.  So you would need to list all the current and future idiosyncrasies of all flavors and outlaw them.  Worse, what you have outlawed may be *the* valid way to represent something in some other dialect, so the specifier needs to jump through hoops to work around that or simply cannot write down the regex needed.  Also, all those features that are subtly different between flavors (starting with ., \s, …) can’t be used.

The approach probably works for [A-Fa-f0-9]+, but becomes icky for anything more complicated quickly.  (And, actually, XSD regexes are very close to what you would come up with, anyway, except for “features” like character class subtraction or block escapes.)

Oh, and defining “normal implementations” is left as an exercise to the reader :-)

What we could do (and that would be quite useful, I think) is *document* the subset of XSD regexes that actually has the same meaning in PCRE2, Java8, JavaScript, .NET and a few more real-world regex languages after adding the necessary anchors in those languages.

Grüße, Carsten

---

From: Robert Varga <[email-masked]>
Date: Fri, 14 Jun 2019 15:04:51 +0200

> Now the main deployability problem with W3C XSD regexes is that they added some functionality that is sorely missing in other dialects, such as character class subtraction, so it is more than an ho…

Having gone through the path of:
- just throw it in java.util.regex.Pattern, with ^$ wrapped
- oh yeah, that does not work for complex patterns
- oh wait, character class references are different
- oh wait, character groups are different
...

I decided to take this exact approach here:
https://git.opendaylight.org/gerrit/73875 . The idea is to contribute
the ANTLR grammar to https://github.com/antlr/grammars-v4/ so that
anybody working with a language supported by ANTLR can get some head start.

This work low priority for me, though, as the current set of hacks work
well enough and a complete solution requires Java 9 anyway -- so anyone
is free to pick it up...

Regards,
Robert