Subject: [netmod] Regular expression implementation issue

From: Jernej Tuljak <[email-masked]>
Date: Mon, 30 Mar 2015 16:12:24 +0200

Hi,

we've been advised that our implementation refuses some regular 
expressions in pattern-stmt args, while pyang allows them. For example:

pattern '[[]]'+;

The Java implementation of XSD 1.0 specs (JAXP1.4, Java1.7), which we 
rely on, refuses the above with: "syntax error in pattern: '[' is 
invalid in a character class. Write '\['."

Following the normative reference for XSD-TYPES in RFC6020, you 
eventually end up in 
http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#nt-charRange, which 
specifies the following:

/A single XML character is a ·character range· that identifies the set 
of characters containing only itself. All XML characters are valid 
character ranges, except as follows://
//
//    The [, ], - and \ characters are not valid character ranges;//
//    The ^ character is only valid at the beginning of a ·positive 
character group· if it is part of a ·negative character group·//
//    The - character is a valid character range only at the beginning 
or end of a ·positive character group·. //
//
/This seems consistent with what the error reports.

I'd like to determine whether we need to correct our implementation.

Note that newer XSD specifications change text for regular expressions, 
so one should be very careful when choosing a re engine for YANG 
validation. I remember having to fix a bug in our code due to usage of 
two different regular expression engine implementations (XSD 1.0 vs. 
1.1), which displayed inconsistent behavior.

This might also be relevant for YANG 1.1 if you are considering updates 
to the normative references section (not sure what IETF practice is in 
this regard). If an update is done, valid regexes might become invalid 
and vice versa (YANG 1.0 vs. YANG 1.1).

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 30 Mar 2015 17:18:01 +0200

> expressions in pattern-stmt args, while pyang allows them. For

Pyang uses a function from the libxml2 module (Python bindings for
libxml2) for checking these patterns. However, this module isn't part of
Python distribution and has to be installed separately. For example, in
Debian/Ubuntu it is the package "python-libxml2".

So, if this module is installed, broken regexp patterns such as the one
below will be rejected. If not, patterns are not checked at all and no
error/warning is issued either, presumably because it caused problems
with other users (Martin?).

Note, however, that the RELAX NG schema generated by the DSDL plugin is
then rejected by both xmlllint and jing validators.

Lada

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Paul Borman <[email-masked]>
Date: Mon, 30 Mar 2015 08:53:00 -0700

Does the standard delegate what is a legal RE to an implementation of RE's
or does the standard specify what an RE is and some implementations are
suitably close?  My reading is that no matter what the implementation of
pyang, or any other tool, the acceptable patterns are defined in
http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs.

No valid YANG should contain any pattern that is not defined by the
aforementioned document, even though some implementations might not flag an
invalid pattern as an error.  A tool used to validate YANG would be
required to reject invalid patterns.

    -Paul

---

From: Jernej Tuljak <[email-masked]>
Date: Mon, 30 Mar 2015 16:17:58 +0200

> expressions in pattern-stmt args, while pyang allows them. For example:

This is a typo, should be:

pattern '[[]]+';

Jernej

> http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#nt-charRange, 
> which specifies the following:
>
> Note that newer XSD specifications change text for regular 
> expressions, so one should be very careful when choosing a re engine 
> for YANG validation. I remember having to fix a bug in our code due to 
> usage of two different regular expression engine implementations (XSD 
> 1.0 vs. 1.1), which displayed inconsistent behavior.
>
> This might also be relevant for YANG 1.1 if you are considering 
> updates to the normative references section (not sure what IETF 
> practice is in this regard). If an update is done, valid regexes might 
> become invalid and vice versa (YANG 1.0 vs. YANG 1.1).
>
> Jernej
>
>
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 30 Mar 2015 20:50:12 +0200

> On 30 Mar 2015, at 17:53, Paul Borman <[email-masked]> wrote:
> 
> Does the standard delegate what is a legal RE to an implementation of RE's or does the standard specify what an RE is and some implementations are suitably close?  My reading is that no matter what the implementation of pyang, or any other tool, the acceptable patterns are defined in http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs.

Yes, that’s right.

> 
> No valid YANG should contain any pattern that is not defined by the aforementioned document, even though some implementations might not flag an invalid pattern as an error.  A tool used to validate YANG would be required to reject invalid patterns.

As I said, just install the libxml2 Python module and all is good. I agree pyang should’t be silent if it is not available, we should change this behaviour.

Lada