Subject: [netmod] string in when and must statement in YANG ABNF Grammar

From: "Bogaert, Bart (Nokia - BE)" <[email-masked]>
Date: Tue, 29 Mar 2016 07:09:52 +0000

Is there a reason why the ABNF for example the when and must statement has
been "restricted" to when-keyword sep string optsep 

As "string"  can't be tokenized no errors are generated by YANG compilers if
this string contains an error, e.g. referred leaf does not exit due to a
typo.  This problem only exposes itself at run-time.  I was wondering why
this string was not broken down into a number of specific parts that define
the when statement so that these kind of errors can be trapped early in the
development process?   I am rather new in this so I did not follow all
discussions that led to the definition of YANG and hence have no idea
whether this was discussed and what were the reasons not to do it this way.

Best regards - Vriendelijke groeten,

Bart Bogaert

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 29 Mar 2016 09:22:53 +0200 (CEST)

> whether this was discussed and what were the reasons not to do it this way.

The arguments to the "when" and "must" statements are XPath 1.0
expressions.  The syntax of XPath 1.0 is not defined by YANG, but by
the XPath 1.0 spec.  This is the reason that the YANG grammar isn't
more specific.

YANG compilers differ in their ability to detect errors in these XPath
expressions.  Some perform more checks than others (unfortunately,
pyang is pretty bad in this regard... (patches are always welcome:)).

[For your particular example, referencing a leaf that doesn't exist is
not an error per se; it is perfectly valid XPath.  But it probably
warrants a warning by the compiler.]

/martin

---

From: "Bogaert, Bart (Nokia - BE)" <[email-masked]>
Date: Tue, 29 Mar 2016 07:56:33 +0000

Martin,

Thanks for this feedback.  A run-time crash of the NETCONF server is a bad
"outcome" of a typo like this...
I checked the W3C pages and there is grammar defined for an XPATH expression
so ultimately the string could be parsed as an XPATH (I guess it is
currently not done).

Any other programming language ensures to eliminate typing errors during the
compilation phase allowing early detection of issues like this.  It is
becoming a problem when the NETCONF server internally stores values (e.g.
identityref) in a format different from what is specified in the string of
the when statement resulting in unexpected options when trying to use the
CLI generated from the model.  In this specific case we were advised to
prefix the leaf (as the leaf was stored in this way...) even though the leaf
was in the local module itself.  This makes development even less
transparent...  When we flagged this we received the following feedback:

Quote:
For YANG 1.1 that is underway it will be defined in the following way:
" The string value of a node of type identityref in a "must" or "when" XPath

expression is the referred identity's qualified name with the prefix
present.  
If the referred identity is defined in an imported module, the prefix in the

string value is the prefix defined in the corresponding "import" statement.

Otherwise, the prefix in the string value is the prefix for the current 
module.""

But as this is within a string that is not parsed (depending on the
compiler) this is something that people need to do and if not done will
still result in run-time problems.

Best regards - Vriendelijke groeten,
Bart Bogaert
System Architect Data-Centric SW Architectures 
Fixed Networks - Broadband Access BU,  Nokia
Contact number +32 3 2408310 (+32 477 673952)

-----Original Message-----
From: EXT Martin Bjorklund [mailto:[email-masked]] 
Sent: 29 March 2016 09:23
To: Bogaert, Bart (Nokia - BE)
Cc: [email-masked]
Subject: Re: [netmod] string in when and must statement in YANG ABNF Grammar

"Bogaert, Bart (Nokia - BE)" <[email-masked]> wrote:
> Is there a reason why the ABNF for example the when and must statement 
> has been "restricted" to when-keyword sep string optsep
> 
> As "string"  can't be tokenized no errors are generated by YANG 
> compilers if this string contains an error, e.g. referred leaf does 
> not exit due to a typo.  This problem only exposes itself at run-time.  
> I was wondering why this string was not broken down into a number of 
> specific parts that define the when statement so that these kind of errors
can be trapped early in the
> whether this was discussed and what were the reasons not to do it this
way.

The arguments to the "when" and "must" statements are XPath 1.0 expressions.
The syntax of XPath 1.0 is not defined by YANG, but by the XPath 1.0 spec.
This is the reason that the YANG grammar isn't more specific.

YANG compilers differ in their ability to detect errors in these XPath
expressions.  Some perform more checks than others (unfortunately, pyang is
pretty bad in this regard... (patches are always welcome:)).

[For your particular example, referencing a leaf that doesn't exist is not
an error per se; it is perfectly valid XPath.  But it probably warrants a
warning by the compiler.]

/martin

---

From: William Ivory <[email-masked]>
Date: Tue, 29 Mar 2016 08:12:35 +0000

Hi Bart,

Maybe you need to use a different NETCONF server if your current one is crashing with badly formed XPATH expressions?  

Our YANG compiler validates XPATH expressions when we load in YANG files, and anything that slips through should certainly not cause a crash at runtime (unless there's a bug of course).  I had assumed others would do likewise - as far as I'm concerned XPATH is an 'extension' of YANG and we validate it with the same level of scrutiny as the YANG itself.

So, I don't think this is anything to do with the 'programming language' - it's all about individual implementations of the YANG / XPATH compiler.

I don't quite follow your point about prefixing a leaf - if you don't explicitly prefix a leaf in an XPATH statement then there are clear (well, after the third or fourth reading!) rules that specify what prefix will be used, depending on whether the must statement is inside a grouping or an augment statement, or in 'plain' YANG.  Is your NETCONF server doing something unexpected?

Regards,

William

-----Original Message-----
From: netmod [mailto:[email-masked]] On Behalf Of Bogaert, Bart (Nokia - BE)
Sent: 29 March 2016 08:57
To: EXT Martin Bjorklund <[email-masked]>
Cc: [email-masked]
Subject: Re: [netmod] string in when and must statement in YANG ABNF Grammar

Martin,

Thanks for this feedback.  A run-time crash of the NETCONF server is a bad "outcome" of a typo like this...
I checked the W3C pages and there is grammar defined for an XPATH expression so ultimately the string could be parsed as an XPATH (I guess it is currently not done).

Any other programming language ensures to eliminate typing errors during the compilation phase allowing early detection of issues like this.  It is becoming a problem when the NETCONF server internally stores values (e.g.
identityref) in a format different from what is specified in the string of the when statement resulting in unexpected options when trying to use the CLI generated from the model.  In this specific case we were advised to prefix the leaf (as the leaf was stored in this way...) even though the leaf was in the local module itself.  This makes development even less transparent...  When we flagged this we received the following feedback:

Quote:

expression is the referred identity's qualified name with the prefix present.  
If the referred identity is defined in an imported module, the prefix in the

Otherwise, the prefix in the string value is the prefix for the current module.""

But as this is within a string that is not parsed (depending on the
compiler) this is something that people need to do and if not done will still result in run-time problems.

Best regards - Vriendelijke groeten,
Bart Bogaert
System Architect Data-Centric SW Architectures Fixed Networks - Broadband Access BU,  Nokia Contact number +32 3 2408310 (+32 477 673952)

-----Original Message-----
From: EXT Martin Bjorklund [mailto:[email-masked]]
Sent: 29 March 2016 09:23
To: Bogaert, Bart (Nokia - BE)
Cc: [email-masked]
Subject: Re: [netmod] string in when and must statement in YANG ABNF Grammar

YANG compilers differ in their ability to detect errors in these XPath expressions.  Some perform more checks than others (unfortunately, pyang is pretty bad in this regard... (patches are always welcome:)).

[For your particular example, referencing a leaf that doesn't exist is not an error per se; it is perfectly valid XPath.  But it probably warrants a warning by the compiler.]

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 29 Mar 2016 10:15:18 +0200 (CEST)

Hi,

> "outcome" of a typo like this...

Sure; but that's a tool issue, not a specification issue.

> so ultimately the string could be parsed as an XPATH (I guess it is

It all depends on the tool you are using.

> still result in run-time problems.

Right; for this particular example, it is better to use the new (YANG
1.1) functions derived-from() and derived-from-or-self().

/martin

---

From: William Lupton <[email-masked]>
Date: Tue, 29 Mar 2016 09:56:10 +0100

It might also be worth noting (see https://github.com/mbj4668/pyang/wiki/InstanceValidation) that "DSDL schemas can be used with generic off-the-shelf XML tools for both syntactic and semantic validation of XML instance documents”. This includes validating “must” statements and so on. William.

> On 29 Mar 2016, at 08:22, Martin Bjorklund <[email-masked]> wrote:
> 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 30 Mar 2016 11:07:53 +0200

William Lupton <[email-masked]> writes:

> It might also be worth noting (see
> https://github.com/mbj4668/pyang/wiki/InstanceValidation) that "DSDL
> schemas can be used with generic off-the-shelf XML tools for both
> syntactic and semantic validation of XML instance documents”. This
> includes validating “must” statements and so on. William.

Right, although XPath expressions as they appear in YANG modules need to
be massaged somewhat before they can be used in Schematron -
e.g. explicit prefixes need to be added where they are missing.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C