Subject: [netmod] YANG 1.1 issue: YIN -> YANG translation breaks text formatting

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 23 Mar 2014 18:36:54 +0100

* YIN -> YANG translation breaks text formatting

** Description

When using YIN as the source format with the prospect of transforming it to YANG (e.g. for publishing in an RFC), it is often impossible to get longer text (in 'description' etc.) properly indented and formatted, especially if the text consists of multiple paragraphs and/or contains itemized lists or other structures.

Further details and example use cases can be found [[https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang][here]].

** Solution

IN YIN, permit mixed content inside the <text> element that is a child of 'contact', 'description', 'organization' and 'reference' statements. That is, the contents of <text> may contain text nodes as well as XML elements from foreign namespaces (any namespace other than the YIN namespace, current module namespace and namespaces of all imported modules).

The default YIN->YANG translation of such a mixed content is similar to XSLT: ignore all XML elements and concatenate the text nodes. However, translators can make use of these elements.

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Sun, 23 Mar 2014 11:22:23 -0700

On Sun, Mar 23, 2014 at 10:36 AM, Ladislav Lhotka <[email-masked]> wrote:

> When using YIN as the source format with the prospect of transforming it
> to YANG (e.g. for publishing in an RFC), it is often impossible to get
> longer text (in 'description' etc.) properly indented and formatted,
> especially if the text consists of multiple paragraphs and/or contains
> itemized lists or other structures.
>
> Further details and example use cases can be found [[
> https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang][here]].
>
> ** Solution
>
> IN YIN, permit mixed content inside the <text> element that is a child of
> 'contact', 'description', 'organization' and 'reference' statements. That
> is, the contents of <text> may contain text nodes as well as XML elements
> from foreign namespaces (any namespace other than the YIN namespace,
> current module namespace and namespaces of all imported modules).
>
> The default YIN->YANG translation of such a mixed content is similar to
> XSLT: ignore all XML elements and concatenate the text nodes. However,
> translators can make use of these elements.
>
>
This seems like a really complex solution for something that is barely used.
I don't know of anybody except you who uses YIN format instead of YANG.

Since YANG only preserves string indentation for a single statement, I
don't see
why mixed mode XML is needed for YIN.  Since XML preserves whitespace for
elements, it seems a parser could process a text block as if it was a
YANG double quoted string. Why is the translation broken?

Andy

---

From: Jernej Tuljak <[email-masked]>
Date: Mon, 24 Mar 2014 08:32:11 +0100

> Further details and example use cases can be found [[https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang][here]].

This is due to the fact that YANG is capable of carrying more meta 
information than YIN (like comments between concatenated quoted strings, 
and string concatenation itself). But I think that's okay and don't see 
it as a problem.

I find it quite strange that you use YIN as the format in which you 
write YANG code. I understand why (read the link you provided), but 
still...I don't think YIN was ever meant for that purpose. You will 
never be able to write YIN with the same "precision" as YANG.

Note that you could probably avoid most formatting issues if you abandon 
the idea of pretty printing your YIN text values, since all whitespace 
within them should be considered as significant. A simpler solution than 
mixed content.

Jernej

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 09:42:51 +0100

On 24 Mar 2014, at 08:32, Jernej Tuljak <[email-masked]> wrote:

> This is due to the fact that YANG is capable of carrying more meta information than YIN (like comments between concatenated quoted strings, and string concatenation itself). But I think that's okay and don't see it as a problem.
> 
> I find it quite strange that you use YIN as the format in which you write YANG code. I understand why (read the link you provided), but still...I don't think YIN was ever meant for that purpose. You will never be 

Strange? I am an Emacs user, and schema-supported editing (auto-completion, on-the-fly validation etc.) provided by its nXML mode gives me far more comfort compared to the yang mode. Of course, the pre-requisite is that the pointy XML stuff doesn’t make me feel sick. :-)

> able to write YIN with the same "precision" as YANG.

I don’t agree. Why not?

> 
> Note that you could probably avoid most formatting issues if you abandon the idea of pretty printing your YIN text values, since all whitespace within them should be considered as significant. A simpler solution than mixed content.

This is how I started and, believe me, it was a pain. The indentation in YIN source is generally different from YANG and every change in the module structure or moving definitions around required manual adjustments of descriptions. So I am now using limited HTML-like markup in YIN, and I wrote an XSLT stylesheet for YIN->YANG translation that supports this markup (and also preserves most comments). It works like charm, the translation is now fully reliable, and I have been using it for preparation of my I-Ds in the past couple of years.

Another use case for this extension would be multi-lingual descriptions that are quite easy to implement using XML markup and off-the-shelf tools. A specialized translator can then generate the YANG module with descriptions in a selected language.

I am not saying it is a critical feature, and I can certainly live without it. On the other hand, solid XML parsers should be able to deal with it quite easily.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 24 Mar 2014 10:13:21 +0100

> I am not saying it is a critical feature, and I can certainly live without it. On the other hand, solid XML parsers should be able to deal with it quite easily.

Lada,

can this perhaps be done by an extension of the YIN format or does it
have to be part of YANG 1.1? Or should YIN perhaps become a separate
document and moved out of YANG 1.1? I am just asking questions to
explore the possible solution space.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 24 Mar 2014 11:08:23 +0100 (CET)

Hi,

I have added this to the issues list as Y24, status NEW.

I agree with Andy and Jernej that this should not be done.  I have two
concerns:

  1.  The translation is not losslee.

  2.  Unless we specify which XML elements are allowed, and their
      meaning, their usage will be implementation specific.
      Translation from YIN to YANG would no longer produce the same
      result by different implementations.

/martin

Ladislav Lhotka <[email-masked]> wrote:
> When using YIN as the source format with the prospect of transforming
> it to YANG (e.g. for publishing in an RFC), it is often impossible to
> get longer text (in 'description' etc.) properly indented and
> formatted, especially if the text consists of multiple paragraphs
> and/or contains itemized lists or other structures.
> 
> Further details and example use cases can be found
> [[https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang][here]].
> 
> ** Solution
> 
> IN YIN, permit mixed content inside the <text> element that is a child
> of 'contact', 'description', 'organization' and 'reference'
> statements. That is, the contents of <text> may contain text nodes as
> well as XML elements from foreign namespaces (any namespace other than
> the YIN namespace, current module namespace and namespaces of all
> imported modules).
> 
> The default YIN->YANG translation of such a mixed content is similar
> to XSLT: ignore all XML elements and concatenate the text
> nodes. However, translators can make use of these elements.
> 
> --

---

From: Jernej Tuljak <[email-masked]>
Date: Mon, 24 Mar 2014 11:09:44 +0100

> Strange? I am an Emacs user, and schema-supported editing (auto-completion, on-the-fly validation etc.) provided by its nXML mode gives me far more comfort compared to the yang mode. Of course, the…

I agree it's a personal preference, but a major selling point of YANG is 
supposedly human readability. XML or any other form of markup can be 
hard to read by humans. And it's definitely harder to edit, IMO.

>> able to write YIN with the same "precision" as YANG.

Try to express the following with YIN. I'm an annoying tool user and the 
type of last comment is significant to me (I'd like the last comment to 
be an EOL comment instead of a multi-line comment when YANG gets 
generated from your YIN). The whitespace around them comments is also 
significant as is the position of the concatenation.

description /* comment after keyword */ "A mess" + /* inline comment */ 
" of a description." /* after argument */ ; // trailing single line comment

This is an example of a valid YANG statement with very specific 
formatting. I don't know why anyone would write YANG like this, but you 
could if you wanted to. Hence "precision".

Like I said. The level of meta information between the two formats is 
different.

> This is how I started and, believe me, it was a pain. The indentation in YIN source is generally different from YANG and every change in the module structure or moving definitions around required manual adjustments of descriptions.

Why? Descriptions are in element values. If these are not pretty 
printed, they can be moved around without losing formatting. Or am I 
missing something? I agree that it must look like a mess though.

> So I am now using limited HTML-like markup in YIN, and I wrote an XSLT stylesheet for YIN->YANG translation that supports this markup (and also preserves most comments). It works like charm, the translation is now fully reliable, and I have been using it for preparation of my I-Ds in the past couple of years.
>

Would your proposal also affect YANG-->YIN conversion?

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 11:15:38 +0100

> explore the possible solution space.

As long as it is a non-standard extension, YANG compilers reject such modules. I don't know how to make it possible unless YIN spec says it is allowed.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 11:43:02 +0100

>  1.  The translation is not losslee.

The current translation is not lossless either.

>      meaning, their usage will be implementation specific.

But at least YANG compilers won’t reject it.

>      result by different implementations.

Yes, even one implementation may be instructed to produce different results (plain text, reStructuredText, markdown).

I assume that the standard version of any module will always be the one that is published, e.g. in an RFC (in YANG syntax). This YIN extension is only intended as an aid for presentation purposes, or for certain freaks that want to write modules in YIN.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 13:47:34 +0100

On 24 Mar 2014, at 11:09, Jernej Tuljak <[email-masked]> wrote:

> I agree it's a personal preference, but a major selling point of YANG is supposedly human readability. XML or any other form of markup can be hard to read by humans. And it's definitely harder to edit, IMO.

That’s why I always translate YIN to YANG before publishing a module. Yes, all things being equal, YIN is harder to edit than YANG but, as I wrote, nXML is superior to yang mode, so unless somebody adds auto-completion, validation etc. to the yang mode, I will probably stick to nXML. 

> Try to express the following with YIN. I'm an annoying tool user and the type of last comment is significant to me (I'd like the last comment to be an EOL comment instead of a multi-line comment when YANG gets generated from your YIN). The whitespace around them comments is also significant as is the position of the concatenation.
> 
> description /* comment after keyword */ "A mess" + /* inline comment */ " of a description." /* after argument */ ; // trailing single line comment
> 
> This is an example of a valid YANG statement with very specific formatting. I don't know why anyone would write YANG like this, but you could if you wanted to. Hence "precision”.

Comments do not carry any info relevant for the data model, and they can be handled in most real-life cases.

> 
> Like I said. The level of meta information between the two formats is different.
> 
>> 
> Why? Descriptions are in element values. If these are not pretty printed, they can be moved around without losing formatting. Or am I missing something? I agree that it must look like a mess though.

If I want to publish a module in the YANG form, then the descriptions, references etc. have to be pretty-printed, and it is difficult to format them automatically if there is no markup.

> Would your proposal also affect YANG-->YIN conversion?

No, unless somebody wants to use some kind of markup in YANG syntax, too.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 24 Mar 2014 14:19:40 +0100

> As long as it is a non-standard extension, YANG compilers reject such modules. I don't know how to make it possible unless YIN spec says it is allowed.

I am confused - a YANG compiler parses YANG not YIN. Or is the
assumption here that a YANG compiler also has to accept YIN?

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 14:30:28 +0100

> assumption here that a YANG compiler also has to accept YIN?

From the text in Sec. 11, it is unclear whether the YIN syntax is mandatory to implement. At any rate, this would be relevant only to those that do implement YIN.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 24 Mar 2014 07:43:40 -0700

On Mon, Mar 24, 2014 at 2:13 AM, Juergen Schoenwaelder <
[email-masked]> wrote:

> > I am not saying it is a critical feature, and I can certainly live
> without it. On the other hand, solid XML parsers should be able to deal
> with it quite easily.
> >
>
> Lada,
>
Indentation is preserved for 1 statement, not across multiple statements.
XML preserves whitespace.  A tool can figure out how to format the YIN
text so it looks like YANG formatting for double quoted strings.

Fancy string concatenation done for formatting is not very important.
I don't want tool developers (like me) to be required to support "round trip
pretty printing".  IMO the YIN format should go away. I never thought
it was useful.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Mar 2014 15:51:28 +0100

On 24 Mar 2014, at 15:43, Andy Bierman <[email-masked]> wrote:

> 
> 
> 
> On Mon, Mar 24, 2014 at 2:13 AM, Juergen Schoenwaelder <[email-masked]> wrote:

Fine with me.

Lada