Subject: [netmod] perfect extraction, tabs, and long lines - oh my

From: Kent Watsen <[email-masked]>
Date: Fri, 21 Sep 2018 18:16:48 +0000

[new subject line]

It is one thing for an editor to use tabs during the creation of text,
and another to publish text with an expectation that consumers will
render the tabs the same way.  Either the source editor converts tabs
to spaces, which is interoperable today, or keep the tabs while
publishing metadata in the text, using some TBD standard, enabling
consumers to use the same tab stops.  

If there were a standard enabling the publishing of text including
tabs, it should work for all artwork, not just artwork that has been
folded.  This is similar to the discussion we had before about having
begin/end markers enabling perfect extractions, in that it is also
something that pertains to all artwork, not just artwork that has 
been folded.  

Thus, there are a total of three problems:
  P1: perfect extraction
  P2: tabs
  P3: long lines

Assuming all thing were solved problems, and assuming that we always
want perfect extraction, the possible combinations for the occurrence
of the other two problems are:
  - no tabs or long lines
  - tabs, but no long lines
  - long lines, but no tabs
  - tabs and long lines

How are they ordered?  Clearly supporting perfect extractions has 
to be the outermost thing, but what about the other two?   Does it
matter?

Thinking about solutions:

 - the solution for long-lines is to use a header (not a footer)
   because it's believed important to prime readers *before* they
   read the text.

 - the solution for perfect-extraction could be either:
     - use both a header-and-footer marker (low tech)
     - or use either a header or a footer that encodes
       something like a "num lines" value into the 
       marker.  (note: footer-only okay since the marker
       is for programmatic processors, not the readers)

 - the solution for tabs could be to use either a header
   or a footer that encodes the tab- stop metadata. (note:
   footer-only okay since the marker is for programmatic
   processors, not the readers)

If tabs were to be supported by the folding solution (note: it
doesn't make sense to talk about "folds being supporting by the
tabbing solution"), then either:

  a) tabs are handled *before* folding, and the folding-solution 
     is aware of the tab-solution (i.e., it is able to process 
     the metadata).

      - everybody nods ;)

  b) the folding-solution is really a folding+tab solution, that is,
     it has a built-in way of handling tabs (i.e., encoding tab stop
     metadata) independent of how tabs are handled for text that has
     not been folded.

      - this may be technically possible, but we should avoid having
        two solutions to solve the tab problem.  We would be better
        off solving the tab-problem directly and then use (a).

  c) the folding-solution folds using the source tab stops, but does
     not itself encode metadata about the tab stops, assuming that
     there is a "promise" that the encoding of the metadata will
     occur in a wrapper layer around it.

      - this feels icky, but it seems viable and, would possible
        allow us to proceed with this draft without having to solve
        the tabbing problem now.

Options:

  1) RFC disallows TABS in both the source-input and folded-output.
     ***This is what we currently have***

  2) RFC disallows TABS only in the folded-output, per RFC 7991,
     leaving it to the folding-logic (the script) to decide if it
     wants to:
      a) disallow TABS in the source input (curr script does this)
      b) detect TABS exist and prompt user for TAB stop info
      c) detect TABS and query environment for cur TAB stop info
         (but tab-stops may differ in the shell the text editor,
         or whatever was used to create the text, right?)

  3) RFC allows TABS, and solves it by depending on a tab-solution,
     as described by (a).

  4) RFC allows TABS, but does not solves it, as described by (c).
     This would probably NOT be allowed from a standardization
     perspective.

Moving to (2) would be easy and probably resolves most concerns
here.  

Moving to (3) is possible, but we would do so only to:

 - support non-IETF use cases

 - or pave the way for an rfc7991bis that could depend on the 
   solutions we define here.  

   That is, rfc7991bis could *allow* long-lines and tabs while
   `xml2rfc` applies the solutions being discussed here only
   for when exporting the "plain-text" format (other formats
   may have better ways to support perfect extractions and/or
   not care about long-lines or tabs).

   PS: as a corollary, realize that when we pre-textualizing
       artwork for XML-based submissions, we are somewhat
       worsening the result for other output formats (not
       "plain-text").

Kent

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 24 Sep 2018 11:28:59 +0200

Hi,

it is quite funny: we are using XML in an area that it wasn't really
designed for - representation of hierarchical data - but, on the other
hand, we *don't* use XML where it could effectively help us avoid
awkward formatting and extraction problems such as those discussed in
this thread.

Since the beginning, I've been writing YANG source in the YIN XML
notation (and most people find it weird). However, when one uses a
schema-aware editor such as nxml mode of emacs, writing YIN is really a
pleasant experience:

- no need to remember YANG syntax, all statements are autocompleted in a
  context-sensitive way

- no need to care about the prescribed order of substatements (for the
  pyang --ietf check)

- (mostly) no need to care about long lines and whitespace.

A schema-aware editor takes care about the first item and XSLT scripts
about the rest. All the tools are available in my GitHub skeleton project

https://github.com/llhotka/YANG-I-D

And as for extracting YANG modules from I-D text: given that I-D
submission format is XML (xml2rfc), the most natural way for including
YANG modules in an I-D would be to use YIN directly instead of
<artwork>. Both formatting and extracting the module would then be
absolutely painless - it is a different XML namespace, so tools should
have no problems with it.

I suspect I won't attract many supporters but I couldn't help myself. It
bothers me that we have all the tools and technologies available but -
because the general opinion is that they are hard to use - we instead
struggle with brittle and tricky technicalities like those below. Isn't
it much harder after all?

Lada

Kent Watsen <[email-masked]> writes:

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Kent Watsen <[email-masked]>
Date: Wed, 26 Sep 2018 14:09:52 +0000

I recommend that we select "option-2" (see bottom).

- it easy to do.
- there's no current support for having tabs in folded output.
- doing so doesn't preclude an "option-3" someday in the future.

The authors will assume that this is WG consensus if there are no 
objections within a week's time.

Kent // co-author

-----Original Message-----

[new subject line]

      - everybody nods ;)

Options:

  3) RFC allows TABS in the folded output, and solves it by 
     depending on a tab-solution, as described by (a).

  4) RFC allows TABS in the folded output, but does not solves it,
     as described by (c). This would probably NOT be allowed from 
     a standardization perspective.

Kent

---

From: Robert Wilton <[email-masked]>
Date: Wed, 26 Sep 2018 15:16:51 +0100

> I recommend that we select "option-2" (see bottom).
Yes, option 2 seems reasonable to me.

Thanks,
Rob

> .

---

From: Kent Watsen <[email-masked]>
Date: Wed, 26 Sep 2018 14:09:52 +0000

- it easy to do.

Kent // co-author

-----Original Message-----

[new subject line]

      - everybody nods ;)

Options:

Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Sep 2018 17:22:43 +0200

> I recommend that we select "option-2" (see bottom).

[...]
 
>          or whatever was used to create the text, right?)

Given that 2) has a)-c), I do not understand what the recommendation
actually is. The recommendation is hopefully 2a) and we are done.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Robert Wilton <[email-masked]>
Date: Wed, 26 Sep 2018 16:28:18 +0100

My interpretation:

Option 2 is to disallow tabs in the output, but leave it to the 
implementation to decide how to handle tabs in the input document, so a 
script would be allowed to do a, b, or c.

Just supporting "option 2(a)" is the same as "option 1":

Perhaps you mean that you prefer option 1?

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Sep 2018 17:41:56 +0200

> Perhaps you mean that you prefer option 1?

1) or 2a) since 2b) and 2c) are not implementable in tools that are
detached from the author and his/her editor and they will ultimately
require to carry some metadata around and this means additional
complexity for no real gain since the output has no TABs anyway.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 26 Sep 2018 17:47:25 +0200 (CEST)

> complexity for no real gain since the output has no TABs anyway.

I agree.  I prefer option 1.  If you do have tabs in your input,
simply translate them before running the folding algorithm.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Wed, 26 Sep 2018 17:38:22 +0000

> actually is. The recommendation is hopefully 2a) and we are done.

For the script that we include in the Appendix, the authors wish to
keep the current "2a" behavior and have no plans to change that.

The only change we're envisioning is to the following paragraph from
Section 6.1 (Automated Folding):

OLD:

   Scan the artwork to ensure the horizontal tab character does not
   appear.  If any horizontal tab character appears, exit (this artwork
   cannot be folded).

NEW:

   Scan the artwork to ensure the horizontal tab character does not
   appear.  If any horizontal tab character appears, either 1) do
   something to support tabs in the folded output (out of scope to
   this draft), 2) do something to convert the horizontal tabs in
   the input to space characters in the folded output, or 3) exit,
   forcing the input provider to convert the horizontal tabs to
   space characters first.

Note: the "do something" phraseology seems too informal, would welcome
suggestions for tightening it up.

Kent // co-author

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Sep 2018 19:45:42 +0200

> suggestions for tightening it up.

This is not an improvement. Just more complexity and noise.

/js

---

From: Kent Watsen <[email-masked]>
Date: Wed, 26 Sep 2018 19:21:49 +0000

> This is not an improvement. Just more complexity and noise.

Disagree.  The RFC should not mandate the tool exits, that would be
overreaching.  How about this simpler language?

NEW:

   Scan the artwork for horizontal tab characters.  If any horizontal
   tab characters appear, either resolve them to space characters or
   exit, forcing the input provider to convert them first.

Kent // co-author

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Sep 2018 21:35:48 +0200

>    exit, forcing the input provider to convert them first.

Sounds good.

/js