Subject: [Tools-discuss] Why do we even have text formats any more?

From: "Martin Thomson" <[email-masked]>
Date: Wed, 28 Jul 2021 11:53:49 +1000

I realize that this might be a little inflammatory as far as subjects go, but bear with me.

There are probably a few narrow cases where rendering plain text is better than HTML.  But what we've been doing for years (thanks to Henrik's great tool) is take text and turn it into HTML using the power of regular expressions.  That's been good, but it's not always reliable (how many errata mention that "Section X of [FOO]" links to Section X of this document?).  It's also been lagging as the text format changed (case in point: lack of a table of contents).

Here's an alternative: style the HTML so that it looks like the text.  I tried this and it worked shockingly well.

Repo: https://github.com/martinthomson/rfc-txt-html
Demo: https://martinthomson.github.io/rfc-txt-html/diff.html

This isn't perfect, but it seems pretty good to me.  Keep in mind that this took only a little bit of time to sketch out. No doubt it can be improved.  The readme has a bunch of things I found, all minor.

I don't think that this is the end of text, but a possible way to limit our use of the htmlizer[1].  People who need to automate access to content might still use text, though I will argue that XML is superior in that regard.   The other thing that comes to mind is diffs: HTML-native diff tools are somewhat less than ideal.  Either way, serving HTML is just better.

Enjoy,
Martin

[1] Though I still see a shocking number of people authoring in XML (or XML-capable input formats) and submitting in text.  But I think we have plans to limit that.

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 27 Jul 2021 19:21:26 -0700

FWIW, I generally do reviews by pulling the .txt file into my editor and
then
cutting and pasting the pieces I want to respond to. Presumably I could find
some way to make that work with HTML if we ditched the text format entirely
though it would probably be somewhat less convenient.

-Ekr

On Tue, Jul 27, 2021 at 6:54 PM Martin Thomson <[email-masked]> wrote:

> There are probably a few narrow cases where rendering plain text is better
> than HTML.  But what we've been doing for years (thanks to Henrik's great
> tool) is take text and turn it into HTML using the power of regular
> expressions.  That's been good, but it's not always reliable (how many
> errata mention that "Section X of [FOO]" links to Section X of this
> document?).  It's also been lagging as the text format changed (case in
> point: lack of a table of contents).
>
> Here's an alternative: style the HTML so that it looks like the text.  I
> tried this and it worked shockingly well.
>
> This isn't perfect, but it seems pretty good to me.  Keep in mind that
> this took only a little bit of time to sketch out. No doubt it can be
> improved.  The readme has a bunch of things I found, all minor.
>
> I don't think that this is the end of text, but a possible way to limit
> our use of the htmlizer[1].  People who need to automate access to content
> might still use text, though I will argue that XML is superior in that
> regard.   The other thing that comes to mind is diffs: HTML-native diff
> tools are somewhat less than ideal.  Either way, serving HTML is just
> better.
>
> Enjoy,
> Martin
>
>
> [1] Though I still see a shocking number of people authoring in XML (or
> XML-capable input formats) and submitting in text.  But I think we have
> plans to limit that.
>
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe):
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Robert Sparks <[email-masked]>
Date: Tue, 27 Jul 2021 21:22:30 -0500

This is worth exploring, but a few things:

First thought - we have some thousands of RFCs that only exist as txt, 
so I'm reading your argument as "for new things", but keep in mind we 
often have to process both old and new things (think diff).

Second thought - diff. You and I have discussed some potential 
candidates for html-diff that would rival the text diff for visual 
inspection, and they're still not-quite there. And when it comes down to 
some things, diffing text is still going to be the best generalizable 
tool. XML-diffing continues to be more of a stretch than it intuitively 
appears. (I think there's an argument here for keeping as much work that 
would involve diffing as we can in a language like markdown, but...)

Third thought - an alternative already brought up (Carsten I think was 
first) to html-ization for the things we have v3 xml for is to create a 
writer for it that builds it from the xml source rather than trying to 
pull things by heuristics from the text. Maybe where you're pointing 
would obviate that, but there may be different decisions to make in that 
writer that would be advantageous.

And finally, to your footnote, raising "why aren't people submitting 
XML?" - I've seen recently that there is fear from some seasoned 
submitters that the processor at the datatracker will get the references 
wrong. This is tied up with working in v2 and the issues we are working 
to correct with bibxml generation. Mitigating that fear will have an 
impact on the xml submission rate, I think.

RjS

On 7/27/21 8:53 PM, Martin Thomson wrote:
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 28 Jul 2021 16:59:00 +1000

On Wed, Jul 28, 2021, at 12:22, Robert Sparks wrote:
> "for new things"

Of course.  Anything that is in v3 format could get this treatment.

> diff

Yeah, I noted that.

It is clear to me that HTML tools are just not good enough.  Oddly, this is not because they couldn't just diff the text, but because they try to do so much more, like capture style or element choice changes as well as text changes.

For this, and for Ekr's use case, or anything where the goal is to remove noise, having text renderings available is useful.  

My pitch here is mostly about the most common case, which is people reading the documents in a web browser (and maybe those who print them too).  Keeping text because people have processes or tools that rely on text is probably necessary, if only because that means being able to present the entire series in a nominally consistent format.

I do think that the htmlizer can go in that case.

> create a writer for it that builds it from the xml source rather than trying to 
> pull things by heuristics from the text. 

Yeah, this is a neat idea, and it would avoid some of the issues I've discovered.  That said, the marginal benefit for the engineering cost involved seems unjustified in light of the HTML restyling option here.

> the processor at the datatracker will get the references wrong. 

Interesting.  Is this because people use some combination of entity references and processing instructions and fear that those will be filled in poorly?  That seems like a reasonable concern.  Part of it might derive from the fact that people have their own (skewed or hacked) local reference caches that they want to use in generating the final output.  I believe that xml2rfc can already produce XML with inline references, which draws from local sources.  Maybe we just need to document that process better for those people.

That and make the entire references infrastructure more reliable.  I routinely have build failures due to reference fetches that fail.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Jul 2021 09:33:26 +0200

On 28. Jul 2021, at 04:22, Robert Sparks <[email-masked]> wrote:
> 
> And finally, to your footnote, raising "why aren't people submitting XML?" - I've seen recently that there is fear from some seasoned submitters that the processor at the datatracker will get the references wrong. This is tied up with working in v2 and the issues we are working to correct with bibxml generation. Mitigating that fear will have an impact on the xml submission rate, I think.

Until recently, submitting TXT was the easy way to work around the mistreatment of hybrid XML by the submission process(*).
It will take some time until that gets out of people’s head.

The htmlizer did a mostly useful service for v2 users that the HTML output did not have: Adding links for section references.
V3 can now do that, but it requires conscious keyboarding(**); htmlizer completely automated that (but got it wrong occasionally).

I wrote a small PoC for a database of places where htmlizer gets it wrong with the necessary substitutions.
The cognitive barrier that v2 is obsolete might prevent that feature from happening.

Grüße, Carsten

(*) People used to submit TXT for most of the life of the I-D repository.
Then it became possible to submit XML along with that, which didn’t add anything except for archiving the XML as well.
But when the submission process started rejecting or mishandling submissions because XML came with them, everybody learned that it is a bad idea to send the XML, and TXT it was, again.

(**) Of course, in kramdown-rfc,

  Section 4.2.1 of {{RFC8949}}

simply becomes

  {{Section 4.2.1 of RFC8949}}

but for people keyboarding XML, manually adding section references is a bit more nightmarish:

From a recent AUTH48 (which still forces me to get in contact with XML):
     <t>IANA has created a new "<xref section="SenML Features" sectionFormat="bare" target="IANA.SENML"
     relative="#senml-features"/>" subregistry within
     the "Sensor Measurement Lists (SenML)" registry <xref target="IANA.SENML" format="default"/> with the

I can’t blame people who don’t want to make that effort.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 28 Jul 2021 11:57:36 +0300

On 2021-7-28, at 5:21, Eric Rescorla <[email-masked]> wrote:
> FWIW, I generally do reviews by pulling the .txt file into my editor and then

w3m -dump https://martinthomson.github.io/rfc-txt-html/rfc9000.html

(There are some paragraph marks in the output that I'm sure can be turned off somehow with an option or further posprocessing.)

Thanks,
Lars

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 28 Jul 2021 16:31:52 +0200

a) As mentioned in before, i would love for datatracker to offer
   the option for users to choose or even upload an html style sheet
   which would then be used for rendering of datatracker drafta/RFC
   for them. This IMHO would be the most easy and useful starting point 
   to leverage the flexibility of HTML.

b) I can not see any way to move away from a pure text editor
   for all core work - writing drafts and commenting on them.
   I can see that this can be done with either a txt or
   a markdown version of drafts/RFCs, not with HTML/XML.

c) RFCdiff is a big thing for me, i guess that can evolve to use other
   imputs than just txt over time.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Wed, 28 Jul 2021 11:42:42 -0400

> There are probably a few narrow cases where rendering plain text is
    > better than HTML.

The time when it's nice to have actual .txt files is when writing code, and
one needs to search for, and then copy and pasting bits of requirement into the code.
But, if I'm really working on such a document, I can run xml2rfc -o text myself.

    > This isn't perfect, but it seems pretty good to me.

I agree: it looks great.
I also agree that we should obsolete the htmlizer.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Russ Housley <[email-masked]>
Date: Wed, 28 Jul 2021 12:00:31 -0400

Others have already said how important diff is to their work.  Review team members, especially checking whether previous comments were addressed, use diff heavily.  I have a script to run diff locally when I am not online.  The online diff and my script depend on text files.

Russ

> On Jul 28, 2021, at 11:42 AM, Michael Richardson <[email-masked]> wrote:
> 
> Signed PGP part
>

---

From: Robert Sparks <[email-masked]>
Date: Wed, 28 Jul 2021 11:15:38 -0500

> I also agree that we should obsolete the htmlizer.

I don't think we can - what do you want to show for older RFCs? I think 
people expect the html-ized version, and we can't render v3 html for those.

Further - the discussion so far is ignoring (or presuming a change in) 
the large set of submissions we get now (as plain text) that do not have 
rfcxml anywhere in their production process.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Jul 2021 18:50:33 +0200

Am 28.07.2021 um 17:42 schrieb Michael Richardson:
 > ...

Why exactly doesn't that work for HTML? Loss of line breaks in prose?

Best regards, Julian

---

From: Michael Richardson <[email-masked]>
Date: Fri, 30 Jul 2021 18:13:21 -0400

> people expect the html-ized version, and we can't render v3 html for

but, if the file doesn't change then the HTML doesn't change, so it's just
static content after it's been run once, isn't it?

    > Further - the discussion so far is ignoring (or presuming a change in) the
    > large set of submissions we get now (as plain text) that do not have rfcxml
    > anywhere in their production process.

I agree that diff creates an important requirement to keep generating and
serving txt, and I'd forgotten that.  (And I'm a big rfcdiff user)

---

From: Michael Richardson <[email-masked]>
Date: Fri, 30 Jul 2021 18:14:41 -0400

> Why exactly doesn't that work for HTML? Loss of line breaks in prose?

When I load the HTML into my programming editor, I see markup, not text.

I don't really want to fish through a browser in another window.
I also don't want paste to include any extra stuff.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 31 Jul 2021 06:31:38 +0200

> I don't really want to fish through a browser in another window.

Ok.

> I also don't want paste to include any extra stuff.

"extra stuff" such as?

Best regards, Julian