Subject: [xml2rfc] assuming that period (.) ends a sentence is sometimes wrong

From: Daniel Kahn Gillmor <[email-masked]>
Date: Fri, 26 Feb 2021 21:44:09 -0500

The toolchain to build draft-ietf-openpgp-crypto-refresh produces XML
that contains:
…
          <li>PGP - Pretty Good Privacy.
PGP is a family of software systems developed by Philip R. Zimmermann from which OpenPGP is based.</li>
…

xml2rfc renders this to text as:

…
   *  PGP - Pretty Good Privacy.  PGP is a family of software systems
      developed by Philip R.  Zimmermann from which OpenPGP is based.
…

It looks like it is assuming that a period (.) ends a sentence, and that
two spaces should follow each sentence.  It is correct about the . after
"Privacy", but it is wrong about the . after "R". :(

This is kind of a dumb nit-pick, but it was noticed during a review on
the mailing list.

Is there a recommended way to fix this?  What XML input would produce a
.txt with only one space after the "R."?

   --dkg

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 27 Feb 2021 03:54:13 +0100

I’d replace the R. Z with R. Z.  
OK, that may be hard to see (an nbsp, U+00A0).
(Actually, it’s not there, as my mail reader swallows it.  Wow.)

Any software that consumes plaintext text should follow the age old convention that a sentence is ended by a period followed by a newline, or a period followed by two spaces.  
Unfortunately, this simple rule has fallen out of fashion…

Grüße, Carsten

> On 27. Feb 2021, at 03:44, Daniel Kahn Gillmor <[email-masked]> wrote:
> 
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Feb 2021 06:23:33 +0100

Am 27.02.2021 um 03:44 schrieb Daniel Kahn Gillmor:
> ...

The simplest possible fix would be to remove the code in xml2rfc
altogether. At some point, the former RSE announced that this would be
the case for text output, but for some reason, this plan was never
written down and executed.

Changing the source to work around the issue in one specifc formatter is
not cool in the first place.

Adding a non-breaking space here will have the desired effect in plain
text output, but will affect other output formats as well.

What *might* be possible in this case is to put the author name into a
<contact> element. (Haven't tried that though).

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Sat, 27 Feb 2021 10:09:27 -0600

> The toolchain to build draft-ietf-openpgp-crypto-refresh produces XML

There are only two possible correct answers, but historically it is
impossible to get agreement on adopting either:

 - mark-up sentences, e.g.,

     <sentence>D. K. G. wrote a post.</sentence>
     <sentence>This follow-up might be controversial.</sentence>

 - follow sentence-ending periods with two spaces (which does not mean
   the the rendered output must also do the same, as it could use a wide
   space instead), e.g.,

     D. K. G. wrote a post.  This follow-up might be controversial.

   i.e., `.  ` as a sort of mark-up

Instead many developers prefer to code up imperfect heuristics for
sentence ending periods.  If you search relevant archives (e.g., this
list's) you'll find that this is a periodic discussion.

I myself am quite used to always following every sentence period with
two spaces.  (In smart phone text input boxes that's a huge pain
because, at least on mine they turn two spaces typed in quick succession
into a period and space.)

Don your asbestos suits now.  Flame war incoming.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Feb 2021 17:35:02 +0100

> impossible to get agreement on adopting either:

At least three :-).

>       <sentence>This follow-up might be controversial.</sentence>

I'd be surprised if people would be willing to do this.

>     i.e., `.  ` as a sort of mark-up

Tricky, because it would change the whitespace handling inside <t>
(where currently multiple white space characters are always equivalent
to a single space).

> list's) you'll find that this is a periodic discussion.

Yes.

> Don your asbestos suits now.  Flame war incoming.

The third answer is: stop trying to. Optimizing the plain text output
format really really is not important.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Sat, 27 Feb 2021 11:30:04 -0600

> > impossible to get agreement on adopting either:

Your third solution is not a solution.  It says stop trying to have
rendered text distinguish between sentence-ending periods and not.  That
is just unacceptable.

Ah, there is a third solution, duh: use a Unicode widespace space after
sentence-ending periods (&emsp;).  It... might be just slightly less
unpleasant than <sentence>:

  Here's an example.&emsp; And here's the next sentence.&emsp; I had to
  follow &amp;emsp; with a space because running on two sentences is
  annoying.

> I'd be surprised if people would be willing to do this.

Right, no one would or should want that.

> (where currently multiple white space characters are always equivalent

Is that XML or xml2rfc forcing that?

> > list's) you'll find that this is a periodic discussion.

Yuck.

> format really really is not important.

Not acceptable.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Feb 2021 18:40:53 +0100

> Is that XML or xml2rfc forcing that?

People rely on whitespace being insignificant inside <t> and similar
elements. For instance, for indentation.

So we would need an algorithm that can distinguish between what's
itentional and what is not.

>> format really really is not important.

I just opened a random ancient RFC I'm familiar with (2068), and it
hasn't two spaces after a sentence ending. At least not consistently.

So exactly when became this a "requirement"? And who determines what
"acceptable" for plain text output?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Sat, 27 Feb 2021 12:01:02 -0600

> elements. For instance, for indentation.

But what makes whitespace in <t> insignificant?  If it's xml2rfc, then
that can be changed.

> hasn't two spaces after a sentence ending. At least not consistently.

So?

> "acceptable" for plain text output?

It's very much a thing in typography to follow sentence-ending periods
with a wide space precisely to distinguish from non-sentence-ending
periods.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Feb 2021 19:14:02 +0100

> But what makes whitespace in <t> insignificant?  If it's xml2rfc, then

Whitespace is insignificant everywhere except in <artwork> and <sourecode>.

People rely on this. Think indentation.

>> hasn't two spaces after a sentence ending. At least not consistently.

So it apparently has been "acceptable" in the past.

> with a wide space precisely to distinguish from non-sentence-ending

If you search for this on the web, you'll find lots of discussions about
this topic, and I don't thing there's universal agreement on that. Fpr
instance, see <https://en.wikipedia.org/wiki/Sentence_spacing>.

In any case: now that there are official HTML versions, the
"readability" of the text variant IMHO is not a big concern anymore. If
you want something that is easy to read, take the HTML or PDF versions.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Sat, 27 Feb 2021 12:33:59 -0600

> People rely on this. Think indentation.

Ah yes.  So &emsp;.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Feb 2021 19:39:10 +0100

That said, RFC 7322 has:

>    *  When a sentence ended by a period is immediately followed by
>       another sentence, there must be two blank spaces after the period.

(<https://tools.ietf.org/html/rfc7322#section-3.2>).

That text is gone in draft-flanagan-7322bis-00, dated March 2017. I'd
love to hear the current TRSE's position on that -- is this change going
to stay?

Best regards, Julian

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Sat, 27 Feb 2021 14:15:42 -0500

On Sat 2021-02-27 10:09:27 -0600, Nico Williams wrote:
>  you'll find that this is a periodic discussion.

har har 😛

> Don your asbestos suits now.  Flame war incoming.

I'm really uninterested in starting (or participating in) a flame war
about anything, much less about punctuation use.

I'm just trying to get the draft i'm collaborating on to render properly
and to not screw up how someone's actual name is written.

I'll go ahead with Carsten's U+00A0 proposal as a workaround for now, i
guess.

        --dkg

---

From: "John Levine" <[email-masked]>
Date: 27 Feb 2021 14:16:43 -0500

>love to hear the current TRSE's position on that -- is this change going

I don't know why it went into 7322 or who removed it in the draft or why.

Personally, I'm with you.  The extra space is an antique tradition from
the days of manual typewriters and I don't see any reason to worry about it
any more.  If you care about legibility, read the HTML version with those
nice proportional typefaces.

R's,
John

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 28 Feb 2021 08:40:04 +1300

On 28-Feb-21 08:16, John Levine wrote:
> I don't know why it went into 7322 

Inertia, I suspect.

> or who removed it in the draft or why.

> nice proportional typefaces.

So unless inertia rules the day, why don't we advance 7322bis and finally remove this pointless rule?

    Brian

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 27 Feb 2021 20:57:20 +0100

On 2021-02-27, at 20:16, John Levine <[email-masked]> wrote:
> any more.  

That statement is often repeated, but sentence spacing actually has nothing to do with typewriters (except that there needed to be a way to emulate it with typewriters, which is the reason that it is often reduced to “single space vs. double space” — not the choice we have today).  This confusion may have influenced the discussion, as the perception indeed is that sentence spacing is a remnant from typewriter days, when it simply was good typography for centuries (actually often applying a number of different spacing regimes after semicolons, colons, and periods).

I think we also have data that may warrant a discussion about the implications of the direction fashion has taken here.

Even Forbes [1] (whose take on science warrants, er, discussion) did not manage to totally dismiss the results of a study that showed the benefits for legibility of extended spaces after sentence ends.

[1]: https://www.forbes.com/sites/tarahaelle/2018/05/09/one-or-two-spaces-after-a-period-that-new-study-cant-tell-you/

As I hinted, the fashion is moving away from this legibility aid [2].
Anecdotically I can say that my reading speed and comprehension benefits mightily if document authors have taken the care to insert wider sentence ends, so for me this is simply a loss of quality.

[2]: https://www.instructionalsolutions.com/blog/one-space-vs-two-after-period

> If you care about legibility, read the HTML version with those

Somewhat ironically, “proportional” (i.e., non-monospaced) typefaces benefit way more from wide sentence end spaces than monospaced ones.  In the latter, the wide period is already adding some visual spacing, which is lost with the thin periods in non-monospaced type.  

Typesetting systems have of course supported sentence spacing since the 1970, and, e.g., troff hat different settings for spaces between words and spaces placed  at a sentence end (.ss setting, first and second parameter; in serious typesetting it was customary to set the first to 1/3 em and the second to 1/2 em, i.e. 1.5 times the inter-word spacing).  But DTP may have put an end to this great period.

For mass-market tools like MS-Word, the legibility feature of sentence spacing may simply take too much effort, and we may want to decide the same for RFC production.  But that doesn’t make it the right decision.

Grüße, Carsten

---

From: "John R Levine" <[email-masked]>
Date: 27 Feb 2021 18:51:30 -0500

> That statement is often repeated, but sentence spacing actually has nothing to do with typewriters (except that there needed to be a way to emulate it with typewriters, which is the reason that it…

Well, yes, but most of us came to it via typewriters.  That's the rule I 
learned at a summer typing class on manual typewriters with blank keytops 
in about 1969.

Having been through the publishing process in a lot of books, I can report 
that no matter how good your tools are, the only way to typeset stuff of 
professional quality is to do hand tweaks where the tools don't get it 
quite right.  For a bunch of reasons we have decided we're not doing that 
and I would prefer not to say oh, but THIS tweak is worth it.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 28 Feb 2021 07:32:00 +0100

On 28. Feb 2021, at 00:51, John R Levine <[email-masked]> wrote:
> 
> Having been through the publishing process in a lot of books, I can report that no matter how good your tools are, the only way to typeset stuff of professional quality is to do hand tweaks where the tools don't get it quite right.  For a bunch of reasons we have decided we're not doing that and I would prefer not to say oh, but THIS tweak is worth it.

For properly doing sentence spacing, what is needed is a way to signal sentence ends.
For 50 years, the convention in keyboarding manuscripts has been that dots at the end of the input line and dots followed by two spaces (here we are actually using two spaces — in the manuscript!) are periods (i.e., sentence ends).
That works exceedingly well.
Authors that keyboard carelessly don’t get proper sentence spacing, but no major disaster happens.

No tweaks needed.

The discussion came up because xml2rfc treated the dot in “Philip R. Zimmermann” as a sentence end.
This is a mere bug, and bugs can be fixed.
I’d send a pull request, but...

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Feb 2021 09:40:13 +0100

> I’d send a pull request, but...

I have opened
<https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/602> wrt the
discrepancy between what rfc7322bis does say (or actually does not say
anymore), and what xml2rfc (in text mode) attempts to do.

Furthermore, it really would be good if there was a plan how to progress
RFC 7322bis (while making sure it is aligned with what the grammar can
express). A place for issue tracking (maybe at
<https://github.com/rfc-format>?) would be good as well.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 28 Feb 2021 13:52:45 +0100

> <https://github.com/rfc-format>?) would be good as well.

If your intention is to amputate functionality, I think 2 needs to come before 1.

Grüße, Carsten

---

From: "John R Levine" <[email-masked]>
Date: 28 Feb 2021 12:14:27 -0500

> That works exceedingly well.

I don't see how that would work in xml2rfc where <t> elements don't 
preserve spacing.

> This is a mere bug, and bugs can be fixed.

It is a hard bug to fix.  You can add a heuristic not to treat a single 
letter followed by a dot as a sentence end, but it's wrong sometimes since 
the single letter word "I" occasionally ends a sentence.

As I said, you can make guesses but you can't fully automate it.

Regards,

---

From: Nico Williams <[email-masked]>
Date: Sun, 28 Feb 2021 11:38:26 -0600

On Sun, Feb 28, 2021 at 12:14:27PM -0500, John R Levine wrote:
> I don't see how that would work in xml2rfc where <t> elements don't preserve
> spacing.

Provided it doesn't also lose alternative Unicode whitespace characters,
using &emsp; is an option.  In a pinch we could have an element to mark
the end of a sentence (<s/>).

> the single letter word "I" occasionally ends a sentence.

Indeed.  "U.S." is a perfect example, because it can appear in the
middle of a sentence or at the end of a sentence, and you can't even use
the case of the first letter of the next word to distinguish because
words naming principal U.S. insitutions are capitalized after "U.S."

---

From: "John R Levine" <[email-masked]>
Date: 28 Feb 2021 12:54:45 -0500

> the end of a sentence (<s/>).

At the end of every sentence?  That's, uh, quite a stretch.  Are we sure 
this problem is worth that much effort by every author?

Regards,

---

From: Nico Williams <[email-masked]>
Date: Sun, 28 Feb 2021 11:59:59 -0600

> this problem is worth that much effort by every author?

If some such markup were only needed when the sentence-ending-period
heuristic would fail, then its usage would be very rare.  Maybe
non-breaking spacing could be used for that?

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Feb 2021 19:49:56 +0100

> non-breaking spacing could be used for that?

Nope.

Please let's not leak "optimizations" for text output into the canonical
format.

Assuming we continue with that 2SP practice (in contrast to what 7322bis
says): there are other alternatives, such as a config file to be
supplied to xml2rfc which continues known exception strings.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Feb 2021 19:56:36 +0100

> supplied to xml2rfc which continues known exception strings.

s/continues/contains/

BR, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 1 Mar 2021 08:51:56 +1300

> this problem is worth that much effort by every author?

Since we're designing on the hoof here, I suggest you'd need a construct like
<literal value="Philip R. Zimmermann"/>.

But much simpler to scrap the double space rule.

    Brian

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 28 Feb 2021 15:55:50 -0500

> But much simpler to scrap the double space rule.

Two things are being muddled here:

1) two spaces at end of sentences in .txt output;

2) how two distinguish sentence endings by xml2rfc in xml input.

There has been *some* discussion of using two spaces in the input for 
(2), but it doesn't work that way now and there are many issues in 
changing it to work that way. It isn't evident to me that it is a 
serious proposal.

*If* we had a reliable method for (2) then I doubt there would be much 
issue with (1). The problem is that the existing method for (2) isn't 
reliable.

I haven't checked, but I presume the current problems (2) are also 
exhibited in html output.

ISTM that the real question is whether authors will be willing to 
manually annotate the xml input to indicate sentence endings. I haven't 
seen any proposal mentioned that I would willingly use on a regular 
basis. I would rather suffer with the existing heuristic.

	Thanks,
	Paul

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 1 Mar 2021 10:59:16 +1300

> exhibited in html output.

Why would they be? The html format has single spaces. (Just checked in
RFC8981, which was announced half an hour ago.)

    Brian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 28 Feb 2021 23:07:35 +0100

> Two things are being muddled here:

Indeed.

> 1) two spaces at end of sentences in .txt output;

And 1a) how to render sentence spacing in the HTML (which, AFAIK, is not helping here).

A sentence spacing of two character positions has been the traditional rendering in plain text.  
That is suboptimal, of course (as many things are in monospaced environments).  

> There has been *some* discussion of using two spaces in the input for (2), but it doesn't work that way now and there are many issues in changing it to work that way. It isn't evident to me that it is a serious proposal.

So this is about sentence detection, not about sentence spacing.
(Which in turn can make use of sentence spacing in the input, but that is orthogonal.)

I’m trying to understand why the traditional method doesn’t work.
There is never any ambiguity with traditional keyboarding (a new line starts after a sentence); except that one has to be careful not to do an input line-break after [.!?] that is within a sentence.

The question is whether there is a need to accommodate multiple sentences per input line for proper sentence detection.
People differ in their style here.
The newline after a sentence rule helps with version control as well, so I have a strong preference for that style.
But for people who like running on on the same input line, the two-space convention has been working well.

XML may not "preserve whitespace”, but what that exactly means here is not clear to me.  
Double spaces in XML input are copied verbatim into the HTML (where they then are swallowed by the HTML processor), so it is not like the processor is not seeing them.

An easy fallback position is to no longer recognize sentence ends within an input line; that certainly solves the Philip R. Zimmermann issue (which shouldn’t be there as we have a <contact> element for natural names, but I digress).

> *If* we had a reliable method for (2) then I doubt there would be much issue with (1). The problem is that the existing method for (2) isn't reliable.

I’d say get rid of the heuristics.

But then, the thinking in the python textwrap module that is being used here to do the heavy lifting is extremely confused, so I don’t know whether that can be parameterized to be sane.  
Xml2rfc certainly tries...

> I haven't checked, but I presume the current problems (2) are also exhibited in html output.

There is no attempt in xml2rfc that I can recognize to have sentence spacing in HTML output.

> ISTM that the real question is whether authors will be willing to manually annotate the xml input to indicate sentence endings.

The answer is: Absolutely not.

> I haven't seen any proposal mentioned that I would willingly use on a regular basis. I would rather suffer with the existing heuristic.

A heuristic causes people to want to work around its limitations.  
Adding a zero-width space after the end of a middle name is a simple workaround for the current heuristic, but, ugh.  
As long as the future of the <contact> element is unclear (and the grammar bugs around it aren’t fixed), I’d also not want to steer people towards that.

Grüße, Carsten

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 28 Feb 2021 23:58:31 -0500

> RFC8981, which was announced half an hour ago.)

I didn't check, but gathered from the discussion that html had big 
spaces of some sort between sentences. That would seem to be the analog 
to two spaces in txt. And if we don't expect that in html then there is 
even less justification for the two spaces in txt.

> But much simpler to scrap the double space rule.

yes.

---

From: Nico Williams <[email-masked]>
Date: Sun, 28 Feb 2021 23:23:31 -0600

> >> <literal value="Philip R. Zimmermann"/>.

Consider these examples:

  It's the biggest stimilus in the U.S. Congress budget history.

Now, with the power of grammar we can parse that as one sentence.

But now try this one:

  Some wonder what is the biggest problem facing the U.S. Congress.

Hmmm.  Well, yes, a stickler for grammar will tell you that parses as
just one sentence.

There is no heuristic you can implement for that case, and wider space
after sentence-ending periods really does help readers parse sentences.

One could say "don't write ambiguous sentences" / "rewrite to avoid
ambiguity".  Like:

  Some wonder what is the biggest problem facing the United States
  Congress.

Well, OK, if there's no desire to do anything about this in xml2rfc, I
guess that will be authors' only resort.

Here's a possible scheme that doesn't count on XML preserving any space
for xml2rfc:

  if (s[i] == '.' && s[i-1] == '.') ||  # E.g., ellipsis
     ('.' && s[i-2] != '.')             # I.e., not a dotted acronym
    if s[i+1] == &nbsp
      # sentence does not end
    else
      # sentence ends
  else if s[i] == '.' && s[i-2] == '.'  # I.e., a dotted acronym
    if s[i+1] == &emsp;
      # sentence ends
    else
      # sentence does not end

Replace &nbsp; and &emsp; with elements if need be.

Nico
--