Subject: Re: [xml2rfc] assuming that period (.) ends a sentence is sometimes wrong

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 09:33:57 +0100

Am 28.02.2021 um 23:07 schrieb Carsten Bormann:
> ...
> So this is about sentence detection, not about sentence spacing.
> (Which in turn can make use of sentence spacing in the input, but that is orthogonal.)
>
> I’m trying to understand why the traditional method doesn’t work.
> There is never any ambiguity with traditional keyboarding (a new line starts after a sentence); except that one has to be careful not to do an input line-break after [.!?] that is within a sentence.
>
> The question is whether there is a need to accommodate multiple sentences per input line for proper sentence detection.
> People differ in their style here.
> The newline after a sentence rule helps with version control as well, so I have a strong preference for that style.
> But for people who like running on on the same input line, the two-space convention has been working well.
>
> XML may not "preserve whitespace”, but what that exactly means here is not clear to me.

XML parsers *do* preserve whitespace in element contents.

The tricky part is changing *applications* that use XML from one content
model (whitespace is insignifant) to another model (whitespace is
"mostly" insignificant) after the fact.

I also fear that we'll end up with *other* edge cases then.

For instance, if I have this source:

<t>
    PGP is a family of software systems developed by Philip R.
    Zimmermann from which OpenPGP is based.
</t>

Is the "R." a sentence end?

> Double spaces in XML input are copied verbatim into the HTML (where they then are swallowed by the HTML processor), so it is not like the processor is not seeing them.

But do they survive the preptool step? If so, that would IMHO be a bug.

> ...

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 11:51:24 +0100

> Is the "R." a sentence end?

Yes.

Don’t do that then [1].

Grüße, Carsten

[1]: http://catb.org/esr/jargon/html/D/Don-t-do-that-then-.html

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 11:58:01 +0100

On 2021-03-01, at 09:33, Julian Reschke <[email-masked]> wrote:
> But do they survive the preptool step?

Sure.

A quick check shows that https://www.rfc-editor.org/rfc/rfc8949.xml has about 318 in-line sentence end marks, and 101 end-of-line ones (this may or may not count ones at the end of paragraphs).

> If so, that would IMHO be a bug.

Can’t speak to that.
I don’t think there is a canonical answer to that question in the XML community.
(This has been technical debt for a third of a century.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 12:12:38 +0100

>> If so, that would IMHO be a bug.

I can only talk about what we *intended* to specify.

The only elements in the V3 grammar which have an implied xml:space
value of "preserve" are <artwork> and <sourcecode>. Thus, whitespace in
any other element is not significant. That also means, that the
preptool, when pretty-printing, should remove it (I might open a ticket
for that).

Changing the defaults for <t> (and similar elements) *would* be a
change. I'm not saying it can't be done, but given the fact that the
Style Guide has removed the requirements for 2SP, and furthermore
<https://www.chicagomanualofstyle.org/qanda/data/faq/topics/OneSpaceorTwo.html>
(the CMO is frequently referenced for RFC style questions), I would be
surprised if there was willingness to do this.

In any case: *if* the outcome was that we want to handle
space-after-sentence-ends differently, we absolutely should do this in
all output formats, not just plain text.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 12:19:48 +0100

On 2021-03-01, at 12:12, Julian Reschke <[email-masked]> wrote:
> value of "preserve" 

What do you believe setting this to default does?

If you can cite a specification, bonus points.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 12:34:49 +0100

> If you can cite a specification, bonus points.

That's why I used the term "intent".

When RFC 7749 and 7791 were done, we already had a huge library of XML
written against RFC 2629, and had at least three existing
implementations. I'm not aware of any existing XML where whitespace
inside <t> was considered significant, nor did any of the
implementations treat it this way.

Speaking as author of RFC 7749 I can only said that it never ever
occured to me that this might be unclear.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 12:59:11 +0100

On 2021-03-01, at 12:34, Julian Reschke <[email-masked]> wrote:
> 
> I'm not aware of any existing XML where whitespace

Idon’tthinkthattreatingwhitespaceasinsignificantisagreatidea.

So what exactly *do* you mean?

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 13:16:35 +0100

> So what exactly *do* you mean?

It means that a sequence of whitespace characters can be collapsed to a
single space character.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 13:46:20 +0100

> It means that a sequence of whitespace characters can be collapsed to a

That is XSLT normalize-space.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 13:57:58 +0100

> That is XSLT normalize-space.

XPATH, to be pedantic.

It's not exactly the same thing in that leading / trailing space in
content can be significant (at least if you consider it to apply to text
child nodes of <t>). For instance,

   <t>see <xref target="foo"/></t>

is not equivalent to

   <t>see<xref target="foo"/></t>

At this point I'm not sure anymore what we're discussing here.

If your point is that RFC 7991bis should be clearer about what XML calls
the "default whitespace handling mode", I agree.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 14:19:16 +0100

> At this point I'm not sure anymore what we're discussing here.

I’m sorry, I’m trying to elicit what you think xml:space=preserve does (or doesn’t that =default does, actually).  It does not mesh with what I learned about XML.

> the "default whitespace handling mode", I agree.

Well, what *DOES* XML call the default whitespace handling mode?

https://www.w3.org/TR/2008/REC-xml-20081126/#sec-white-space

> The value "default" signals that applications' default white-space processing modes are acceptable for this element; the value "preserve" indicates the intent that applications preserve all the white space.

(This is in a context of:

> In editing XML documents, it is often convenient to use "white space" (spaces, tabs, and blank lines) to set apart the markup for greater readability. Such white space is typically not intended for inclusion in the delivered version of the document. On the other hand, "significant" white space that should be preserved in the delivered version is common, for example in poetry and source code.

The intent certainly is not to get rid of all whitespace.  There is no talk about the whitespace normalization you have in mind, either.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 1 Mar 2021 14:35:59 +0100

> I’m sorry, I’m trying to elicit what you think xml:space=preserve does (or doesn’t that =default does, actually).  It does not mesh with what I learned about XML.

xml:space=reserve: all whitespace is significant; as used in HTML's
<pre> element.

xml:space=default: whatever the "application" deems significant (AFAICT
from the spec). In our case, the "application" is the xml2rfc processor.
Also, AFAICT, when M. T. Rose developed this, he used the same
processing model as HTML.

> The intent certainly is not to get rid of all whitespace.  There is no talk about the whitespace normalization you have in mind, either.)

Yes. Again, if your point is that the default handling should be
described in RFC 7991bis, I'm totally with you.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 14:55:24 +0100

> processing model as HTML.

Right, which is fine with me.  My copy of xml2rfcv1 also does sentence spacing, pretty much on the model that I described.

My point is that nothing in XML or existing specifications speaks against sentence detection and sentence spacing.  The setting of xml:space is completely irrelevant, as the handling of whitespace is left to the application in either case.

(This does not prejudice whether we want to do sentence spacing in the future, of course.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 1 Mar 2021 09:48:59 -0600

On Mon, Mar 01, 2021 at 02:55:24PM +0100, Carsten Bormann wrote:
> Right, which is fine with me.  My copy of xml2rfcv1 also does sentence
> spacing, pretty much on the model that I described.
> 
> My point is that nothing in XML or existing specifications speaks
> against sentence detection and sentence spacing.  The setting of
> xml:space is completely irrelevant, as the handling of whitespace is
> left to the application in either case.

Thanks for clarifying that for everyone.

> (This does not prejudice whether we want to do sentence spacing in the

Agreed.

IMO it would be good to have at least a way to handle the cases where no
extra spacing hurts readability, and to assume they are few and rare.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 2 Mar 2021 10:10:55 +0100

> (This does not prejudice whether we want to do sentence spacing in the future, of course.)

-> <https://github.com/rfc-format/draft-iab-xml2rfc-v3-bis/issues/198>

Best regards, Julian