Subject: [xml2rfc] Non-determinism (Re: Unable to run xml2rfc --help)

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Jan 2021 13:54:48 +0100

While it might be a special need to actually compare generated HTML between different runs, but when testing an entire authoring chain, it is a bit unnerving to have changes like

-          <li class="toc ulEmpty compact" id="section-toc.1-1.12">
+          <li class="ulEmpty toc compact" id="section-toc.1-1.12">

between runs that are meant to produce identical output.

My python-fu is not sufficient to suggest a way to hide the randomized Python behavior from this output (replace Dictionary by OrderedDict?, but I thought that was the default now since Python 3.7); could somebody else please step in.

Grüße, Carsten

> On 2020-12-29, at 16:44, Carsten Bormann <[email-masked]> wrote:
> 
> That appears to be a rather funny bug.
> Apparently, the ASCII neutral minus hyphen is in the list of keys. Depending on how the Python hashing random number generator is seeded, that hyphen is going to a position where it can’t be in the regexp or where it can. True Heisenbug. 
> (The hyphen needs to be protected to go into a character class, even when the resulting character range seems to be syntactically acceptable.)
> 
> Sent from mobile, sorry for terse
> 
>> On 29. Dec 2020, at 11:22, Roger Price <[email-masked]> wrote:
>> 
>> On Mon, 28 Dec 2020, Carsten Bormann wrote:
>> 
>>> On 2020-12-25, at 15:24, Roger Price <[email-masked]> wrote:
>>>> python3.5
>>> but this got my attention: Python 3.5 is end-of-life; did you try a newer Python?
>> 
>> I changed the xml2rfc shebang to #!/usr/bin/python3.8 and ran Python 3.8.1 but it made no difference.
>> 
>>> I’m out of educated guesses (at least until you are showing more of the traceback),
>> 
>> The full Python error message is:
>> 
>> Traceback (most recent call last):
>>  File "/usr/bin/xml2rfc", line 7, in <module>
>>    from xml2rfc.run import main
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/__init__.py", line
>>    14, in <module>
>>    from xml2rfc.parser import  XmlRfcError, CachingResolver, XmlRfcParser, XmlRfc
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/parser.py",
>>    line 20, in <module> from xml2rfc.writers import base
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/__init__.py",
>>    line 2, in <module> from xml2rfc.writers.base import RfcWriterError
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/base.py",
>>    line 30, in <module>
>>    from xml2rfc.util.unicode import ( punctuation, unicode_replacements,
>>    unicode_content_tags, bare_unicode_tags,
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/util/unicode.py",
>>    line 260, in <module>
>>    punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
>>  File "/usr/lib/python3.5/re.py", line 224, in compile
>>    return _compile(pattern, flags)
>>  File "/usr/lib/python3.5/re.py", line 293, in _compile
>>    p = sre_compile.compile(pattern, flags)
>>  File "/usr/lib/python3.5/sre_compile.py", line 536, in compile
>>    p = sre_parse.parse(p, flags)
>>  File "/usr/lib/python3.5/sre_parse.py", line 829, in parse
>>    p = _parse_sub(source, pattern, 0)
>>  File "/usr/lib/python3.5/sre_parse.py", line 437, in _parse_sub
>>    itemsappend(_parse(source, state))
>>  File "/usr/lib/python3.5/sre_parse.py", line 575, in _parse
>>    raise source.error(msg, len(this) + 1 + len(that))
>> sre_constants.error: bad character range …-‚ at position 3
>> 
>> In file unicode.py at line 260
>> 
>> punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
>> 
>> I added the two lines
>> 
>> import sys
>> print("unicode.py: list(punctuation.keys())) = {}".format(list(punctuation.keys())))
>> 
>> On runs when Python crashes I see outputs:
>> 
>> unicode.py: list(punctuation.keys()) = ['−', '‐', '′', '\u2002', '’', '´',
>>                 '\u2003', '‚', '–', '\u2009', '”', '-', '“', '‘', '…', '„', '—']
>> unicode.py: list(punctuation.keys()) = ['„', '”', '—', '–', '…', '-', '\u2009',
>>                 '‘', '’', '\u2003', '−', '´', '\u2002', '‚', '′', '‐', '“']
>> 
>> When the runs succeed I see outputs:
>> 
>> unicode.py: list(punctuation.keys()) = ['−', '\u2003', '\u2009', '‘', '“', '‚',
>>                 '-', '”', '—', '’', '′', '\u2002', '‐', '–', '…', '´', '„']
>> unicode.py: list(punctuation.keys()) = ['\u2002', '‐', '′', '\u2003', '−', '´',
>>                 '“', '”', '„', '–', '-', '‘', '’', '…', '‚', '\u2009', '—']
>> 
>> I do not understand why such a list of constants has to be so random. I'm not sure where to look next, but if you would like to see further traces of other functions, just ask. Roger_______________________________________________
>> xml2rfc mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/xml2rfc
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Julian Reschke <[email-masked]>
Date: Mon, 4 Jan 2021 14:03:51 +0100

Am 04.01.2021 um 13:54 schrieb Carsten Bormann:
> ...

See <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/553>.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Jan 2021 14:21:01 +0100

> See <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/553>.

Ah, thank you, Julian!

So I don’t need to write a bug report.

But I can confirm: The order does change between runs on the same platform (Python 3.9 on macOS 10.15.7 (19H114)).

I don’t quite agree that non-deterministic output (which is visible as such all the way to the HTML published by the RPC — see below for RFC89xx) is a minor bug.
It should be possible for third parties to check the published derived versions against their own builds, without the need to rerun the tool until the output happens to match.

Grüße, Carsten

rfc8900.html:   50 class="toc compact ulEmpty"
rfc8901.html:   30 class="compact ulEmpty toc"
rfc8902.html:   27 class="ulEmpty toc compact"
rfc8904.html:   22 class="ulEmpty toc compact"
rfc8905.html:   13 class="ulEmpty compact toc"
  14 class="ulEmpty toc compact"
rfc8906.html:   56 class="compact ulEmpty toc"
rfc8907.html:   63 class="compact toc ulEmpty"
rfc8908.html:   22 class="compact toc ulEmpty"
rfc8909.html:   27 class="ulEmpty toc compact"
rfc8910.html:   24 class="ulEmpty compact toc"
rfc8914.html:   44 class="compact toc ulEmpty"
rfc8915.html:   61 class="toc compact ulEmpty"
rfc8916.html:   30 class="compact ulEmpty toc"
rfc8917.html:   17 class="ulEmpty toc compact"
rfc8918.html:   10 class="compact ulEmpty toc"
  10 class="toc ulEmpty compact"
rfc8919.html:   43 class="compact toc ulEmpty"
rfc8920.html:   42 class="toc ulEmpty compact"
rfc8921.html:   81 class="ulEmpty compact toc"
rfc8922.html:   46 class="toc ulEmpty compact"
rfc8923.html:   27 class="ulEmpty compact toc"
  21 class="ulEmpty toc compact"
rfc8924.html:   51 class="compact toc ulEmpty"
rfc8925.html:   23 class="toc ulEmpty compact"
rfc8926.html:   58 class="ulEmpty compact toc"
rfc8927.html:   47 class="toc ulEmpty compact"
rfc8928.html:   51 class="ulEmpty toc compact"
rfc8929.html:   36 class="toc compact ulEmpty"
rfc8930.html:   22 class="toc ulEmpty compact"
rfc8931.html:   42 class="ulEmpty compact toc"
rfc8932.html:   67 class="toc compact ulEmpty"
rfc8933.html:   21 class="toc compact ulEmpty"
rfc8934.html:   55 class="ulEmpty compact toc"
rfc8935.html:   34 class="ulEmpty compact toc"
rfc8936.html:   38 class="toc ulEmpty compact"
rfc8937.html:   16 class="toc ulEmpty compact"
rfc8938.html:   11 class="compact ulEmpty toc"
  30 class="ulEmpty compact toc"
rfc8939.html:   37 class="toc compact ulEmpty"
rfc8940.html:   16 class="ulEmpty toc compact"
rfc8943.html:   18 class="compact toc ulEmpty"
rfc8944.html:   15 class="ulEmpty toc compact"
rfc8945.html:   51 class="ulEmpty toc compact"
rfc8947.html:   29 class="compact toc ulEmpty"
rfc8948.html:   24 class="ulEmpty compact toc"
rfc8949.html:   97 class="ulEmpty compact toc"
rfc8950.html:   21 class="ulEmpty compact toc"
rfc8951.html:   28 class="compact toc ulEmpty"
rfc8952.html:   53 class="ulEmpty toc compact"
rfc8953.html:   28 class="toc ulEmpty compact"
rfc8954.html:   21 class="toc ulEmpty compact"
rfc8955.html:   56 class="toc ulEmpty compact"
rfc8956.html:   30 class="ulEmpty compact toc"
rfc8958.html:   12 class="compact toc ulEmpty"
rfc8960.html:   12 class="toc ulEmpty compact"
  10 class="ulEmpty compact toc"
rfc8961.html:   16 class="compact ulEmpty toc"
rfc8970.html:   22 class="toc compact ulEmpty"
rfc8971.html:   20 class="ulEmpty compact toc"

---

From: Julian Reschke <[email-masked]>
Date: Mon, 4 Jan 2021 14:31:34 +0100

> But I can confirm: The order does change between runs on the same platform (Python 3.9 on macOS 10.15.7 (19H114)).

Same here: Python 3.6 on Cygwin.

> It should be possible for third parties to check the published derived versions against their own builds, without the need to rerun the tool until the output happens to match.

+1

Best regards, Julian

---

From: Roger Price <[email-masked]>
Date: Mon, 4 Jan 2021 15:14:23 +0100 (CET)

On Mon, 4 Jan 2021, Julian Reschke wrote:

>> I don’t quite agree that non-deterministic output (which is visible as such 
> all the way to the HTML published by the RPC — see below for RFC89xx) is a 
> minor bug.
>> It should be possible for third parties to check the published derived 
> versions against their own builds, without the need to rerun the tool until 
> the output happens to match.
>
> +1

One of the problems with Python is that Python developers seem to enjoy changing 
Python behaviour.  In 3.7 dicts are ordered, but for how long?  Wouldn't it be 
simpler in the long run for xml2rfc to replace dict with "from collections 
import OrderedDict" ?

Roger

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Jan 2021 15:24:20 +0100

Ex-BDFL has declared it a feature:

https://mail.python.org/pipermail/python-dev/2017-December/151283.html

but somehow it doesn’t seem to help here.

Grüße, Carsten

> On 2021-01-04, at 15:14, Roger Price <[email-masked]> wrote:
> 
>> all the way to the HTML published by the RPC — see below for RFC89xx) is a minor bug.
>> versions against their own builds, without the need to rerun the tool until the output happens to match.
>> 
>> +1
> 
> One of the problems with Python is that Python developers seem to enjoy changing Python behaviour.  In 3.7 dicts are ordered, but for how long?  Wouldn't it be simpler in the long run for xml2rfc to replace dict with "from collections import OrderedDict" ?
> 
> Roger_______________________________________________
> xml2rfc mailing list
> [email-masked]