Subject: [Tools-discuss] Xml2rfc regression?: rfc2629.dtd

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jan 2022 02:15:20 +0100

Error: Can not access local file: /ietf/jsonpath-wg/internet-draft/rfc2629.dtd
Error: Can not access local file: /ietf/jsonpath-wg/internet-draft/rfc2629.dtd
Error: Unable to parse the XML document: draft-ietf-jsonpath-base.xml
 None: Line 0: Can not access local file: /ietf/jsonpath-wg/internet-draft/rfc2629.dtd

When did this happen?

I updated xml2rfc yesterday, but I didn’t see this regression until today.

(Maybe I broke something on my system…)

Grüße, Carsten

---

From: Kesara Rathnayake <[email-masked]>
Date: Tue, 25 Jan 2022 15:17:20 +1300

> (Maybe I broke something on my system…)

With xml2rfc release 3.12.0, includes from the local filesystem are 
disabled. To allow that you have to use `--allow-local-file-access`.

Note that `SYSTEM` inclusions are allowed without the new flag, as long 
as the included file is in the templates directory.
The templates directory can be configured by using `--template-dir`.

   --Kesara
> 
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss

-- 
Kesara Rathnayake
Senior Software Development Engineer - IETF LLC
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jan 2022 07:24:03 +0100

Hi Kesara,

thanks for the super-quick response!

> On 2022-01-25, at 03:17, Kesara Rathnayake <[email-masked]> wrote:
> 
> Note that `SYSTEM` inclusions are allowed without the new flag, as long as the included file is in the templates directory.

There is indeed an rfc2629.dtd under
lib/python3.9/site-packages/xml2rfc/templates
(I’m assuming that this is where my xml2rfc thinks its (default) template directory is.)

However, there also is an rfc2629.dtd in the local directory where the draft’s .xml is (this is needed so we can extract elements from the xml without xmlstarlet complaining).  Xml2rfc apparently finds the local one but then decides it shouldn’t allow access to it, instead of using the template one.

So, essentially, almost all builds that use xml tools for extracting components from the XML file are now broken.

I did an emergency fix in kramdown-rfc 1.5.25 that substitutes

<!DOCTYPE rfc [
  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;">

for

<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [

if --v3 (kdrfc -3) is given.
This leaves v2 high and dry, but those files are more likely to use the predefined character entities and less likely to use additional XML tooling and this a local rfc2629.dtd.
The emergency fix also makes the predefined character entities (except the four that now seem to be canonical for the RFC editor) unusable with v3 — this is a breaking change (I didn’t check how many drafts this breaks), but ultimately possibly a good one.  If you need a workaround, add:

doctype-reference: 'SYSTEM "rfc2629.dtd"'
or
doctype-reference: 'SYSTEM “rfc2629-xhtml.ent"'

to the YAML (and make sure you don’t have a local copy in the directory).

(FYI: When I looked today in /archive/id, 
of about 53451 XML I-Ds, approximately 
49185 XML I-Ds contained matches for
<!DOCTYPE rfc SYSTEM "rfc2629.dtd”
or similar and approximately 1870 contained matches for
<!DOCTYPE rfc SYSTEM "rfc2629-xhtml.ent”>
which probably has the same problem with a local copy.
So this is not a theoretical concern.)

Grüße, Carsten

---

From: "Martin Thomson" <[email-masked]>
Date: Tue, 25 Jan 2022 18:00:19 +1100

> I did an emergency fix in kramdown-rfc 1.5.25 that substitutes

Thank you.  Consider making that permanent.  Having to build an XML catalogue just to do simple XML processing (extracting author names, say) is an annoying hurdle to clear.  Inline declarations for entities is probably unnecessary for the output of a tool (UTF-8 does just work), but I do find it easier to distinguish various types of space if entities are used when it isn't U+20.

Can you push your change please?

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jan 2022 08:35:31 +0100

> On 25. Jan 2022, at 08:00, Martin Thomson <[email-masked]> wrote:
> 
> Thank you.  Consider making that permanent.  

I think with the “doctype-reference” escape valve the fix can stay in.

> Having to build an XML catalogue just to do simple XML processing (extracting author names, say) is an annoying hurdle to clear.  

I agree; kramdown-rfc just kept chugging along with most of the first ~50000 XML drafts out there, and this is now gone.

> Inline declarations for entities is probably unnecessary for the output of a tool (UTF-8 does just work), but I do find it easier to distinguish various types of space if entities are used when it isn't U+20.

Yes.  I wish Emacs were better in showing these types (of course, I'll just need to configure it right :-).

I have no idea what made the four entities that are now inserted by the v2v3 conversion so special; these of course highlight the inconsistent application of RFC 7997.
(Ceterum censeo: I want my dashes…)

> Can you push your change please?

Sorry, done.

Grüße, Carsten

---

From: Kesara Rathnayake <[email-masked]>
Date: Wed, 26 Jan 2022 14:39:10 +1300

> So, essentially, almost all builds that use xml tools for extracting components from the XML file are now broken.

I have captured this bug here: 
https://trac.ietf.org/trac/xml2rfc/ticket/703 and working on a fix.
I think this was a long-existing bug where xml2rfc looked at the source 
directory first instead of the templates directory and only surfaced 
because of recent local file access restrictions.

   --Kesara

> _______________________________________________
> Rfc-markdown mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfc-markdown

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jan 2022 13:39:49 +1300

> On 25/01/2022, at 8:00 PM, Martin Thomson <[email-masked]> wrote:
> 

For info, the proposed new XML templates do not use DTD files any longer and have the entities inline.  See for example https://github.com/ietf-authors/rfcxml-templates-and-schemas/blob/main/draft-rfcxml-general-template-standard-00.xml  

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 27 Jan 2022 01:43:06 +0100

> For info, the proposed new XML templates do not use DTD files any longer and have the entities inline.  See for example https://github.com/ietf-authors/rfcxml-templates-and-schemas/blob/main/draft-…

Makes a lot of sense.

Is there any documentation why exactly these four character entities were chosen?

  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;”>

(This would also be useful to indicate their intended use.)

Grüße, Carsten

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jan 2022 13:53:31 +1300

> On 27/01/2022, at 1:43 PM, Carsten Bormann <[email-masked]> wrote:
> 

In https://authors.ietf.org/en/templates-and-schemas#character-entities it explains "The templates above include the following character entities for invisible typographics characters directly in the source with a DOCTYPE statement as follows:"  (I’ve just seen the extraneous ’s’).   

Is that sufficient or is more explanation needed?

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 27 Jan 2022 02:02:06 +0100

> On 2022-01-27, at 01:53, Jay Daley <[email-masked]> wrote:
> 

Several aspects are not covered.

* Why are these four in this list, and not others?

* Why are these non-ASCII characters allowed outside artwork in RFCXMLv3, contrary to RFC 7997?

* For actual documentation, you would expect a usage example for each of them, and maybe some documentation what xml2rfc does with them.  (This needn’t be here, but needs to be somewhere.)

Oh, and neither nbsp nor nbhy are invisible; they just look too similar to other characters in certain editors that I would not recommend :-)

(Oh, and the warning at the top should point to an explanation of unprep.)

BTW, what is the current authoritative source repo for xml2rfc?

Grüße, Carsten

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jan 2022 18:24:38 +1300

> On 27/01/2022, at 2:02 PM, Carsten Bormann <[email-masked]> wrote:
> 
> 

(For what follows, I’m very happy for you or anyone else to suggest text and edits. Ideally this site would be community edited and maintained)

> * Why are these four in this list, and not others?

I think the implicit reason, that you can’t tell they are there if you use the Unicode character is reasonably obvious, but I’m happy to elucidate. 

> * Why are these non-ASCII characters allowed outside artwork in RFCXMLv3, contrary to RFC 7997?

I don’t know the answer to that. 
> * For actual documentation, you would expect a usage example for each of them, and maybe some documentation what xml2rfc does with them.  (This needn’t be here, but needs to be somewhere.)

Good idea. Got any suggestions?

> Oh, and neither nbsp nor nbhy are invisible; they just look too similar to other characters in certain editors that I would not recommend :-)

Ah yes. 
> (Oh, and the warning at the top should point to an explanation of unprep.)

Yes we need a proper page about prep/unprep. Want to write one? Until then we have a brief paragraph about that on another page that’s not that useful https://authors.ietf.org/en/rfcxml-overview
> BTW, what is the current authoritative source repo for xml2rfc?

Robert is the authoritative source of info on this but from our chat earlier today I believe it is still trac/svn with the aim of moving it to GitHub in the next few weeks. 

Jay

-- 
Jay Daley 
IETF Executive Director
[email-masked]