Subject: [Tools-discuss] help - old xml2rfc error is back

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 08:36:50 -0500

Hi,

I'm on a new ubuntu 20.4 install (under wsl) and am getting the 
previously reported (resolved?) error:

Error: Unable to parse the XML document: draft-ietf-detnet-mpls.xml
  draft-ietf-detnet-mpls.xml: Line 1498: Unable to resolve external 
request: "reference.RFC.2119"

the version info is:

 > pip3 show xml2rfc
Name: xml2rfc
Version: 3.3.0
Summary: Xml2rfc generates RFCs and IETF drafts from document source in 
XML according to the IETF xml2rfc v2 and v3 vocabularies.
Home-page: https://tools.ietf.org/tools/xml2rfc/trac/
Author: Henrik Levkowetz
Author-email: [email-masked]
License: BSD-3-Clause
Location: /home/lberger/.local/lib/python3.8/site-packages
Requires: six, appdirs, pyyaml, setuptools, requests, pyflakes, lxml, 
configargparse, html5lib, pycountry, intervaltree, jinja2, 
google-i18n-address, kitchen
Required-by:

I had no problems under wsl and ubuntu 18.04

Is there a known solution to this?

Thanks,

Lou

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Nov 2020 14:40:17 +0100

Hi Lou,

> Is there a known solution to this?

Could you provide the xml source, please?

	Henrik

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 08:47:23 -0500

Hi Henrik,

Thanks for the quick response.  I did a 'sudo apt install python3-pip; 
pip3 install xml2rfc' yesterday (it's a new computer) so is whatever is 
from the public repo.  Is this sufficient?

Lou

---

From: Julian Reschke <[email-masked]>
Date: Tue, 3 Nov 2020 14:47:30 +0100

Am 03.11.2020 um 14:36 schrieb Lou Berger:
> ...

How do you include the reference?

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Nov 2020 15:19:47 +0100

Hi Lou,

> from the public repo.  Is this sufficient?

No, I think I really need your xml source: draft-ietf-detnet-mpls.xml

	Henrik

> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]
>

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 09:23:23 -0500

Sure, no problem. https://tools.ietf.org/id/draft-ietf-detnet-mpls-13.xml

Lou

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 09:30:19 -0500

FYI - just some extra data...

I went on to a very old install, and was successful using python 2 and 
xml2rfc Version: 2.22.3 -- it worked as expected.

On the same machine I then upgraded to 3.3.0 and got the same error

Error: Unable to parse the XML document: draft-ietf-detnet-ip-over-mpls.xml
  draft-ietf-detnet-ip-over-mpls.xml: Line 651: Unable to resolve 
external request: "reference.RFC.2119"

Lou

---

From: Julian Reschke <[email-masked]>
Date: Tue, 3 Nov 2020 15:38:03 +0100

> Sure, no problem. https://tools.ietf.org/id/draft-ietf-detnet-mpls-13.xml

So

   xml2rfc -v --html draft-ietf-detnet-mpls-13.xml

gives:

   Parsing file draft-ietf-detnet-mpls-13.xml
   Resolving entity... /projects/xml2rfc/rfc2629.dtd
   Resolving entity... /projects/xml2rfc/rfc2629-xhtml.ent
   Resolving entity... /projects/xml2rfc/rfc2629-other.ent
   Resolving entity... /projects/xml2rfc/rfc2629.dtd
   Resolving entity... /projects/xml2rfc/rfc2629-xhtml.ent
   Resolving entity... /projects/xml2rfc/rfc2629-other.ent
   Resolving include...
https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119
    ... 301 https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119
    ... 404
https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119
   URL retrieval failed with status code 404 for
'https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119'
   Resolving include...
https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119
    ... 404
https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119
   URL retrieval failed with status code 404 for
'https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119'

...appending ".xml" to the processing instruction seems to fix it. Not
sure whether this is a change in xml2rfc or the reference file server.

FWIW, you wouldn't have this problem if you either supplied a full URI
for the reference, or just pasted it into your source code once and for
all (it's immutable anyway, right?).

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Nov 2020 15:56:59 +0100

Hi Lou,

> Sure, no problem. https://tools.ietf.org/id/draft-ietf-detnet-mpls-13.xml

Thank you.

This is a legacy (v2) schema document, without full reference include
paths.

With the transition to versions 3.x of xml2rfc, the default is to use the
schema v3 output formatters, and in order to do so, the v2 source is
converted to v3 schema internally.

One difference in the v2 and v3 parsers is that v3 uses XInclude to handle
inclusion, and one aspect of the XInclude specification is that it does
not permit the use of fallback locations for XInclude URLs; they have to
be complete and correct, or fail.

(I initially implemented the XInclude handling with the same kind of
fallbacks that exist in the v2 parser, but was told that it broke the
XInclude spec. and had to go.)

There are 3 ways around this; two which you can use immediately, and one
which I'll have to look at including in a future version of the v2-to-v3
converter:

  * Provide complete explicit URLs for included references:  Instead of
    <?rfc include="reference.RFC.2119"?> use
    <?rfc include="https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119"?>

  * Use the --legacy switch to process the source with the v2 parser and
    formatter instances

  * (For me) maybe modify the xml2rfc v2 to v3 converter to supply missing
    reference URL paths for recognized reference file names.

Best regards,

	Henrik

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 10:46:52 -0500

Hi Henrik,

adding the --legacy flag did the trick.  Is there someway to auto-detect 
use of the v2 schema -- for those who are like me and just use what has 
worked in the past without any attention to, or interest in, 
schemas/versions?

Thanks again,

Lou

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Nov 2020 16:52:14 +0100

Hi Lou,

> worked in the past without any attention to, or interest in, 

Not for the 3.x releases.  For that, you should install the last of the 2.x
series -- it should be good enough for most purposes for quite some time.

Best regards,

	Henrik

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 10:57:42 -0500

Thanks for the assist.  I'll change my scripts to use --legacy.

Lou

---

From: Michael Richardson <[email-masked]>
Date: Tue, 03 Nov 2020 14:48:05 -0500

I had the same problem yesterday, ten minutes to the deadline.
I estimate I have it once a week.

I just curl the relevant file and put it into ~/.local/xmlrfc.
I find this:
  obiwan-[~](2.6.6) mcr 17666 %ls -l .cache/xml2rfc/reference.RFC.8765*
  -rw-r--r-- 2 mcr mcr 1064 Aug 28 16:53 .cache/xml2rfc/reference.RFC.8765
  -rw-r--r-- 2 mcr mcr 1064 Aug 28 16:53 .cache/xml2rfc/reference.RFC.8765.xml

to be some kind of pathology that I don't understand.

It's unclear to me from what Hendrik said about the fallback locations if the
root cause problems we are running into are related to cloudflare caching, or
something else.   Clearly it sounds to me like xml2rfc's fallback processing
has been hiding some other instability.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Nov 2020 08:50:45 +1300

Lou,

Why not just switch to v3? At some point that will be necessary anyway, so why not this week?

Whenever I update a draft now, I simply convert it to v3 format and carry on.

There are a couple of gotchas:
1. You need to change <vspace /> manually to something else.
2. The conversion tool includes the current version number in I-D citations; I regard that as a bug and strip them out.

Regards
   Brian Carpenter

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 4 Nov 2020 00:52:47 +0100

I just did this at -29 for my ACP draft, and it was quite painfull and only acceptable
because i didn't have to do a lot of changes afterwards. I did include this as my main
concern against xmlv3 in the LLC questionaire:

The conversion from v2 to v3 replaces user preferred choices with options that
are more difficult to edit manually - even though i think v3 has wqually easily
manually editable choices.

For example, all biography is inlined instead of using the best reference format.
Or title lines are split in two with name tag separated.

Aka: Ideally you would want the v3 conversion to change as little as necessary, at least
as an option. Of course you may always have another tool to convert any v3 into whatever
may be a preferred v3 representation. But thats a different thing.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Lou Berger <[email-masked]>
Date: Tue, 3 Nov 2020 19:13:49 -0500

Kudos to the tools team for --legacy mode which did the trick.

I guess my response to the origin question is -- if it's working why 
would I want to spend my oversubscribed time on a formatting tool 
conversion to produce the same results.

Also a note of special appreciation to Henrik for his professionalism 
and typical fine support on this, I'm sure, very trying day!

Lou

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 4 Nov 2020 01:40:54 +0100

> Kudos to the tools team for --legacy mode which did the trick.

Yepp.

> I guess my response to the origin question is -- if it's working why would I
> want to spend my oversubscribed time on a formatting tool conversion to
> produce the same results.

YOu shouldn't unless at some point in time (hopefully not now) maintenance
of backward compatiblity is considered to be too expensvie for tooling.

I did the upgrade because i wanted to use a nw v3 feature (contact for contributor section).

Next step is really an exit strategy to work directly with XML (markup).

> Also a note of special appreciation to Henrik for his professionalism and
> typical fine support on this, I'm sure, very trying day!

Will become more trying for us when Henrik leaves next year. But i guess we deserve it.
*sigh*

Cheers
   Toerless

-- 
---
[email-masked]

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 4 Nov 2020 15:10:29 +0100

Hi Brian, Lou,

> 1. You need to change <vspace /> manually to something else.

Yes.  Running the v2 to v3 converter (xml2rfc --v2v3) will convert some
instances of <vspace/>, but not all -- the correct replacement varies so
much with what the author intended that I didn't find a way to sort out
all cases programmatically.  A good approach might be to run the conversion
once, and take note of the warnings about <vspace/> instances not converted;
then handle those manually before re-running the conversion.  I think the
cases that _are_ handled are handled in a reasonable manner.

> 2. The conversion tool includes the current version number in I-D
> citations; I regard that as a bug and strip them out.

Right.  There is a bit of a complication here in that the v2 entity
references are resolved by the parser before the v2v3 conversion starts,
so the information about version-agnostic or version-specific URLs is
lost early on.  What I'll do for the next release is to by default
create XInclude statements to the version-agnostic URLs, and add a
switch to retain the version-specific URLs as an alternative.

Best regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 4 Nov 2020 17:29:59 +0100

> switch to retain the version-specific URLs as an alternative.

Which is a nice demonstration of the hard-learned general principle that syntactic features (include) should never be used to encode semantic content (here: bibliographic reference).

(Not addressed to anyone specifically because that child is already solidly embedded in the bottom of the well.)

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 4 Nov 2020 18:13:51 +0100

Hi Michael,

> I just curl the relevant file and put it into ~/.local/xmlrfc.

Umm?  I would think you mean ~/.cache/xml2rfc above, not ".local"?
Otherwise there's a lot of missing information here.

> has been hiding some other instability.

The v2 processing tries to both provide path, URL, and extension fallbacks,
and that could also be reflected in the cache.  If you have some drafts
referring to "reference.RFC.8765" and others to "reference.RFC.8765.xml"
I think you might see what you see above.

Rather than trying to _now_ figure out all the ins and outs of the old v2
processing, I'd really recommend simply providing the full URLs of references
in any new work, and transitioning to v3 and XInclude.

Regards,

	Henrik

---

From: Michael Richardson <[email-masked]>
Date: Wed, 04 Nov 2020 14:02:46 -0500

>> I had the same problem yesterday, ten minutes to the deadline.  I
    >> estimate I have it once a week.
    >>

    > Otherwise there's a lot of missing information here.

Yes, my bad typing rather than copy&pasting.

    >> I find this: obiwan-[~](2.6.6) mcr 17666 %ls -l
    >> .cache/xml2rfc/reference.RFC.8765* -rw-r--r-- 2 mcr mcr 1064 Aug 28
    >> 16:53 .cache/xml2rfc/reference.RFC.8765 -rw-r--r-- 2 mcr mcr 1064 Aug
    >> 28 16:53 .cache/xml2rfc/reference.RFC.8765.xml
    >>
    >> It's unclear to me from what Hendrik said about the fallback locations
    >> if the root cause problems we are running into are related to
    >> cloudflare caching, or something else.  Clearly it sounds to me like
    >> xml2rfc's fallback processing has been hiding some other instability.

    > The v2 processing tries to both provide path, URL, and extension
    > fallbacks, and that could also be reflected in the cache.  If you have
    > some drafts referring to "reference.RFC.8765" and others to
    > "reference.RFC.8765.xml" I think you might see what you see above.

okay.

    > Rather than trying to _now_ figure out all the ins and outs of the old
    > v2 processing, I'd really recommend simply providing the full URLs of
    > references in any new work, and transitioning to v3 and XInclude.

I'm using kramdown for mostly everything, so I'm gonna let Carsten do that.
The output is v2 for now, and I upgrade it with --v2v3 in every place I control.

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 4 Nov 2020 22:18:43 +0100

Hi Michael,

> The output is v2 for now, and I upgrade it with --v2v3 in every place I control.

Ok, I see how that limits some options.  Is it not possible to use full URLs 
references, though?

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 4 Nov 2020 22:28:44 +0100

> Ok, I see how that limits some options.  Is it not possible to use full URLs 

There is never a need to use references in the XML with kramdown-rfc.
(Use `stand_alone: true`)
If you do want to use unresolved references in the XML (`stand_alone: false`) for some reason, kramdown-rfc generates entity declarations and not PIs or xinclude.  Entity declarations need full URIs and therefore use them:

<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
<!ENTITY RFC0793 SYSTEM "https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.0793.xml">
...

So I don’t think kramdown-rfc is part of the problem here.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 04 Nov 2020 18:37:57 -0500

Carsten Bormann <[email-masked]> wrote:
    >>>> Rather than trying to _now_ figure out all the ins and outs of the
    >>>> old v2 processing, I'd really recommend simply providing the full
    >>>> URLs of references in any new work, and transitioning to v3 and
    >>>> XInclude.
    >>>
    >>> I'm using kramdown for mostly everything, so I'm gonna let Carsten do
    >>> that.  The output is v2 for now, and I upgrade it with --v2v3 in
    >>> every place I control.
    >>
    >> Ok, I see how that limits some options.  Is it not possible to use
    >> full URLs references, though?

    > (Use `stand_alone: true`) If you do want to use unresolved references
    > in the XML (`stand_alone: false`) for some reason, kramdown-rfc
    > generates entity declarations and not PIs or xinclude.  Entity
    > declarations need full URIs and therefore use them:

    > So I don’t think kramdown-rfc is part of the problem here.

I mostly go with stand_alone: true now.

I do not remember where I had problems, it might have been older documents
which were in XML only.

But, my original point stands: it seems that the fallback process in v2
processing of xml2rfc has been hiding some instability in our archives of reference.*

---

From: Lou Berger <[email-masked]>
Date: Thu, 5 Nov 2020 01:47:26 +0000

Henrik,

Thank you for the info.

Lou

----------
On November 4, 2020 9:10:44 AM Henrik Levkowetz <[email-masked]> wrote:

Hi Brian, Lou,

Lou,

Best regards,

Henrik

Regards
Brian Carpenter

Hi Henrik,

Thanks again,

Lou

Hi Lou,

Thank you.

Best regards,

Henrik

Lou

Hi Lou,

Hi Henrik,

Henrik

Lou

Hi Lou,

Hi,

Name: xml2rfc
Version: 3.3.0

Henrik

Please report datatracker.ietf.org<http://datatracker.ietf.org> and mailarchive.ietf.org<http://mailarchive.ietf.org>
bugs at http://tools.ietf.org/tools/ietfdb
or send email to [email-masked]

Please report tools.ietf.org<http://tools.ietf.org> bugs at
http://tools.ietf.org/tools/issues
or send email to [email-masked]

Please report datatracker.ietf.org<http://datatracker.ietf.org> and mailarchive.ietf.org<http://mailarchive.ietf.org>
bugs at http://tools.ietf.org/tools/ietfdb
or send email to [email-masked]

Please report tools.ietf.org<http://tools.ietf.org> bugs at
http://tools.ietf.org/tools/issues
or send email to [email-masked]

Please report datatracker.ietf.org<http://datatracker.ietf.org> and mailarchive.ietf.org<http://mailarchive.ietf.org>
bugs at http://tools.ietf.org/tools/ietfdb
or send email to [email-masked]

Please report tools.ietf.org<http://tools.ietf.org> bugs at
http://tools.ietf.org/tools/issues
or send email to [email-masked]