Subject: [Tools-discuss] Where are the IETF draft XML ?

From: Toerless Eckert <[email-masked]>
Date: Wed, 16 Jun 2021 16:18:43 +0200

Is it possible that the new world of moving from  tools.ietf.org to datatracker.ietf.org
just made it more difficult to get to the XML of IETF drafts ?

e.g.: https://datatracker.ietf.org/doc/html/draft-ietf-anima-autonomic-control-plane-27
(or any other draft for that matter)
shows no xml, and of course i do know there was when this was available from tools.ietf.org.

Hasn't been that long that draft access on tools.ietf.org was retired/redirected to datatracker,
that's why i only notice this now.

RFCs on datatracker have their xml shown, but that is sourced from rfc-editor.

Cheers
    Toerless

---

From: Lars Eggert <[email-masked]>
Date: Wed, 16 Jun 2021 17:28:29 +0300

On 2021-6-16, at 17:18, Toerless Eckert <[email-masked]> wrote:
> shows no xml

The [Tracker] link takes you to https://datatracker.ietf.org/doc/rfc8994/, which has an "XML" button under "Formats".

Thanks,
Lars

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Jun 2021 16:32:10 +0200

> (or any other draft for that matter)

Not true:
<https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-cache-16>

> ...

It's only available if it was submitted by the authors. Was it?

Best regards, Julian

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 16 Jun 2021 16:33:25 +0200

> RFCs on datatracker have their xml shown, but that is sourced from rfc-editor. 

I was asking about xml for drafts.

Cheers
    Toerless

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Jun 2021 16:36:00 +0200

Am 16.06.2021 um 16:32 schrieb Julian Reschke:
>> e.g.:
>> https://datatracker.ietf.org/doc/html/draft-ietf-anima-autonomic-control-plane-27
>>

Aha.

What seems to happen is that the XML link disappears for archived drafts.

Try

https://www.ietf.org/archive/id/draft-ietf-anima-autonomic-control-plane-27.xml

Best regards, Julian

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 16 Jun 2021 16:51:32 +0200

Updating subject...

Yes, i always submitted the XML.

Theory:
  That draft-ietf-httpbis-cache-16 is not an RFC yet.  Seems like datatracker does
  not offer links to the XML of drafts once they have become RFC. 

The XML files for drafts do seem to be still directly accessible if you know the path:
   https://www.ietf.org/archive/id/draft-ietf-anima-autonomic-control-plane-27.xml

I would call this a datatracker bug. No XML, no TXT rendering provided for drafts.

How the heck am i going to do research work into the past,
i often had to do rfcdiff against older drafts. That requires access
to draft text versions, it will not work with just the html'ized versions.

Do i need to rsync the whole tools.id repo now to do this ?!

*sigh*

Toerless

On Wed, Jun 16, 2021 at 04:32:10PM +0200, Julian Reschke wrote:
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss

-- 
---
[email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Wed, 16 Jun 2021 18:08:44 +0300

Hi,

> Do i need to rsync the whole tools.id repo now to do this ?!

If you think this is a missing feature in the datatracker, please submit a ticket? https://trac.ietf.org/trac/ietfdb/newticket

Also, I'd like to ask you to adopt a more constructive attitude.

Thanks,
Lars

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 16 Jun 2021 17:57:35 +0200

> Do i need to rsync the whole tools.id repo now to do this ?!

I don’t think any Indonesian websites will help here.

Run this from your crontab:

cd
rsync -abvz rsync.ietf.org::internet-drafts/ std/id
rsync -abvz --exclude '/tar/*' ftp.rfc-editor.org::rfc-ed-all/ std/rfc
rsync -avzH rsync.iana.org::assignments std/iana
rsync -avzH xml2rfc.tools.ietf.org::xml2rfc.bibxml std/bib
rsync -avzH rfc-editor.org::refs std/bib/rfc
rsync -a --include=/refer\* --exclude="*" std/bib/rfc/ .cache/xml2rfc
kramdown-rfc-cache-i-d-bibxml

That will significantly reduce the time you need to click around in random web interfaces.

Grüße, Carsten

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 16 Jun 2021 18:04:40 +0200

> If you think this is a missing feature in the datatracker, please submit a ticket? https://trac.ietf.org/trac/ietfdb/newticket

Thanks. I was asking yesterday already in another thread and nobody seemed to know the URL,
i remember i opened a bunch of cases in the past but couldn't find the URL anymore (bad housekeeping i know)
Maybe the URL is a bit hidden...

Created:
  Regresson bug: html'ized text renderings miss hotlinks
    https://trac.ietf.org/trac/ietfdb/ticket/3307

  Regression: tools->datatracker: no access to drafts XML from datatracker
    https://trac.ietf.org/trac/ietfdb/ticket/3308

  Regression: tools->datatracker: no access to drafts TXT from datatracker
    https://trac.ietf.org/trac/ietfdb/ticket/3309

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 17 Jun 2021 03:38:11 +0200

On Wed, Jun 16, 2021 at 05:57:35PM +0200, Carsten Bormann wrote:
> I don???t think any Indonesian websites will help here.

So you are not registering your draft names as domains under .ID ? How lame.

Kidding aside:

Your rsync list you have does not sync drafts that are not current, e.g.: expired
or that have become RFC. Those where the ones i was primarily inquiring about.

That full set of past drafts can be synced from
rsync.tools.ietf.org::tools.id.  Given how tools.ietf.org is likely condemned
and/or maybe not maintained well anymore, i just checked:

  -  rsync.ietf.org::id-archive seems to have almost all the same files,
     just missing very few.

  -  There is also rsync.ietf.org::xml2rfc.bibxml hopefully replacing 
     xml2rfc.tools.ietf.org::xml2rfc.bibxml that you list below.

And yes, you and i can use rsync, but do you think that should be the
only choice to get to all current and past draft files ?

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 17 Jun 2021 08:14:41 +0200

> or that have become RFC. Those where the ones i was primarily inquiring about.

Actually, it does (because it runs periodically, it syncs them while they are still current).
But I agree that doesn’t help if you haven’t been doing this for a decade or more.

>     xml2rfc.tools.ietf.org::xml2rfc.bibxml that you list below.

Thanks — these are useful updates for my list.
(Tools team:
It would be useful to have some documentation for what these are supposed to be, beyond the single-line summaries you get with
rsync rsync.ietf.org::
)

> only choice to get to all current and past draft files ?

To get all drafts?  Yes, rsync is the choice.

To get any specific draft, say, in XML form?

https://www.google.com/search?q=draft-ietf-anima-autonomic-control-plane-27+xml
➔
Tracker
➔
Xml

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Jun 2021 08:48:10 +0200

> https://www.google.com/search?q=draft-ietf-anima-autonomic-control-plane-27+xml

Yes, but that's what's broken right now. XML is only linked for RFCs and
current drafts.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 17 Jun 2021 11:14:04 +0200

> Yes, but that's what's broken right now. XML is only linked for RFCs and

Got it.

(I had missed the fact that the tracker link doesn’t point to the specific version.)

Now if I could figure out why

is a 404 in Safari (with “not found” page), but not in Chrome, and curl -I says

$ curl -I https://www.ietf.org/archive/id/draft-ietf-anima-autonomic-control-plane-27.xml
HTTP/2 200
date: Thu, 17 Jun 2021 09:07:25 GMT
content-type: text/xml
last-modified: Thu, 02 Jul 2020 12:52:54 GMT
etag: W/"60775-5a974e310b908-gzip"
vary: Accept-Encoding
strict-transport-security: max-age=31536000; preload
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
content-security-policy: frame-ancestors 'self' ietf.org *.ietf.org meetecho.com *.meetecho.com gather.town *.gather.town
x-frame-options: ALLOW-FROM ietf.org *.ietf.org meetecho.com *.meetecho.com gather.town *.gather.town
access-control-allow-origin: *
access-control-allow-headers: *
cf-cache-status: HIT
age: 66843
expires: Thu, 17 Jun 2021 10:07:25 GMT
cache-control: public, max-age=3600
cf-request-id: 0abad2bc0f00004e44deac1000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 660b20a678ff4e44-FRA

Oh.  Safari says:

[Error] Failed to load resource: the server responded with a status of 404 () (rfc2629.xslt, line 0)

And shows *that* error page.
The wonders of the Web.

Grüße, Carsten

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 17 Jun 2021 21:11:30 -0500

I'm doing my best to ask the smartest stupid questions I can phrase ... :-)

> https://www.ietf.org/archive/id/draft-ietf-anima-autonomic-control-plane-27.xml

So, I happen to have added two new references (in kramdown-rfc2629, but
let's not talk about my tool choices for now).

They are

  I-D.hurst-quic-rtp-tunnelling:
  I-D.rtpfolks-quic-rtp-over-quic:

  I-D.hurst-quic-rtp-tunnelling: works fine - it's not expired (yet).
  I-D.rtpfolks-quic-rtp-over-quic: is a couple of years old, so expired,
and gives me roughly

spencer@Spencer-Dell:/mnt/c/users/spencer/documents/github/rtp-quic-sdp$
make
cat draft-dawkins-rtp-quic-sdp.md  | kramdown-rfc2629 --v3 |
lib/add-note.py | xml2rfc -q -s 'Setting consensus="true" for IETF STD
document' --rfc-base-url https://datatracker.ietf.org/doc/html/
--id-base-url https://datatracker.ietf.org/doc/html/
--cache=/home/spencer/.cache/xml2rfc --v2v3 /dev/stdin -o /dev/stdout
>draft-dawkins-rtp-quic-sdp.xml
Traceback (most recent call last):

          5: from /usr/local/bin/kramdown-rfc2629:23:in `<main>'

          4: from /usr/local/bin/kramdown-rfc2629:23:in `load'

          3: from
/var/lib/gems/2.7.0/gems/kramdown-rfc2629-1.4.15/bin/kramdown-rfc2629:467:in
`<top (required)>'
           2: from
/var/lib/gems/2.7.0/gems/kramdown-2.3.1/lib/kramdown/document.rb:116:in
`method_missing'
           1: from
/var/lib/gems/2.7.0/gems/kramdown-2.3.1/lib/kramdown/converter/base.rb:109:in
`convert'
 /var/lib/gems/2.7.0/gems/kramdown-2.3.1/lib/kramdown/converter/base.rb:109:in
`encode!': U+00F6 from UTF-8 to US-ASCII
(Encoding::UndefinedConversionError)

Error: Unable to parse the XML document: /dev/stdin

 make: *** [lib/main.mk:69: draft-dawkins-rtp-quic-sdp.xml] Error 1

make: *** Deleting file 'draft-dawkins-rtp-quic-sdp.xml'

spencer@Spencer-Dell:/mnt/c/users/spencer/documents/github/rtp-quic-sdp$

(I know this, because I tried adding them one at a time)

If the expired draft is the one that blows up, is that perhaps related to
the discussion about the XML link disappearing for archived drafts?

I checked where Julian pointed me, and
https://www.ietf.org/archive/id/draft-rtpfolks-quic-rtp-over-quic-01.xml
does exist there.

Best,

Spencer

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 05:15:10 +0200

On 18. Jun 2021, at 04:11, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> I-D.rtpfolks-quic-rtp-over-quic

Probably less about being expired that about being old enough to have a charset problem.
I can’t reproduce that, though.
Can you send me the reference.I-D.rtpfolks-quic-rtp-over-quic.xml that is in .refcache (or in ~/.cache/xml2rfc)?

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 11:49:52 +0200

> Can you send me the reference.I-D.rtpfolks-quic-rtp-over-quic.xml that is in .refcache (or in ~/.cache/xml2rfc)?

Well, before I get that, I note that my cached copy of that says “Joerg” and the one I get now from datatracker says “Jörg” — ö is U+00F6.

So we do have a smoking gun here.

(1) I need to find out why the beyond-ASCII character causes a problem for you.

(2) People should be alert that we have non-transliterated names in references now (even expired ones), and there might be interesting consequences.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jun 2021 12:05:03 +0200

> (2) People should be alert that we have non-transliterated names in references now (even expired ones), and there might be interesting consequences.

We had those before, and, unless I'm missing something, that change was
supposed to be reverted...

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 12:36:04 +0200

On 2021-06-18, at 12:05, Julian Reschke <[email-masked]> wrote:
> supposed to be reverted…

Not sure I still believe that.

The bibxml files need to be useful in a v2 and a v3 environment; I don’t think that either has a problem with Latin names.  (Kramdown-rfc in “coding: ascii” mode has a problem, but that may simply no longer be a useful mode of operation.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jun 2021 13:24:10 +0200

> The bibxml files need to be useful in a v2 and a v3 environment; I don’t think that either has a problem with Latin names.  (Kramdown-rfc in “coding: ascii” mode has a problem, but that may simply…

Wasn't there agreement that the reference files should reflect how the
names are actually spelled on the referenced document?

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 13:29:46 +0200

> names are actually spelled on the referenced document?

OK, got it.

The notable thing for tools authors is that this indeed sets us up for a future where bibxml files are no longer pure-ASCII, so heroic attempts to support pre-UTF-8 working environments completely stop making sense.

Grüße, Carsten

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 18 Jun 2021 06:54:10 -0500

Hi, Carsten,

Thank you and Julian for the clues!

I don't seem to have the character strings "ascii" or "coding:" in my local
repository directory, including "not in my markdown".

My markdown is attached.

You' may be curious about this - I'm running

kramdown version 2.3.1
kramdown-rfc2629 version 1.4.15
xml2rfc version 3.8.0
gem version 3.1.2
ruby version 2.7.0p0

(and, obviously other software, but I don't think it got that far)

Best,

Spencer

On Fri, Jun 18, 2021 at 6:30 AM Carsten Bormann <[email-masked]> wrote:

> The notable thing for tools authors is that this indeed sets us up for a
> future where bibxml files are no longer pure-ASCII, so heroic attempts to
> support pre-UTF-8 working environments completely stop making sense.
>
> Grüße, Carsten
>
> List info (including how to Unsubscribe):
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 13:58:14 +0200

> On 2021-06-18, at 13:54, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> I don't seem to have the character strings "ascii" or "coding:" in my local repository directory, including "not in my markdown". 

Actually, it says:

coding: us-ascii

(line 13).  Commenting out that line make it work for me.

Clearly, it doesn’t make sense to have an option the main semantics of which now is to break the build.

May I ask: Where did you get that “coding: us-ascii” line from?

Grüße, Carsten

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 18 Jun 2021 07:15:51 -0500

Hi, Carsten,

On Fri, Jun 18, 2021 at 6:58 AM Carsten Bormann <[email-masked]> wrote:

> > I don't seem to have the character strings "ascii" or "coding:" in my
> local repository directory, including "not in my markdown".
>
> Actually, it says:
>
> coding: us-ascii
>

Thank you for the eye test, and I'll refrain from sending email before 7 am
local time  in the future, because that's clearly not helpful!

(I just checked again, and windows explorer search STILL doesn't find those
character strings)

The markdown is also working for me now.

Thank you for that.

> May I ask: Where did you get that “coding: us-ascii” line from?

I'd hate to think how many generations of that markdown file might have
been represented in setting up my current repository, but I know it's not
an intentional insertion on my part.

I've already removed it from another of my draft repos, so thank you for
that one, too!

Best,

Spencer

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jun 2021 14:29:27 +0200

On 2021-06-18, at 14:15, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> I've already removed it from another of my draft repos, so thank you for that one, too!

Yes, that is the recommended way forward.

I put in a fix into 1.4.17 that still allows the use of `coding: (us-)ascii` in case people really still need this with their dusty decks, but that feature from around 2012-01-14 clearly is “NOT RECOMMENDED” now.

Thanks for the alert!

Now for reverting the datatracker-generated bibxml…

$ curl https://datatracker.ietf.org/doc/bibxml3/draft-rtpfolks-quic-rtp-over-quic/xml
[…]
      <author fullname="Jörg Ott">
	 <organization>Technische Universitaet Muenchen</organization>
[…]

(Of course, the current state gets doubly funny with the transliterated “Universitaet Muenchen” in there…)

Grüße, Carsten