Subject: [xml2rfc] Problem with submitting xml

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 09:56:53 -0400

I am submitting the attached xml draft at 
https://datatracker.ietf.org/submit/

I get errors like:

/tmp/draft-moskowitz-drip-operator-privacy-08-lx98crix.xml(0): Error: 
IDREF attribute target references an unknown ID "I-D.ietf-drip-reqs", at 
None

I have no problem with the local xml2rfc with the --v3 option or online 
at: https://xml2rfc.tools.ietf.org

What is submit doing wrong???

thanks

---

From: Julian Reschke <[email-masked]>
Date: Wed, 6 Oct 2021 16:09:05 +0200

> What is submit doing wrong???

The attached document looks good to me (so far :-), but it seems to be
different from the one you are trying to submit... (judging from the
filename).

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 10:18:49 -0400

> different from the one you are trying to submit... (judging from the

This is strange.  I tried it yet again, and got a different file name there.

/tmp/draft-moskowitz-drip-operator-privacy-08-mzufzex8.xml(13): Error: 
Invalid document before running preptool.

In the browse line, the "proper" file name is showing.

Something is adding that 8 character piece to the name.

---

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 10:25:05 -0400

And I just tried it from a Win system also with Firefox, and got the 
same failure.

Of course with a different 8 char addition to the file name.

On 10/6/21 10:18 AM, Robert Moskowitz wrote:
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Jennifer Richards <[email-masked]>
Date: Wed, 6 Oct 2021 11:25:44 -0300

> Something is adding that 8 character piece to the name.

The Django file upload mechanism will add the random string like that to 
a filename if there's a filename collision.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 6 Oct 2021 16:30:10 +0200

> Something is adding that 8 character piece to the name.

Ah.

Probably on purpose, to avoid race conditions in the file system.

It seems that the online version fails to resolve the xi:include for
that symbol. Getting
<https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs/xml> and
pasting it into the source (minus the XML declaration) fixes it for me.
(In case you're in a hurry to submit).

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 10:33:29 -0400

> (In case you're in a hurry to submit).

Well not really in a hurry.  I got all day.   :)
Really all week.  Still.

It works with https://xml2rfc.tools.ietf.org/ but not the submit tool?

Seems something wrong there.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 6 Oct 2021 16:36:50 +0200

Am 06.10.2021 um 16:33 schrieb Robert Moskowitz:
> ...

Absolutely.

Best regards, Julian

---

From: Jennifer Richards <[email-masked]>
Date: Wed, 6 Oct 2021 11:40:50 -0300

> Probably on purpose, to avoid race conditions in the file system.

The first upload for a given name will appear as uploaded, only 
subsequent files will be given the random suffix.

It seems likely that something is not being cleaned up properly and the 
datatracker is not properly handling the case where the saved filename 
is modified. I think that would cause it to ignore the new file and keep 
trying to process the original file.

---

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 11:26:31 -0400

>> Seems something wrong there.

So what might be the problem?  ietf-drip-reqs is not yet approved for 
RFC and thus maybe no longer available as a draft.  Why is the submit 
script not getting that draft's info?

draft-ietf-drip-rid worked back on 9-14 to reference the reqs draft...  :)

---

From: Julian Reschke <[email-masked]>
Date: Wed, 6 Oct 2021 18:12:38 +0200

Am 06.10.2021 um 17:26 schrieb Robert Moskowitz:
> ... >> Absolutely.

It *is* strange. With a local install of xml2rfc and --expand, the
following line

   <xi:include
href="https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs/xml"/>

apparently gets expanded to:

>      <reference anchor="I-D.ietf-httpbis-priority" xml:base="https://datatracker.ietf.org/doc/bibxml3/draft-ietf-httpbis-priority/xml">
>         <front>
>           <title>Extensible Prioritization Scheme for HTTP</title>
>           <author fullname="Kazuho Oku">
>             <organization>Fastly</organization>
>           </author>
>           <author fullname="Lucas Pardue">
>             <organization>Cloudflare</organization>
>           </author>
>           <date month="September" day="24" year="2021"/>
>           <abstract>
>             <t>   This document describes a scheme for prioritizing HTTP responses.
>    This scheme expresses the priority of each HTTP response using
>    absolute values, rather than as a relative relationship between a
>    group of HTTP responses.
>
>    This document defines the Priority header field for communicating the
>    initial priority in an HTTP version-independent manner, as well as
>    HTTP/2 and HTTP/3 frames for reprioritizing the responses.  These
>    share a common format structure that is designed to provide future
>    extensibility.
>
>             </t>
>           </abstract>
>         </front>
>         <seriesInfo name="Internet-Draft" value="draft-ietf-httpbis-priority-05"/>
>         <format type="TXT" target="https://www.ietf.org/archive/id/draft-ietf-httpbis-priority-05.txt"/>
>       </reference>

which of course produces reference errors.

After clearing the cache (xml2rfc -C) everything seems to be ok.

The simplest explanation would be that at some point the past,
<https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs/xml> had
content for a different draft. Or there's a strange bug in xml2rfc.

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Wed, 6 Oct 2021 12:55:42 -0400

I changed:

     <xi:include 

to:

     <xi:include 
href="https://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-drip-reqs.xml"/>

And it works.

Why?

On 10/6/21 12:12 PM, Julian Reschke wrote:
>>      <reference anchor="I-D.ietf-httpbis-priority" 
>> xml:base="https://datatracker.ietf.org/doc/bibxml3/draft-ietf-httpbis-priority/xml">
>>         <front>
>>         <format type="TXT" 
>> target="https://www.ietf.org/archive/id/draft-ietf-httpbis-priority-05.txt"/>
>>       </reference>
>
>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 6 Oct 2021 20:20:39 +0200

> href="https://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-drip-reqs.xml"/>

If the theory about a bad file/cache entry is correct, then of course
changing to a different URI will solve that.

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Wed, 6 Oct 2021 17:54:07 -0500

Right this second, 
https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs/xml 
expands to what looks like the right thing.

There is a possible hint in the existence of a file named simply 'xml' 
in the xml2rfc cache directory that the datatracker submit function 
uses, that has the odd content Julian points to (it's from 23Sep fwiw).

Also reference.I-D.ietf-drip-reqs.xml in that cache directory has 
content that reflects the drip-reqs draft.

We'll go look through the xml2rfc code to see what codepath that might 
result in reading/writing a filename as simply "xml", which is my best 
current guess as to what Bob ran into.

RjS

---

From: Robert Sparks <[email-masked]>
Date: Wed, 6 Oct 2021 18:12:46 -0500

(And I did remove that stray file from the cache)

---

From: Robert Sparks <[email-masked]>
Date: Wed, 6 Oct 2021 18:47:21 -0500

Ah -

So, the reference bob should have been using is 
https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs.xml

that is _dot_ xml instead of _slash_ xml.

That the /xml variant worked at the datatracker was unintended, and it's 
directly leading to the issue that's appearing in cache.

RjS

p.s. - the thing that let the slash work at the bibxml endpoint is the 
failure to escape the . in this regexp:

     url(r'^bibxml3/%(name)s(?:-%(rev)s)?.xml$' % settings.URL_REGEXPS, 
views_doc.document_bibxml),

On 10/6/21 6:12 PM, Robert Sparks wrote:
>> There is a possible hint in the existence of a file named simply 
>> 'xml' in the xml2rfc cache directory that the datatracker submit 
>> function uses, that has the odd content Julian points to (it's from 
>> 23Sep fwiw).
>>
>> We'll go look through the xml2rfc code to see what codepath that 
>> might result in reading/writing a filename as simply "xml", which is 
>> my best current guess as to what Bob ran into.
>>
>> RjS
>>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 7 Oct 2021 03:27:33 +0200

On 7. Oct 2021, at 01:47, Robert Sparks <[email-masked]> wrote:
> 
> Ah -
> 
> So, the reference bob should have been using is https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs.xml
> 
> That the /xml variant worked at the datatracker was unintended, and it's directly leading to the issue that's appearing in cache.

Thanks.  I just pushed a fix for kramdown-rfc that no longer uses the /xml form (1.5.8).

In case some documentation needs to be updated, I tried to find where I picked up the /xml form.
I used it in a private email I sent *to* Henrik on 2020-02-11.
I haven’t found the message where I actually learned about the datatracker feature (which must have used /xml).

(BTW: kramdown-rfc-cache-i-d-bibxml was bulk-updating from …/xml all the time, and I never saw that caching glitch we had today.)

Grüße, Carsten

> 
> RjS
> 
> p.s. - the thing that let the slash work at the bibxml endpoint is the failure to escape the . in this regexp:
> 
>     url(r'^bibxml3/%(name)s(?:-%(rev)s)?.xml$' % settings.URL_REGEXPS, views_doc.document_bibxml),
> 
> 
>>> Right this second, https://datatracker.ietf.org/doc/bibxml3/draft-ietf-drip-reqs/xml expands to what looks like the right thing.
>>> 
>>> There is a possible hint in the existence of a file named simply 'xml' in the xml2rfc cache directory that the datatracker submit function uses, that has the odd content Julian points to (it's from 23Sep fwiw).
>>> 
>>> Also reference.I-D.ietf-drip-reqs.xml in that cache directory has content that reflects the drip-reqs draft.
>>> 
>>> We'll go look through the xml2rfc code to see what codepath that might result in reading/writing a filename as simply "xml", which is my best current guess as to what Bob ran into.
>>> 
>>> RjS
>>>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 7 Oct 2021 08:54:48 +0200

> views_doc.document_bibxml),

Thanks a lot for the quick analysis!

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 7 Oct 2021 11:53:52 +0200

> Thanks a lot for the quick analysis!

I think we knew about this bug (accepting /xml) for a while, but it wasn’t fixed because the /xml form was in use.  We didn’t know that this can lead to the caching bug, and given that we now have experienced it, the choice is

(1) disabling /xml, maybe with some transition time
(2) fixing the caching bug

The caching bug doesn’t seem to occur in normal operation of kramdown-rfc, which has been using /xml by default for about half a year now (fixed in 1.5.8).
I suspect an interaction between the xinclude function of the submission interface and the caching logic made this bug appear with Bob’s input — the submission input generated by kramdown-rfc output does not exercise xinclude, so we wouldn’t have noticed the caching bug.

Note that kramdown-rfc does not typically create an error when it gets the wrong answer (if the caching bug were triggered by it), as it ignores the anchor= attribute (replacing it with an anchor chosen by the author, which for historical reasons defaults to what *should* be in the reference XML).  But of course the wrong output should be visible in the draft’s bibliography.

Grüße, Carsten

---

From: Robert Moskowitz <[email-masked]>
Date: Thu, 7 Oct 2021 06:05:57 -0400

I got a comment on the latest ver and I will be making an update soon. 
Hopefully this week, after I figure out the best way to fix the crypto.  
I really don't want to use CFB16, but I might not have a choice...

So I will also make this fix.

On 10/6/21 7:47 PM, Robert Sparks wrote:
> That the /xml variant worked at the datatracker was unintended, and 
> it's directly leading to the issue that's appearing in cache.
>
> RjS
>

---

From: Robert Moskowitz <[email-masked]>
Date: Thu, 7 Oct 2021 10:10:46 -0400

Just successfully submitted ver -09 using .xml

I don't know what I did to get the /xml variant, and you said, it worked 
so I just was working with it for some time...

thanks for the assistance all.