Subject: [dispatch] RFC 3896 and 3987 vs WHATWG URL Living Standard

From: Larry Masinter <[email-masked]>
Date: Sat, 5 Jun 2021 15:35:05 -0700

Recent progress on WHATWG's URL spec led me to suggest a BOF on the topic
for IETF 111

Provide a succinct grammar for valid URL strings
<https://github.com/whatwg/url/issues/479> . Issue #479 . whatwg/url
(github.com)

But I didn't get an ack, and perhaps this belongs in DISPATCH/ART etc.

--

https://LarryMasinter.net https://interlisp.org

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 Jun 2021 22:21:14 -0400

--On Saturday, June 5, 2021 15:35 -0700 Larry Masinter
<[email-masked]> wrote:

> Recent progress on WHATWG's URL spec led me to suggest a BOF
> on the topic for IETF 111
> 
> <https://github.com/whatwg/url/issues/479> . Issue #479 .
> whatwg/url (github.com)  
> 

Larry,

This is a question rather than a statement of preference because
I have not thought about it in some time, but are you thinking
it may be time to either revise or deprecate 3896 and 3897?  I
may not have enough information but it seems clear to me that,
in terms of the specs to which implementers are paying
attention, the action has shifted toward WHATWG (and, to a
lesser extent, W3C) and away from the IETF documents.

If deprecating the IETF URI and IRI specs is really the question
(or part of it), I'd think either a BOF or a well-prepared
presentation and a rather large block of time in DISPATCH would
be appropriate.

best,
  john

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 8 Jun 2021 16:37:14 +1000

My sense right now is that it's not (yet) appropriate to deprecate 3986/7.

Even if we could ignore the non-Web folks using URIs and IRIs (and there are perhaps good arguments for considering that, IMO, although I know that others have strong feelings, and there's a lot of history here), the current WHATWG specification doesn't specify them -- it specifies how to get from an arbitrary string to what a browser considers a URL to be. It doesn't specify grammar, and it doesn't cover many of the things that 3986 in particular does.

The issue that Larry links to seems to be taking tentative steps towards aligning the grammar in 3986/7 with what WHATWG does, which is great. If that's incorporated, I think there's still a significant gap that would need to be filled before we can deprecate. Even if that gap were to be filled, I question whether the WHATWG is an appropriate home for the specification -- although obviously the W3C has come to peace with HTML living there.

Looking at this from a different angle, we could also ask ourselves whether 3986/7 should be updated to align with WHATWG URL. I think the answer to that is likely to be no -- not only is the delta apparently very small (according to the comments on the linked issue), alignment would mean doing things like considering <> to be valid content in URIs. Again, what they're specifying is how to get from an arbitrary string to a URI, not the syntax itself.

Actually, one useful thing we *could* do would be to replace section 1.1.3 of RFC3986 with something that actually helps people understand the differences between all of these things. If we ceded 'URL' to the WHATWG, leaving 3986 to define URI, it might help.

Cheers,

> On 6 Jun 2021, at 12:21 pm, John C Klensin <[email-masked]> wrote:
> 
> 
> 
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch

--
Mark Nottingham   https://www.mnot.net/

---

From: Alwin Blok <[email-masked]>
Date: Tue, 8 Jun 2021 14:43:39 +0200

Hello, 

I have recently subscribed to this mailing list after seeing Larry Masinter's message on the issue tracker of the WHATWG URL Standard <https://github.com/whatwg/url/issues/479#issuecomment-855303022>. 

I have done a lot of research on the WHATWG URL standard. Some of the results can be seen on my personal GitHub page here <https://github.com/alwinb/url-specification> and here <https://github.com/alwinb/spec-url>. I'm doing that work as an individual, I'm not part of a WHATWG steering group. I mention that just in case, to prevent confusion. 

I believe that an addendum to RFC3986 (URI) and RFC3987 (IRI) that specifies the WHATWG parsing and resolution behaviour, would be _very_ valuable. Another, maybe preferable option would be to combine RFC3986 (URI) and RFC3987 (IRI) into a new document that specifies elementary operations that can be easily combined to accurately and exactly reproduce the WHATWG behaviour. 

Personally I think that the WHATWG standard would improve greatly if the 'basic-url-parser' is refactored and split up into spearate parsing, resolving, normalising and encoding operations. I am doubtful that this will happen at all, though I do estimate that over time support for relative URLs will have to be added. 

I have thought a lot about how to resolve the, difficult situation and I've come to the conclusion that the WHATWG and the IETF can actually (I am serious) complement each other well here. The style of the IETF is more suitable for specifying the elementary operations and the deeper structure, whereas the WHATWG is limited in that area and can instead focus on the step-by-step instructions in pseudocode and on the specification of the web APIs.

The most important thing will be, that:

1) Any new IETF effort at an updated RFC3986/ RFC3987 specifies elementary operations that can be used by the WHATWG to implement the WHATWG behaviour, accurately and exactly, and

2) That the WHATWG acknowledges the new IETF effort, and explicitly points out that the behaviour is equivalent to theirs. This is similar to what has been done in the WHATWG Encoding Standard <https://encoding.spec.whatwg.org/#utf-8> for UTF8. I will add that personally I think that it is good for readers, but also as a gesture to the IETF, to make such a statement a bit more prominent.

All of this, is just an outline, of course, and it does require a lot of effort, still. 
But I believe that it can be done and that it is worthwhile. 

Regards,
Alwin Blok

---

From: Larry Masinter <[email-masked]>
Date: Wed, 9 Jun 2021 17:39:58 -0700

The question is: is there enough interest to have a BOF at IETF 111 to
discuss
a plan for moving forward? I don't imagine starting out with a presentation
of
a solution. 

  I'd probably want to focus on "minimum viable product":
What things could we do first with minimum effort that would improve the
current state?  A replacement for RFCs 3986 and 3987 wouldn't be my
first step. Maybe a short standards-track "UPDATES" that points to/contains
deltas or a new grammar that is more consistent with (tested)
implementations.
And a pointer to WHATWG-URL, whether normative or not I'm not sure.

That's a little more than Mark's "Section 1.1.3" because it makes normative
changes but (apparently) minimal ones.

I was encouraged to see WHATWG paying attention to feedback from curl
and node.js as implementations of URLs, not just browsers.

Larry

> -----Original Message-----
> From: Mark Nottingham <[email-masked]>
> Sent: Monday, June 7, 2021 11:37 PM
> To: John C Klensin <[email-masked]>
> Cc: Larry Masinter <[email-masked]>; [email-masked]
> Subject: Re: [dispatch] RFC 3896 and 3987 vs WHATWG URL Living Standard
> 
> Even if we could ignore the non-Web folks using URIs and IRIs (and there
are
> perhaps good arguments for considering that, IMO, although I know that
> others have strong feelings, and there's a lot of history here), the
current
> WHATWG specification doesn't specify them -- it specifies how to get from
> an arbitrary string to what a browser considers a URL to be. It doesn't
specify
> grammar, and it doesn't cover many of the things that 3986 in particular
does.
> 
> The issue that Larry links to seems to be taking tentative steps towards
> aligning the grammar in 3986/7 with what WHATWG does, which is great. If
> that's incorporated, I think there's still a significant gap that would
need to be
> filled before we can deprecate. Even if that gap were to be filled, I
question
> whether the WHATWG is an appropriate home for the specification --
> although obviously the W3C has come to peace with HTML living there.
> 
> Looking at this from a different angle, we could also ask ourselves
whether
> 3986/7 should be updated to align with WHATWG URL. I think the answer to
> that is likely to be no -- not only is the delta apparently very small
(according
> to the comments on the linked issue), alignment would mean doing things
> like considering <> to be valid content in URIs. Again, what they're
specifying
> is how to get from an arbitrary string to a URI, not the syntax itself.
> 
> Actually, one useful thing we *could* do would be to replace section 1.1.3
of
> RFC3986 with something that actually helps people understand the
> differences between all of these things. If we ceded 'URL' to the WHATWG,
> leaving 3986 to define URI, it might help.
> 
> Cheers,
> 
> 
> > --On Saturday, June 5, 2021 15:35 -0700 Larry Masinter <[email-masked]>
> > wrote:
> >
> >> Recent progress on WHATWG's URL spec led me to suggest a BOF on the
> >> topic for IETF 111
> >>
> > This is a question rather than a statement of preference because I
> > have not thought about it in some time, but are you thinking it may be
> > time to either revise or deprecate 3896 and 3897?  I may not have
> > enough information but it seems clear to me that, in terms of the
> > specs to which implementers are paying attention, the action has
> > shifted toward WHATWG (and, to a lesser extent, W3C) and away from the
> > IETF documents.
> >
> > If deprecating the IETF URI and IRI specs is really the question (or
> > part of it), I'd think either a BOF or a well-prepared presentation
> > and a rather large block of time in DISPATCH would be appropriate.
> >
> > best,
> >  john
> >
> >
> >

---

From: John C Klensin <[email-masked]>
Date: Thu, 10 Jun 2021 00:33:46 -0400

Larry, 

Without trying to answer your question, two small cautions.
While WHATWG has chosen, probably for good reason, to
concentrate on URLs, as you know 3986 is about URIs more
generally.  It includes some sweeping requirements that are
arguably unreasonable or over-constraining for non-URL URIs
generally and URNs in particular.  One of the reasons the URNbis
WG got RFCs 8241 and 8254 out and then essentially gave up was
connected to those constraints and I gather that there has been
some development and deployment outside IETF that made progress
by ignoring restrictive interpretations of 3986.  If it is worth
the sort of work you suggest to recognize changes consistent
with WHATWG work, is it not equally worth considering updates
that would create better alignment with URN practice and demands
on URN extensions?

Not directly related, but my understanding is that work
elsewhere has moved considerably beyond RFC 3987 (fwiw, a
Proposed Standard while 3986 is an Internet Standard).  It seems
to me likely that we would want to treat 3986 and 3987 quite
differently, maybe even retiring the latter in favor of a
pointer to work done elsewhere.

I don't know how to peek into a can of worms without opening it.

    john

--On Wednesday, June 9, 2021 17:39 -0700 Larry Masinter
<[email-masked]> wrote:

> The question is: is there enough interest to have a BOF at
> IETF 111 to discuss
> a plan for moving forward? I don't imagine starting out with a
> presentation of
> a solution. 
> 
> What things could we do first with minimum effort that would
> improve the current state?  A replacement for RFCs 3986 and
> 3987 wouldn't be my first step. Maybe a short standards-track
> "UPDATES" that points to/contains deltas or a new grammar that
> is more consistent with (tested) implementations.
> That's a little more than Mark's "Section 1.1.3" because it
> makes normative changes but (apparently) minimal ones.
> 
> I was encouraged to see WHATWG paying attention to feedback
> from curl and node.js as implementations of URLs, not just
> browsers.

---

From: Alwin Blok <[email-masked]>
Date: Sun, 13 Jun 2021 14:01:31 +0200

> On 10 Jun 2021, at 02:39, Larry Masinter <[email-masked]> wrote:
> 
> A replacement for RFCs 3986 and 3987 wouldn't be my

Since I am new to the IETF process, is there a link that I can follow with more information about this?
Specifically, a description of what is an UPDATES, in this case. 

> On 8 Jun 2021, at 08:37, Mark Nottingham <[email-masked]> wrote:
> 
> The issue that Larry links to seems to be taking tentative steps towards aligning the grammar in 3986/7 with what WHATWG does, which is great. If that's incorporated, I think there's still a significant gap that would need to be filled before we can deprecate. 

I don’t know if it will be incorporated. I feel quite a lot of resistance to it. My impression may be wrong though and maybe it will work out just fine. 

> although obviously the W3C has come to peace with HTML living there

Oh…. They should up their game and publish a much simpler but behaviourally equivalent specification of the html parsing chapter only. 

> Looking at this from a different angle, we could also ask ourselves whether 3986/7 should be updated to align with WHATWG URL. I think the answer to that is likely to be no -- not only is the delta apparently very small (according to the comments on the linked issue)

It can maybe just be descriptive, as in: this is the grammar and the behaviour that the WHATWG implicitly specifies. 

The delta between RFC3987 and WHATWG URL is not small. You cannot directly compare the documents, the style of the WHATWG prevents that. But the standard can be (after so much effort…) characterised. That characterisation could be published as an IETF document. That would be great! The diff between that, and RFC3987 should be relatively small. 

> alignment would mean doing things like considering <> to be valid content in URIs. Again, what they're specifying is how to get from an arbitrary string to a URI, not the syntax itself.

The WHATWG explicitly defines valid URLs to always have a scheme. But then it implicitly defines ‘loose URL’s and ‘loose relative URL’s via their ‘basic-url-parser’ function. 

That function specifies, how to get from an arbitrary input string and optionally an ‘URL record’ to a resolved-normalised-URL-record, or failure. 

If you break down that function, you see that it implicitly specifies:

- A grammar for valid URLs and one for loose URLs (incorporating valid and invalid)
- Grammars for relative references
- An encoding normal form for loose URLs
- Several slightly different resolution operations

They don’t seem to be attentively aware of that themselves. 
The solution was pointed out lucidly by David Sheets in 2012:

<https://mailarchive.ietf.org/arch/msg/ietf/iX5a8Bn3JJO98O1Z6BShkRSApY0/>

Kind regards,
-Alwin Blok