Subject: [apps-discuss] URL definitions and draft-ruby-url-problem

From: Paul Hoffman <[email-masked]>
Date: Thu, 18 Dec 2014 08:20:41 -0800

This seems like an important document for us to look at, and possibly adopt. Section 3 is pretty scary, and section 4 seems like a very reasonable solution.

--Paul Hoffman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 19 Dec 2014 19:41:02 +0100

* Paul Hoffman wrote:
>This seems like an important document for us to look at, and possibly 
>adopt. Section 3 is pretty scary, and section 4 seems like a very 
>reasonable solution.

I have reviewed this document. Sections 1 and 2 seem reasonable to me.
Section 3 has

   The main problem is conflicting specifications that overlap but don't
   match each other.

   Additionally, the following are issues that need to be resolves to
   make URL processing unambiguous and stable.

   o  Nomenclature: over the years, a number of different sets of
      terminology has been used.  URL / URI / IRI is not the only
      difference.  [tantek-slice] chronicles a number of differences.

The latter refers to differences among APIs for manipulating resource
identifiers. I do not think that is a problem and does not need solving.

   o  Parameterization: standards in this area need to define such
      matters as normalization forms and values for parameters such as
      UseSTD3ASCIIRules.

Where the relevant standards allow implementations to choose options, it
is usually because there are good reasons to do so. Implementers ought
to document their choices properly, and it is a good thing when similar
implementations make the same choices, it might even be useful to have a
specification saying "Web browsers must use these options: A, B, C", but
that would just be a matter of doing it. So I am not sure what problem
needs solving in this regard.

   o  Interoperability: even after accounting for the above, there is a
      demonstrable lack of interoperability across popular libraries and
      browsers.  [whatwg-interop] identifies a number of such
      differences.

There are different classes of problems in this regard, e.g. there may
be existing requirements that are widely ignored, there may be ambiguous
requirements interpreted differently across implementations, there may
be implementation-defined behavior that varies across implementations in
a harmful manner, or there may be widely deployed behavior where further
standardisation might be useful, to mention a few. I think it would be
helpful to discuss these classes of problems separately.

   o  Specific scheme definitions: some UR* scheme definitions are
      woefully out of date, incomplete, or don't correspond to current
      practice, but updating their definitions is unclear.  This
      includes "file:", for which there is a current effort, but there
      are others which need review (including 'ftp:', 'data').

An open question here seems to be how to separate concerns. Can updating
specifications for individual schemes be done independently? If so, that
also would seem to simply require somebody doing it, so I am not sure of
the problem indicated here.

As for the "Outline of Potential Solution" in section 4, I agree that a
plan should be built. How many specifications, for instance, should we
have, what would be their scope, and what would be their contents? With-
out a plan, considering the other points in the section seems premature;
perhaps some of them are reasonable things to do independently of any
plan; in that case, they could simply be done.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Sam Ruby <[email-masked]>
Date: Fri, 19 Dec 2014 14:47:09 -0500

> I have reviewed this document. Sections 1 and 2 seem reasonable to me.

Thanks!

> identifiers. I do not think that is a problem and does not need solving.

Can I ask why not?

To be clear, I'm not suggesting that everybody adopt the same terms.  I 
am suggesting that somewhere we document what terms are in use and how 
they map.

> needs solving in this regard.

First, it is not just web browsers.  It is runtime libraries that 
accompany various programming languages.  It is code embedded in word 
processors and web servers.  I don't believe that having the URLs mean 
different things based on the context is in any body's best interests.

At a minimum, we should consider clearly documenting the set of URLs 
that are may be interpreted differently in different contexts.  Perhaps 
we should identify those URLs as non-conforming.

However, that's not enough.  Consider ICANN approved non-ASCII domain 
names.  Different RFC 3986 compliant libraries handle https URLs with 
such hostnames differently.  We can't simply tell people to avoid such.

> helpful to discuss these classes of problems separately.

I agree!  To seed the discussion, I offer the following web page with a 
number of interesting test cases:

https://url.spec.whatwg.org/interop/test-results/

Feel free to propose additional test cases, suggest categorizations, or 
changes to either specs or to libraries, or even additional libraries 
that should be included.

> the problem indicated here.

One thing that is important to recognize is that every modern 
programming language is going to have a URI or URL parse function, 
method, subroutine, or whatever.  While it may make sense to allow new 
schemes to impose additional validity requirements specific to their 
scheme or additional semantics, it is in everybody's best interests if 
we define a standard way in which URLs which make use of unknown URL 
schemes are to be parsed.

Note: I am *NOT* suggesting that what currently is in the WHATWG URL 
Living Standard is that definition.  I think we need something a bit 
closer to what RFC 3986 defines.

I welcome input on what that behavior should be.  Let's work on it 
together.  Meanwhile, 
https://www.w3.org/Bugs/Public/show_bug.cgi?id=27233 is open on this issue.

> plan; in that case, they could simply be done.

There is indeed a big chicken and egg problem here.

My best guess at this moment is that RFC 3987 needs to be retired and 
work needs to start on a RFC 3986bis, and that probably needs a new 
Working Group.  And that needs the discussion we are currently having to 
happen.

I will say that I'm willing to work with anybody, anywhere.  If work on 
a RFC 3986bis starts, I will contribute.

I also am not looking to make this somebody else's problem.  If it means 
I need to step up to be that editor, I will do that too.  And in the 
process, I will actively encourage others to contribute, and by that I 
mean GitHub pull requests.

But I will also say that while we should be having the high level 
discussion at this time and in parallel to the technical discussions 
about how various classes of input strings should be parsed, it really 
is the latter (parser) discussion that we should be doing the deep dive on.

Once we have a starter set list of proposed changes, we can circle back 
and determine whether a set of errata is sufficient or if the overhead 
of an entire Working Group is required.

However, that is just my preference.  Should you be interested in 
suggesting changes to draft-ruby-url-problem, I encourage pull requests 
for that too.  Julian Reschke has already submitted a few.  You could be 
next!

- Sam Ruby

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 19 Dec 2014 21:56:02 +0100

>am suggesting that somewhere we document what terms are in use and how 

Well, as an example, RFC 3986 defines the term "fragment identifier". If
we document that some APIs call it `hash` and APIs vary in whether they
include the `#` separator, who could do something easily that is hard to
do at the moment?

>such hostnames differently.  We can't simply tell people to avoid such.

Identifying problematic sets of resource identifiers with unstable in-
terpretation would be very useful. We could then decide what to do about
those cases.

>closer to what RFC 3986 defines.

Agreed.

>of an entire Working Group is required.

Sounds good to me.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 22 Dec 2014 19:12:42 +0900

I have the following comments on draft-ruby-url-problem-00:

Overall: I think it's good to have such a document because a lot of 
people keep mentioning problems related to URIs/IRIs/URLs but often 
without enough background or in fragments that then get forgotten. If 
this document can clearly call out these problems, then it will be very 
valuable. This is separate from whether this document should eventually 
be published; it could be published after things have settled down, or 
just be abandoned.

Abstract: I'd make sure that URI and IRI are also mentioned, because 
they clearly are the terminology used widely in the IETF.

Details:

1. Brief History of URL standards
(http://tools.ietf.org/html/draft-ruby-url-problem-00#section-1)

"Although it was quickly determined that it was desirable to allow 
non-ASCII characters, shoehorning utf-8 into ASCII-only systems was 
unacceptable; at the time Unicode was not so widely deployed."

I'd prefer a less emotional word than "shoehorning", and a more 
technical word than "unacceptable". Also, UTF-8 doesn't need to be 
mentioned here, what matters is Unicode.

Also, it would be good to know what "at the time" refers to. If it 
refers to 1994, the statement about Unicode is certainly true, but for 
that date, the statement "it was quickly determined that it was 
desirable to allow  non-ASCII characters" is clearly false. Actually, 
even in 2005, and even after that, there were/are people who claim that 
IRIs should be just "user interface elements", or that ASCII only would 
be okay. Otherwise, the introduction in RFC 3987 (see 
http://tools.ietf.org/html/rfc3987#section-1.1) could have been a lot 
shorter.

"The IRI-to-URI transformation specified in [RFC3987] had options; it 
wasn't a deterministic path."

If this is seen as relevant, please move it to section 3, because it's 
way too small a detail to keep in a section entitled "Brief History of 
URL standards". Then please explain what the options are, and what the 
problems are that you see with these options.
(Please note that the choice between using ToASCII and UTF-8->%-encode 
in the conversion, which may be one of the options you refer to, stems 
from the fact that RFC 3986 allows both %-encoding and Punycode, and 
therefore if this is a problem, this should be mentioned for RFC 3986.)

"The URI-to-IRI transformation was also heuristic, since there was no 
guarantee that %xx-encoded bytes in the URI were actually meant to be 
%xx percent-hex-encoded bytes of a utf8 encoding of a Unicode string."

Again, if this is seen as relevant, please move it to section 3, because 
it's way too small a detail to keep in a section entitled "Brief History 
of URL standards". Also, please change "utf8" to "UTF-8". [As far as I'm 
aware, this isn't a problem; although it's a heuristic, it's one that 
works extremely well, and one that browsers use all the time.]

"... IDNA specs ([RFC3490] and [RFC5895]) did not fully addressed IRI 
processing."

It would be good to know what this is about. UTS-46 is about how to 
bridge the differences between IDNA2003 and IDNA2008.

2. Current Organizations and Specs in Development
(http://tools.ietf.org/html/draft-ruby-url-problem-00#section-2)

"Documents sitting needing update, abandoned now, are three drafts 
([iri-3987bis], [iri-comparison], and [iri-bidi-guidelines]), which were 
originally intended to obsolete [RFC3987]."

The problems with Bidi IRIs should definitely be mentioned in section 3. 
They are well known among experts. What's not known is the solution for 
these problems. The solution given in RFC 3987 has some obvious errors 
(how to handle combining marks); it's general approach also probably can 
be improved on, but it's not sure why.

Removing the bidi issues, the problem with the drafts as currently 
abandoned is that besides the very necessary work of fixing errors,..., 
there was also an attempt to rewrite some of the more general text. My 
conclusion from that exercise has been that this isn't productive at 
all. If I restart on these documents, I'll go back to the original RFC 
3987 and integrate errata. Based on this experience, I also strongly 
suggest that in the event of an attempt to update RFC 3986, updates are 
done on a strict "need-to" base, rather than attempting to write 
everything from scratch.

"This work is based on [UTS-46], and is intented to obsolete both 
[RFC3986] and [RFC3987]."

s/intented/intended/

UTS-46 doesn't fully cover [RFC3986] and [RFC3987], so I'd suggest 
splitting the sentence to avoid potential misunderstandings.

I have read the phrase "is intended to obsolete both [RFC3986] and 
[RFC3987]", but in particular in the document at hand, it would be good 
to say what that's actually supposed to mean. Very specifically, a 
non-IETF document cannot formally obsolete an IETF document. If (a) the 
intent is to bring this document to a level of completeness and 
consensus that would allow the IETF to obsolete the above RFCs with a 
stump RFC, then it would be good to say so explicitly. On the other 
hand, if (b) the intent is to allow browser makers and browser-related 
specs to not have to consult these RFCs, that should be said explicitly.
[My current impression is that the goal is supposed to be close to (a), 
but what's been done is mostly close to (b).]

4. Outline of Potential Solution

s/outout/output/

"Other than responsing to any feedback that may be provided, no changes 
to any Unicode Consortium product is required."

My understanding is that the Unicode Consortium plans to adapt UTS-46 as 
registries (e.g. DENIC) move from IDNA2003 to IDNA2008. So this sentence 
is not necessarily correct.

Regards,   Martin.

On 2014/12/19 01:20, Paul Hoffman wrote:
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 22 Dec 2014 20:02:28 +0900

Hello Sam,

> am suggesting that somewhere we document what terms are in use and how

Good. If that's the case, can you make sure the document says so? 
Otherwise, many people will interpret it the way Björn did.

On top of that, doesn't Tantek's blog post already do that? (except for 
some newer libraries, but we could ask him to add these)

> such hostnames differently.  We can't simply tell people to avoid such.

Is that specific to httpS? Is that security related?

> changes to either specs or to libraries, or even additional libraries

I don't see Ruby (the programming language). I see addressable, and 
there's a Ruby Gem by that name (http://rubygems.org/gems/addressable), 
but it would be good to have the builtin functionality, too.

Also, on that page, it would be helpful to have a column with the result 
that was tested against.

Also, I don't see any true IRIs. The wide-character digits are a very 
special case.

> schemes are to be parsed.

Yes, definitely. That's the whole idea behind the "generic" in RFC 
3986's title.

> closer to what RFC 3986 defines.

That sounds great.

> work needs to start on a RFC 3986bis,

Can you explain why you are making this difference between RFC 3986 and 
3987? As one of the coauthors of RFC 3987, I'm very familiar with its 
shortcomings, but I don't know of any that couldn't be fixed with an update.

> and that probably needs a new

RFC 3986/3987 where done without a WG, but these days, there is very 
little the IETF does without a WG. In that sense, I agree.

On the other hand, the experience with the IRI WG has shown that trying 
to pretend urgency and form a WG before some serious draft work may not 
work for a topic where (as I said in a previous mail) interest is spread 
wide but thin.

> for that too.  Julian Reschke has already submitted a few.  You could be

[Sorry, I prefer to comment in email, because then people can follow the 
discussion. I hope you'll take this input, too. On the other hand, if 
you prefer github, I'd indicate that in 
http://tools.ietf.org/html/draft-ruby-url-problem-00.]

Regards,   Martin.

---

From: Sam Ruby <[email-masked]>
Date: Mon, 22 Dec 2014 07:23:55 -0500

Agreed.

I also believe I agree with pretty much all of the below, and feel it is 
valuable.  After all, the current draft is the product of a Sunday 
afternoon's work based largely on a blog post that Larry wrote over 
three months ago.

Some of the feedback you provide is immediately actionable (example: 
s/intented/intended/).  Some of it requires somebody to actually make a 
proposal (example: less emotional word than "shoehorning").  Some of it 
requires more research (example: this sentence is not necessarily correct.)

Ultimately these items should result in updates to the draft and/or 
issues being recorded.  I'd encourage you to make pull requests and/or 
directly open issues.  If that doesn't happen, at a minimum, I will 
address the first set.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Mon, 22 Dec 2014 07:55:08 -0500

> Is that specific to httpS? Is that security related?

It applies to any scheme that includes a hostname, including file.

> but it would be good to have the builtin functionality, too.

Such would not be difficult to add.  At the moment, my focus has been on 
libraries that are actively updated and/or are advertised as spec compliant.

If adding the Ruby library interests you, can I ask that you help with 
the capture of the data?  The following is already pretty close to what 
I would need:

https://github.com/webspecs/url/blob/develop/evaluate/testaddressable.rb

The test data used by the program can be found here:

https://url.spec.whatwg.org/interop/urltestdata.json

> Also, on that page, it would be helpful to have a column with the result

The table is pretty wide, but I could certainly add a checkboxes that 
hide or show specific columns.

Just so I am clear on what it is that you are proposing: you would like 
to see a column that includes the value for "href" for the selected 
baseline?  Probably after the column titled "Base" and before the column 
titled "User Agents with differences"?

> Also, I don't see any true IRIs. The wide-character digits are a very

I encourage you to propose specific test cases.  The master source for 
test cases across these efforts is here:

https://github.com/w3c/web-platform-tests/blob/master/url/urltestdata.txt

> shortcomings, but I don't know of any that couldn't be fixed with an

You may indeed be better informed than I am on this subject.  Your 
guess, therefore, may be different.

> work for a topic where (as I said in a previous mail) interest is spread

There is a bit of a chicken and egg problem here.

I'm actively updating the WHATWG URL document, attempting to resolve the 
differences found in the test-results mentioned above.  If those changes 
can be done in RFC 3986 and/or RFC 3987, and the WHATWG URL document 
could reference the results, I'll do that.

If there are no plans to update those RFCs, then I see the WHATWG URL 
document continuing on its current path.

> http://tools.ietf.org/html/draft-ruby-url-problem-00.]

These do not need to be mutually exclusive options.  Participating 
exclusively via email is not a problem.  I'd encourage concrete 
suggestions.  As an example, I'm not sure how to deal with comments like 
"I don't see any true IRIs.".  By contrast, I very much do know how to 
deal with pull requests that propose specific changes/additions/deletions.

- Sam Ruby

---

From: Larry Masinter <[email-masked]>
Date: Mon, 22 Dec 2014 17:35:56 +0000

FYI

Some comments
http://lists.w3.org/Archives/Public/www-archive/2014Dec/0004.html
I'll either update the document or at least submit these as issues.

Note also continued discussion of [url] in http://lists.w3.org/Archives/Public/public-ietf-w3c/2014Dec/