Subject: Re: [apps-discuss] Fwd: FW: New Version Notification for draft-kerwin-file-scheme-13.txt

From: Matthew Kerwin <[email-masked]>
Date: Thu, 1 Jan 2015 22:42:38 +1000

Thanks for the comments; responses inline.

On 10 December 2014 at 05:54, Larry Masinter <[email-masked]> wrote:

>  ​​
>
>
> First comment:
>
>  Please align with the special processing for “file:” in:
> https://specs.webplatform.org/url/webspecs/develop/
>
>
> That spec (developed in sync with WHATWG URL) focuses on
>
> parsing and processing of base + relative, when the scheme
>
> (or base scheme) is  “file:”, while the kerwin-file-scheme
> talks more about mapping between local file paths and “file:”
>
> IRIs (URLs).
>
> But the overlap is significant, and I hope could be eliminated,
> if the direction of moving the webplatform.org spec to
> obsolete 3987 is accepted.
>
> I'm not sure how to approach this request. Do you have any specific
details about what's not in alignment, or could be better? Or are you
suggesting I hold off doing anything until the webplatform.org draft
matures some more? (I refer to that big red box that says "...the parsing
rules in this section have not enjoyed wide review, and therefore are more
likely to be subject to change than other parts of this specification")

I guess I could add support for backslashes. I suspect that would end up in
a non-normative appendix, along with the "|" drive letter separator (as
suggested in an earlier message, and currently being worked into the next
draft.)

Second comment:
>
>
>
> You describe the relationship between local file names
> and ‘file:’ URLS for “UNIX-like”, “DOS- or Windows-based”
> and “VMS Files-11”. These categories aren’t precise or
> exhaustive, and file system processing of file names can
> vary depending on version, language pack installed in OS,
> as well as network protocol (Nfs?).
> *​​*
>
> I would rather see a more extensible organization of
> the specification by establishing a template, and then
>
> separately documenting, on a file-system by file-system
> basis, how to translate and match, based not on the
> URI but on the parsed components resulting from
> URL-parsing. This would allow new or different file
> systems to be documented.
>
> That makes sense, and is an appealing suggestion. I will experiment with
this, and see if I can come up with a new version that incorporates it.

Third comment:
>
>
>
> The translation between file names and URLs and things
> like them happen in several specs:
>
>
>
> content-disposition for file download, multipart/form-data
> for file upload, by common HTTP servers, when packaging files
>
> together into archive types as being discussed by those proposing
>
> a new ‘archive’ top level media type.
>
>
>
> It would be beneficial if we converge these, as it reduces
> interoperability
> to have incompatible methods of translation for local files names depending
> on the categorization.
>
>
I've had a quick look at some of the specs (notably RFC 2388, 6266, 5987),
and while I agree that we don't want to have too many string-to-filename
subsystems, I don't really see what approach to take to unifiy them, or
what it would really buy us. For the most part, everything either says
"decode percent-encoded UTF-8" or doesn't say anything (implying
ASCII/ISO-8859-1?) -- how they then map to the local file system is never
touched.

​Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Sam Ruby <[email-masked]>
Date: Thu, 01 Jan 2015 09:21:21 -0500

On 01/01/2015 07:42 AM, Matthew Kerwin wrote:
> On 10 December 2014 at 05:54, Larry Masinter <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     First comment:
>     ____
>
>     https://specs.webplatform.org/url/webspecs/develop/____
>
>     That spec (developed in sync with WHATWG URL) focuses on ____
>     parsing and processing of base + relative, when the scheme____
>     talks more about mapping between local file paths and “file:”____
>     IRIs (URLs). ____
>
>     if the direction of moving the webplatform.org
>     <http://webplatform.org> spec to
> suggesting I hold off doing anything until the webplatform.org
> <http://webplatform.org> draft matures some more? (I refer to that big
> red box that says "...the parsing rules in this section have not enjoyed
> wide review, and therefore are more likely to be subject to change than
> other parts of this specification")

A few concrete examples are here:

http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13511.html

It is my hope that by comparing notes we can converge.

> I guess I could add support for backslashes. I suspect that would end up
> in a non-normative appendix, along with the "|" drive letter separator
> (as suggested in an earlier message, and currently being worked into the
> next draft.)

I don't understand the thought process here. The current Internet-Draft 
takes care to document a select few browser specific syntaxes, but many 
others (including the ones you mention above) are not included.  What is 
the selection criteria you are using?

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Thu, 01 Jan 2015 15:57:48 +0000

Matthew,

I've finally been prompted to read through your 
https://tools.ietf.org/html/draft-kerwin-file-scheme-13.  I think it's great 
that this is being tackled (again), and I'm hoping that there will be enough 
pragmatism this time round to achieve consensus on something that's an 
improvement over the status quo, even if not perfect.

My comments follow...

Section 2: windows-path

I'm pretty sure I've come across '/c:/rest-of-path' as a mapping of Windows file 
paths (I'm not sure where; I may even have used it myself.  It makes some 
logical sense to me in that on Windows (non-UNC-naming) the drive letters are 
effectively the top-level directories in a single-root file system.)

But that would subsumed by 'path-absolute'.  [Later] I see you have this covered 
as an example, and more.

Section 2: Inclusion of "|"

As others have raised, there's an question of how to deal with common extensions 
to normal URI syntax, and separation of a preferred core for file: URI syntax 
from documentation of common variations.  If possible, my preference would be to 
treat non-conformance with RFC3986, like this, as a common variation.  I think 
'core' file URIs should preferably be fully RFC3986 conformant so that existing 
generic parsers will handle them.

[Later] I see you have this covered too.

I think the document might be clearer in its intent if at the start of section 
2, and in the syntax productions, it were more clearly signaled that the syntax 
described here includes commonly-used forms that go beyond RFC3986 syntax.

Section 2: local vs non-local

See below.  I think this is a non-relevant distinction.  e.g. Is a remote file 
system mounted on a Unix-style mount point supposed to include an authority?

Section 3: methods on file URIs

I'm not sure I agree that translation to filename is the  *only* operation that 
can be done using a file: URI.  And I have concerns about use of the term 
"method" here, which is strongly suggestive of HTTP "methods".

Specifically, dereferencing:  I have frequently used APIs that retrieve the 
entity referenced by a URI, including file: URIs.  Indeed, in much of my code, 
this is central to unifying web and local file access.   I think retrieval is an 
operation that is very naturally performed on a file URI, with results broadly 
equivalent to an HTTP GET.  Some analogue to HTTP PUT is also conceivable.

I think the issue here is not that the nature of operations that can be 
performed is restricted, but the details of how to perform them, which will vary 
from system to system.

You also say "The local file system API can only be used if the file URI has a
    blank (or absent) authority and the path, when transformed to the
    local system's conventions, is not a UNC string."

A common case I've come across is where the authority is "localhost", in which 
case it seems reasonable to allow local file access based the file: URI.

More generally, it seems to me that the difference between local and remote 
files (with or without host) is in the mechanism that might be used to access 
the file.  In some cases, it may be the same, by virtue of being handled by the 
underlying system's file system.  And in some cases, a filename that appears to 
have no host may in fact be remote (cf. remote file system, mounted on Unix-like 
file system).  I think that trying to make this distinction raises more 
questions than it answers.

Section 3.1 (esp step 4):

I'm a bit confused by the intent here.  I think it's reasonably clear that

   file://example.org/c:/path/segments
   file:///c:/path/segments

would be expected, but what about:

   file:/c:/path/segments
vs
   file:c:/path/segments

I think you prefer the latter.

But I think it would be more consistent, and would avoid potential ambiguities 
in relative reference resolution, to prefer the former;  i.e. to consider that 
drives are like top-level sub-directories of a root "/".  I think this section 
could also be thereby simplified a little.

<aside on the reference resolution point above>

Consider that I have a relative reference:

     c:/foo/bar

to be resolved against base URI:

     file:

the result is

     file:c:/foo/bar

But if it is to be resolved against a base URI:

     file:d:/

What should be the resolved URI?  Per RFC3986, it would be

     file:d:/c:/foo/bar

which doesn't make sense if c:, d: are interpreted as Windows drive letters.

You can't tell here if the reference starting with "c:..." is intended to be an 
absolute or relative path.  RFC3986 says relative.  So there's no way to know if 
the desired result above should be

     file:c:/foo/bar

If the leading "/" is always included, this ambiguity can be resolved 
consistently with RFC3986.  i.e.

     c:/foo/bar   is a relative path
     /c:/foo/bar  is absolute path

</aside>

Section 3.3, steps 5, 6:

I think it is necessary to say something about %-escaping here.  I've been 
bitten in the past by doing roughly what you describe here, with filenames that 
happen to contain (say) '#' characters.

Section 6:

Can you please include a full registration template for the file: scheme here. 
cf.  https://tools.ietf.org/html/rfc4395#section-5.4

(Many of the fields will just refer to other sections of the document)

...

Again, thanks for tackling this.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 01 Jan 2015 16:17:16 +0000

On 01/01/2015 12:42, Matthew Kerwin wrote:
>>   ââ
>>   Please align with the special processing for âfile:â in:
>> (or base scheme) is  âfile:â, while the kerwin-file-scheme
>> talks more about mapping between local file paths and âfile:â
>>
>> IRIs (URLs).
>>

For some reason, I didn't see Larry's original comment.  Probably overwhelmed. 
But I've just looked at https://specs.webplatform.org/url/webspecs/develop/ and:

(a) noticed the same issue that Matthew just noted about the "big red box".

(b) The syntax diagram appears to go against RFC3986 by excluding URIs of the form:

     file:///<local-path>

corresponding to a blank reg-name for the authority host field:

     authority     = [ userinfo "@" ] host [ ":" port ]

     host          = IP-literal / IPv4address / reg-name

     reg-name      = *( unreserved / pct-encoded / sub-delims )

(excerpted from https://tools.ietf.org/html/rfc3986#appendix-A)

This is a form I use extensively in my code, and it's explicitly discussed in 
the Wikipedia entry for file: URIs, so I guess I'm not alone in this - 
http://en.wikipedia.org/wiki/File_URI_scheme)

>> and âfile:â URLS for âUNIX-likeâ, âDOS- or Windows-basedâ
>> and âVMS Files-11â. These categories arenât precise or
>> *ââ*

+1

But I'll also note that for practical use on modern systems, the similarity with 
Unix file naming is really useful.  Python's I/O libraries interpret "/" in 
filenames as directory path separators, even on Windows.  This means that I can 
(and do) use relative path names as relative references that work with respect 
to a file: URI base, or (say) an HTTP base URI.  This helps me to unify 
accessing both local and web resources.

Probably this consideration has no place in the file: scheme specification ... 
just sayin'.

And it doesn't really help with more exotic file systems like VMS - but surely, 
these are a tiny minority presence on the web these days, so I'm not sure how 
much concern is justified here.

>> a new âarchiveâ top level media type.
>>
>>
>>

I think this also relates to my comment above about the value of being able to 
unify local file and web resource accessing.  I think unifying URIs and 
Unix-style path names is a useful start, and maybe add extra information for 
selected other file systems (notably Windows).

IMO, having a clear story for Unix/Linux and Windows would address the vast 
majority of actual systems for which this might be an issue.

...

W.r.t. the 'archive' media type that Larry mentions.  There may be value in 
having a common fragment syntax that allows reference to files within an 
archive.  And it would maker sense (to me) if such a syntax could re-use 
elements a of the URI path syntax and be interpreted in a way that is similar to 
file: URI paths.  For more background on this, see 
http://w3ctag.github.io/packaging-on-the-web/#fragment-identifiers, specifically 
the 'url=...' option.

Specifically, consider that a directory tree is to be packaged as an archive. 
Each file in that tree has a relative reference w.r.t. the base of that tree, 
which when resolved would be a file: URI.  It would be convenient if those 
relative references can also be used as part of fragement identifier referring 
to the same file within the packaged archive.

What I think this all points to is having a uniform representation of paths 
within a file: URI that is amenable to common RFC3986-style relative reference 
resolution in a variety of scenarios.  I don't think there's any need to call 
out all these cases explicitly, but to take them in to account when considering 
design options.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 01 Jan 2015 17:04:29 +0000

[Further to my previous, below...]

Oops.  My bad.

Of course, per RFC 3986,

     c:/foo/bar

is an absolute URI with scheme "c".  The relative reference form would need to be:

     ./c:/foo/bar
or
     c%3A/foo/bar

But this raises a different, albeit less compelling, reason to prefer

    file:/c:/foo/bar

to avoid mistakes of trying to treat c:/foo/bar as a reference relative to 
file:// rather than an absolute URI.

#g
--

---

From: Sam Ruby <[email-masked]>
Date: Thu, 01 Jan 2015 12:29:01 -0500

On 01/01/2015 11:17 AM, Graham Klyne wrote:
> For some reason, I didn't see Larry's original comment.  Probably
> overwhelmed. But I've just looked at
> https://specs.webplatform.org/url/webspecs/develop/ and:
>

I added that big red box.  I encourage you to follow the following links:

http://lists.w3.org/Archives/Public/public-webapps/2014OctDec/0526.html

https://mailarchive.ietf.org/arch/msg/apps-discuss/KyxFLw2FCwv7CSpiWAmLr8Ha7vc

It is my hope that by working together I can feel confident enough to 
remove that red box.  As it is, I don't feel that either spec matches 
widely deployed applications.

> This is a form I use extensively in my code, and it's explicitly
> discussed in the Wikipedia entry for file: URIs, so I guess I'm not
> alone in this - http://en.wikipedia.org/wiki/File_URI_scheme)

Why do you come the conclusion that it excludes such URIs?  Authority 
can consist of solely a host:

https://specs.webplatform.org/url/webspecs/develop/#authority

And a host can be an empty string:

https://specs.webplatform.org/url/webspecs/develop/#host

Additionally, there is a reference implementation and test results:

https://url.spec.whatwg.org/interop/test-results/

An example with an empty authority field:

https://url.spec.whatwg.org/interop/test-results/48db81fd23

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Thu, 01 Jan 2015 18:47:45 +0000

> I added that big red box.  I encourage you to follow the following links:

Sam,

Thanks for the links.  My thoughts in response...

> http://lists.w3.org/Archives/Public/public-webapps/2014OctDec/0526.html

That message appears to be concerned primarily with web browser APIs.  While an 
important constituency, they are not the only one.  My own work, which informs 
my personal interest in this, is more concerned with web server based 
applications, which might be characterized as "middleware" - i.e. acting as both 
web servers and clients.

With reference to the final comment in that email, I think it would be great if 
we can agree a form of file: URI usage that works for everyone, so that the URL 
parsing spec doesn't have to say anything specific about file: URIs.

> https://mailarchive.ietf.org/arch/msg/apps-discuss/KyxFLw2FCwv7CSpiWAmLr8Ha7vc

This raises two points for me:

1. 'file: should be seen as an "escape hatch"'.

I disagree.  For me, the value of file: URIs is to provide a file naming 
structure that can be used in libraries that unify local and web access to 
resources.  So I think it's important that file: URIs follow common URI syntax 
and resolution mechanisms, even if their interpretation for the purposes of 
dereferencing, etc., is defined locally.

(This is not to impose "normative statements on OS vendors for their own 
software" - unless the vendors choose to use URIs natively for file naming.)

2. Use of vendor-specific documentation

I agree with this, specifically: "The right way ... is to write the RFC in such 
a way that OS-specific variations are not required for RFC-compliance in the 
first place"

So it's clear to me that there are aspects of file: URI handling that are 
local-context dependent (e.g. how to actually dereference).  But I think other 
activities (such as relative reference resolution should be possible without 
regard to the underlying file system implementation - and this is the level of 
commonality that a file: URI scheme RFC should aim to provide.

> It is my hope that by working together I can feel confident enough to remove
> that red box.  As it is, I don't feel that either spec matches widely deployed
> applications.
>

Fair enough.  Which suggests to me that focusing on a single focused spec and 
aligning around that might be a productive way to tackle this.

> Why do you come the conclusion that it excludes such URIs?  Authority can
> consist of solely a host:
>

Sorry, I didn't follow through.  I saw "host" with no bypass track, and was 
incorrectly assuming that "host" could not be empty.  I'm happy to accept your 
assurance that file:/// is allowed in your syntax, and to withdraw that 
particular comment.

<aside>
A problem for me is (as I've said before in other forums) that RFC3986 is a 
perfectly good specification of URI syntax and I don't see the need to consult 
any other.  So why should I put energy into so doing?  I make this point on the 
presumption that I'm not the only one who is OK with RFC 3986.

Now, if the community decides that some other spec is the True Pronouncement 
about URI syntax, I shall have to reconsider.  But I don't see why I should be 
asked to put energy into reviewing a specification which doesn't give me 
anything I don't already have.

This doesn't mean I oppose this specification in its goals to cover areas that 
are not covered by RFC3986.  But, speaking personally, I'd really like to be 
assured that any valid RFC3986 URI will be acceptable according to the syntax 
you describe.  That way I don't have to read the other document if I don't want 
the extra capabilities it offers.
</aside>

#g
--

---

From: Sam Ruby <[email-masked]>
Date: Thu, 01 Jan 2015 14:08:22 -0500

On 01/01/2015 01:47 PM, Graham Klyne wrote:
> That message appears to be concerned primarily with web browser APIs.
> While an important constituency, they are not the only one.  My own
> work, which informs my personal interest in this, is more concerned with
> web server based applications, which might be characterized as
> "middleware" - i.e. acting as both web servers and clients.

There are multiple, valid, points of view.  My intent was not to pose 
that link in isolation.

> With reference to the final comment in that email, I think it would be
> great if we can agree a form of file: URI usage that works for everyone,
> so that the URL parsing spec doesn't have to say anything specific about
> file: URIs.

That would indeed be a great outcome.

> structure that can be used in libraries that unify local and web access
> to resources.  So I think it's important that file: URIs follow common
> URI syntax and resolution mechanisms, even if their interpretation for
> the purposes of dereferencing, etc., is defined locally.
>

I'll contrast "can be used in libraries that..." (immediately above), 
and "works for everyone" (previous point).

If the intent of draft-kerwin-file-scheme is only to be valid in a 
subset of libraries, then it should say so.  If it is intended to also 
match other user agent behaviors (e.g. browsers), then we collectively 
have considerably more work to do.

> I agree with this, specifically: "The right way ... is to write the RFC
> in such a way that OS-specific variations are not required for
> RFC-compliance in the first place"
>
> So it's clear to me that there are aspects of file: URI handling that
> are local-context dependent (e.g. how to actually dereference).  But I
> think other activities (such as relative reference resolution should be
> possible without regard to the underlying file system implementation -
> and this is the level of commonality that a file: URI scheme RFC should
> aim to provide.

I'll note that draft-kerwin-file-scheme includes such constructs as 
windows-path and unc-path.

>> remove
> Fair enough.  Which suggests to me that focusing on a single focused
> spec and aligning around that might be a productive way to tackle this.

What I am focused on is the following question: what should a 
"URI.parse" method do?  In some ways that question is more general (in 
that is isn't file: scheme specific).  In some ways that question is 
more focused (in that it doesn't attempt to describe the operating 
system specific interpretations of the results).

> <aside>
> A problem for me is (as I've said before in other forums) that RFC3986
> is a perfectly good specification of URI syntax and I don't see the need
> to consult any other.  So why should I put energy into so doing?  I make
> this point on the presumption that I'm not the only one who is OK with
> RFC 3986.
>
> Now, if the community decides that some other spec is the True
> Pronouncement about URI syntax, I shall have to reconsider.  But I don't
> see why I should be asked to put energy into reviewing a specification
> which doesn't give me anything I don't already have.
>
> This doesn't mean I oppose this specification in its goals to cover
> areas that are not covered by RFC3986.  But, speaking personally, I'd
> really like to be assured that any valid RFC3986 URI will be acceptable
> according to the syntax you describe.  That way I don't have to read the
> other document if I don't want the extra capabilities it offers.

I have evidence that RFC 3986 doesn't match a variety of user agent 
behavior.  Agents that aren't limited to browsers, but also to libraries 
that are used by what you would consider "middleware".

Here is a filtered list of test results that only considers RFC 3986 
valid URI references as inputs:

https://url.spec.whatwg.org/interop/test-results/?filter=valid

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 10:26:19 +0100

> https://url.spec.whatwg.org/interop/test-results/?filter=valid

Checking the first test 
<https://url.spec.whatwg.org/interop/test-results/d7d52bebd0?filter=valid>:

   "http://example\t.\norg"

This is not a valid URI reference.

I understand that this is work-in-progress, but please do not pretend we 
have a large set of valid URI tests that expose problems when in fact we 
do not.

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 04:46:57 -0500

> have a large set of valid URI tests that expose problems when in fact we

I'm not seeing what you are seeing.

If you go to the following page:

Test '0' is indeed the test you are describing.

Selecting 'showing only valid URI references', filters out that line, 
showing only tests 1, 2, 7, 8, 9, ...

The filter used can be found here:

https://gist.github.com/think49/770087

Patches welcome.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 10:48:50 +0100

> https://gist.github.com/think49/770087

Indeed; it was a caching problem (Firefox had previous results in the 
cache).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 11:01:23 +0100

> https://url.spec.whatwg.org/interop/test-results/?filter=valid

So I'm now looking at the first valid entry with UA differences:

<https://url.spec.whatwg.org/interop/test-results/19b44e58a2?filter=valid>

which tests:

   http://user:pass@foo:21/bar;par?b#c

This one is valid per the RFC 3986 ABNF and it contains a "userinfo" 
component.

RFC 3986:

"...Use of the format "user:password" in the userinfo field is 
deprecated. Applications should not render as clear text any data after 
the first colon (":") character found within a userinfo subcomponent 
unless the data after the colon is the empty string (indicating no 
password). Applications may choose to ignore or reject such data when it 
is received as part of a reference and should reject the storage of such 
data in unencrypted form. The passing of authentication information in 
clear text has proven to be a security risk in almost every case where 
it has been used...." -- 
<http://greenbytes.de/tech/webdav/rfc3986.html#rfc.section.3.2.1.p.1>

RFC 7230:

"The URI generic syntax for authority also includes a deprecated 
userinfo subcomponent ([RFC3986], Section 3.2.1) for including user 
authentication information in the URI. Some implementations make use of 
the userinfo component for internal configuration of authentication 
information, such as within command invocation options, configuration 
files, or bookmark lists, even though such usage might expose a user 
identifier or password. A sender MUST NOT generate the userinfo 
subcomponent (and its "@" delimiter) when an "http" URI reference is 
generated within a message as a request target or header field value. 
Before making use of an "http" URI reference received from an untrusted 
source, a recipient SHOULD parse for userinfo and treat its presence as 
an error; it is likely being used to obscure the authority for the sake 
of phishing attacks." -- 
<http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.2.7.1.p.8>

Looking at the test results suggests that

a) only the Python implementation is really broken (it probably follows 
RFC 2396 which treated params as a separate component; Firefox had a 
similar problem until a few years ago; but that was fixed by me)

b) IE gets away with treating the presence of userinfo as an error; it 
probably would be a good idea if other UAs followed that example.

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 05:43:37 -0500

> similar problem until a few years ago; but that was fixed by me)

FWIW, there was a problem with my script that captured test results, I 
wasn't capturing username and password, and this is now fixed:

https://github.com/webspecs/url/commit/72b286722209771b4a18a53a3493c20e88d95736

But that is likely separate from what you are commenting on, namely the 
problem with an incomplete pathname.

For Firefox, I had to visit the following page and then hit refresh 
before the data would show up in test-results page:

https://url.spec.whatwg.org/interop/useragent-results/python

> probably would be a good idea if other UAs followed that example.

Thanks!

This is exactly the type of discussion I am looking for.  Please continue!

There is an open bug on this issue:

https://www.w3.org/Bugs/Public/show_bug.cgi?id=27516

I've added a link to your message in that bug, and indicated that at a 
minimum, such usage should be considered a conformance error.

On the test results page, you can filter by conforming URLs.  This is a 
proper subset of set of URIs that are valid per the RFC 3986 ABNF.

It is my hope that we can work together to define a set of conformance 
criteria which defines a set of URLs for which there is reasonable 
interop; even if that means changing specifications or implementations.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 13:19:58 +0100

> problem with an incomplete pathname.

FWIW, it could also be an incorrect use of Python's urllib.parse 
(<https://docs.python.org/3.0/library/urllib.parse.html>) which is 
documented to extract the last path segment's parameter as a separate 
component. So test code will need to properly reconstruct the path with 
that information.

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 08:31:28 -0500

> component. So test code will need to properly reconstruct the path with

I'm inclined to think otherwise as Python's implementation treats the 
';' as a delimiter and not a part of the path.  As an example, the 
following two calls produce identical results:

urlparse('http://foo/a/b')
urlparse('http://foo/a/b;')

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 14:44:32 +0100

> urlparse('http://foo/a/b;')

Actually I was wrong; special handling of parameters dates back to RFC 
1808 and was removed in RFC 2396, see 
<https://tools.ietf.org/html/rfc1808#section-2.1>:

> <scheme>://<net_loc>/<path>;<params>?<query>#<fragment>

So this is what these implementations try to do.

To reconstruct the path, given that API, you'll need to re-append the 
params. And yes, the problem with Python's implementation probably is 
that you can't distinguish an empty params component from absence of 
that component. This should be raised as a bug report against the Python 
libraries.

(My point being that this -- when properly used -- affects only the edge 
case of a path ending in a ";")

Best regards, Julian

---

From: Graham Klyne <[email-masked]>
Date: Fri, 02 Jan 2015 14:27:11 +0000

On 01/01/2015 19:08, Sam Ruby wrote:
> I'll contrast "can be used in libraries that..." (immediately above), and "works
> for everyone" (previous point).
>
> If the intent of draft-kerwin-file-scheme is only to be valid in a subset of
> libraries, then it should say so.  If it is intended to also match other user
> agent behaviors (e.g. browsers), then we collectively have considerably more
> work to do.

I never made that claim.

I also think it is not the role of the URI spec to describe "agent behaviour" 
(beyond relative reference resolution, if that is a "behaviour").

I think it is the role of the URI spec to:

(a) define what constitutes a valid URI, and URI reference, and

(b) describe how to combine a valid base URI with a valid URI reference to yield 
a valid resulting URI.

Which is, of course, what RFC3986 does (how well is up for discussion).

I fully accept that there may be desirable agent behaviours that are not covered 
here, and that an additional document may be desired to describe these, 
particularly where the behaviours impact interoperability.

> I'll note that draft-kerwin-file-scheme includes such constructs as windows-path
> and unc-path.

Sure, but I'm not sure what point you're making here.

I would want to see all such system-specific forms conform to standard URI 
syntax, and to yield the desired results when resolved using a standard 
resolution algorithm.

> What I am focused on is the following question: what should a "URI.parse" method
> do?  In some ways that question is more general (in that is isn't file: scheme
> specific).  In some ways that question is more focused (in that it doesn't
> attempt to describe the operating system specific interpretations of the results).

For me, the question of what URI.parse *does* goes beyond what the core URI spec 
needs to define.  But I agree about operating system specific behaviours of 
file: URIs being outside the desirable scope of that core spec.

> I have evidence that RFC 3986 doesn't match a variety of user agent behavior.
> Agents that aren't limited to browsers, but also to libraries that are used by
> what you would consider "middleware".
>
> Here is a filtered list of test results that only considers RFC 3986 valid URI
> references as inputs:
>

I took a brief look, but haven't delved into the details of your results.  At 
that superficial level, the list suggests to me that there are many cases where 
implementations are buggy, and in different ways.  It doesn't tell me what are 
the problems in RFC3986.

In a brief sampling, I couldn't see any divergence which is likely to be 
resolvable by changing the URI spec.

#g
--

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 09:40:13 -0500

> case of a path ending in a ";")

Given that it isn't documented as such, I would consider that to be more 
of a workaround than proper use, but it isn't worth arguing over:

https://github.com/webspecs/url/commit/020175339c39166201f2a30109ca0b48371b455c

Thanks for opening the issue against Python!

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 10:18:55 -0500

On 01/02/2015 09:27 AM, Graham Klyne wrote:
>> subset of

I'm confused.  Let me try again.

I am making the claim that draft-kerwin-file-scheme does "work for 
everyone".  Either there needs to be considerably more work done, or it 
needs to reduce its scope.

> I also think it is not the role of the URI spec to describe "agent
> behaviour" (beyond relative reference resolution, if that is a
> "behaviour").
>
> (b) describe how to combine a valid base URI with a valid URI reference
> to yield a valid resulting URI.
>

I, indeed, would like to discuss that topic.

> I fully accept that there may be desirable agent behaviours that are not
> covered here, and that an additional document may be desired to describe
> these, particularly where the behaviours impact interoperability.

I would like to discuss that topic too.

Whether that document is separate or not will depend on the outcome of 
the discussion as to whether RFC 3986 matches current, deployed 
applications.

> Sure, but I'm not sure what point you're making here.

Let me try to make that point in a different way then.  I'm skeptical 
that Apple will be interested in implementing any portion of a 
specification that is specific to Microsoft Windows.  This goes back to 
the point of trying to build a specification that "works for everyone".

> I would want to see all such system-specific forms conform to standard
> URI syntax, and to yield the desired results when resolved using a
> standard resolution algorithm.

We are going to need to qualify that statement considerably.

Here is an example of a system-specific form: "C:\Program Files (x86)".

A strategy that is more likely to be successful would be to identify 
URIs as being completely system independent, and URLs as being mostly 
system independent, and for there to be a well known and documented 
mechanism for converting from URLs to URIs.  Even that is not likely to 
be completely achieved -- the conversion may end up being (at least 
partially) system dependent, but in such cases we should be able to 
define the problematic set of the inputs as non-conforming.

An example of a restriction we should consider: valid schemes must have 
at least two characters.

>> "URI.parse" method
> For me, the question of what URI.parse *does* goes beyond what the core
> URI spec needs to define.  But I agree about operating system specific
> behaviours of file: URIs being outside the desirable scope of that core
> spec.

Can I get you to explain what you mean by this.  We can ignore operating 
system specific behaviors for the moment.  I would think that the basic 
operation of identifying the scheme, path, fragment, etc for a given 
input is exactly what a URI spec needs to define.  Why do you think 
otherwise and/or what am I missing?

>> behavior.
> I took a brief look, but haven't delved into the details of your
> results.  At that superficial level, the list suggests to me that there
> are many cases where implementations are buggy, and in different ways.
> It doesn't tell me what are the problems in RFC3986.

We can agree that implementations don't match RFC 3986.  In such cases, 
where the bug is would be need to be determined on a case by case basis.

> resolvable by changing the URI spec.

I encourage you to spend more time with that data.  An example of a 
concrete problem is handing of hosts in a UTS-46 compliant manner.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 16:37:06 +0100

> the discussion as to whether RFC 3986 matches current, deployed

It could be separate, even if we find out that we want to update RFC 3986.

> otherwise and/or what am I missing?

RFC 3986 does that for valid URIs, as far as I can tell. If you take the 
non-normative regexp in the appendix into account, it even des that for 
many more strings.

> concrete problem is handing of hosts in a UTS-46 compliant manner.

Which test, specifically?

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 10:38:23 -0500

> needs to reduce its scope.

Oops.  Missing an important word.

I am making the claim that draft-kerwin-file-scheme does NOT "work for
everyone".

>>>> according to the syntax you describe.  That way I don't have to read public-webapps <[email-masked]>
>>>> the
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 10:58:18 -0500

> It could be separate, even if we find out that we want to update RFC 3986.

Agreed.

The case where being separate would be made more difficult would be the 
case where the IETF did not agree to update RFC 3986.

> non-normative regexp in the appendix into account, it even des that for

If you limit it to pure US ASCII inputs, it certainly gives precise 
answers.  The question is whether it gives accurate answers.

> Which test, specifically?

Most of the tests in the range of 242..263 deal with IDNA issues in some 
manner or another.  A particularly fun one is test 261:

https://url.spec.whatwg.org/interop/test-results/e99bce2c50

<aside>ordinal test numbers may change over time.  The hash, however, 
will not</aside>.

I encourage you to scan the text of RFC 3986 for the term "IDNA".  In 
the decade between when RFC 3986 was published and today, things have 
progressed a bit here.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jan 2015 17:54:11 +0100

> the decade between when RFC 3986 was published and today, things have

I'm fully aware that RFC 3986 is likely incorrect wrt IDNA advice.

However, the test that you cite does not use a valid URI, so I don't 
think it can be used as example that something in RFC 3986 is incorrect.

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jan 2015 13:39:45 -0500

> think it can be used as example that something in RFC 3986 is incorrect.

Since you agree that RFC 3986 is likely incorrect wrt IDNA advice, can I 
get you to suggest a test case that demonstrates this?

The master source for these tests is here:

https://github.com/w3c/web-platform-tests/blob/master/url/urltestdata.txt

A pull request would be appreciated.  A detailed suggestion with the 
same information (a source string, a base URI, and what the expected 
values for each of the components) also works.

  - - -

Railing this up a level, Roy Fielding believes that a WG would need to 
be chartered in order to update RFC 3986.  See the bottom of the 
following email:

http://lists.w3.org/Archives/Public/public-ietf-w3c/2014Dec/0088.html

Per that email, I tend to agree.

On any specific issue, I don't know what the outcome should be -- 
whether RFC 3986 should be updated, or whether implementations should 
change.  What I am looking for is to get together the set of people who 
are interested in that discussion and for us to come to a consensus 
together.

I'm willing to do the leg-work, whether that be running tests or 
updating specs, or filing bugs against implementations.

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Sat, 03 Jan 2015 12:10:46 +0000

Sam,

Rather than continue the blow-by-blow exchange of points, let me try and respond 
in one place where I think we stand:

1. I agree that draft-kerwin-file-scheme *should* work for everyone.  And that 
probably means reducing the scope of anything there that may be considered 
normative.

    But I also believe it can be useful to document other behaviours 
*informatively*.  I think this is discussed elsewhere and anticipate evolution 
in this direction.  This could mean, to develop your example, describing 
Microsoft Windows specific behaviours without any expectation that such 
behaviours would be implemented by Apple.

2. I think our disagreement may lie primarily in the area of what should be the 
scope of RFC 3986 or its successor.  There is (I claim) a substantial developer 
community who are familiar with RFC3986 as it stands, and creating a new 
document to cover the same material with enlarged scope is unnecessary and 
disruptive.  The additional scope coverage could be in a new document that 
builds upon what RFC3986 does specify.

    Part of this disagreement is that I don't think the URI core spec needs to 
describe the operation of splitting a URI into its components.  (I regard 
https://tools.ietf.org/html/rfc3986#appendix-B as merely informative.)  The 
syntax specification is sufficient to associate substrings of a well-formed URI 
with named syntax productions.

    I also think the core URI spec should not be describing how to turn non-URI 
strings (some of which may be system-dependent forms) into valid URIs.  (Except 
URI-references, which are covered by the resolution specification.)

3. Where there is divergence between implementations and RFC3986, these indeed 
should be considered on a case-by-case basis, but with (IMO) the presumption 
that RFC3986 is correct.  I.e. it is for those who think there is a problem with 
RFC3986 to make the case.

    You ask me to spend time with your data.  That's a big ask.  If some 
implementers think there is a problem with RFC 3986 then I think it is they who 
should be making the specific arguments about where RFC3986 is problematic.  I 
accept that UTS-46 host names is such an area, concerning which there should be 
a considered and focused debate (and about which I have insufficient knowledge 
to make a meaningful contribution).

4. I agree with your point about qualifying my statement about system-dependent 
forms conforming to core URI syntax.  While forms such as "C:\Program Files 
(x86)" might be described as variations, I don't think they should be considered 
to be valid URIs.

I'm supportive of the strategy you outline (repeated here for ease of 
reference), which I don't think is so different from what I've argued for:

[[
A strategy that is more likely to be successful would be to identify URIs as 
being completely system independent, and URLs as being mostly system 
independent, and for there to be a well known and documented mechanism for 
converting from URLs to URIs.  Even that is not likely to be completely achieved 
-- the conversion may end up being (at least partially) system dependent, but in 
such cases we should be able to define the problematic set of the inputs as 
non-conforming.
]]
-- https://mailarchive.ietf.org/arch/msg/apps-discuss/lNsLrE3xDYpo2GyvgHzGGZv-PLI

Where I may diverge is that I don't think the "well known and documented 
mechanism for converting from URLs to URIs" should be part of the URI 
specification (cf. my point 2 above).

5. The previous point also begs the question of what should be covered by the 
file: scheme document.  I think it may be appropriate to describe some commonly 
occurring system-dependent file: URL forms, but I'm less convinced that this is 
the place to describe how to map them to URIs.  Any normative specification of 
file: URI formats should be restricted to forms that comply fully with RFC3986.

#g
--

On 02/01/2015 15:18, Sam Ruby wrote:
> I am making the claim that draft-kerwin-file-scheme does "work for everyone".
> Either there needs to be considerably more work done, or it needs to reduce its
> scope.
>
> Whether that document is separate or not will depend on the outcome of the
> discussion as to whether RFC 3986 matches current, deployed applications.
>
> Let me try to make that point in a different way then.  I'm skeptical that Apple
> will be interested in implementing any portion of a specification that is
> specific to Microsoft Windows.  This goes back to the point of trying to build a
> specification that "works for everyone".
>
> A strategy that is more likely to be successful would be to identify URIs as
> being completely system independent, and URLs as being mostly system
> independent, and for there to be a well known and documented mechanism for
> converting from URLs to URIs.  Even that is not likely to be completely achieved
> -- the conversion may end up being (at least partially) system dependent, but in
> such cases we should be able to define the problematic set of the inputs as
> non-conforming.
>
> An example of a restriction we should consider: valid schemes must have at least
> two characters.
>
> Can I get you to explain what you mean by this.  We can ignore operating system
> specific behaviors for the moment.  I would think that the basic operation of
> identifying the scheme, path, fragment, etc for a given input is exactly what a
> URI spec needs to define.  Why do you think otherwise and/or what am I missing?
>
>>>> <aside>
> We can agree that implementations don't match RFC 3986.  In such cases, where
> the bug is would be need to be determined on a case by case basis.
>
> I encourage you to spend more time with that data.  An example of a concrete
> problem is handing of hosts in a UTS-46 compliant manner.
>
>> #g
>> --
>
> - Sam Ruby
>

---

From: Sam Ruby <[email-masked]>
Date: Sat, 03 Jan 2015 08:09:08 -0500

On 01/03/2015 07:10 AM, Graham Klyne wrote:
>
> 3. Where there is divergence between implementations and RFC3986, these
> indeed should be considered on a case-by-case basis, but with (IMO) the
> presumption that RFC3986 is correct.  I.e. it is for those who think
> there is a problem with RFC3986 to make the case.

You ask me to presume that RFC 3986 is correct.  That's a big ask. 
Particularly given that there is no clear path provided for updating RFC 
3986.  For context, that's a decade old spec that I did not participate 
in the development of, and I have clear data that shows that popular 
parsing libraries -- client, sever, and everywhere in the middle -- 
don't implement.

> implementers think there is a problem with RFC 3986 then I think it is
> they who should be making the specific arguments about where RFC3986 is
> problematic.  I accept that UTS-46 host names is such an area,
> concerning which there should be a considered and focused debate (and
> about which I have insufficient knowledge to make a meaningful
> contribution).

I'm looking for people who are willing to look at data and work with 
implementors and document the result.  I am open to the possibility of 
some or all of that resulting documentation residing at the IETF.

I'll note that I neither created these test cases nor implemented any of 
the libraries (other than my reference implementation which is an 
attempt to track the evolving specification) in question.  I'm not going 
into this with any per-conceived notion as to who is right or who is 
wrong.  I am open to working with everybody, including people who 
believe that their preferred implementation or their preferred 
specification is right.

This is my starting point:

Feedback on pretty much everything on those pages is what I am looking 
for.  Examples would be: tests that should be added, implementations 
that should be added, bugs in how I am interpreting the results, 
different ways to present the results, changes to one or more specs that 
should be made based on these results, and changes that should be made 
to implementations based on these results.

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Sat, 03 Jan 2015 17:03:38 +0000

On 03/01/2015 13:09, Sam Ruby wrote:
> You ask me to presume that RFC 3986 is correct.  That's a big ask. Particularly
> given that there is no clear path provided for updating RFC 3986.  For context,
> that's a decade old spec that I did not participate in the development of, and I
> have clear data that shows that popular parsing libraries -- client, sever, and
> everywhere in the middle -- don't implement.

Presumption:  "An idea that is taken to be true on the basis of probability"
-- http://www.oxforddictionaries.com/definition/english/presumption

(Note: not "assumption".  I mean like "presumption of innocence")

For precisely the reasons you state (i.e. it's been around and in use for a 
while, and it has been the primary reference source for implementers who care 
about standards) I don't think it's a big ask.  But if you feel differently, 
then so be it.

#g
--

---

From: Sam Ruby <[email-masked]>
Date: Sat, 03 Jan 2015 12:54:12 -0500

On 01/03/2015 12:03 PM, Graham Klyne wrote:
>> Particularly

If you wish to give me a language lesson, the please permit me to do 
likewise.

Evidence: "the available body of facts or information indicating whether 
a belief or proposition is true or valid."

(Note: not "faith".  I mean like "scientific evidence")

> For precisely the reasons you state (i.e. it's been around and in use
> for a while, and it has been the primary reference source for
> implementers who care about standards) I don't think it's a big ask.
> But if you feel differently, then so be it.

It indeed has been around for a while.  As have a large number of 
implementations.  Many of which predate RFC 3986.

A large number of well-intentioned implementations do indeed give lip 
service to RFC 3986 compliance.  You may chose to accept those claims on 
face value.  As I have evidence to the contrary, sadly, I no longer do.

I intend to work with implementors, providing patches and/or new 
implementations along the way.  And I'll continue to document and 
publish findings.  One such place I have published such work is at the  W3C:

   http://www.w3.org/TR/url/

Meanwhile, I'm actively inquiring as to whether there are others who 
would be willing to help update RFC 3986 with me.  If that is not meant 
to be, so be it.

- Sam Ruby

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 03 Jan 2015 18:56:50 +0000

Hi Sam,

>   http://www.w3.org/TR/url/

I have at least one question about how you (or W3C, or any of us)
plan to head towards some reasonable level of completeness with
that work. (This may be a bit of an aside in the current discussion,
or maybe not, I'm not sure.)

The draft at the URL above includes [1], which is a risibly small
and fixed (?) subset of an IANA registry. [2] What's the plan for
making that sensible? I would assume pointing at the IANA registry
is the simple and obvious fix there, but am puzzled as to why that
hasn't been done in the few years this text has been around.

Is that just an oversight? Or is your work really only covering
exactly that particular subset of schemes? Or something else?

Thanks,
S.

[1] http://www.w3.org/TR/url/#relative-scheme
[2] https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml

---

From: Sean Leonard <[email-masked]>
Date: Sat, 03 Jan 2015 11:35:12 -0800

> hasn't been done in the few years this text has been around.

+1
Yeah man, just use the IANA registry...

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sat, 03 Jan 2015 12:06:11 -0800

Basically this entire reply is a big +1 in the hopes of converging on 
consensus and moving onwards.

On 1/3/2015 4:10 AM, Graham Klyne wrote:
> Sam,
>
> Rather than continue the blow-by-blow exchange of points, let me try 
> and respond in one place where I think we stand:
>
>
> 1. I agree that draft-kerwin-file-scheme *should* work for everyone.  
> And that probably means reducing the scope of anything there that may 
> be considered normative.
+1
> *informatively*.  I think this is discussed elsewhere and anticipate 
> evolution in this direction.  This could mean, to develop your 
> example, describing Microsoft Windows specific behaviours without any 
> expectation that such behaviours would be implemented by Apple.
+½

As I have stated previously, I do not believe that a Standards-Track RFC 
on this topic should enumerate local forms (i.e., Windows vs. Mac OS X 
vs. VMS vs. whatever). But informative text does not seem harmful to me.

>
>
> 2. I think our disagreement may lie primarily in the area of what 
> should be the scope of RFC 3986 or its successor.  There is (I claim) 
> a substantial developer community who are familiar with RFC3986 as it 
> stands, and creating a new document to cover the same material with 
> enlarged scope is unnecessary and disruptive. The additional scope 
> coverage could be in a new document that builds upon what RFC3986 does 
> specify.
>
>    Part of this disagreement is that I don't think the URI core spec 
> needs to describe the operation of splitting a URI into its 
> components.  (I regard https://tools.ietf.org/html/rfc3986#appendix-B 
> as merely informative.)  The syntax specification is sufficient to 
> associate substrings of a well-formed URI with named syntax productions.
+1
>
>    I also think the core URI spec should not be describing how to turn 
> non-URI strings (some of which may be system-dependent forms) into 
> valid URIs.  (Except URI-references, which are covered by the 
> resolution specification.)
+1
>
>
> 3. Where there is divergence between implementations and RFC3986, 
> these indeed should be considered on a case-by-case basis, but with 
> (IMO) the presumption that RFC3986 is correct.  I.e. it is for those 
> who think there is a problem with RFC3986 to make the case.
+1
>
>    You ask me to spend time with your data.  That's a big ask.  If 
> some implementers think there is a problem with RFC 3986 then I think 
> it is they who should be making the specific arguments about where 
> RFC3986 is problematic.  I accept that UTS-46 host names is such an 
> area, concerning which there should be a considered and focused debate 
> (and about which I have insufficient knowledge to make a meaningful 
> contribution).
+1. UTS-46 host names shouldn't affect the file: scheme any differently 
than anything else.

>
>
> 4. I agree with your point about qualifying my statement about 
> system-dependent forms conforming to core URI syntax.  While forms 
> such as "C:\Program Files (x86)" might be described as variations, I 
> don't think they should be considered to be valid URIs.
>
> mechanism for converting from URLs to URIs.  Even that is not likely 
> to be completely achieved -- the conversion may end up being (at least 
> https://mailarchive.ietf.org/arch/msg/apps-discuss/lNsLrE3xDYpo2GyvgHzGGZv-PLI
+1

>
> Where I may diverge is that I don't think the "well known and 
> documented mechanism for converting from URLs to URIs" should be part 
> of the URI specification (cf. my point 2 above).
+1 it's a local convention issue

>
>
> 5. The previous point also begs the question of what should be covered 
> by the file: scheme document.  I think it may be appropriate to 
> describe some commonly occurring system-dependent file: URL forms, but 
> I'm less convinced that this is the place to describe how to map them 
> to URIs.  Any normative specification of file: URI formats should be 
> restricted to forms that comply fully with RFC3986.
+1

There could be some value in having a "normative" file: URI 
format...which if anything, probably means that it is:
file://{HOST}/{root stuff}/{directories...}/{filename}

In an abstract non-system-specific way, that fully complies with RFC 3986.

Individual systems (Windows, Mac OS) can have other forms (e.g., \ 
[which doesn't comply with 3986 and thus is not a URI] or : [which does 
comply and therefore is a URI, but isn't analyzable in the 3986 
p-component way]), and those other forms are acceptable for their use, 
but they aren't the "normative" form (whether or not they are "URIs") 
and that's okay.

The end.

Sean

---

From: Sam Ruby <[email-masked]>
Date: Sat, 03 Jan 2015 15:46:02 -0500

> exactly that particular subset of schemes? Or something else?

This is a valid question, and the subject of an open bug:

https://www.w3.org/Bugs/Public/show_bug.cgi?id=27233

So the short answer is: it is a known issue, and suggestions are welcome.

The longer answer isn't all that much longer.  Given that every modern 
programming language (and for that matter, every browser) will have a 
part of their runtime library a concept of either a URI or a URL, and a 
method to parse a string into such a structure, the question you pose is 
equivalent to: "how should URI.parse methods handle unknown schemes"?

Possible answers include: treat the content as hierarchical, and treat 
the content as opaque.  There may be other answers.

What there probably needs to be is a sane default, and a way to register 
new schemes.  At the moment, the URL Working Draft treats unknown 
schemes as opaque.  The bug suggests that hierarchical might be a better 
choice.

As to registration, at the moment that is undefined.  The spec literally 
says "..." at this point:

   http://www.w3.org/TR/url/#url-writing

The hope is to work together with the authors of the following 
Internet-Draft:

   https://tools.ietf.org/html/draft-ietf-appsawg-uri-scheme-reg

This is mentioned in bullet 3 of the following section:

   https://tools.ietf.org/html/draft-ruby-url-problem-00#section-4

Meanwhile, patches are welcome!  It may be that there are certain URI 
schemes that defy conventional classification (file: certainly comes to 
mind, there may be others) that need to be specified explicitly in the 
specification.

The easiest way to participate is to propose tests in the form of input 
strings, base strings, and expected results.  That data will be added to:

And I'll use that data to update:

Should you feel inclined to suggest changes to the URL spec, I'd 
encourage you to look at the following which contains an incomplete but 
significantly reworked parser:

That repository has a bunch of other things, including the evaluation 
scripts and a reference implementation.  More information can be found here:

https://github.com/webspecs/url#the-url-standard

- Sam Ruby

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 03 Jan 2015 21:05:02 +0000

Hiya,

> So the short answer is: it is a known issue, and suggestions are welcome.

My suggestion: a) reference the IANA registry for schemes and b) the
same for port numbers and then c) list which subsets of those the
additions to 3986 (*) in the w3c document cover. That is, I don't think
you need to try provide additional rules/text for all schemes but you
do need to specify which you cover.

(*) I'm assuming a sensible model where [1] ends up as a set of
additional specifications on top of 3986 (or on top of 3986+errata
or on top of a 3986bis if one ends up being needed).

   [1] http://www.w3.org/TR/url/

>   https://tools.ietf.org/html/draft-ietf-appsawg-uri-scheme-reg

That's a good plan. It's (welcome) news to me though, but good to
hear. Are the editors of [1] and that draft in contact already?

> The easiest way to participate 

That depends on who you are I guess:-) For me, that is not the best
way (nor easiest) to participate.

Cheers,
S.

> is to propose tests in the form of input

---

From: Julian Reschke <[email-masked]>
Date: Sun, 04 Jan 2015 09:43:13 +0100

> values for each of the components) also works.

My understanding is that whatever IDNA-related problems might be in the 
spec, they won't be observable with parsing tests alone; we would need 
to observe that actually gets on the wire when (for instance) the HTTP 
request is made (such as what the DNS lookup is).

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 22:22:59 +1000

On 2 January 2015 at 00:21, Sam Ruby <[email-masked]> wrote:

> ​
> ​

​To be honest, a big reason for using RFC 3986 as a normative reference is
so I don't have to deal with things like hostnames. If those definitions
are good enough for the shiny new HTTP/1.1 (which actually uses hostnames)
then surely they're also sufficient for file: (which often doesn't).

BTW who on earth writes localhost as "2130706433"? Or more significantly,
who on earth *accepts* that as a valid URL?

​
> the selection criteria you are using?
For the most part: things that aren't silly (big-endian integer encoding of
IPv4 addresses), things that aren't widely touted as obsolete (like
backslashes[1]), and as much as possible, things that go against the parent
standards (like trying to jimmy "/c|/" into RFC 3986).

Importantly, I don't want to include every codepath in every implementation
that exists, but I don't want the definition I write to *not* work in any
implementations.

[1] http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 23:02:20 +1000

On 2 January 2015 at 01:57, Graham Klyne <[email-masked]> wrote:

> https://tools.ietf.org/html/draft-kerwin-file-scheme-13.  I think it's
> great that this is being tackled (again), and I'm hoping that there will be
> enough pragmatism this time round to achieve consensus on something that's
> an improvement over the status quo, even if not perfect.
>
> My comments follow...
>
>
> Section 2: windows-path
>
> I'm pretty sure I've come across '/c:/rest-of-path' as a mapping of
> Windows file paths (I'm not sure where; I may even have used it myself.  It
> makes some logical sense to me in that on Windows (non-UNC-naming) the
> drive letters are effectively the top-level directories in a single-root
> file system.)
>
> But that would subsumed by 'path-absolute'.  [Later] I see you have this
> covered as an example, and more.
>
>
Yes, the ABNF is currently being changed to remove a lot of unnecessary
redundancy and confusion like this, and also hopefully to resolve the
following point.

> As others have raised, there's an question of how to deal with common
> extensions to normal URI syntax, and separation of a preferred core for
> file: URI syntax from documentation of common variations.  If possible, my
> preference would be to treat non-conformance with RFC3986, like this, as a
> common variation.  I think 'core' file URIs should preferably be fully
> RFC3986 conformant so that existing generic parsers will handle them.
>
> I think the document might be clearer in its intent if at the start of
> section 2, and in the syntax productions, it were more clearly signaled
> that the syntax described here includes commonly-used forms that go beyond
> RFC3986 syntax.
>
>
​The current plan is to move it to a non-normative appendix as a "commonly
used but invalid" thingy, so the normative construct can be simpler.​

> See below.  I think this is a non-relevant distinction.  e.g. Is a remote
> file system mounted on a Unix-style mount point supposed to include an
> authority?
>
>
Probably not. My intention was to distinguish things that don't have an
authority (can use local filesystem calls to access) from those that
require an authority (must be opened some other way, like a different
syscall, or by implementing some other protocol). This part of the language
would benefit from having more peoples' input.

> I'm not sure I agree that translation to filename is the  *only* operation
> that can be done using a file: URI.  And I have concerns about use of the
> term "method" here, which is strongly suggestive of HTTP "methods".
> ​
> ​
>
> ​
> ​

​RE the only operation: rather than deal with trying to define (or
reasonably leave unspecified) the set of operations that are made available
by the local filesystem API, I went a bit reductive. I'm not sure of a
better way forward, other than to make the language less strong.

RE the term "method": actually that's my OO programming background shining
through, but yes, BCP 35 or 115 or whatever it's called uses "operations"
so I can change to conform with that.

​
>
> ​​
> Specifically, dereferencing:  I have frequently used APIs that retrieve
> the entity referenced by a URI, including file: URIs.  Indeed, in much of
> my code, this is central to unifying web and local file access.   I think
> retrieval is an operation that is very naturally performed on a file URI,
> with results broadly equivalent to an HTTP GET.  Some analogue to HTTP PUT
> is also conceivable.
> ​​
>
> ​
> ​

Yes, that's fair, I was being overly reductive. Presumably file: URIs ought
to support the full CRUD suite, assuming the underlying filesystem allows
it.

​
> performed is restricted, but the details of how to perform them, which will
> vary from system to system.
>
> A common case I've come across is where the authority is "localhost", in
> which case it seems reasonable to allow local file access based the file:
> URI.
> ​​
>
> ​
> ​

​I've swung back and forth on this particular topic. The issue is that some
folk (including Chrome) treat "file://localhost/foo" as \\localhost\foo
(i.e. a samba file share served from this machine), and when it comes to
choosing one way or the other, this time I went with Chrome.

This is a change (potentially a big one?) from 1738, but I guessed it was
reasonable.

​
>
> More generally, it seems to me that the difference between local and
> remote files (with or without host) is in the mechanism that might be used
> to access the file.  In some cases, it may be the same, by virtue of being
> handled by the underlying system's file system.  And in some cases, a
> filename that appears to have no host may in fact be remote (cf. remote
> file system, mounted on Unix-like file system).  I think that trying to
> make this distinction raises more questions than it answers.
>
>
That may be the case. Is it better to do a 1738 and not say anything useful
about a non-blank, non-localhost authority?

> But I think it would be more consistent, and would avoid potential
> ambiguities in relative reference resolution, to prefer the former;  i.e.
> to consider that drives are like top-level sub-directories of a root "/".
> I think this section could also be thereby simplified a little.
>
>
​To be perfectly honest, if someone can propose something about the root
that makes my life easier, I'll welcome it with open arms. My struggle has
been with addressing the UNIX root directory, and now you're suggesting I
also include it in Windows. I guess if it's ubiquitous it saves me the
burden of trying to describe it, or explain when not to use it.​

> bitten in the past by doing roughly what you describe here, with filenames
> that happen to contain (say) '#' characters.
>
>
Isn't that covered by "Transform the directory name to a path segment (
[RFC3986], <https://tools.ietf.org/html/rfc3986#section-3.3> Section 3.3
<https://tools.ietf.org/html/rfc3986#section-3.3>) as per Section 2 of
[RFC3986] <https://tools.ietf.org/html/rfc3986#section-2>."?

>
> Section 6:
>
> Can you please include a full registration template for the file: scheme
> here. cf.  https://tools.ietf.org/html/rfc4395#section-5.4
>
It used to be like that until version 12, when somebody (I can't remember
who) suggested it could be simplified to what it currently is.

> Again, thanks for tackling this.
​No worries, thanks for the comments.​

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 23:11:22 +1000

> widely deployed applications.
So... are you advocating we make file: non-standard? That seems to be what
both linked posts suggest. That would go with Dave Thaler's earlier
proposition, that we move it to Historical, and get rid of it.

Not exactly what I had in mind coming into this.

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 23:20:43 +1000

On 3 January 2015 at 22:10, Graham Klyne <[email-masked]> wrote:

> Sam,
>
> Rather than continue the blow-by-blow exchange of points, let me try and
> respond in one place where I think we stand:
>
>
> 1. I agree that draft-kerwin-file-scheme *should* work for everyone.  And
> that probably means reducing the scope of anything there that may be
> considered normative.
>
> evolution in this direction.  This could mean, to develop your example,
> describing Microsoft Windows specific behaviours without any expectation
> that such behaviours would be implemented by Apple.
>
>
This is in the works. I've had a computer die, and been in full time
vacation mode for the past month, so I'm a bit behind schedule, but I'll
have something for folks to look at as soon as I can.

> system-dependent forms conforming to core URI syntax.  While forms such as
> "C:\Program Files (x86)" might be described as variations, I don't think
> they should be considered to be valid URIs.
>
> A strategy that is more likely to be successful would be to identify URIs
> as being completely system independent, and URLs as being mostly system
> converting from URLs to URIs.  Even that is not likely to be completely
> achieved -- the conversion may end up being (at least partially) system
> dependent, but in such cases we should be able to define the problematic
> set of the inputs as non-conforming.
> ]]
Is there a reference somewhere that distinguishes URIs and URLs the way you
guys are here? I thought URL was still a subset of URI. Has that changed?

>
> 5. The previous point also begs the question of what should be covered by
> the file: scheme document.  I think it may be appropriate to describe some
> commonly occurring system-dependent file: URL forms, but I'm less convinced
> that this is the place to describe how to map them to URIs.  Any normative
> specification of file: URI formats should be restricted to forms that
> comply fully with RFC3986.
>
>
Indeed. The normative core of the spec will be the pure RFC 3986-compatible
parts; the non-normative appendices will include bad things like pipe
characters and backslashes and such. I think it would be helpful to
*suggest* how to map those to valid URIs, at the very least so people can
update their old documents.

---

From: Sam Ruby <[email-masked]>
Date: Sun, 04 Jan 2015 08:32:57 -0500

On 01/04/2015 07:22 AM, Matthew Kerwin wrote:
>
> On 2 January 2015 at 00:21, Sam Ruby <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     http://www.ietf.org/mail-__archive/web/apps-discuss/__current/msg13511.html
>     <http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13511.html>
>
> ​To be honest, a big reason for using RFC 3986 as a normative reference
> is so I don't have to deal with things like hostnames. If those
> definitions are good enough for the shiny new HTTP/1.1 (which actually
> uses hostnames) then surely they're also sufficient for file: (which
> often doesn't).
>
> BTW who on earth writes localhost as "2130706433"? Or more
> significantly, who on earth *accepts* that as a valid URL?

I cited three specific examples in:

All three of the below are accepted by Chrome, and possibly others:

file://2130706433/foo
file://[2001::1]/foo
file://０Ｘｃ０．０２５０．０１/foo

Note that RFC 3986 requires square brackets around IPv6 names, but your 
draft does not.  The final example involves IDNA processing. 
Unfortunately, IDNA processing can't be avoided, nor does RFC 3986 
specify it.

>     I don't understand the thought process here. The current
>     Internet-Draft takes care to document a select few browser specific
>     syntaxes, but many others (including the ones you mention above) are
>     not included.  What is the selection criteria you are using?
>
> For the most part: things that aren't silly (big-endian integer encoding
> of IPv4 addresses), things that aren't widely touted as obsolete (like
> backslashes[1]), and as much as possible, things that go against the
> parent standards (like trying to jimmy "/c|/" into RFC 3986).
>
> Importantly, I don't want to include every codepath in every
> implementation that exists, but I don't want the definition I write to
> *not* work in any implementations.
>

You have a definition for a "windows-path" and a "unc-path".  These are 
unlikely to work on Android or OS/X.

Firefox does not appear to support host for file: URIs.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Sun, 04 Jan 2015 08:47:20 -0500

On 01/04/2015 08:11 AM, Matthew Kerwin wrote:
>
>
> On 2 January 2015 at 03:29, Sam Ruby <[email-masked]
>         https://specs.webplatform.org/__url/webspecs/develop/
>         <https://specs.webplatform.org/url/webspecs/develop/> and:
>
>     http://lists.w3.org/Archives/__Public/public-webapps/__2014OctDec/0526.html
>     <http://lists.w3.org/Archives/Public/public-webapps/2014OctDec/0526.html>
>
>     https://mailarchive.ietf.org/__arch/msg/apps-discuss/__KyxFLw2FCwv7CSpiWAmLr8Ha7vc
>     <https://mailarchive.ietf.org/arch/msg/apps-discuss/KyxFLw2FCwv7CSpiWAmLr8Ha7vc>
>
>     It is my hope that by working together I can feel confident enough
>     to remove that red box.  As it is, I don't feel that either spec
>     matches widely deployed applications.
>
> So... are you advocating we make file: non-standard? That seems to be
> what both linked posts suggest. That would go with Dave Thaler's earlier

Some indeed are proposing essentially that.  I'm not:

https://www.w3.org/Bugs/Public/show_bug.cgi?id=27518#c1

What I will say is that if you drop normative statements concerning 
things that aren't interoperable (like hosts and windows paths), you end 
up with a rather small specification.

- Sam Ruby

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 23:47:52 +1000

> unlikely to work on Android or OS/X.

They can (and should) parse them. Then they can (and should) fail to open
them. What's wrong with that?

To get rid of "windows-path", as I said earlier tonight, if someone can
give me better text to deal with roots then by all means please do so.

​
> Firefox does not appear to support host for file: URIs.

Firefox does a couple of things in its own way. I've wavered back and forth
about whether it's "file://host/path" or "file:////host/path" (or
"file://///host/path" as Firefox says), and in the end I went with the one
that seems to be fairy widely (if not universally) supported, and in the
spirit of how URIs work in the modern age. There's an open bug somewhere
about how Firefox interprets UNC-type file: URIs. Perhaps if this draft
gets published, in some form or other, that will be the catalyst required
to get said bug fixed/resolved.

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 4 Jan 2015 23:54:26 +1000

On 4 January 2015 at 23:47, Sam Ruby <[email-masked]> wrote:

>
> What I will say is that if you drop normative statements concerning things
> that aren't interoperable (like hosts and windows paths), you end up with a
> rather small specification.
>
>
I'm working on it. <​
https://github.com/phluid61/internet-drafts/blob/15-separate-norm-inform/file-scheme/draft.md

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Jan 2015 17:56:52 +0100

> the content as opaque.  There may be other answers.

Such as: treat everything the same (as per RFC 3986).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Jan 2015 17:57:54 +0100

> would be willing to help update RFC 3986 with me.  If that is not meant

I am willing to help to update RFC 3986 once we actually know precisely 
what needs to be done.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 08 Jan 2015 10:27:59 +0900

> I am willing to help to update RFC 3986 once we actually know precisely

Same here for RFC 3987. I know some things that need to be done, but I 
get the impression that others think other things need to be done, 
without being able to understand what exactly.

Regards,   Martin.

---

From: Graham Klyne <[email-masked]>
Date: Thu, 08 Jan 2015 12:15:18 +0000

On 03/01/2015 17:54, Sam Ruby wrote:
> Meanwhile, I'm actively inquiring as to whether there are others who would be
> willing to help update RFC 3986 with me.  If that is not meant to be, so be it.

I would be happy to review specific problem statements and proposals for updates 
relating to RFC 3986, and to help with drafting of same where I feel able.

#g
--

---

From: t.petch <[email-masked]>
Date: Sat, 10 Jan 2015 11:23:12 +0000

----- Original Message -----
From: "Sam Ruby" <[email-masked]>
To: "Matthew Kerwin" <[email-masked]>
Sent: Sunday, January 04, 2015 1:32 PM
> >     A few concrete examples are here:
http://www.ietf.org/mail-__archive/web/apps-discuss/__current/msg13511.h
tml
<http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13511.html
> Unfortunately, IDNA processing can't be avoided, nor does RFC 3986

Sam

I part company with you here.  IDNA processing can be avoided - don't do
it!

Many of the IETF standards have no i18n in general, or IDNA in
particular, just as the early standards' views of security would have
hackers licking their lips with glee.

Retrospectively, security has been added to most protocols, i18n has
been added to many.  The lack of security is a show stopper for most,
but the lack of i18n limits the scope but still renders it useful for
many so Matthew's I-D I see as a good step forward, all it needs to say
is on this is that i18n (or IDNA) is for further study and we would
still have a valuable RFC.

I note that kitten is currently wrestling with normalisation in the
context of Kerberos, where a loophole could generate a major security
breach, and seem to be heading towards saying, if you are not an expert
in this area, do not attempt to implement it (my words) i.e. this
i18n/IDNA/normalisation area is not well enough engineered to be relied
on in some areas.

Tom Petch

<snip>

---

From: Sam Ruby <[email-masked]>
Date: Sat, 10 Jan 2015 11:39:30 -0500

> still have a valuable RFC.

If that is all it says, then I think the result would be confusing.  If 
it went further and said that it does not represent what a number of 
popular implementations do (including, at a minimum, current browsers), 
that would be fine.  If it went that way, it should also remove other 
mentions of what those browsers do.

> i18n/IDNA/normalisation area is not well enough engineered to be relied

Saying that this is a specification for Kerberos would be fine.

- Sam Ruby

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Jan 2015 17:33:18 +0000

t.petch Sent: Saturday, January 10, 2015 3:23 AM

> I part company with you here.  IDNA processing can be avoided - don't
> do it!
> 

I'm not sure, but I think you're arguing that the "file:" URL scheme not specify the behavior of dealing with non-ASCII filenames. I don't think a document that didn't spec something would meet the criteria for Proposed Standard. 

There is a serious proposal to make some of the options of handling "file:" into "implementation dependent", but with leaving some kinds of relative path names interoperable, and  a discussion about how to structure the document so that implementations can be considered, registered, or described independently. About whether some of the specifications needed for normative reference are proprietary.

We actually have data about how implementations respond. There is an opportunity to make this corner of the web more openly specified so application writers can build platform-independent code using an interoperable subset, and platform builders and maintainers can know what to code to.

"for further study" is not generally found in IETF standards, although it is a technique used in academic papers and seen in other SDOs. We've studied. We're making progress. Keep going.

Larry
--
http://larry.masinter.net

---

From: Sean Leonard <[email-masked]>
Date: Sat, 10 Jan 2015 10:15:07 -0800

[SNIPS]
> still have a valuable RFC.
(+1...?)

Look here's the deal as I see it. URIs are made of the following 
*characters* only:
ALPHA DIGIT - . _ ~
: / ? # [ ] @
! $ & ' ( ) * + , ; =
%

That's it.

Folks are confusing the "web browser text box at the top" with URIs. Web 
browsers can display and accept whatever characters they want in the 
address box. How they do that is not an RFC 3986 issue. It might be 
governed by some de-jure or de-facto Web "standards" (or RFC 3987?), but 
it's not directly relevant to URIs. The end.

Nowadays it's not even a "URL bar" or "address bar". It's variously 
called an "omnibox" and has a lot of code running behind it to do 
context-sensitive things with lots of additional user experience 
widgets. For example if you type "? ietf" in Chrome (and others...I 
believe Internet Explorer 6 had this) it's going to do something that 
has nothing to do with URIs.

With respect to file: URIs, if this is a draft about file: URIs, stick 
with what URIs are. See RFC 3986. (Notably, an advantage of clean 
specifications around RFC 3986 is that you can broadly detect what 
things are NOT URIs. E.g., "? ietf" is NOT a URI so a magic box can put 
it in a different category, and a command-line tool can do something 
intelligent that is not related to URI retrieval.)

Thanks,

Sean

---

From: t.petch <[email-masked]>
Date: Sat, 10 Jan 2015 18:24:29 +0000

----- Original Message -----
From: "Larry Masinter" <[email-masked]>
To: "t.petch" <[email-masked]>; "Sam Ruby"
<[email-masked]>; "Matthew Kerwin" <[email-masked]>
Cc: "IETF Apps Discuss" <[email-masked]>
Sent: Saturday, January 10, 2015 5:33 PM
> I'm not sure, but I think you're arguing that the "file:" URL scheme
not specify the behavior of dealing with non-ASCII filenames. I don't
think a document that didn't spec something would meet the criteria for
Proposed Standard.
>
> There is a serious proposal to make some of the options of handling
"file:" into "implementation dependent", but with leaving some kinds of
relative path names interoperable, and  a discussion about how to
structure the document so that implementations can be considered,
registered, or described independently. About whether some of the
specifications needed for normative reference are proprietary.
>
> We actually have data about how implementations respond. There is an
opportunity to make this corner of the web more openly specified so
application writers can build platform-independent code using an
interoperable subset, and platform builders and maintainers can know
what to code to.
>
> "for further study" is not generally found in IETF standards, although
it is a technique used in academic papers and seen in other SDOs. We've
studied. We're making progress. Keep going.

Larry,

Indeed we do not but we do have Applicability Statements which seem to
me to be semantically the same.

Thinking laterally, I see that
http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-35#section-12
is in the RFC Editor's Queue and so has IETF consensus.  And by our IPR
rules we can lift any text we like from it, and it seems a very good
summary of the situation, perhaps a bit long for our purposes but the
basis of a way forward. (not just for file:// but for any other scheme
or indeed for other IETF work).

I am fine with vendor extensions - where would a vendor be if they did
not have them? - but am resistant to making them a standard, rather a
non-normative appendix - some WGs would have a separate I-D for an
implementation repoirt but I find that overcomplex.  Equally, if such
examples cover three or more widely differing approaches, I find the
argument that we have omitted Brand X spurious.  Let Brand X raise their
own ISE submission, do not let them delay the production of a useful
RFC.

Tom Petch

---

From: t.petch <[email-masked]>
Date: Sat, 10 Jan 2015 18:50:52 +0000

----- Original Message -----
From: "Sean Leonard" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, January 10, 2015 6:15 PM

Sean

Indeed it does.  Firing up my trusty IE6 and entering '? ietf', I am
taken to bing with q=ietf and there is all I need.

Great vendor extension but no way should a Standards Track RFC go to
such a level of detail (not that you were suggesting it should)..

Tom Petch

---

From: Sam Ruby <[email-masked]>
Date: Sat, 10 Jan 2015 14:21:34 -0500

> Look here's the deal as I see it. URIs are made of the following

I call strawman.  Nobody here mentioned a quote "web browser text box at 
the top" end-quote.

> intelligent that is not related to URI retrieval.)

I'm working on a specification how strings that can found in the href 
attribute of HTML elements like <link>, <base>, and <a>.  Some of those 
strings may start with "file:".

I think it would be a good idea for us to explore whether there is a 
relationship between those strings and RFC 3986.  I'll go further and 
state that it is in nobody's best interest for there to be overlapping 
and conflicting standards in this area.

- Sam Ruby

---

From: Sean Leonard <[email-masked]>
Date: Sat, 10 Jan 2015 12:28:17 -0800

On 1/10/2015 11:21 AM, Sam Ruby wrote:
> I call strawman.  Nobody here mentioned a quote "web browser text box 
> at the top" end-quote.
>
> attribute of HTML elements like <link>, <base>, and <a>.  Some of 
> those strings may start with "file:".
>
http://www.w3.org/TR/html401/struct/links.html#h-12.1

http://www.w3.org/TR/html401/types.html#type-uri

"This specification uses the term URI as defined in[URI] 
<http://www.w3.org/TR/html401/references.html#ref-URI>(see also[RFC1630] 
<http://www.w3.org/TR/html401/references.html#ref-RFC1630>)."

[URI] = RFC 2396
"Uniform Resource Identifiers (URI): Generic Syntax" 
<http://www.ietf.org/rfc/rfc2396.txt>, T. Berners-Lee, R. Fielding, L. 
Masinter, August 1998. Note that RFC 2396 updates[RFC1738] 
<http://www.w3.org/TR/html401/references.html#ref-RFC1738>and[RFC1808] 
<http://www.w3.org/TR/html401/references.html#ref-RFC1808>.

 From the DTD:

<!ATTLIST A
...
   href  <http://www.w3.org/TR/html401/struct/links.html#adef-href>         %URI;  <http://www.w3.org/TR/html401/sgml/dtd.html#URI>           #IMPLIED  -- URI for linked resource --

...

<!ENTITY %URI  "CDATA  <http://www.w3.org/TR/html401/types.html#type-cdata>"
     -- a Uniform Resource Identifier,
        see[URI]  <http://www.w3.org/TR/html401/references.html#ref-URI>
     -->

**********
HTML5:

http://www.w3.org/TR/html5/links.html#attr-hyperlink-href

The|href|attribute on|a 
<http://www.w3.org/TR/html5/text-level-semantics.html#the-a-element>|and|area 
<http://www.w3.org/TR/html5/embedded-content-0.html#the-area-element>|elements 
must have a value that is avalid URL potentially surrounded by spaces 
<http://www.w3.org/TR/html5/infrastructure.html#valid-url-potentially-surrounded-by-spaces>.

A string is avalid URL potentially surrounded by spacesif, 
afterstripping leading and trailing whitespace 
<http://www.w3.org/TR/html5/infrastructure.html#strip-leading-and-trailing-whitespace>from 
it, it is avalid URL 
<http://www.w3.org/TR/html5/infrastructure.html#valid-url>.

AURLis avalid URLif it conforms to the authoring conformance 
requirements in the URL standard.[URL] 
<http://www.w3.org/TR/html5/references.html#refsURL>

[URL]

    *Note:*URLs can be used in numerous different manners, in many
    differing contexts. For the purpose of producing strict URLs one may
    wish to consider[RFC3986]
    <http://www.w3.org/TR/html5/references.html#refsRFC3986>[RFC3987]
    <http://www.w3.org/TR/html5/references.html#refsRFC3987>. The W3C
    URL specification defines the term URL, various algorithms for
    dealing with URLs, and an API for constructing, parsing, and
    resolving URLs. Developers of Web browsers are advised to keep
    abreast of the latest URL developments by tracking the progress
    ofhttps://url.spec.whatwg.org/. We expect that the W3C URL draft
    will evolve along the Recommendation track as the community
    converges on a definition of URL processing.

    Most of the URL-related terms used in the HTML specification
    (*URL*,*absolute URL
    <http://www.w3.org/TR/html5/infrastructure.html#absolute-url>*,*relative
    URL
    <http://www.w3.org/TR/html5/infrastructure.html#relative-url>*,*relative
    schemes*,*scheme component*,*scheme
    data*,*username*,*password*,*host*,*port*,*path*,*query*,*fragment*,*percent
    encode
    <http://www.w3.org/TR/html5/infrastructure.html#percent-encode>*,*get the
    base*, and*UTF-8 percent encode
    <http://www.w3.org/TR/html5/infrastructure.html#utf-8-percent-encode>*)
    can be straightforwardly mapped to the terminology of[RFC3986]
    <http://www.w3.org/TR/html5/references.html#refsRFC3986>[RFC3987]
    <http://www.w3.org/TR/html5/references.html#refsRFC3987>.
    The|URLUtils
    <http://www.w3.org/TR/html5/infrastructure.html#urlutils>|(formerly
    known as|URL|) collection of attributes (e.g.|href|and|protocol|)
    and its required definitions (*input
    <http://www.w3.org/TR/html5/forms.html#the-input-element>*,*query
    encoding*,*url*,*update steps*,*set the input*) are considered
    common practice nowadays. Some of the URL-related terms are still
    being refined (e.g.*URL parser
    <http://www.w3.org/TR/html5/infrastructure.html#url-parser-0>*,*parse errors*,*URL
    serializer*,*default encode set
    <http://www.w3.org/TR/html5/infrastructure.html#default-encode-set>*, and*percent
    decode
    <http://www.w3.org/TR/html5/infrastructure.html#percent-decode>*).

    As a word of caution, there are notable differences in the manner in
    which Web browsers and other software stacks outside the HTML
    context handle URLs. While no changes would be accepted to URL
    processing that would break existing Web content, some important
    parts of URL processing should therefore be considered as
    implementation-defined (e.g. parsing file: URLs or operating on URLs
    that would be syntax errors under the[RFC3986]
    <http://www.w3.org/TR/html5/references.html#refsRFC3986>[RFC3987]
    <http://www.w3.org/TR/html5/references.html#refsRFC3987>syntax).

    URL <http://www.w3.org/TR/url/>(URL:http://www.w3.org/TR/url/), E.
    Arvidsson, M.[tm] Smith. W3C.

******************************

---

From: Sam Ruby <[email-masked]>
Date: Sat, 10 Jan 2015 16:05:57 -0500

>> and conflicting standards in this area.

I'm not sure what point you are trying to make as all you have provided 
are links and excerpts.  But I'll try to provide some background on the 
information you quoted.

>         see[URI]  <http://www.w3.org/TR/html401/references.html#ref-URI>

An obsolete W3C recommendation indeed points to an obsolete RFC.

Permit me to introduce myself.  I'm one of the co-chairs of the W3C HTML 
Working group.  I am indeed aware of this work.

>     Arvidsson, M.[tm] Smith. W3C.

I'm indeed aware of much of that text as I contributed to it.  I'm also 
actively working to make that text unnecessary.

Since that has been published, I've personally updated two of the 
documents mentioned in that prose: https://url.spec.whatwg.org/ and 
http://www.w3.org/TR/url/.

I've also gathered a fair amount of data looking into interop issues:

More background can be found here:

- Sam Ruby

---

From: Sean Leonard <[email-masked]>
Date: Sat, 10 Jan 2015 15:13:21 -0800

On 1/10/2015 1:05 PM, Sam Ruby wrote:
>
> Permit me to introduce myself.  I'm one of the co-chairs of the W3C 
> HTML Working group.  I am indeed aware of this work.

Pleased to meet you!

[various other text]

I believe that the relevant prompting text is:
---
I'm working on a specification how strings that can found in the href
attribute of HTML elements like <link>, <base>, and <a>.  Some of
those strings may start with "file:".

---

Let us agree that RFC 3986 === "URI". A URI is supposed to be made up of 
characters *entirely within* the repertoire I previously posted (taken 
from RFC 3986).

The issue at hand is that a lot of software uses Unicode, and in 
particular, there are protocol slots and data entry fields that are 
labeled "URI" or "URL" in some fashion, but do not limit themselves to 
the RFC 3986 repertoire. I.e., in <a href=""> etc. you can have Unicode 
characters, so the question is, what should you do with them?

That is a very interesting question and one that I do not particularly 
care to wade into. So, I guess I am sidestepping the prompt. :)

But the subject of this thread is about registering the file: URI scheme 
under the current rules in force, which are RFC 3986 and RFC 4395. (And, 
I suppose, RFC 3987.)

So it seems to me that the first and "most important" part of 
draft-kerwin-file-scheme should clearly state how file: URIs work, 
according to the character repertoire permitted by RFC 3986.

Then if the author wants to get into the other issues, i.e., dealing 
with the presence of non-URI characters (including Unicode characters 
that keep the string following IRI), that can be done in a subsequent 
section. (Which can be a "normative" section of the main text, if we can 
get agreement on all of that.)

Note that RFC 4395 says:

      2.6 <http://tools.ietf.org/html/rfc4395#section-2.6>.
      Internationalization and Character Encoding

    When describing URI schemes in which (some of) the elements of the
    URI are actually representations of human-readable text, care should
    be taken not to introduce unnecessary variety in the ways in which
    characters are encoded into octets and then into URI characters; see
    RFC 3987  <http://tools.ietf.org/html/rfc3987>  [6  <http://tools.ietf.org/html/rfc4395#ref-6>] andSection 2.5 of RFC 3986  <http://tools.ietf.org/html/rfc3986#section-2.5>  [5  <http://tools.ietf.org/html/rfc4395#ref-5>] for guidelines.  If URIs
    of a scheme contain any text fields, the scheme definition MUST
    describe the ways in which characters are encoded, and any
    compatibility issues with IRIs of the scheme.

******

The text says not to introduce "unnecessary variety", but in the file: 
case, maybe some variety is natural and therefore necessary.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sat, 10 Jan 2015 15:27:52 -0800

On 1/10/2015 11:21 AM, Sam Ruby wrote:
>>> particular,
[SNIP]
> I call strawman.  Nobody here mentioned a quote "web browser text box 

It's not really a strawman: there are different experiences at play. I 
do a lot of security work; in certificates, a URI (GeneralName -> 
uniformResourceIdentifier) is stored in an IA5String data type, which is 
US-ASCII only. Therefore unlike a blob of HTML, arbitrary Unicode 
characters cannot possibly be present. The example 
<file://０Ｘｃ０．０２５０．０１/foo> provided literally cannot exist in 
that format; it would be treated as a decoding error.

Under the (obsolete...but still widely used) HTML 4.01, it says <a 
href=""> is supposed to have only %URI characters, so the presence of 
non-URI characters is either an error (if the DTD is strictly enforced), 
or results in undefined/implementation-defined behavior (which is what 
actually happens, since the DTD almost never gets strictly enforced...by 
web browsers). I suppose that the newer standards are intended to clean 
that up...

Sean

---

From: Sam Ruby <[email-masked]>
Date: Sat, 10 Jan 2015 18:32:18 -0500

>> HTML Working group.  I am indeed aware of this work.

Likewise!

> characters *entirely within* the repertoire I previously posted (taken

Let's not.  I believe that there needs to be a RFC 3986bis created, and 
therefore a Working Group chartered.

As a concrete example, I believe that the repertoire of characters 
available for fragment identifiers should be expanded.

And related to the original discussion, domains containing Unicode 
characters do now exist, and IDNA specifies the mapping of those 
characters to the set of characters that RFC 3986 allows.  RFC 3986 
needs to be updated to cover this.

> according to the character repertoire permitted by RFC 3986.

This very question changes when the possibility of a RFC 3986bis enters 
the discussion.

>     RFC 3987  <http://tools.ietf.org/html/rfc3987>  [6
> <http://tools.ietf.org/html/rfc4395#ref-6>] andSection 2.5 of RFC 3986
> <http://tools.ietf.org/html/rfc3986#section-2.5>  [5
> <http://tools.ietf.org/html/rfc4395#ref-5>] for guidelines.  If URIs

I actually would argue that the advice RFC 4395 provides be followed. 
And that's precisely why IDNA is relevant here.

- Sam Ruby

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 11 Jan 2015 10:05:36 +1000

> therefore a Working Group chartered.
So, if and when such a group is chartered, and if it's subsequently
determined that work produced by that group should pause all ongoing work
with specifying URI schemes, and if a new 'file' spec isn't already
published before that, then of course we'd roadblock a new 'file'
draft/spec.

But we're not there yet. I've no intention to rewrite 3986, nor do I want
to publish a draft that counters it. Least of all 'file', which has such a
history of going against the grain -- the whole point of doing this is to
pull it back into the flock.

---

From: Sean Leonard <[email-masked]>
Date: Sat, 10 Jan 2015 18:23:21 -0800

On 1/10/2015 4:05 PM, Matthew Kerwin wrote:
> On 11 January 2015 at 09:32, Sam Ruby <[email-masked] 
> <mailto:[email-masked]>>wrote:
>
>     Let's not.  I believe that there needs to be a RFC 3986bis
>     created, and therefore a Working Group chartered.
>
>
> determined that work produced by that group should pause all ongoing 
> work with specifying URI schemes, and if a new 'file' spec isn't 
> already published before that, then of course we'd roadblock a new 
> 'file' draft/spec.
>
> But we're not there yet. I've no intention to rewrite 3986, nor do I 
> want to publish a draft that counters it. Least of all 'file', which 
> has such a history of going against the grain -- the whole point of 
> doing this is to pull it back into the flock.

+1...my sentiments exactly...keep the train moving on the existing rails.

Sean

---

From: t.petch <[email-masked]>
Date: Sun, 11 Jan 2015 11:22:02 +0000

----- Original Message -----
From: "Sam Ruby" <[email-masked]>
To: "Sean Leonard" <[email-masked]>; <[email-masked]>
Sent: Saturday, January 10, 2015 11:32 PM
> needs to be updated to cover this.

An interesting idea.  I have seen masses of Working Groups chartered
while I have been contributing to the IETF and for all, I do not know
where the first impetus comes from.  Certainly it is up to the IESG to
approve, approve the charter and so on, and most start with two
IESG-approved BoFs at IETF meetings and these BoFs seem to come from WG
Chairs or authors whose work I know well.  The IETF website lists
current BoFs.  There is a deadline for requesting BoFs prior to any
meeting such as IETF92 but I am not sure when that is.  If you want it
to happen, that is what needs to happen.

I get the impression that the worst outcome is forming a WG which then
decays, produces nothing, costs a lot, so the IESG looks for an awful
lot of enthusiasm to start with, knowing that most of it will be gone
within the year.

Tom Petch

<snip>

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 08:18:20 -0500

> to happen, that is what needs to happen.

2015-02-06 (Friday): Cut-off date for BOF proposal requests to Area 
Directors at UTC 23:59.

http://www.ietf.org/meeting/important-dates-2015.html

> lot of enthusiasm to start with, knowing that most of it will be gone

By spring, I should be able to get browser vendors together in a room 
and we should be able to review the following results together:

Based on those discussions, the following specs will be updated:

https://url.spec.whatwg.org/
http://www.w3.org/TR/url/

I also plan to work with library authors for various popular programming 
languages with the intent of making their URI/URL libraries more web 
compatible.  This work may take various forms: bug reports, patches, or 
even new libraries.

This work is all being done in the open, and I'm making an attempt to 
reach out to everybody who may be interested in participating.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 15:51:59 +0100

> available for fragment identifiers should be expanded.

Could you provide a bit more information? Example for a character 
missing? And the reason to add it?

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 10:14:51 -0500

> missing? And the reason to add it?

More details available here:

https://www.w3.org/Bugs/Public/show_bug.cgi?id=27252

As luck would have it, I had a chance to discuss this with Tim 
Berners-Lee in person on Wednesday.  A typical URI parsing library will 
treat everything after the hash mark as a fragment.

The current proposal can be found here:

https://specs.webplatform.org/url/webspecs/develop/#fragment

Here are the proposed set of conforming characters:

https://specs.webplatform.org/url/webspecs/develop/#url-code-points

Additionally, the presence of a percent sign not immediately followed by 
two ASCII hex digits would also be a conformance error.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 16:53:40 +0100

> two ASCII hex digits would also be a conformance error.

Now suffering from information overflow.

We were discussing RFC 3986. Which *ASCII* characters that are currently 
forbidden in fragment identifiers do you want to allow?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 11 Jan 2015 11:04:39 -0500

One more point of data - 
  https://indiewebcamp.com/fragmention

Note that they’re using a second #, which isn’t conformant to the URI ABNF.

Cheers,

> On 11 Jan 2015, at 10:14 am, Sam Ruby <[email-masked]> wrote:
> 
> As luck would have it, I had a chance to discuss this with Tim Berners-Lee in person on Wednesday.  A typical URI parsing library will treat everything after the hash mark as a fragment.
> 
> Additionally, the presence of a percent sign not immediately followed by two ASCII hex digits would also be a conformance error.
> 
>> Best regards, Julian
> 
> - Sam Ruby
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 11:19:25 -0500

> forbidden in fragment identifiers do you want to allow?

We seem to be in a loop.

To you, RFC 3986 (including potential errata and/or bis work) implies ASCII.

I point out that that restriction does not seem to make sense for fragments.

You return back to asking about ASCII.

To help break this loop, permit me to turn this question around. 
Restricting the scope of the conversation to fragments, why do you 
believe it makes sense to limit such to ASCII only?  What problems do 
non-ASCII characters in fragments cause?

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 17:20:29 +0100

> Note that they’re using a second #, which isn’t conformant to the URI ABNF.

Well, this is new work, right. Why don't they use something that *is* 
conforming?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 11 Jan 2015 11:25:08 -0500

I and others have brought that up. What’s interesting is that they say it’s reasonably interoperable with deployed implementations.

Cheers,

> On 11 Jan 2015, at 11:20 am, Julian Reschke <[email-masked]> wrote:
> 
> Well, this is new work, right. Why don't they use something that *is* conforming?
> 
> Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 17:28:38 +0100

> I point out that that restriction does not seem to make sense for

Actually, you did not. Instead you pointed to lots of material somewhere 
else that I didn't want to parse just to find out what you're proposing.

> You return back to asking about ASCII.

Because that's what RFC 3986 is concerned with.

> non-ASCII characters in fragments cause?

I fail to see how fragments are special here compared to, say, the path.

I fully agree that work on what we used to call IRIs is something that 
needs to be done. However right now I'm trying to figure out what's 
wrong with RFC *3986*.

I hear you saying that URIs should allow non-ASCII characters in certain 
places. This may break code where these characters are put on the wire 
(such as HTTP) or stored in places that do not allow non-ASCII 
characters (say, a database column).

The fact that it's hard to extend the URI character repertoire beyond 
ASCII is why the IETF attempted to do it in a separate spec/construct. 
I'm not convinced that the situation has changed sufficiently to 
invalidate that approach.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 17:29:45 +0100

> I and others have brought that up. What’s interesting is that they say it’s reasonably interoperable with deployed implementations.

Let me guess: "deployed implementations" == "what current browsers do".

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 11 Jan 2015 11:35:38 -0500

… and Curl, IIRC. 

> On 11 Jan 2015, at 11:29 am, Julian Reschke <[email-masked]> wrote:
>

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 11:45:08 -0500

On 01/11/2015 11:29 AM, Julian Reschke wrote:
>> I and others have brought that up. What’s interesting is that they say
>> it’s reasonably interoperable with deployed implementations.
>>
>> Cheers,
>> ...
>

Your sarcasm is not appreciated.

I encourage you to actually look at test results:

https://url.spec.whatwg.org/interop/test-results/7b83ef3682

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 17:52:19 +0100

> … and Curl, IIRC.

I'm sure it is true for many implementations.

However, given the URI spec, scanning for the "#" from the *end* would 
be entirely conforming and reasonable, and code like that would be broken.

Also, people shouldn't forget that there already is a framework for 
using fragment identifiers that are not based on document anchors; and 
that is XPointer. There's a reason why certain characters were 
disallowed in anchors (such as HTML's @id); it's to preserve an 
extension point for new addressing schemes.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 17:58:16 +0100

Sam,

I was referring to what Mark linked to (the use of a leading "#" inside 
a fragment identifier). The test above tests something else.

Looking at *that* test: it tests a non-ASCII character in a fragment 
(not allowed per RFC 3986). The tests show that it's accepted, but some 
implementations subsequently UTF8-encode-then-percent-escape.

So non-ASCII is indeed not interoperable, unless you normalize the 
result. UTF8PercentEscaping non-ASCII characters *before* putting them 
inside a URI's fragment identifier would be another test; it would be 
interesting to see whether the implementations consistently treat the 
escaping.

Best regards, Julian

---

From: t.petch <[email-masked]>
Date: Sun, 11 Jan 2015 17:43:20 +0000

----- Original Message -----
From: "Sam Ruby" <[email-masked]>
To: "t.petch" <[email-masked]>; "Sean Leonard"
<[email-masked]>; <[email-masked]>
Sent: Sunday, January 11, 2015 1:18 PM

Sam

The sense I get from your e-mails is a concern with the http: scheme and
perhaps not with other ones.  As I am sure you know, there are hundreds
of registered schemes, several of which are key to the operation of the
Internet so in a sense, the http: scheme is almost insignificant in the
scheme(!) of things.  Perhaps RFC3986 could be thought of as an object
class from which hundreds of specific schemes have inherited their
generic syntax.

To make an incompatible change to RFC3986, allowing separator characters
to appear where previously prohibited or allowing additional characters
not to be percent encoded, would need evaluating against all those
hundreds of schemes to verify that the schemes still work, an
interesting challenge.

Of course, if a proposed change, and I am unclear what specific changes
you propose, is upwards compatible, allowing all interpreters of other
schemes to continue unaffected, then this would not apply.

Tom Petch

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 13:54:50 -0500

> implementations subsequently UTF8-encode-then-percent-escape.

At some point, I would like to discuss what characters should be allowed 
in a fragment, and would like RFC 3986 either to match the set we come 
up with, or for RFC 3986 to be obsoleted by a specification that does.

I've been proceeding by testing actual implementations and actually 
talking to at least two authors of RFC 3986.  And, trust me, I plan to 
have conversations with the third author.

Each time I try to have this discussion, you keep returning to what RFC 
3986 currently says as of this moment.

If and when you are ready to talk about what RFC 3986 or its successors 
should say at some point in the future, I'll be here.

Meanwhile, it is my intention to produce a spec that matches reality, 
and to do so by meeting with people who produce tools and meeting them 
half-way.  Meeting them half-way doesn't mean re-quoting what a decade 
old specification says, it means determining whether the specs need to 
be updated or the implementations need to be updated.

On this particular test, I believe that it is quite likely that firefox, 
galimatias and rusturl will be updated to match the URL specification 
this year.  It is my hope that safari will too.

Getting agreement isn't easy, and it doesn't happen quickly.  It, 
however, does require people who participate to agree that change is 
possible.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 14:18:13 -0500

On 01/11/2015 12:43 PM, t.petch wrote:
> perhaps not with other ones.

I'm not sure where you got that impression, but I will assure you that 
that is not the case.

I'd like the following Perl expression:

   URI->new_abs(input, base)->scheme . ":"

... to produce the same value as the following JavaScript expression:

   new URL(input, base).protocol

... for all sets of input and bases.

Whether that means getting Perl to change, one or more web browsers to 
change, or one or more specs to change (and quite possibly, a 
combination of the above), is yet to be determined.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 22:41:28 +0100

> 3986 currently says as of this moment.

I keep returning to URIs as things consisting of ASCII code points.

If you want to talk about URI-like identifiers that can contain 
non-ASCII characters, we'll need to discuss RFC *3987*. As far as I can 
recall, there's more or less agreement that this needs to happen, but 
that's IMHO a different conversation than the one about RFC 3986.

Best regards, Julian

---

From: <[email-masked]>
Date: Sun, 11 Jan 2015 21:24:27 +0000

What really concerns me about the apparent objective of making all URL parsers produce identical results from the set of tests produced by Whatwg, is that it seems to ignore the fact that URLs are created and consumed in very different contexts.

When text is entered in the address bar of a web browser and it is parsed as a URL, then I understand the desire to be as tolerant as possible of errors.  I see minimal danger in making an educated guess as to what the user intended because the browser will likely correct the error and show the user what the corrected URL is, before making the safe request.

However, when I use the System.Uri class to parse some text that I find in a HTTP response or configuration file, it would really concern me if the parser started making significant assumptions about how to generate valid URLs from text that doesn’t conform to the syntax of RFC 3986.

I would be really interested in seeing the development of a specification that describes the optimum way of converting a string that doesn't conform to RFC 3986 syntax into one that does.  That I can get behind.  Changing .NET's System.Uri to make best guesses, I can't.

Darrel

---

From: Sean Leonard <[email-masked]>
Date: Sun, 11 Jan 2015 14:10:11 -0800

[Adding pkix@]
On the two intertwined points:

On 1/11/2015 8:28 AM, Julian Reschke wrote:
> Actually, you did not. Instead you pointed to lots of material 
> somewhere else that I didn't want to parse just to find out what 
> you're proposing.
>
> I hear you saying that URIs should allow non-ASCII characters in 
> certain places. This may break code where these characters are put on 
> the wire (such as HTTP) or stored in places that do not allow 
> non-ASCII characters (say, a database column).
>
> https://url.spec.whatwg.org/interop/test-results/7b83ef3682

***

I fully agree with Julian on the matter of US-ASCII for URIs. URIs (RFC 
3986) are only made of US-ASCII characters.

If someone wishes to extend URIs (as opposed to IRIs or whatever) to 
include non-US-ASCII characters, that's a problem for web browsers and 
all other Internet software alike. This goes exactly to my point about 
protocol slots.

Certificates, CRLs, and other security objects are just as fundamentally 
a part of the Web (and web browser) infrastructure as HTML. In 
X.509/PKIX security objects, the GeneralName uniformResourceIdentifier 
construct is US-ASCII only (IA5String). If you extend "URIs" to be 
beyond US-ASCII, RFC 5280 has to be updated...and all the security 
libraries that depend upon it. Just because HTML(5) can be served as 
UTF-8 or use &amp; encoding or whatever, doesn't make the problem go away.

Does the URL Interop test-results explicitly test for certificates? I 
suggest attempting to put some non-US-ASCII characters in a GeneralName 
protocol slot (say, for revocation) and see what happens.

HTML 4.01 is at least consistent in saying (for its time) that hrefs and 
other things are URIs. For interoperable behavior, use US-ASCII 
characters only and stick with % encoding.

The security angle brings up another problem: the interoperable 
transcription of URIs across systems. The ASCII range is a limited 
repertoire, so it is easy to write it out unambiguously on paper, 
display it on a TV screen, say it over the radio or a public service 
announcement, or memorize it on your smartphone, in order to type it 
into your web browser, the command-line, or any other system of choice.

If you allow the enormous (and ever-expanding) range of Unicode 
characters in "URIs", all of those use cases become fundamentally 
ambiguous, inviting homograph attacks. Which smiley face out of nearly a 
hundred smiley emoji do you mean when you say "http://foo.com/😋" ?? How 
about an URI containing "ῗ" (U+1FD7 GREEK SMALL LETTER IOTA WITH 
DIALYTIKA AND PERISPOMENI)--what composition or decomposition mode? What 
if the combining accent mark code points are in a different order?

***
I have empathy for what Sam/the W3C wants, since the HTML protocol slots 
basically beg to be filled with Unicode strings like <a 
href="http://zh.wikipedia.org/wiki/巴泰勒米·波岡達"> (instead of <a 
href="http://zh.wikipedia.org/wiki/%E5%B7%B4%E6%B3%B0%E5%8B%92%E7%B1%B3%C2%B7%E6%B3%A2%E5%B2%A1%E9%81%94">).

But maybe the more interoperable approach is to define a format and 
mechanism (e.g., IRIs, or something like IRIs v2) to map /from ///the 
Unicode-capable protocol slots, /to/ the well-standardized RFC 3986 URI 
format.

My 2¢.

Sean

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 17:11:51 -0500

> I keep returning to URIs as things consisting of ASCII code points.

Indeed you do.  :-(

> that's IMHO a different conversation than the one about RFC 3986.

Let me compare what you suggest should happen with what Roy Fielding 
suggests should happen.  For whatever reason, Roy's post didn't make the 
web archive, but my reply did, so I will Roy quote from:

https://lists.w3.org/Archives/Public/public-ietf-w3c/2014Dec/0088.html

 >>> The problem with RFC3987 was that it tried to define a new
 >>> addressing format instead of simply defining an arbitrary
 >>> reference and how to get from there to an interoperable URI. It
 >>> did not work because it wasn't written to handle arbitrary input
 >>> and could not keep up with changes in IDNA.

and

 > This does not mean there isn't value in coming up with a consistent
 > reference parsing model for HTML that reflects the browser
 > environment and results in a consistent URL DOM and address display.
 > It simply doesn't change what is in RFC3986, nor does it escape the
 > fact that a browser is still dependent on RFC3986 to produce a valid
 > URI out of whatever it happens to parse when it eventually chooses to
 > use that URI in an IETF protocol like HTTP.

I'd like to explore what Roy is suggesting: layering URLs on URIs, 
without going through a URL layered on IRI layered on URI approach. 
Doing so may require changes to both the definition of URLs and URIs.

In particular, I'd like to see how close we can come to a goal where the 
output of URL parsing followed by URL stringification is always a valid URI.

There is no question that a change to the "ASCII-ness" of schemes is 
clearly a non-starter.  But I would hope that we could discuss whether a 
change to the "ASCII-ness" of fragments is a possibility.  In 
particular, I don't believe that there is any reason why a URL parser 
(in a browser or elsewhere) needs to percent encode fragments.

It may come to pass that Roy is the only person who shares the goals 
that he described.  In which case, the end result will be an appendix to 
the URL standard which describes the real world use cases that RFC 3986 
does not address which prevent a proper layering.  Additionally, there 
would be a description of what the set of outputs from a URL 
parse/stringification process does produce, i.e., a description of what 
RFC 3986 should have said.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 17:24:06 -0500

> protocol slot (say, for revocation) and see what happens.

The topic is character repertoire for fragment identifiers, not schemes. 
  I am not suggesting that we allow non-US-ASCII protocols.

I am willing to accept that GeneralName's that are expressed in the form 
of a URI must be ASCII-only.

The next question is: are all possible URIs valid GeneralNames?

Here are two examples:

data:image/gif;base64,R0lGODlhyAAiALM...DfD0QAADs=
https://www.ietf.org/#related

Are both valid GeneralNames?  If not, why not?

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Jan 2015 23:38:46 +0100

> Doing so may require changes to both the definition of URLs and URIs.

I'm totally fine with what Roy suggests. The point I was trying to make 
was not that we necessarily need to touch RFC 3987, but whatever 
introduces non-ASCII into identifiers ought to happen in something 
layered on top of RFC 3986.

> (in a browser or elsewhere) needs to percent encode fragments.

a) I still do not see how fragments are special in general. They *are* 
special in the context of browsers and HTTP, but that doesn't mean we 
can generalize from that.

b) The *parser* never needs to percent encode anything. A *sender* would 
need to percent-encode (*) if fragments stay the way thy are. Q: how is 
that a problem in practice?

((*) Maybe it's time to coin a term for the 
utf-8-encode-then-uri-percent-escape thingy)

> that he described.  In which case, the end result will be an appendix to

I believe I share these goals.

> RFC 3986 should have said.

Which brings us back to the question what exactly is wrong with RFC 
3986. So far I heard "what it says about IDNA". What else?

Best regards, Julian

---

From: Sean Leonard <[email-masked]>
Date: Sun, 11 Jan 2015 14:44:30 -0800

On 1/11/2015 2:24 PM, Sam Ruby wrote:
> The topic is character repertoire for fragment identifiers, not 
> schemes.  I am not suggesting that we allow non-US-ASCII protocols.
>
> I am willing to accept that GeneralName's that are expressed in the 
> form of a URI must be ASCII-only.
>

Yes. Use case: Certificate Image. RFC 6170 (previously RFC 3709).

> https://www.ietf.org/#related

Yes. Use case: Certification Practice Statement (CPS) Pointer 
(Certificate Policies, id-qt-cps). Section 4.2.1.4 of RFC 5280. I have 
seen fragments in there in the wild.

Both examples are valid.

Note: the examples I provide here are actually examples where the URIs 
are encoded directly in IA5Strings; there is no GeneralName CHOICE. But 
the premise still stands. Others with PKIX experience can point to 
systems/protocols where uniformResourceIdentifier is in use with the 
appropriate.

One offhand example that I was able to find, via a Google search, is the 
GS1 Certificate Profile Standard v2.0 
<http://www.gs1.org/gsmp/kc/epcglobal/cert/cert_2_0-standard-20100610.pdf>. 
That happens to use GeneralName. It's a bit elliptical, and refers to 
RFC 4043 (which I rarely see), but...it's there.

Sean

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 21:23:57 -0500

> RFC 4043 (which I rarely see), but...it's there.

Thanks!  This input is helpful.  I don't see where it clearly states 
that fragments are allowed, but then again, I don't see where it says 
that they would be disallowed.  That coupled with your statement that 
you have seen them in practice (however rare) is valuable input.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Sun, 11 Jan 2015 21:28:59 -0500

> that a problem in practice?

Again, it is my hope/expectation that by year end every single one of 
the parsers that I have looked out (with the possible/probable exception 
of Perl) will NOT percent encode fragments.

This means that virtually every parser will not be RFC 3986 compliant by 
your definition (or, to follow your lead, RFC 3986 is irrelevant to them 
as they are not "*senders*").

Furthermore, absolutely no specification in existence would cover the 
gap between what parsers do and what RFC 3986 requires.

Forgetting me, the WHATWG, the W3C, and everything else; I would think 
that that would be a problem.

Finally, I will openly say that I don't understand your distinction.  I 
know what it means to send a HTTP request (which you concede is 
special), but I have no idea what it means to *send* a data: URI.

RFC 2616 (and its successors) are about *sending* (among other things). 
  RFC 3986 is about syntax and a process for resolving relative 
references.  There are three occurrences of the word "send" in RFC 3986. 
  Two are examples.  The final occurrence relates to the necessity for 
there to be a base URI when sending a relative reference.

> 3986. So far I heard "what it says about IDNA". What else?

I have published a lot of test values that I am looking for people to 
work through with me.  Cooperation amongst browser vendors in this 
effort is increasing.  I have yet to find anybody on this list willing 
to look at this data.

The fact that you either will not or can not "hear" that parsers are not 
RFC 3986 compliant, coupled with the fact that you either will not or 
can not work with the test results I have published is not working for me.

- Sam Ruby

---

From: Nico Williams <[email-masked]>
Date: Sun, 11 Jan 2015 23:30:34 -0600

On Sun, Jan 11, 2015 at 11:38:46PM +0100, Julian Reschke wrote:
> b) The *parser* never needs to percent encode anything. A *sender*
> would need to percent-encode (*) if fragments stay the way thy are.
> Q: how is that a problem in practice?

Exactly.

Protocol slots that limit content to US-ASCII must use appropriate
encoding, which means that the code putting non-ASCII content into such
slots should do the necessary encoding.  For IRIs this is described in
RFC3987.

Whether non-ASCII Unicode forms may appear in slots where they are
allowed is a much more interesting question.  RFC3987 says "yes".

E.g., IRIs in SANs MUST be encoded into URIs per-RFC3987 because RFC5280
defines the uniformResourceIdentifier arm of the GeneralName CHOICE as
IA5String.  If we were to add an IRI OtherName for PKIX then a non-ASCII
then IRIs could appear _there_ in their Unicode, unencoded form.

The location bar of the browser *is* a protocol element of sorts:
because it's involved in cut-n-paste [local, but still real] protocols,
because of JavaScript.

The location bar therefore is a Unicode-aware slot[*] and MUST accept
IRIs (Unicode), but it also MUST accept ASCII encoded IRIs (because of
one might be pasted in).  Because it's also a UI element, the location
bar MUST display Unicode.

[*] Because it's a UI element, and because a string and strings are
    Unicode strings in JavaScript.  I admit this is hand-waving: there
    is no requirement that a browser support JavaScript, and there can
    be no requirement that the UI facing part of it use Unicode when the
    locale doesn't.  A browser could accept non-Unicode, non-ASCII in a
    non-Unicode, non-ASCII locale, but since at some point the rubber
    has to meet the road and an ASCII URI will be needed, the browser
    effectively has to do the encodings that RFC3987 says to do, which
    means the browser must deal in Unicode, which means we might as well
    speak of the location bar as being Unicode-aware even in cases where
    the locale uses a Unicode-incompatible codeset.

> I believe I share these goals.

Me too.  I don't see why there's any controversy here.  What am I
missing?

> 3986. So far I heard "what it says about IDNA". What else?

IMO nothing, not even that.

I don't want to see punycoded/pct-encoded strings in my UIs -- no one
does[**].  But that doesn't mean that protocol elements shouldn't have
to see them either.  Yes, sometimes encoded strings will leak where they
shouldn't (and then someone has to work to plug the leak).

[**] End users anyways.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sun, 11 Jan 2015 23:46:16 -0600

On Sun, Jan 04, 2015 at 10:22:59PM +1000, Matthew Kerwin wrote:
> To be honest, a big reason for using RFC 3986 as a normative reference

The file scheme is a bit special though in that a hostname could appear
outside the authority part of the URI, right?  Hosts appearing there
should be dealt with as any other non-authority part of an IRI when the
IRI must be fed into a URI slot.

OTOH, RFCs 3986 and 3987 apply, so if a hostname is to appear as an
authority in a file scheme URI/IRI then you know what to do (because
those RFCs tell you what to do).

> who on earth *accepts* that as a valid URL?

The browser location bar certainly shouldn't.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 12 Jan 2015 00:06:42 -0600

> that exists, but I don't want the definition I write to *not* work in any

I suspect we'll have to make some choices then.

> [1] http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx

Yes, there's an aliasing problem when IRIs are encoded onto URIs where
some of the URI consumers simply decode into bytes in non-UTF-8,
non-ASCII locales.  This is not something that the IETF ignored, it's
just something that the IETF couldn't fix.

Many filesystems allow just about any byte values other than the
traditional path component separator character(s) (ASCII slash and, for
Windows, _also_ backslash), and so we have an interop problem.

I don't think we should have this discussion every time a filesystem-
related protocol comes up!  We should have a spec we can reference from
specs dealing with filesystems.

We've had this discussion in quite a few contexts (e.g., WebDAV, NFSv4,
even PKCS#11 URIs just now; and probably many more).

The facts are:

 - filesystems exist with non-Unicode, non-ASCII object names (files,
   directories, symlinks, ...), and/or which allow new such names to be
   created at any time;

 - interpretation of such octet strings is necessarily locale-specific;

 - non-Unicode, non-ASCII locales exist and will continue to for some
   time;

 - often the only entities that know a filesystem object name's codeset
   are end users;

 - we have Internet protocols that need to interface to these
   filesystems;

 - we also have local interfaces involved that aren't Internet
   protocols.

(It gets worse.  We even have a filesystem that uses an inconvenient
Unicode normalization.  But the above facts will do for now.)

The thing to do is to demand [normalized] Unicode on the wire.  We can't
really achieve that, but we (the IETF) demand it anyways and note the
problems that result from not knowing what codeset is used for a given
character string because we only know it as an octet string.  (Those
problems don't arise from demanding UTF-8 on the wire.)

We do this because the only alternative that can interop is to tag these
strings with codeset metadata, but since a) we lack that metadata (see
facts above), and b) it would be *much* more work to start carrying that
metadata everywhere than it would be to convert to/from Unicode at the
boundaries where we *do* have that metadata... we do the simpler thing
of demanding Unicode in the middle.

The same applies to the file scheme.  Yes, you can get in trouble if you
mix non-ASCII names with locales with various different codesets, so
what?  We should still say that RFCs 3986 and 3987 apply.

Longer term, what should really happen is that filesystem _APIs_ that
just-use-8 today should start doing conversion to/from Unicode so that
"the middle" (e.g., the VFS layer, the wire, ...) can carry Unicode and
the "edges" (the filesystem's on-disk format, the APIs) can carry
whatever is locally appropriate, and then end users should clean up as
necessary (tools for this are common).

This is a bit like boiling a Great lake: do we really expect C runtimes
for various OSes to do this?  I'd like to expect that, yes, though I
won't hold my breath.  But it's better than the alternative (see above).

Nico
--

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 12 Jan 2015 16:28:18 +1000

On 12 January 2015 at 12:28, Sam Ruby <[email-masked]> wrote:

> know what it means to send a HTTP request (which you concede is special),
> but I have no idea what it means to *send* a data: URI.
> ​​
>
> ​
> ​

Provide it​ to a parser. E.g. by writing it in a HTML document or CSS file.

As with any URIs, the UA parser interprets the sequence of bytes in the
document (or from a UI element, etc.) into a "URI object", and uses that
URI object to resolve the resource. Whether that be by sending a HTTP
request or constructing a virtual object or whatever is not really relevant
at this point.

​
> RFC 3986 is about syntax and a process for resolving relative references.
> There are three occurrences of the word "send" in RFC 3986.  Two are
> examples.  The final occurrence relates to the necessity for there to be a
> base URI when sending a relative reference.
> ​​
> ​
> ​
>
> ​
>

Actually 2616 and co. have two sides: defining the 'http' and 'https' URI
schemes (inheriting from the generic syntax from 3986), and the actual http
protocol.

The HTTP RFCs don't ever talking about "sending" URIs, they send requests
which are generated from URIs.
 The overlap is that a HTTP request includes some parts of the URI in its
fields. HTML is a URL
​"
sender
​"
, if you want to define such a thing, as is the location bar, etc.

> RFC 3986 compliant, coupled with the fact that you either will not or can
> not work with the test results I have published is not working for me.
>
>
Parser compliance has two parts:

1) For any given input deemed "valid"​ by the spec, all compliant parsers
interpret it the same way. For RFC 3986 this is a bit hazy, because it
defines a generic syntax but the outputs are defined by scheme-specific
specs that inherit (/extend) the generic syntax. The practical outputs of
parsing a 'http' URI, for example, are defined by RFC 7230 -- ultimately it
results in a valid HTTP request.

2) For any given input deemed "invalid" by the spec with a specific failure
mode, all compliant parsers fail in the same way. This is a bit trickier,
because it's often not well defined or has to be inferred. As a general
rule, we usually say, "anything that doesn't match the spec isn't a URI."

This is handy because it makes it easier for something like the awesomebar
to determine whether a given input is a http URI (etc.) or not.

I'm guessing this is where you come in. It seems that what you want is to
identify and standardise common behaviours for a bunch of those "not a URI
(but probably should be interpreted as a URL)" inputs. I don't think
revving 3986 is the right way to do this.

For one: RFC 3986 provides the basis for a lot of URI schemes, so changing
it in an incompatible way leaves a lot of those schemes up in the air.

For two: as in HTTP, the syntax elements are sometimes used in protocol
artefacts. That makes them pretty rigid.

A better way would be to define a new "thing", and provide instructions for
how to translate to/from RFC 3986-style URIs. That's what RFC 3987 did with
IRIs. I'm pretty sure that's pretty close to what you're already doing with
the URL spec. Rather than fight RFC 3986, how about treating it as a target?

---

From: Julian Reschke <[email-masked]>
Date: Mon, 12 Jan 2015 07:32:23 +0100

> of Perl) will NOT percent encode fragments.

...when doing what? A parser parses a URI into components. Are you 
referring to the act of return the fragment identifier as result of the 
parsing process?

> as they are not "*senders*").

Can you give a concrete example where this affects a URI that is 
currently valid per RFC 3986?

> special), but I have no idea what it means to *send* a data: URI.

I used "sender" as in "software that generates a URI".

> effort is increasing.  I have yet to find anybody on this list willing

Ahem. I think you got concrete feedback from Mark, Björn, and myself 
(and maybe others).

> can not work with the test results I have published is not working for me.

Sam, could you for once leave the meta-level and provide an example 
(remember: this thread was about fragment identifiers)? And yes, I'm 
still talking about URIs that are currently valid per RFC 3986.

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 12 Jan 2015 16:48:26 +1000

> those RFCs tell you what to do).
​Hmm, I think there's an issue I should file[1]. So far I've hand-waved the
authority-in-path issue by deferring to Microsoft's UNC definition, but it
seems people would prefer to use the 'host' ABNF construct to detect
UNC-in-path URIs, and then have guidance for translating the host to
something UNC accepts (or not) before dereferencing.

​Is that right?​

​[1]: https://github.com/phluid61/internet-drafts/issues/4​

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 12 Jan 2015 18:11:40 +0900

> To you, RFC 3986 (including potential errata and/or bis work) implies

Sam - As far as I have looked at your spec, the main difference between 
fragments and the rest of an URL is that fragments don't get %-encoded 
but stay as Unicode when parsing.

> I point out that that restriction does not seem to make sense for

 From an user point, the restriction doesn't make sense for any part of 
an UR?. From a protocol point, there are protocols/formats where slots 
are defined (and implemented) to only accept ASCII. These 
protocols/formats cannot be upgraded just by changing the definition. 
Therefore, it seems that even an RFC 3986 bis, however that will look, 
will have to have a definition/production/name for the form that is 
ASCII only.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 12 Jan 2015 10:20:21 +0100

Exactly.

Another thing to consider is what we gain by allowing non-ASCII in a 
single place in the identifier as opposed to the confusion caused of 
having different rules for the different parts.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 12 Jan 2015 18:29:46 +0900

> into your web browser, the command-line, or any other system of choice.

Many other writing systems are equally limited, and are easier to use 
for those using them natively. Some writing systems in East Asia aren't 
that limited, but have other mechanisms to just about allow the same 
things to happen.

> hundred smiley emoji do you mean when you say "http://foo.com/😋" ??

Homograph attacks can happen, but only at points where different actors 
can add to the same namespace. There are exceptions, but usually, the 
path part isn't such an exception.

Also, people using smileys in their IRIs and hoping that these get 
easily copied are just shooting themselves in the foot. Just because it 
turns out that using some Unicode characters in IRIs isn't necessarily a 
good idea doesn't mean that we should exclude them all.

> How

Very very much so. The former is readable (although there are better 
examples than foreign names such as Barthélemy Boganda) to a significant 
part of the world's population; the later is gibberish for everybody.

> Unicode-capable protocol slots, /to/ the well-standardized RFC 3986 URI

With IRIs, that's essentially what we have. (of course I don't want to 
imply that the IRI spec cannot be improved upon)

Regards,   Martin.

---

From: <[email-masked]>
Date: Mon, 12 Jan 2015 16:03:38 +0000

> From: Sam Ruby
> … 

> effort is increasing.  I have yet to find anybody on this list willing 

I have looked at the test data.  Specifically I have looked at the comparisons between the .NET framework (csharp) URI parser and the reference implementation.

I’m not sure how you ever expect to obtain a consistent result between a library that returns errors along with a parsed result and one that throws an exception and aborts parsing upon encountering an error.  Is it your expectation that the .Net framework stops throwing exceptions when URI parsing?

Also, regarding escaping of fragment identifiers, in your csharp test program you use Uri.Fragment, which is explicitly documented to return an escaped fragment.  If the developer does not wish the fragment to be escaped then they are free to use,

Uri.GetComponents(UriComponents.Fragment, UriFormat.Unescaped)

Different languages do things in different ways.  Trying get every URL parsing library to behave identically just feels like an ocean boiling exercise. 

Darrel

---

From: David Singer <[email-masked]>
Date: Mon, 12 Jan 2015 09:11:45 -0800

> On Jan 11, 2015, at 8:04 , Mark Nottingham <[email-masked]> wrote:
> 
> One more point of data - 

Interesting.  When I wanted a URL scheme that had nested fragments, I had to use “*” for the second, with the rule that when de-nesting, the first * after the # that was being interpreted has to be replaced by #.

The use-case was ‘package formats’.  Given a packaging system that can pack anything, one can use the fragment syntax to indicate some component of the package. But what do you do if that component has a fragment syntax already and you want to use it?

e.g. http://www.example.com/package.pkg#whatsit.htm*anchor

>> Additionally, the presence of a percent sign not immediately followed by two ASCII hex digits would also be a conformance error.

David Singer
Manager, Software Standards, Apple Inc.

---

From: Nico Williams <[email-masked]>
Date: Mon, 12 Jan 2015 11:19:43 -0600

On Mon, Jan 12, 2015 at 04:48:26PM +1000, Matthew Kerwin wrote:
> Hmm, I think there's an issue I should file[1]. So far I've hand-waved
> the authority-in-path issue by deferring to Microsoft's UNC
> definition, but it seems people would prefer to use the 'host' ABNF
> construct to detect UNC-in-path URIs, and then have guidance for
> translating the host to something UNC accepts (or not) before
> dereferencing.

I have no specific preference here.  But you *must* pick one since
otherwise we'd have two different ways of encoding IDNs in file URIs.

If I had to pick... I think I'd want the hostname in the UNC to be
treated as a URI authority (which means that IDNs would have to be
ToASCII()'ed).  But I could also go the other way.

Would either alternative lead to ambiguities?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 12 Jan 2015 11:43:00 -0600

> You ask me to presume that RFC 3986 is correct.  That's a big ask.

I think you may be missing some important things.

URIs are restricted to US-ASCII because there are protocol slots where
that's all that's allowed.

IRIs are not restricted to US-ASCII, of course, and there is a mapping
from IRIs to URIs for the purpose of being able to pass IRIs where only
URIs are permitted.

There are "URI slots" and "IRI slots", and mostly an implementor need
only keep track of which is which, and what locale is used for user
input and display, and convert accordingly.  (Yes, there are ambiguities
that result as described below; please read on.)

This is critical.  We (the IETF) cannot say that "hey, well, so sorry,
but any protocol slots that only permitted US-ASCII will now have to
magically permit UTF-8".  Full stop.  Full stop because: to go make such
a decision we'd have to boil the oceans (fix all existing URI slots to
be IRI slots -- fix as-deployed) or simply choose to ignore them.  The
latter is politically difficult: lots of implementors would be upset,
and anyways, flag days are the sort of "nuclear option" we try to avoid
at the IETF.

But I think mostly you're failing to understand that all an implementor
has to do is keep track of what kind of a slot anything is.

Your mistake is VERY understandable: we're discussing filesystem issues,
and filesystems are the premier example of an application where existing
software historically lost, *regularly still* loses track of character
strings' charset and encoding metadata.

If so many apps just-use-whatever, why-can't-URIs-too-damn-it is a fair
reaction.  But it is not, IMO, a good reaction.

Accepting just-use-whatever will lead to the situation getting worse.

Use-and-expect-Unicode will tend to make the situation better, even
though non-Unicode-and-non-ASCII will tend to leak from time to time,
leading to aliasing issues.  (This approach will tend to put pressure to
perform codeset and encoding conversions at the edges where the required
metadata is available.  E.g., system call stubs, and so on.)

Given this, if you accept that use-and-expect-Unicode is the better
approach to dealing with filesystem object naming, then you can probably
accept the need to keep track of what is a URI slot and what is an IRI
slot, and convert as necessary.  This is basically what RFCs 3986 and
3987 already require, and it's what others are telling you.

There is a lot of context here.  NFSv4 is a good place to start, because
there the issues are very stark, but RFC3530 just papered over them, and
we're periodically had to re-address them.

See also:

https://www.ietf.org/mail-archive/web/apps-discuss/current/msg13710.html

Nico
--

---

From: Sam Ruby <[email-masked]>
Date: Mon, 12 Jan 2015 12:55:57 -0500

On 01/12/2015 11:03 AM, [email-masked] wrote:
> *> From:* Sam Ruby <mailto:[email-masked]>
> *> *
> *> *Again, it is my hope/expectation that by year end every single one of
> *> *the parsers that I have looked out (with the possible/probable
> exception
> *> *of Perl) will NOT percent encode fragments.
> *> *…
> *> *
> *> *I have published a lot of test values that I am looking for people to
> *> *work through with me.  Cooperation amongst browser vendors in this
> *> *effort is increasing.  I have yet to find anybody on this list willing
> *> *to look at this data.
>
> I have looked at the test data.  Specifically I have looked at the
> comparisons between the .NET framework (csharp) URI parser and the
> reference implementation.

Thanks for doing this!

> I’m not sure how you ever expect to obtain a consistent result between a
> library that returns errors along with a parsed result and one that
> throws an exception and aborts parsing upon encountering an error.  Is
> it your expectation that the .Net framework stops throwing exceptions
> when URI parsing?

At the moment, I'm gathering data and sharing results.

> Also, regarding escaping of fragment identifiers, in your csharp test
> program you use Uri.Fragment, which is explicitly documented to return
> an escaped fragment.  If the developer does not wish the fragment to be
> escaped then they are free to use,
>

Fixed.  Thanks!

https://github.com/webspecs/url/commit/1ac9fb4279f90f6c349ee8954159ea48e99cbdef

https://url.spec.whatwg.org/interop/test-results/730127041a

> Different languages do things in different ways.  Trying get every URL
> parsing library to behave identically just feels like an ocean boiling
> exercise.

In the case of browsers and HTML, different browsers doing different 
things was a major pain point, and large strides have been made over the 
past several years on addressing this.

I hope to make things better (more compatible, more interoperable) for 
developers.  If the best I can do is document what options need to be 
used to get compatible results, then that's what I will do.

- Sam Ruby

---

From: Nico Williams <[email-masked]>
Date: Mon, 12 Jan 2015 12:15:37 -0600

On Mon, Jan 12, 2015 at 06:29:46PM +0900, "Martin J. Dürst" wrote:
> examples than foreign names such as Barthélemy Boganda) to a
> significant part of the world's population; the later is gibberish
> for everybody.

The href attribute in HTML is one thing.  "_All_ URI/IRI slots" is
another.

I don't object to versioning protocols and data formats to change
specific URI slots into IRI slots.

Thus I have no objection to -say- the href attribute in HTML being made
into an IRI slot (with or without versioning of HTML -- that being an
issue for W3C, though if it were the IETF we'd want to do something
better than a flag day).

I object to any proposal that implies that all existing URI slots must
now [magically] be able to carry Unicode non-ASCII character data.

> >Unicode-capable protocol slots, /to/ the well-standardized RFC 3986 URI

We already have this: RFC 3987.

> With IRIs, that's essentially what we have. (of course I don't want
> to imply that the IRI spec cannot be improved upon)

Exactly.

Terminology is a problem here.  In conversation and these e-mail
threads, I really want to refer to IRIs as "URIs".  But in the context
of RFCs 3986 and 3987 it is critical to be careful to say "URI" in
reference to "slots" accepting US-ASCII only, and "IRI" in reference to
slots that notionally[*] accept Unicode.

It would be very convenient if we only needed to be so careful when
pairing "URI" and "IRI" with the word "slot".

[*] I say notionally because related UI elements generally can only
    handle an equivalent subset when used in non-Unicode locales.

Nico
--

---

From: Dave Cridland <[email-masked]>
Date: Mon, 12 Jan 2015 22:24:51 +0000

Sam,

I'm getting really confused by what I suspect are differences in
terminology.

>> I keep returning to URIs as things consisting of ASCII code points.
I think that is their current definition, and I think that - outside of
HTML at least - changing that definition would be hugely problematic. X.509
is one case where the existing syntactic definitions simply force ASCII.
Also, I think this doesn't matter, of which more later...

> > reference parsing model for HTML that reflects the browser
Note that he makes the distinction between a "reference" and a URI.

> I'd like to explore what Roy is suggesting: layering URLs on URIs, without
> going through a URL layered on IRI layered on URI approach. Doing so may
> require changes to both the definition of URLs and URIs.
>
>
No, Roy is talking about References. References, of which the most popular
form discussed is a Relative Reference (commonly - but incorrectly - called
a relative URL), I understand to be "Things which can be put into an HREF
attribute", loosely. Probably also a good model for interpreting random
crap a user stuff into the address bar, too.

One could reasonably argue that URIs are a proper subset of References.
Let's assume that, if you don't object.

> output of URL parsing followed by URL stringification is always a valid URI.
URLs I understand to be a proper subset of URIs; it's simply a URI which
has protocol information for resolving the resource. The distinction is
messy, but the subsetness isn't.

IRIs are not a subset of URIs; instead they're a subset of References
(assuming you're happy with URIs being such a subset). I think they have an
intersection with URIs (ie, some URIs, at least, are also IRIs).

> change to the "ASCII-ness" of fragments is a possibility.  In particular, I
> don't believe that there is any reason why a URL parser (in a browser or
> elsewhere) needs to percent encode fragments.
>
>
Well, you're clearly wrong, as has been discussed. However, I don't see any
reason why we cannot have arbitrary unicode codepoints in a Reference
whether or not it includes a fragment. We need it in domain names, paths,
XMPP addresses, and plenty of other things, as well as fragments in HTTP
References. Moreover, I think, judging from things you've said, that you
also actually want Refernces to be uplifted to Unicode - not actual URIs.

The current specification for doing so is the IRI spec, as Julian has
pointed out a few times. If you think this specification is wrong (or could
be done better, or even *is* practically done better by existing URL
parsers) then I'm all ears.

So just to summarize - if your goal is to have unicode in the (fairly
obvious) places in a Reference in HTML, the browser address bar, and
similar places that usually have a Reference rather than a URI, then I'm
entirely behind you. If your goal is to have consistent parsing, I'm also
100% with you.

If your goal is genuinely to redefine URI syntax and break every
application aside from the web, I'm utterly opposed.

Dave.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 13 Jan 2015 11:44:55 +0900

> e.g. http://www.example.com/package.pkg#whatsit.htm*anchor

Did you ever think about/check out

http://www.example.com/package.pkg#whatsit.htm%23anchor ?

(# being ASCII 0x23)

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 13 Jan 2015 12:11:11 +0900

> intersection with URIs (ie, some URIs, at least, are also IRIs).

IRIs are a superset of URIs, by definition. So all URIs are IRIs.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 13 Jan 2015 17:59:48 +0900

> issue for W3C, though if it were the IETF we'd want to do something

There was no flag day for HTML. IRIs didn't exist (neither the spec nor 
the name) when HTML 4.0 became a Recommendation in 1997 
(http://www.w3.org/TR/REC-html40-971218/). But there's implementation 
advice in that spec at 
http://www.w3.org/TR/REC-html40-971218/appendix/notes.html#h-B.2.1 that 
amounts to essentially the same conversion as the one from IRIs to URIs 
defined later in RFC 3987. Browsers took up that hint sooner or later.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 13 Jan 2015 19:05:52 +0900

> or indeed for other IETF work).

Probably for 'file:', because both 'file:' and nfs deal with file 
systems. But hopefully NOT for other IETF work, in particular not for 
work newly started. There are much better ways to do 
internationalization than the mess we have to deal with (for legacy 
reasons) with file systems.

Regards,   Martin.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 13 Jan 2015 11:50:28 +0000

> IRIs are a superset of URIs, by definition. So all URIs are IRIs.

Thanks; I've never been entirely clear on IRIs and confess to not having
read the spec as far as I can recall.

Please do correct anything else I've got wrong; I doubt that's the only
thing.

---

From: Nico Williams <[email-masked]>
Date: Tue, 13 Jan 2015 11:25:56 -0600

On Tue, Jan 13, 2015 at 07:05:52PM +0900, "Martin J. Dürst" wrote:
> systems. But hopefully NOT for other IETF work, in particular not
> for work newly started. There are much better ways to do

It's good to see pragmatism as to filesystems.

We do have a mess to work with.  Sometimes we can identify boundaries
where conversions (e.g., RFC3987) can be applied to make things better.
Other times we can identify such boundaries but have no hope of getting
conversions applied (e.g., filesystems).

Nico
--

---

From: Graham Klyne <[email-masked]>
Date: Wed, 14 Jan 2015 23:03:44 +0000

On 11/01/2015 22:11, Sam Ruby wrote:
> There is no question that a change to the "ASCII-ness" of schemes is clearly a
> non-starter.  But I would hope that we could discuss whether a change to the
> "ASCII-ness" of fragments is a possibility.  In particular, I don't believe that
> there is any reason why a URL parser (in a browser or elsewhere) needs to
> percent encode fragments.

Changing the ASCII-ness of URI fragments would almost certainly break a fair 
amount of my code.  Maybe there is a general consensus that the gain is worth 
the pain for "marginal" applications such as mine (though I'm not hearing it)...

But I can't help feeling that the place to attack this would be in IRIs, which 
turn have a defined mapping to ASCII-only URIs.

(also, +1 to pretty much everything I've read that Julian has posted on this topic.)

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Wed, 14 Jan 2015 23:09:57 +0000

On 11/01/2015 22:24, Sam Ruby wrote:
> The topic is character repertoire for fragment identifiers, not schemes.  I am
> not suggesting that we allow non-US-ASCII protocols.

*Some* protocols include URIs with fragments.  (The restriction on not sending 
the fragment when dereferencing a URI is a specific to the way resource 
retrieval is handled on the web.)

Even HTTP allows URIs with fragment identifiers in, say, Link: headers.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Jan 2015 10:40:48 +0000

I've been thinking about our discussion, and why we appear to talk past each 
other.  I think it lies in part with this:

On 11/01/2015 19:18, Sam Ruby wrote:
> Whether that means getting Perl to change, one or more web browsers to change,
> or one or more specs to change (and quite possibly, a combination of the above),
> is yet to be determined.

My view:

1. IN SCOPE: The normative scope of RFC3986 is (a) to define what is a valid URI 
and URI reference, and (b) to describe the result of combining a valid URI (i.e. 
a base URI) with a URI reference to obtain a valid URI corresponding to that 
combination.  Everything else (to a first approximation) is explanatory or 
descriptive.

2. OUT OF SCOPE: it is not the place of RFC3986 to define API behaviour.

I think your desire to have a unified API behaviour to access URI components is 
fine, just not in the scope of RFC3986.

Your test data show that different implementations do different things with URI 
strings, and this may be problematic in some contexts.  But I don't think that, 
of themselves, they show that the problem is with RFC 3986.

I use URI parsers in my current work that I know are not entirely conformant 
with RFC3986, in the sense that they may accept URIs that are not strictly valid 
according the syntax defined.  I believe these divergences show up in your test 
data.  But this has not caused any interoperability problems for me, as long as 
valid URIs are accepted.  For me, it is not an issue that PERL and Javascript 
APIs may behave slightly differently, but I accept that it may be a concern for 
others.

3. SUGGESTION:

Rather than proposing to update/replace RFC3986 directly, extending the scope to 
include API behaviour, construct a new specification that builds upon (rather 
than replacing) RFC 3986 normative content to describe a consistent API 
behaviour.  If doing this exposes any specific problems in the RFC3986 normative 
content, propose specific updates.

#g
--

(FWIW, I wrote an early RFC 3986 implementation that was absorbed into the 
Haskell libraries - 
http://hackage.haskell.org/package/network-2.1.0.0/docs/Network-URI.html - the 
API is not without its objectors, but I believe it conforms to the essentials of 
RFC3986 articulated above.  I'm not sure if this is included in your test data.)

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 07:12:50 -0500

On 01/15/2015 05:40 AM, Graham Klyne wrote:
> I've been thinking about our discussion, and why we appear to talk past
> each other.  I think it lies in part with this:
>
>> change,
> 1. IN SCOPE: The normative scope of RFC3986 is (a) to define what is a
> valid URI and URI reference, and (b) to describe the result of combining
> a valid URI (i.e. a base URI) with a URI reference to obtain a valid URI
> corresponding to that combination.  Everything else (to a first
> approximation) is explanatory or descriptive.
>
>
> I think your desire to have a unified API behaviour to access URI
> components is fine, just not in the scope of RFC3986.
>
> Your test data show that different implementations do different things
> with URI strings, and this may be problematic in some contexts.  But I
> don't think that, of themselves, they show that the problem is with RFC
> 3986.
>
> I use URI parsers in my current work that I know are not entirely
> conformant with RFC3986, in the sense that they may accept URIs that are
> not strictly valid according the syntax defined.  I believe these
> divergences show up in your test data.  But this has not caused any
> interoperability problems for me, as long as valid URIs are accepted.
> For me, it is not an issue that PERL and Javascript APIs may behave
> slightly differently, but I accept that it may be a concern for others.

I see divergences in results even when inputs are valid according to RFC 
3986 (note the use of the filter in the below):

> 3. SUGGESTION:
>
> Rather than proposing to update/replace RFC3986 directly, extending the
> scope to include API behaviour, construct a new specification that
> builds upon (rather than replacing) RFC 3986 normative content to
> describe a consistent API behaviour.  If doing this exposes any specific
> problems in the RFC3986 normative content, propose specific updates.

I proposed that the set of characters allowed in a fragment be changed 
to match the following:

When I did so, people freaked out, piled on, and generally made points 
unrelated to what was actually proposed.  My takeaway: this mailing list 
is not a receptive place for people who identify specific problems and 
proposing specific updates.  Forgive me if I'm not exactly eager to do 
that again.

So, I will continue to construct a new specification that describes a 
consistent API behavior.  But until and unless there can be a 
constructive discussion about how to construct a specification in such a 
way that it builds upon RFC 3986, it simply won't be built upon RFC 
3986.  Instead, it will likely have an appendix that describes the 
differences between what it describes are what RFC 3986 describes are.

That being said, I remain open to the idea of building upon RFC 3986. 
This can happen one of two ways:

1) People proposing specific updates to the specification I am working 
on to make this happen.

2) People being receptive and actively soliciting ways in which RFC 3986 
could be changed in order to make this layering possible.

Meanwhile, these meta discussions aren't going to get us there, nor am I 
predisposed to try again with proposing specific updates to RFC 3986.

Perhaps a better place to start is with Haskell.  See below.

> #g
> --
>
> (FWIW, I wrote an early RFC 3986 implementation that was absorbed into
> the Haskell libraries -
> http://hackage.haskell.org/package/network-2.1.0.0/docs/Network-URI.html
> - the API is not without its objectors, but I believe it conforms to the
> essentials of RFC3986 articulated above.  I'm not sure if this is
> included in your test data.)

It is not, but could be.  Haskell isn't one of the languages that I 
happen to know at the moment.  I'm confident that I could quickly learn 
enough Haskell to write the program I need to, but I think it would be a 
good learning experience for somebody to take the first step.

What I am looking for is a program that reads the following test data 
from a file on disk:

   https://url.spec.whatwg.org/interop/urltestdata.json

This program only needs to look at the "input" and "base" values.  It 
needs to parse the base, resolve the input against that base, and then 
extract the data that would be produced if the following API were 
implemented:

   https://specs.webplatform.org/url/webspecs/develop/#api

I realize that the latter is a lot to absorb, and perhaps a better way 
to understand what the output should look like would be to look at 
results from other implementations:

   https://github.com/webspecs/url/tree/develop/evaluate/useragent-results

Doing this may mean that you need to implement some parts of the API in 
your program instead of it being baked into the underlying 
implementation.  That's OK.  Examples might include appending a ':' to a 
scheme to produce a protocol, or splitting an authority into a username 
and password.

I believe that if you would be willing to take a first stab at such a 
program, we can explore whatever differences that exposes, and whether 
or not the proposed API should change, and as a consequence, what the 
implications for RFC 3986 would be.

- Sam Ruby

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 12:35:29 +0000

On 15 January 2015 at 12:12, Sam Ruby <[email-masked]> wrote:

> I proposed that the set of characters allowed in a fragment be changed to
> match the following:
>
> unrelated to what was actually proposed.  My takeaway: this mailing list is
> not a receptive place for people who identify specific problems and
> proposing specific updates.  Forgive me if I'm not exactly eager to do that
> again.
>

Well, people told you that changing the code-points allowed in a URI,
whether in the fragment portion or any other part, was impractical. In
particular, people repeatedly stated that the fragment was not a special
thing in most cases. So when you say that this mailing lists isn't
receptive to specific updates, I think that's an amusing kind of
generalization, given you were provided with a number of very specific
cases where your proposal wouldn't fly, including within HTTP and X.509
(both well-used on the web, I believe), and you've not answered any of
them. I don't think anyone freaked out - in fact, I'm pretty sure nobody
has freaked out since some time in the '60's - I think people voiced their
objections.

Also, I saw it suggested at least once that one solution to your goal - I
don't see it as a problem so much as a desirable enhancement - would be to
declare the slots for URIs in HTML, and the browser address bar, to be IRI
slots. You'd probably still have to down-convert to URIs for HTTP etc, but
still.

Now I don't make any claim of knowing much about IRIs, but based on Martin
Dürst's comment, that seems to fit your goals even better than I'd expected.

Therefore, two questions:

1) Do you disagree that X.509 and/or HTTP's use of URIs (including those
with a fragment identifier) preclude changing URIs to include non-ASCII? If
so, on what basis?

2) Why are IRIs (and moving the web, in general, towards IRIs where
practical) not a solution here?

Dave.

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 08:14:50 -0500

On 01/15/2015 07:35 AM, Dave Cridland wrote:
>
> On 15 January 2015 at 12:12, Sam Ruby <[email-masked]
>     I proposed that the set of characters allowed in a fragment be
>     changed to match the following:
>
>     https://specs.webplatform.org/__url/webspecs/develop/#url-__code-points
>     <https://specs.webplatform.org/url/webspecs/develop/#url-code-points>
>
>     When I did so, people freaked out, piled on, and generally made
>     points unrelated to what was actually proposed.  My takeaway: this
>     mailing list is not a receptive place for people who identify
>     specific problems and proposing specific updates.  Forgive me if I'm
>     not exactly eager to do that again.
>
>
> has freaked out since some time in the '60's - I think people voiced
> their objections.
>
> Also, I saw it suggested at least once that one solution to your goal -
> I don't see it as a problem so much as a desirable enhancement - would
> be to declare the slots for URIs in HTML, and the browser address bar,
> to be IRI slots. You'd probably still have to down-convert to URIs for
> HTTP etc, but still.
>
> Now I don't make any claim of knowing much about IRIs, but based on
> Martin Dürst's comment, that seems to fit your goals even better than
> I'd expected.

Let's just agree that neither of us know much about IRIs.

> with a fragment identifier) preclude changing URIs to include non-ASCII?
> If so, on what basis?

I don't know enough about X.509.

Meanwhile, I do know that there are a number of implementations that 
claim to be RFC 3986 compliant that do not percent encode non-ASCII 
characters contained in fragments.

I do not consider it to be a desirable outcome for both those 
implementations to continue to claim to be RFC 3986 compliant and for 
X.509 to make a similar claim.

I will work with those implementations to come up with a definition that 
they are compliant with.  I welcome people who know about topics such as 
X.509 to participate in those discussions.

As to HTTP, in brief discussions with Mark Nottingham, it seemed likely 
that there are a number of places within HTTP (in particular, places 
where fragments may be included) that "dirty URIs" are common.

> practical) not a solution here?

All I can say is that I personally am not actively exploring that at 
this time.  I welcome people who know about this topic to participate in 
this discussion.

Given the pervasive lack of awareness of IRI's, I don't believe there 
would be much interest in an appendix describing the differences between 
W3C URLs and IETF IRIs.  I do continue to believe that there would be 
interest in such an appending describing the differences between W3C 
URLs and IETF URIs.

- Sam Ruby

P.S.  For whatever reason, the IETF definition of URL hasn't stuck.  The 
common definition for URL doesn't include "Justin Bieber" or whatever 
people are search for today, but does include bookmarklets:

http://en.wikipedia.org/wiki/Bookmarklet

To the extent that they do know of the term URI, they treat it as a 
synonym for URL.  I find it more helpful to distinguish "dirty URLs" 
(i.e., inputs to parsing functions) and "clean URLs" (i.e., the output 
of parsing and stringifying).  It would be ideal if all clean URLs were 
valid URIs, but at the moment, that's not the case.

As a concrete, current, and actionable topic, apparently it is desirable 
and common practice to include square brackets in the search/query 
component of "clean URLs":

https://bugzilla.mozilla.org/show_bug.cgi?id=1121826

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jan 2015 15:31:04 +0100

> characters contained in fragments.

That is a case of "garbage in, garbage out", no? (where "garbage" means 
"not a valid URI").

Or do you have a case where a library accepted a RFC3986-valid fragment 
identifier containing percent-escapes and *removed* that escaping?

> this time.  I welcome people who know about this topic to participate in

If the goal is to widen the set of characters allowed in identifiers, 
than the right thing to revise/use/build upon might be indeed RFC 3987.

> interest in such an appending describing the differences between W3C

Ignoring IDNA etc for a second, you can see IRIs as a superset of URIs 
where the allowable character repertoire has been extended to allow most 
of Unicode.

> common definition for URL doesn't include "Justin Bieber" or whatever

Whose definition of "URL" *does* include "Justin Bieber"?

> http://en.wikipedia.org/wiki/Bookmarklet

It's syntactically a URI (Björn attempted to write down a scheme 
definition once), so I'm not surprised people call it a URL.

> https://bugzilla.mozilla.org/show_bug.cgi?id=1121826

I disagree. A recipient that trips over percent-encoding is broken.

Best regards, Julian

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 14:45:52 +0000

> Let's just agree that neither of us know much about IRIs.
I'd stress I'm not the one writing a new URI spec. Maybe you should be
looking into them, before claiming expertise in this domain. You're coming
across as willfully ignorant if you don't.

I've now read the specification, at least, and while I make no claim of
expertise I'm reasonably confident that this handles everything you could
possibly want WRT what one might call "internationalized URLs"; well beyond
merely unencoded UTF-8 fragments.

Moreover, it seems that XML, and therefore XHTML, natively support IRIs in
their anyURI slots, which surprised me - slightly embarrassing given I use
XML quite a bit. I have no idea if HTML5 allows IRIs natively; even after
reading the spec it's woefully unclear to me, though it does include a
reference, and the existence of XHTML5 suggests to me that it most likely
does.

I would note it's giving you a well-defined process for taking a "URL"
class instance and calling a "toASCII()" on it to get an ASCII URI, to
express it in API terms. Moreover, I believe that a "W3C URL", as you
worryingly put it, is in fact an IRI by definition already.

So really, it took me a couple of hours of reading the specs to decide that
existing "URL" parsers are probably designed to handle IRIs, really. In
which case, I think there's a blindingly obvious path forward for you.

Also, congratulations for apparently demanding "internationalized URLs" for
use in HTML5 when apparently it already does these via the IRI spec.

You know, the one you've ignored our suggestions to look at.

> https://bugzilla.mozilla.org/show_bug.cgi?id=1121826

Speaking personally, I have no objection to this change at all. The
problems with adding non-ASCII codepoints to URIs isn't that some parser
buried in X.509 isn't expecting them, it's that the containing syntax
cannot hold those legally - no URI parser gets a chance to look at these.
Which ASCII codepoints are allowed in which parts shouldn't upset anything.

Dave.

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 15 Jan 2015 14:53:15 +0000

Sam Ruby writes:

> As a concrete, current, and actionable topic, apparently it is
> desirable and common practice to include square brackets in the
> search/query component of "clean URLs":
>

So I want to take this (small) part of your message to try to
understand your implicit requirements process.

The bug report referenced there has a few key factual nuggets:

  1) 3986 [1] does not include '[' and ']' among the URL code points;

  2) The WHATWG URL document is tricky on this point as it stands
     today (2015-01-15): ']' and '[' are not allowed in queries, it is
     a parse error if they do, but the implicit error recovery step
     taken when they _do_ appear is to include them in the parsed URL
     without escaping them.

  3) IE8 and Chrome? do not %-encode '[' or ']' in the query string,
     whereas Firefox35 does %-encode;

On this basis you assert "apparently it is desirable and common
practice to include square brackets in ... 'clean URLS'".

My concern here is _not_ with the specifics of how square brackets are
_or_ should be handled, in browsers or in specs, but rather with how
we should make decisions about such questions.  What can we conclude
based on the facts listed above in this regard?  Insofar as the two
documents involved overlap, they agree -- a string with '[' and/or ']'
in what would otherwise be the query part of a URI/URL is not in fact
a URI/URL -- it fails to conform to the respective definitions
thereof.

The WHATWG document goes further, into a space which is evidently
out-of-scope for 3986, and defines an error recovery procedure for
such strings.  That procedure, it should be noted, _does_
percent-encode some other non-allowed characters when they appear in
the query part, including e.g. '<' and '>'.

Standardising error recovery is clearly a reasonable goal, once the
decision not to take a draconian approach is made.  But in the
(manifest) absence of agreement between existing implementations as to
what that error recovery should be, how should a choice be made?

We now return to your statement, quoted above, as it appears to
contain an answer to that question -- might makes right.  That is,
because Chrome and IE don't %-encode as part of error recovery, and
between them they have more market share than Firefox, Firefox loses,
and must stop doing it.  Obviously that's a caricature of an argument,
not an actual argument.  I'm confident you _have_ an actual argument
in mind, but I can't figure out what it might be.  Please enlighten
me.  Note that "because the WHATWG spec says so" is not sufficient,
because there must be a reason _why_ it says so.

ht

[1] https://tools.ietf.org/html/rfc3986
[2] https://url.spec.whatwg.org/

**Appendix**

The relevant bits of the WHATWG URL document are as follows:

      "A query must be zero or more *URL units*." [3]
      "The *URL units* are *URL code points* and percent-encoded bytes." [3]
      "The URL code points are ASCII alphanumeric, "!", ... [a list
       not including '[' or ']']" [3]

      "[if c is '[' or ']'], run these steps:
         1. If c is not a URL code point and not "%", *parse error*.
         2. ...
         3. Append c to buffer.

       [after processing all the characters into the buffer]
        ...
          3. For each byte in buffer run these subsubsteps:
             1. If byte is less than 0x21, greater than 0x7E, or is
             one of 0x22, 0x23, 0x3C, 0x3E, and 0x60, append byte,
             percent encoded, to url’s query.

          2. Otherwise, append a code point whose value is byte to
          url’s query." [4]

  The 'must' in the first quote above is a 2119 'must'.

[3] https://url.spec.whatwg.org/#url-writing
[4] https://url.spec.whatwg.org/#query-state
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:00:21 -0500

> identifier containing percent-escapes and *removed* that escaping?

Just so that we are understanding one another: this is a case where some 
software (in particular a browser) attempted to produce valid URIs per 
RFC 3986 and things broke as a result.

 From this perspective, percent encoding square brackets in query 
strings is the thing that is broken.  And observing that parsers will 
retain that brokenness isn't helpful.

One possible outcome: this will be yet another case where following RFC 
3986 is not recommended.  I intend to capture a list of such cases, and 
produce a recommendation of what should be done instead.

> than the right thing to revise/use/build upon might be indeed RFC 3987.

If an example of a problem I'm facing is that escaping square brackets 
in query/search strings causes problems, I'm not likely to find the 
solution to that problem in IRIs.

At the current time, it appears that the only option available to me is 
to define something that is neither a superset nor a subset of RFC 3986, 
and document why that is the case.  In particular, document why 
attempting to follow RFC 3986 will cause interoperability problems.

I've participated on this list in the hopes that we collectively could 
come up with a better solution than that.

> Whose definition of "URL" *does* include "Justin Bieber"?

I've been told on this very mailing list just a few days ago that folks 
have been confusing this very thing:

http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13648.html

It is replies like that one, and your comment below that is causing me 
to rapidly lose interest in trying to build on RFC 3986.

> I disagree. A recipient that trips over percent-encoding is broken.

You are certainly entitled to have that opinion.  And you have a RFC 
that you can point to.  But I will say that entering into a discuss with 
that mindset is less than helpful.

If you can help change their minds, that would be great.  If not, I 
would encourage you to keep an open mind.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:10:09 -0500

First, a general comment.  Why does this list attract people whose first 
resource is to go to caricature and sarcasm?

Second, I don't know what the right answer is.  I just know that real 
people have sincerely attempted to follow RFC 3866 and encountered real 
problems with doing so.

I'd personally like to see RFC 3986 changed to address problems like 
this one, and for the W3C specification of API behavior to build upon 
this changed definition.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jan 2015 16:10:55 +0100

> RFC 3986 and things broke as a result.

A broken recipient didn't work, right. What needs to be fixed is the 
recipient, then.

> retain that brokenness isn't helpful.

So, you're actually asking for two things, right?

1) allow raw "[" and "]" in queries (and paths, presumably)

2) stop requiring recipients to remove percent-encoding before comparison

1) is an interesting discussion to have; it maybe true that the URI spec 
is too defensive, as "[" and "]" are only special in the authority part.

What scares me are the implications of 2) though.

> produce a recommendation of what should be done instead.

Can you clarify what you think about 2)? If you don't make that change 
as well, that recipient will continue to be broken.

> solution to that problem in IRIs.

That is true.

> attempting to follow RFC 3986 will cause interoperability problems.

The interop problems are caused because some recipients rely on broken 
behavior in some senders. One way to fix this is to fix the recipient.

> to rapidly lose interest in trying to build on RFC 3986.

I re-read Sean's mail and still don't get what you're referring to.

Best regards, Julian

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 15:28:13 +0000

> resource is to go to caricature and sarcasm?

If that's a reference to me, my first recourse was to suggest you read the
relevant specifications and keep to the same terminology as the technical
literature. My second recourse was to to attempt to clarify your position,
since you hadn't responded to the first.

I only did sarcasm on the third try - I've not yet done caricature.

Please, read the IRI spec, and the HTML5 spec, and then tell me why you
think that IRIs are not *already* the solution.

I don't mean "could be", at this stage I hold they already are, and that
"URL parsers" are in fact IRI parsers, and this is what you want.

Dave.

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:32:06 -0500

> A broken recipient didn't work, right. What needs to be fixed is the

That indeed is one point of view.  Another point of view is that a 
sender attempted to follow a broken standard, and what needs to be fixed 
is the standard.

I'm not saying which point of view is right, I'm merely pointing out 
that there are different points of view.

In order to make progress, it would be helpful if everyone involved 
recognized that there are multiple, valid, points of view.

> 1) allow raw "[" and "]" in queries (and paths, presumably)

I'd like to have that discussion for queries at least.

> 2) stop requiring recipients to remove percent-encoding before comparison

I don't believe I've asked to have that discussion.

> What scares me are the implications of 2) though.

Again, I don't see why 2) is necessary.

> as well, that recipient will continue to be broken.

You seem to be assuming a lot, and it will take peeling back a number of 
layers of the onion to get to where we are understanding each other.

For starters, RFC 3986 essentially punts on topics like normalization 
and comparison:

https://tools.ietf.org/html/rfc3986#page-38

In particular, it explicitly acknoledges the possibility of simple 
string comparison:

https://tools.ietf.org/html/rfc3986#section-6.2.1

I conclude that RFC 3986, as it is currently written, does NOT require 
recipients to remove percent-encoding before comparison, and as such, 
I'm curious as to why you are suggesting that this requirement be removed?

> I re-read Sean's mail and still don't get what you're referring to.

Here's the quote: "Folks are confusing the "web browser text box at the 
top" with URIs."  It continues with an example: 'For example if you type 
"? ietf"'

- Sam Ruby

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 15 Jan 2015 07:42:35 -0800

No hat, but my co-chair hat is nearby.

> resource is to go to caricature and sarcasm?

Although this list is far from unique in this regard, I do agree that it
shouldn't be a tactic in anyone's toolbelt here.

> I'd personally like to see RFC 3986 changed to address problems like this
> one, and for the W3C specification of API behavior to build upon this
> changed definition.
>

I see a lot of chatter from multiple participants about what oughta
happen.  Has anyone put pen to paper yet to propose an update or
replacement document that we can really look at?

-MSK

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:43:09 -0500

On 01/15/2015 10:28 AM, Dave Cridland wrote:
> On 15 January 2015 at 15:10, Sam Ruby <[email-masked]
>     First, a general comment.  Why does this list attract people whose
>     first resource is to go to caricature and sarcasm?
>
> If that's a reference to me, my first recourse was to suggest you read
> the relevant specifications and keep to the same terminology as the
> technical literature. My second recourse was to to attempt to clarify
> your position, since you hadn't responded to the first.
>

Excerpt from RFC 3986:

query         = *( pchar / "/" / "?" )
pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"
unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded   = "%" HEXDIG HEXDIG
sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

Can you point out where in the IRI specification it says that '[' is a 
valid character to include in a query?

Julian has agreed[1] that "If an example of a problem I'm facing is that 
escaping square brackets in query/search strings causes problems, I'm 
not likely to find the solution to that problem in IRIs."

> "URL parsers" are in fact IRI parsers, and this is what you want.

I will claim that what exists as "URL parsers", and for that matter "URI 
parses" are NOT IRI parsers.  Nor are the URI parsers.  What they parse 
is neither a superset nor a subset of either URIs or IRIs.

My goal is to work with authors of such parsers to remove as much as I 
can from the right-most column of the following:

If, in the process, I can come up with a definition that is either a 
proper superset or a proper subset of an existing RFC, that would be great.

- Sam Ruby

[1] http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13782.html

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jan 2015 16:43:23 +0100

> Again, I don't see why 2) is necessary.

if 2) isn't necessary, the receiver mentioned in the bug report 
continues to be broken (in that it doesn't handle percent-encoding 
properly).

> https://tools.ietf.org/html/rfc3986#section-6.2.1

Yes. But it also says it will cause false negatives.

> I'm curious as to why you are suggesting that this requirement be removed?

Just because the list of potential comparisons mentions char-by-char 
doesn't mean it's *always* the right thing to do (it is for instance for 
XML namespace names).

It would be helpful if you actually stated where you want to go from 
here. If you don't do 2), a recipient will have to undo percent-encoding 
before doing comparisons on components. The software mentioned in the 
Mozilla bug does not, so it's broken as per the current spec.

> top" with URIs."  It continues with an example: 'For example if you type

Ah, ok. I thought you were referring to some participant on this list.

Yes, there are people who don't know what a URL or a URI is. Users in 
generally do not *need* to know. And even among those who do understand 
the terms it is not uncommon to say "URL" when what they mean is "what 
does into a/@href".

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:56:26 -0500

On 01/15/2015 10:42 AM, Murray S. Kucherawy wrote:
>
>     Second, I don't know what the right answer is.  I just know that
>     real people have sincerely attempted to follow RFC 3866 and
>     encountered real problems with doing so.
>
>     this one, and for the W3C specification of API behavior to build
>     upon this changed definition.
>

I was told (perhaps bad advice?) that the first step would be to publish 
a problem statement, which I have done:

   https://tools.ietf.org/id/draft-ruby-url-problem-01.txt

I'd like to see some future revision of that document be accepted and 
published as an RFC.

And that the next step would be a BOF:

Based on what I see, and discussions I have held, I'd like to proceed as 
outlined here:

https://lists.w3.org/Archives/Public/public-whatwg-archive/2014Nov/0000.html

What that means in practice is reconciling the "authoring advice" and 
"railroad diagrams" as they appear here:

   https://specs.webplatform.org/url/webspecs/develop/#url-writing
   https://specs.webplatform.org/url/webspecs/develop/#rule-url

With the collected ABNF for URI:

   https://tools.ietf.org/html/rfc3986#appendix-A

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 10:59:46 -0500

> Ah, ok. I thought you were referring to some participant on this list.

Just so it is clear: what I was pointing out is a pure strawman that was 
posted by a participant on this list.  That plus sarcasm plus 
caricatures all contribute to making this a not very welcome environment 
for discussion.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jan 2015 17:12:37 +0100

Sam,

I really really honestly don't know why you are upset about what Sean 
wrote in that mail, or by my request for clarification.

In email conversations it *does* happen that people talk past each 
other, have different backgrounds with respect to terminology etc. The 
best way we can do here is to stay polite, and to ask for clarification 
in order to understand the other side.

Right now, I'm completely confused about what you were trying to say 
when you wrote that sentence I asked about.

Best regards, Julian

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 16:16:19 +0000

> sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"
That's the URI spec, not the IRI spec...

> escaping square brackets in query/search strings causes problems, I'm not
> likely to find the solution to that problem in IRIs."
>
>
*sigh*

One argument at a time. I've already stated that I do not have any problem
in principle with changing the rules on reserved characters in queries in
this respect; moreover I'd be fine with "[]" in a number of places. It
makes the specification slightly more complex, but I can live with that.

Personally, I feel that on finding an unambiguous character that
technically should have been %-encoded, passing it through seems fine -
it's a bit of a pest to specify "unambiguous", but I think that's a
paragraph or two.

With respect to having non-ASCII characters in fragments (or anywhere
else), can you *please* read the IRI spec and consider how that satisfies
(or fails to satisfy) your requirements?

> parses" are NOT IRI parsers.  Nor are the URI parsers.  What they parse is
> neither a superset nor a subset of either URIs or IRIs.
>
>
OK, I'll trust you that they don't parse only entirely valid IRIs. But
modulo the odd reserved character not being %-encoded, I'm pretty sure
they're the same.

The reason I'm claiming that "URL parsers" are in fact aiming to parse IRIs
is because they are, in general, aiming to parse "things that live in HREF
attributes" - and these are IRIs and IRI relative-references as far as I
can tell, so naturally the parsers have gravitated to that (if not by
observation of the specifications).

> My goal is to work with authors of such parsers to remove as much as I can
> from the right-most column of the following:
>
That's a great goal.

By the way, when I switch to "conforming URLs", I see nothing there on
first glance that would appear to be an invalid IRI reference. I think
that's important. Also I think many of your "invalid URIs" are actually
valid. For example, I don't see why "http://www.example.com/foo/%2e" is
invalid; it's over-using %-encoding but that's legal if undesirable.

> proper superset or a proper subset of an existing RFC, that would be great.
OK. I don't really see why you want to discard all existing RFCs and write
your own. But I'm pretty sure you're after 99% of IRI.

---

From: Sean Leonard <[email-masked]>
Date: Thu, 15 Jan 2015 08:26:30 -0800

[SNIP]

On 1/15/2015 5:14 AM, Sam Ruby wrote:
> this time.  I welcome people who know about this topic to participate 
> in this discussion.
>
> would be much interest in an appendix describing the differences 
> between W3C URLs and IETF IRIs.  I do continue to believe that there 
> would be interest in such an appending describing the differences 
> between W3C URLs and IETF URIs.

Ummmmm...............

RDF, a W3C Recommendation, moved to IRIs *exclusively* in RDF v1.1 (25 
Feb 2014). See <http://www.w3.org/TR/rdf11-concepts/>.

With all due respect to the W3C's activity, it seems that ought to be 
sufficient institutional knowledge amongst its participants to make use 
of IRIs and to distinguish them from URIs [RFC3986].

To quote from RDF Section 3.2:
*URIs and IRIs:*IRIs are a generalization ofURIs[RFC3986 
<http://www.w3.org/TR/rdf11-concepts/#bib-RFC3986>] that permits a wider 
range of Unicode characters. Every absoluteURIand URL is anIRI, but not 
everyIRIis anURI. When IRIs are used in operations that are only defined 
for URIs, they must first be converted according to the mapping defined 
insection 3.1 <http://tools.ietf.org/html/rfc3987#section-3.1>of 
[RFC3987 <http://www.w3.org/TR/rdf11-concepts/#bib-RFC3987>]. A notable 
example is retrieval over the HTTP protocol. The mapping involves UTF-8 
encoding of non-ASCII characters, %-encoding of octets not allowed in 
URIs, and Punycode-encoding of domain names.

It would be nice if the W3C maintains similar consistency across all of 
its specifications...

Kind regards,

Sean

---

From: Sean Leonard <[email-masked]>
Date: Thu, 15 Jan 2015 08:35:25 -0800

It seems that many folks have some heated opinions, probably related to 
different backgrounds or technical experience on the matter.

May I suggest that this topical area be included in a structured 
discussion on the agenda at IETF 92, either in APPSAWG or another 
appropriate forum, where the parties can be in one room and understand 
each others' perspectives.

Cheers,

Sean

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 11:38:43 -0500

On 01/15/2015 11:16 AM, Dave Cridland wrote:
>
>     Can you point out where in the IRI specification it says that '[' is
>     a valid character to include in a query?
>
>     Julian has agreed[1] that "If an example of a problem I'm facing is
>     that escaping square brackets in query/search strings causes
>     problems, I'm not likely to find the solution to that problem in IRIs."
>
> *sigh*
>
> One argument at a time. I've already stated that I do not have any
> problem in principle with changing the rules on reserved characters in
> queries in this respect; moreover I'd be fine with "[]" in a number of
> places. It makes the specification slightly more complex, but I can live
> with that.

So discussing how to change RFC 3986 to better match what is commonly 
deployed is indeed a possibility.

> else), can you *please* read the IRI spec and consider how that
> satisfies (or fails to satisfy) your requirements?

I'm not personally convinced that layering URLs on top of IRIs which, in 
turn is layered on top of URIs solve more problems than it introduces. 
In addition, it clearly doesn't solve the problem mentioned above.

> modulo the odd reserved character not being %-encoded, I'm pretty sure

Given that IRIs are a superset of URIs, and there are demonstrable 
differences between how such parsers handle even pure ASCII strings, 
then the claim that they are the same are IRIs is clearly not true.

> The reason I'm claiming that "URL parsers" are in fact aiming to parse
> IRIs is because they are, in general, aiming to parse "things that live
> in HREF attributes" - and these are IRIs and IRI relative-references as
> far as I can tell, so naturally the parsers have gravitated to that (if
> not by observation of the specifications).

If what you are saying is that they both attempt to achieve the same 
goal, then I will agree.  The recommendation I got from IETF area 
directors, W3C tag members, and liaisons was to propose URL as a 
replacement for RFC 3987.

>     My goal is to work with authors of such parsers to remove as much as
>     I can from the right-most column of the following:
>
>     https://url.spec.whatwg.org/__interop/test-results/
>     <https://url.spec.whatwg.org/interop/test-results/>
>
> That's a great goal.

Thanks!

> invalid; it's over-using %-encoding but that's legal if undesirable.

That is entirely intentional.  I have been working to make conforming 
URLs be a proper subset of valid IRIs, and where possible URIs. 
Allowing square brackets in query/search strings is an example where 
that isn't currently possible.

As to valid URIs, I'm using the following code for validation:

If you know of a better test I can perform (preferably in JavaScript so 
it can run in a browser), please let me know.

> OK. I don't really see why you want to discard all existing RFCs and
> write your own. But I'm pretty sure you're after 99% of IRI.

Please don't.  Strawmen on this list are getting rather tiring:

https://yourlogicalfallacyis.com/strawman

I am not discarding all existing RFCs.  Based on advice I have received 
to date from people I trust, I don't believe that IRIs solve the 
problems I'm trying to address.  Your cursory read of that spec hasn't 
changed my mind.  Others have successfully dissuaded me from pursuing 
that goal further.

If people are amenable to discussing changes to RFC 3986, perhaps we 
could work together.

- Sam Ruby

---

From: <[email-masked]>
Date: Thu, 15 Jan 2015 16:15:06 +0000

Sam,

When processing query string parameters, it is normal that % encoded values be decoded.  This is the point of the %encoding, to allow reserved characters to be contained in parameter values.  If a recipient of a URI does not percent decode parameter values before using them, then they are going to be using invalid data.

Therefore, I don't see how any code that was previously correct would be broken by Firefox deciding to follow RFC 3986 and % encode the square brackets.

However, if you feel that RFC 3986 should be changed, I wouldn’t object.

Darrel

Sent from Surface

From: Sam Ruby
Sent: ‎Thursday‎, ‎January‎ ‎15‎, ‎2015 ‎10‎:‎56‎ ‎AM
To: Murray S. Kucherawy
Cc: IETF Apps Discuss

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Jan 2015 16:48:14 +0000

On 15/01/2015 15:56, Sam Ruby wrote:
> I was told (perhaps bad advice?) that the first step would be to publish a 
> problem statement, which I have done:
>
Taking a look at section 4 of that document, here are my thoughts (with premise 
my previous comments about Scope of RFC3986):

...

o  Nomenclature: over the years, a number of different sets of
       terminology has been used.

This may be a problem, but I don't see it as a problem specifically with RFC3986.

And, while it may be confusing, I don't see it as a root cause of 
interoperability problems.

...

o  Deterministic parsing and transformation:

I don't see this as a problem with RFC3986.

...

o  Parameterization: standards in this area need to define such
       matters as normalization forms and values for parameters such as
       UseSTD3ASCIIRules.

I'm not entirely sure what this means, so I may be missing something.  But I 
don't see this as a problem with RFC3986.

...

    o  Interoperability: even after accounting for the above, there is a
       demonstrable lack of interoperability across popular libraries and
       browsers.

I don't see this demonstrates a problem with RFC3986, though I'd be open to 
consider specific claims that it is.

If developers or dependent standards writers choose to not follow an established 
standard, I don't see that writing a new standard solves anything.  
(http://xkcd.com/927/ anyone?  I apologize if that seems like sarcasm, but I 
think it makes a valid point.)

...

    o  Stability: Before any standard document can be marked as
       obsoleted, the requirements other specs that normatively reference
       the to-be-obsoleted standard need to be considered, to avoid
       dangling references.

I don't see this as a problem with RFC3986.

But if the proposal is to obsolete RFC3986, I think it's a major problem for all 
the other standards that depend on it.

...

    o  IDNA: [RFC3490] defines processing for 'IDN-aware domain name
       slots' (where "the host portion of the URI in the src attribute of
       an HTML <IMG> tag" is given as an example.  Later, "IDNA is
       applicable to all domain names in all domain name slots".  So in
       mailto:user@host, is the host a IDN-aware domain name slot?  A
       domain name slot at all?

I don't see this as a problem with RFC3986.

...

    o  Bidi URLs: The problems with writing URLs using characters from
       right-to-left languages are well-known among experts; what is not
       known is a solution for these problems.  The solution given in
       [RFC3987] has some obvious errors (how to handle combining marks);
       it's general approach also probably can be improved on, but it's
       not sure how.

I don't see this as a problem with RFC3986 (which is, by design, restricted to 
USASCII characters).

...

    o  Specific scheme definitions: some UR* scheme definitions are
       woefully out of date, incomplete, or don't correspond to current
       practice, but updating their definitions is unclear.  This
       includes 'file:', for which there is a current effort, but there
       are others which need review (including 'ftp:', 'data:').

I don't see this as a problem with RFC3986.

...

I fear I sound like a broken record here.  I don't deny there are problems in 
all the areas you mention, but for the most part I don't see them as problems 
with RFC 3986.  I think we need to look beyond RFC3986, and build upon it, the 
solutions that can address the problems.

If it turns out there is a consensus that there are changes that should be made 
to base URI syntax (e.g. the recent discussion around '[' and ']') then fine, 
lets discuss those as possible evolutionary updates, but with careful 
consideration for applications that do take note of RFC3986.  Personally, I 
think that introducing '[' and ']' now as valid URI characters would be a 
violation of Postel's robustness principle 
(http://en.wikipedia.org/wiki/Robustness_principle).

#g
--

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 17:19:32 +0000

> deployed is indeed a possibility.
You would need to, I think, change both RFC 3986 and RFC 3987, but no,
there's no fundamental objection from me. Discarding them both out of hand
would, I think, be a serious mistake, as would using specific localized
problems to declare them entirely irrelevant.

> turn is layered on top of URIs solve more problems than it introduces. In
> addition, it clearly doesn't solve the problem mentioned above.
>
>
Multiple layers do actually solve problems.

But I don't think you need to.

I think that - while there's tweaks to the specification that can (and
perhaps should) be made, IRIs fit your essential requirements for syntax.
That is, you can, more or less, ignore URIs, and concentrate on IRIs for
parsing purposes. Then you add a "toASCII" operation on your object, and
you're done.

Parsing them consistently, especially for some error cases, is a different
matter, and something a new W3C/WHATWG spec can usefully address.

Put another way your "URLs" - and really, it'd be much easier if we could
at least agree to use the existing terminology - are essentially IRIs.

> differences between how such parsers handle even pure ASCII strings, then
> the claim that they are the same are IRIs is clearly not true.
>
>
"modulo the odd reserved character not being %-encoded", I said.

If there's other differences, you've not mentioned them, and I've yet to
see them.

> If what you are saying is that they both attempt to achieve the same goal,
> then I will agree.  The recommendation I got from IETF area directors, W3C
> tag members, and liaisons was to propose URL as a replacement for RFC 3987.
>
Which Area Directors? I honestly don't recall this being raised before, and
I find it surprising it went unremarked.

> That is entirely intentional.  I have been working to make conforming URLs
> be a proper subset of valid IRIs, and where possible URIs. Allowing square
> brackets in query/search strings is an example where that isn't currently
> possible.
>
>
OK, but how do you know if they're conformant to the IRI spec if you don't
read it?

> If you know of a better test I can perform (preferably in JavaScript so it
> can run in a browser), please let me know.
>
>
Huh. I don't immediately see why the example I gave isn't valid according
to that code either (unless it's case sensitivity in HEXDIGIT). I do see
the example I gave is valid according to RFC 3986, though.

I imagine you could persuade https://github.com/hildjj/node-abnf to run in
a browser, and avoid hand-crafted regexps.

> https://yourlogicalfallacyis.com/strawman
That's not a strawman, it's hyperbole, but okay, I can play that game.

You're discarding RFC 3987 without even reading it, and your stated desire
is to discard RFC 3986, writing a new specification without reference.
That's neither hyperbole nor a strawman, that is genuinely what I
understand your goal to be.

> I am not discarding all existing RFCs.  Based on advice I have received to
> date from people I trust, I don't believe that IRIs solve the problems I'm
> trying to address.  Your cursory read of that spec hasn't changed my mind.
> Others have successfully dissuaded me from pursuing that goal further.
>
>
https://yourlogicalfallacyis.com/appeal-to-authority

So, you're being willfully ignorant, then, albeit on the poor advice of
others.

If your aim is to replace RFC 3987, you really need to read it and decide
for yourself on its own merits.

Speaking entirely personally, the thing that discomforts me about it is the
name. I can't ever think of typing an IRI into a web browser, despite the
fact that's exactly what I do.

> If people are amenable to discussing changes to RFC 3986, perhaps we could
> work together.
>
>          Dave.
>>
>
> - Sam Ruby
>

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 15 Jan 2015 17:19:38 +0000

Sam Ruby writes:

> Just so that we are understanding one another: this is a case where
> some software (in particular a browser) attempted to produce valid
> URIs per RFC 3986 and things broke as a result.

Ah, so perhaps here is the missing argument!  I didn't see anything in

which suggested that anything had broken.  All I saw was the OP
calling attention to the difference between different browser
behaviours, and stating his preference for the no-escaping
alternative.  Did I miss something?

Or [per private communication], if your the problem is not with the
error recovery as such, but rather with the current consensus position
that '[' and ']' are invalid in query in the first place, _that_
becomes the point on which I'd like to hear what kind of arguments
have convinced you that, as between the following possible situations,
what you would prefer is option (3):

   [using 'I_URI' for URIs as defined in 3986, and 'W_URL' for URLs as
   defined in [1]]

  1) (the _status quo_): Square brackets are not valid in the query
     part of I_URI or W_URL; [1] specifies non-%-encoded error recovery;
     browsers differ;

  2) (what I thought you wanted): Square brackets are not valid in the query
     part of I_URI or W_URL; [1] specifies non-%-encoded error recovery;
     browsers all implement non-%-encoded error recovery;

  3) (what I now think you want): Square brackets are valid in the
     query part of I_URI and W_URL, i.e. 3986 is superseded with an
     appropriate change to the relevant productions and [1] is
     modified to similar effect; no error-recovery is needed.

ht

[1] https://url.spec.whatwg.org/, as of 2015-01-15

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Jan 2015 17:32:40 +0000

On 15/01/2015 12:12, Sam Ruby wrote:
> I see divergences in results even when inputs are valid according to RFC 3986
> (note the use of the filter in the below):
>

As far as I can tell, based on a cursory scan, the divergences are in API 
behaviour.  Which I claim is not in scope of RFC3986, so it's not an RFC3986 
problem.

I explicitly said that your desire for consistent API behaviour is reasonable. 
I just don't think that updating RFC3986 is the best way to get there.

> I proposed that the set of characters allowed in a fragment be changed to match
> the following:
>
> When I did so, people freaked out, piled on, and generally made points unrelated
> to what was actually proposed.  My takeaway: this mailing list is not a
> receptive place for people who identify specific problems and proposing specific
> updates.  Forgive me if I'm not exactly eager to do that again.

I guess there are a lot of people here (myself included) who are quite heavily 
invested in what RFC3986 has to say about URIs, and for whom the change you 
propose would cause pain.  That was the gist of my response.  I think these are 
legitimate concerns that you have to respect if you want to discuss changes to 
RFC3986.  Making changes to an established standard without upsetting a deployed 
base is HARD!

Which is why I try to suggest building on top of what exists.

It may be that (e.g. for href values in HTML5) you need an identifier string 
that is not a URI (e.g. to allow non-ASCII characters in fragment identifiers). 
  If that's what you need then they are not URIs as we currently know them, and 
they can no longer be fully interoperable with software that does work with URIs 
as defined by RFC3986.  So what are the choices:

1. Change the definition of URI and screw the deployed base.  Sometimes that is 
the right answer, but for a 10-year-old widely used standard I suspect this 
isn't one of those cases.

2. Change your spec/software to work within the constraints of that RFC3986 
allows.  It seems the community you're talking to has already dismissed this 
option, which is a choice for them to make.

3. Use something other than URIs.  (IRIs, URLs, whatever.  Let's call them 
BOBs.)  If you want these BOBs to benefit from the deployed use of URIs, then 
you need to define a procedure for mapping BOBs to URIs, and software that uses 
BOBs in contexts that expect URIs will need to apply that mapping.

>
> So, I will continue to construct a new specification that describes a consistent
> API behavior.  But until and unless there can be a constructive discussion about
> how to construct a specification in such a way that it builds upon RFC 3986, it
> simply won't be built upon RFC 3986.  Instead, it will likely have an appendix
> that describes the differences between what it describes are what RFC 3986
> describes are.

I guess it's ultimately a power struggle.  Are the new applications that use 
BOBs more important to the world than all the old applications that use URIs? 
Maybe.

(a) If so, you may be able to deploy BOBs and effectively cut off from the rest 
of the world who use URIs, or force them to upgrade to be part of your new world 
order.

(b) But if the new BOB-using applications also need the features of the deployed 
systems who are disinclined to upgrade, then they will need to bend.

>
> That being said, I remain open to the idea of building upon RFC 3986. This can
> happen one of two ways:
>
> 1) People proposing specific updates to the specification I am working on to
> make this happen.
>
> 2) People being receptive and actively soliciting ways in which RFC 3986 could
> be changed in order to make this layering possible.

That sounds to me a lot like the options (1) and (a) above.  It implies that the 
existing base of URI users have to engage with your solution or deal with 
bifurcation of resource locators in use on the Internet and the Web.

To my mind, it's clearly *possible* to define consistent API behaviour that 
respects the URI syntax as described by RFC3986.  But that may mean excluding 
some forms of identifier that are considered important by the community you're 
advocate for, such as unescaped UTF-8 characters in fragment identifiers.

In a sense, I think you may be right, that this (the IETF) may be the wrong 
community for this discussion, as the biggest loser if bifurcation happens will 
be the Web community.  Given the centrality of URIs to the Web, I think these 
options should be discussed by W3C TAG.

#g
--

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 12:39:50 -0500

On 01/15/2015 12:32 PM, Graham Klyne wrote:
>
> In a sense, I think you may be right, that this (the IETF) may be the
> wrong community for this discussion, as the biggest loser if bifurcation
> happens will be the Web community.  Given the centrality of URIs to the
> Web, I think these options should be discussed by W3C TAG.

FYI, I flew up to NY and met with the W3C TAG last week.  I received 
only encouragement and constructive suggestions.

Separate from that, I don't see bifurcation as a necessary outcome.  It 
certainly isn't my first choice.

- Sam Ruby

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Jan 2015 18:05:38 +0000

On 15/01/2015 17:39, Sam Ruby wrote:
> FYI, I flew up to NY and met with the W3C TAG last week.  I received only
> encouragement and constructive suggestions.

Good!

> certainly isn't my first choice.

I guess the devil is in the details.

If, for example, you want to introduce non-ASCII characters into URI fragments, 
I can't see how bifurcation can be avoided.  How damaging that turns out to be 
is hard to say.  I think a lot of the positioning around IRIs was primarily an 
attempt to avoid this while introducing localized identifiers.  Which is what I 
was describing as BOBs in my previous message.

But, as you say, far better we don't go down that path.

I think that if we had a clear enumeration of areas where RFC3986 updates might 
be proposed, it would be an easier discussion to have.  But as I said before, 
changing an existing standard with a deployed base is hard.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Jan 2015 18:16:34 +0000

Much as I'd love to take a stab at this, I haven't used Haskell myself for a 
while and don't know if I have a working tool chain installed.  Maybe if I stop 
reading and writing emails I might be able to find some time to have a look!

I'm sure it wouldn't take too long to construct the framework you describe, once 
I have the basic tool chain up and running.  But don't hold your breath!

BTW, my implementation would almost certainly be found wanting in some of your 
behavioural consistency tests.

#g
--

On 15/01/2015 12:12, Sam Ruby wrote:
> It is not, but could be.  Haskell isn't one of the languages that I happen to
> know at the moment.  I'm confident that I could quickly learn enough Haskell to
> write the program I need to, but I think it would be a good learning experience
> for somebody to take the first step.
>
> What I am looking for is a program that reads the following test data from a
> file on disk:
>
> This program only needs to look at the "input" and "base" values.  It needs to
> parse the base, resolve the input against that base, and then extract the data
> that would be produced if the following API were implemented:
>
> I realize that the latter is a lot to absorb, and perhaps a better way to
> understand what the output should look like would be to look at results from
> other implementations:
>
> Doing this may mean that you need to implement some parts of the API in your
> program instead of it being baked into the underlying implementation.  That's
> OK.  Examples might include appending a ':' to a scheme to produce a protocol,
> or splitting an authority into a username and password.
>
> I believe that if you would be willing to take a first stab at such a program,
> we can explore whatever differences that exposes, and whether or not the
> proposed API should change, and as a consequence, what the implications for RFC
> 3986 would be.
>
> - Sam Ruby

---

From: t.petch <[email-masked]>
Date: Thu, 15 Jan 2015 17:56:08 +0000

----- Original Message -----
From: "Sam Ruby" <[email-masked]>
To: "Dave Cridland" <[email-masked]>
Cc: "Graham Klyne" <[email-masked]>; <[email-masked]>
Sent: Thursday, January 15, 2015 3:43 PM

> sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"

I do not think that it does (unless percent encoded).

>From RFC3987

  iquery         = *( ipchar / iprivate / "/" / "?" )
  ipchar         = iunreserved / pct-encoded / sub-delims / ":"
                  / "@"
  iunreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar
  ucschar        = %xA0-D7FF / %xF900-FDCF / %xFDF0-FFEF
                  / %x10000-1FFFD / %x20000-2FFFD / %x30000-3FFFD
                  / %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD
                  / %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD
                  / %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD
                  / %xD0000-DFFFD / %xE1000-EFFFD
  iprivate       = %xE000-F8FF / %xF0000-FFFFD / %x100000-10FFFD

where sub-delims is taken from RFC3986.

If by [ and ] you mean x'5B' and x'5D', then they remain invalid in an
IRI query.

That said, [ and ] are interesting in that RFC3986, as updated by
RFC6874,  reserves them purely for IPv6 addresses, which I always saw as
a statement that IPv6 was a work in progress whose ramifications were
unknown (still true today).

So for everyone has followed RFC3986, then relaxing the rules on [ and ]
could just be a question of asking the v6ops and 6man IETF WGs if they
saw problems in allowing them in a query or fragment.  Of course, for
those who have not followed RFC3986 and used them for other purposes,
then it is a question of looking at all the hundreds of schemes based on
RFC3986 and evaluating the impact of such a relaxation; or perhaps a
question of saying that if you have not followed RFC3986, well you took
a risk.

Tom Petch

> Julian has agreed[1] that "If an example of a problem I'm facing is
that
> [1]
http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13782.html

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 13:27:46 -0500

>       modified to similar effect; no error-recovery is needed.

Good question.  Please forgive the lengthy answer.

Lets posit the existence of a W_URL parser.  Given a string, it produces 
an object.  One of the operations that object performs is 
"stringification" (an IDL term that I don't particularly care for which 
essentially means serialization to a string form).

  - - -

If you look at the sequence of operations: parse followed by 
stringification, you get a string transformation.

This can be an identity transform.  If you want, try the following in 
Firefox or Chrome's console:

   new URL('http://example.com/').href == 'http://example.com/'

The result will be true.

The result can be some transformation.  For example:

   new URL('HTTP://EXAMPLE.COM/PATH').href == 'http://example.com/PATH'

Finally, the result can be to give up.

   new URL('http://example.com:port/')

Both Firefox and Chrome throw an exception.

  - - -

Now if you take a look at that transformation, we can discuss the what 
should be desirable characteristics of the set of outputs.

The first characteristic that I would like to suggest is that the 
outputs round trip, and by that I mean that if you feed that output back 
into the transformation, you get the same result back.

The second characteristic is that it actually works.  This isn't a 
precisely specified criteria, but it means that if you attempt to use 
the result it doesn't break anything.  One example of this would be 
placing the string 'http://ietf.org/' into the href attribute of an HTML 
<a> element, rendering it on the screen, and having somebody click on 
the link.  If this goes to the right place, then all is good.  The 
definition of "works" for data: W-URLs, and for javascript: W-URLs are 
different, but that's not an obstacle at this level of discussion.

  - - -

At this point, I'm troubled.  Despite the set of desirable 
characteristics I have specified, I have a well defined set which is 
neither a superset of I-URI's nor I-IRI's, nor a proper subset of either.

An example of something that is not a conforming URL, but is both a 
valid URI and a valid IRI:

   http://257.257.257.257/

Here is something that round-trips a URL parse/stringify transformation, 
and actually works, despite being an invalid URI and invalid IRI:

   https://www.google.com/?q=[z]

To further compound this, RFC 3986 claims that the following strings are 
to be treated as equivalent (and perhaps even preferred as they are 
deemed to be 'valid'):

   https://www.google.com/?q=%5Bz%5D
   https://www.google.com/?q=%5bz%5d

Indeed in many, and generally most, situations, those are considered 
equivalent.  But this is not necessarily so.  To it's credit RFC 3986 
does describe a Comparison Ladder.  Unfortunately it does so using terms 
like "false positives" and "false negatives" when in fact, they aren't 
false at all.

  - - -

So much for a deep dive.  Returning to the surface: I'd like to 
simultaneously loosen up the definition of validity for I-URI's in some 
places while tightening it up in others.  This is a work in progress, so 
which I can't say exactly where just yet in each case, I can say that 
now would be a good time for interested parties to participate.

- Sam Ruby

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 15 Jan 2015 11:30:30 -0800

>   https://tools.ietf.org/id/draft-ruby-url-problem-01.txt

It's not bad advice.  Drafts are roughly free.  :-)

> Based on what I see, and discussions I have held, I'd like to proceed as

If you're looking at doing a BoF, I think that aligns with what I was going
to suggest:

Given the current number of participants and especially the more passionate
ones, I'm wondering if this shouldn't have a dedicated tightly-focused
working group to house this work separately from apps-discuss (which is the
home of both APPSAWG and the APP area general traffic).  Have you thought
of writing up a charter to define the work?

Such a working group could include your problem statement draft as its
first deliverable.

I'm also curious to hear, and probably see in such a charter, why this work
belongs in the IETF and not the W3C.  I'm not making an argument for
either, just don't have the history to understand that part of it.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 15 Jan 2015 11:40:03 -0800

On Thu, Jan 15, 2015 at 8:35 AM, Sean Leonard <[email-masked]> wrote:

> appropriate forum, where the parties can be in one room and understand each
> others' perspectives.
>

Thanks for the reminder: I need to make the general call for agenda items
for the upcoming meeting.

For this particular item, I'm concerned that there are several people (Sam
in particular) participating on this topic that don't usually attend IETF
meetings.  Unless most or all of them will attend, I'm not sure that would
be a good use of our usually-precious face time.  I'm reasonably sure Sean
and Julian will be there, but what about the rest?  Otherwise, continuing
to use the list to figure out a path forward here seems more appropriate.

-MSK

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 16:23:35 -0500

On 01/15/2015 02:30 PM, Murray S. Kucherawy wrote:
> On Thu, Jan 15, 2015 at 7:56 AM, Sam Ruby <[email-masked]
>     I was told (perhaps bad advice?) that the first step would be to
>     publish a problem statement, which I have done:
>

Indeed :-)

>     I'd like to see some future revision of that document be accepted
>     and published as an RFC.
>
>     http://www.ietf.org/meeting/__important-dates-2015.html
>     <http://www.ietf.org/meeting/important-dates-2015.html>
>
>     Based on what I see, and discussions I have held, I'd like to
>     proceed as outlined here:
>
> If you're looking at doing a BoF, I think that aligns with what I was
> going to suggest:

To be clear: I'm perfectly willing to do a BoF, but it is unclear to me 
what it would accomplish.

> Given the current number of participants and especially the more
> passionate ones, I'm wondering if this shouldn't have a dedicated
> tightly-focused working group to house this work separately from
> apps-discuss (which is the home of both APPSAWG and the APP area general
> traffic).  Have you thought of writing up a charter to define the work?

I do believe that a WG is in order.  See related discussion at the 
bottom of the following:

Is there a how-to for writing an IETF charter?  Trust me, there isn't 
any suggestion that I won't aggressively follow up on, so if you want me 
to write up a charter and attend a BoF, not a problem.

> Such a working group could include your problem statement draft as its

That works for me.

> I'm also curious to hear, and probably see in such a charter, why this
> work belongs in the IETF and not the W3C.  I'm not making an argument
> for either, just don't have the history to understand that part of it.

I'm venue agnostic.  In fact, I have already written up a draft W3C Charter:

http://rawgit.com/webspecs/url/develop/docs/url-charter.html

If we could get the interested parties here to join that WG, that would 
be great.  One obstacle is that the W3C Invited Expert agreement as 
currently written would prevent IETF participants from making use of 
their own work in IETF drafts.  I'm working to change that.

That being said, I think that if the W3C were to embark on defining a 
concept that overlaps with URIs but it not either a proper superset or a 
proper subset of that concept, then it would be best if we were to find 
a way for the IETF to endorse such.

Not knowing how to make that happen, I would suggest a division of 
labor.  Roughly thus:

1) Have the W3C take responsibility for what once was IRIs, but will be 
known as URLs.  Ideally the IETF would license or assign copyright for 
RFC 3987 in such a way that much of it could be reused.  This WG would 
also document a parsing API.  One constraint would be that all outputs 
produced are valid URIs.

2) Have the IETF charter a WG with responsibility for creating an RFC 
3986bis.  It would be responsible for updating the nomenclature for 
aligning with the new (and, in some sense, original) definition for the 
term URL.  The intent would also be to update the definition of the term 
URI to align with the validity constraints defined in conjunction with 
the W3C.

I don't work for a browser vendor, but my expectation is that browser 
vendors would have an interest in seeing everything that they produce 
for http: and https: be considered valid.  If anything that they produce 
is a problem, then lets discuss it.  If it isn't a problem and isn't 
currently defined as valid by RFC 3986, then that should be changed.

As for schemes other than http and https, browser vendors aren't likely 
to feel as strongly, so if there is a better definition then they I 
don't expect much of a problem.

The one exception is file:, and a part of this discussion should be 
whether this hypothetical new WG would assume responsibility for what 
now is draft-ietf-appsawg-file-scheme.  My proposal would be that it is 
assigned this responsibility.

- Sam Ruby

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 Jan 2015 22:09:18 +0000

> Good question.  Please forgive the lengthy answer.
Lengthy is fine.

> an object.  One of the operations that object performs is "stringification"
> (an IDL term that I don't particularly care for which essentially means
> serialization to a string form).
>
>  - - -
>
> The first characteristic that I would like to suggest is that the outputs
> round trip, and by that I mean that if you feed that output back into the
> transformation, you get the same result back.
>
>
OK; you're saying that for any URL string S, new URL(new URL(S).href).href
== new URL(S).href

Your tests seem to imply that for any URL string S, there is one and only
one correct value of (new URL(S).href); is that also the case?

To put it another way, your tests imply a canonical form for a URI, and
that seems much more useful.

> precisely specified criteria, but it means that if you attempt to use the
> result it doesn't break anything.  One example of this would be placing the
> string 'http://ietf.org/' into the href attribute of an HTML <a> element,
> rendering it on the screen, and having somebody click on the link.  If this
> goes to the right place, then all is good.  The definition of "works" for
> data: W-URLs, and for javascript: W-URLs are different, but that's not an
> obstacle at this level of discussion.
>
>
I can readily appreciate the sentiment here; like many things, it's hard to
specify. But what you're saying is that for any URL string S, where (new
URL(S).href) does not produce an error, the result must be a semantically
valid URL string? (i'm being delightfully wooly about "semantically valid").

>  - - -
>
> At this point, I'm troubled.  Despite the set of desirable characteristics
> I have specified, I have a well defined set which is neither a superset of
> I-URI's nor I-IRI's, nor a proper subset of either.
>
> An example of something that is not a conforming URL, but is both a valid
> URI and a valid IRI:
>
>   http://257.257.257.257/
>
>
OK... I don't think that's valid, but it's probably valid according to the
ABNF only (ignoring the prose). I think specifying that in the formal
syntax would be uncontentious - in fact, draft-ietf-iri-3987bis-13 already
makes this change.

Similarly, http://127.0.0.1:65536/ is also not valid, but because the
concept of port is not tied to a specific transport protocol, and might not
be restricted to a 16-bit range, that's a harder one to capture in pure
ABNF - that is, the permissable values of port are arguably scheme specific.

Both these are issues of what I called semantic validity above. Others
probably exist.

>   https://www.google.com/?q=[z]
My gut feeling is that allowing this to be valid is a possible change, as
I've said. I appreciate this is at odds with RFC 3986, but if URL parsers
do generally ignore the error that's fine.

> to be treated as equivalent (and perhaps even preferred as they are deemed
> to be 'valid'):
>
> equivalent.  But this is not necessarily so.  To it's credit RFC 3986 does
> describe a Comparison Ladder.  Unfortunately it does so using terms like
> "false positives" and "false negatives" when in fact, they aren't false at
> all.
>
>
I think that this would disappear, largely, with a canonical form. RFC 3986
currently says that pct-encoded should use upper case - that's a lower case
"should" - and if we take this to be a canonical form, then the latter
canonicalizes to the former, and we win. Of course, if [] is made legal in
the query string it'd canonicalize to the previous example.

Comparison of URIs then become a case of comparing the canonicalized
versions of both.

This is problematic in the case of ports - since http://www.google.com:80/
and http://www.google.com/ both refer to the same thing, but to have a
single canonical form for both would require the parser to be aware of the
default port for the scheme. But maybe unknown schemes cannot be
canonicalized, or maybe we have two canonical forms. (Scheme-aware and
scheme-agnostic; obviously you'd specify which you want and where).

> which I can't say exactly where just yet in each case, I can say that now
> would be a good time for interested parties to participate.
>
>  ht
>>
>
> - Sam Ruby
>

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 17:55:06 -0500

First: thanks!  This is exactly the discussion I've been hoping to have!

More below.

On 01/15/2015 05:09 PM, Dave Cridland wrote:
> On 15 January 2015 at 18:27, Sam Ruby <[email-masked]
>     Lets posit the existence of a W_URL parser.  Given a string, it
>     produces an object.  One of the operations that object performs is
>     "stringification" (an IDL term that I don't particularly care for
>     which essentially means serialization to a string form).
>
>       - - -
>
>     This can be an identity transform.  If you want, try the following
>     in Firefox or Chrome's console:
>
>        new URL('http://example.com/').__href
>     <http://example.com/').href> == 'http://example.com/'
>
>     The result will be true.
>
>        new URL('HTTP://EXAMPLE.COM/PATH')__.href
>     <HTTP://EXAMPLE.COM/PATH').href> == 'http://example.com/PATH'
>
>        new URL('http://example.com:port/'__)
>
>     Now if you take a look at that transformation, we can discuss the
>     what should be desirable characteristics of the set of outputs.
>
>     outputs round trip, and by that I mean that if you feed that output
>     back into the transformation, you get the same result back.
>
>
> OK; you're saying that for any URL string S, new URL(new
> URL(S).href).href == new URL(S).href

Yes.

> Your tests seem to imply that for any URL string S, there is one and
> only one correct value of (new URL(S).href); is that also the case?

That is indeed the current design point.  I will share that there is 
some sentiment (not universally shared) to make parts of the file: 
scheme to be implementation defined.

> that seems much more useful.

Indeed!

I will note that I've intentionally shied away from using that term as 
some people already have ideas on what canonicalization means.  In fact, 
until I started looking at this data, I too had different ideas.

>     precisely specified criteria, but it means that if you attempt to
>     use the result it doesn't break anything.  One example of this would
>     be placing the string 'http://ietf.org/' into the href attribute of
>     an HTML <a> element, rendering it on the screen, and having somebody
>     click on the link.  If this goes to the right place, then all is
>     good.  The definition of "works" for data: W-URLs, and for
>     javascript: W-URLs are different, but that's not an obstacle at this
>     level of discussion.
>
>
> I can readily appreciate the sentiment here; like many things, it's hard
> to specify. But what you're saying is that for any URL string S, where
> (new URL(S).href) does not produce an error, the result must be a
> semantically valid URL string? (i'm being delightfully wooly about
> "semantically valid").

That's indeed what I am trying to say.

To get to that point, however requires wide review.  If people can 
demonstrate that any given result is widely problematic (for some 
definition of "wide"), then it would make sense to adjust the parsing 
algorithm to avoid such values.

An example to illustrate: non-ASCII host names are a problem.  IDNA 
(including such things as punycode) is a strategy for converting Unicode 
host names to ASCII.

> OK... I don't think that's valid, but it's probably valid according to
> the ABNF only (ignoring the prose). I think specifying that in the
> formal syntax would be uncontentious - in
> fact, draft-ietf-iri-3987bis-13 already makes this change.

Is is indeed valid (this was discussed recently on this very list).  I 
would hope that it is uncontentious.

> concept of port is not tied to a specific transport protocol, and might
> not be restricted to a 16-bit range, that's a harder one to capture in
> pure ABNF - that is, the permissable values of port are arguably scheme
> specific.

I agree.  I'm OK with not being able to express all of the conditions 
with pure ABNF.  It is the round-tripping and other properties that 
matter to me.

As an aside, my current draft considers the above valid, but there is an 
open bug on it.  The problem isn't deciding that such is invalid, the 
problem is deciding how to canonicalize it.

Looking just at four browsers, we have four different results:

https://url.spec.whatwg.org/interop/test-results/c6c0953a53?select=current

If you expand this to include non-browsers, most don't have a problem 
with such values:

https://url.spec.whatwg.org/interop/test-results/c6c0953a53

> Both these are issues of what I called semantic validity above. Others

Indeed.  Here are a few examples that are worth discussing:

>     Here is something that round-trips a URL parse/stringify
>     transformation, and actually works, despite being an invalid URI and
>     invalid IRI:
>
> My gut feeling is that allowing this to be valid is a possible change,
> as I've said. I appreciate this is at odds with RFC 3986, but if URL
> parsers do generally ignore the error that's fine.

Thanks!

>     To further compound this, RFC 3986 claims that the following strings
>     are to be treated as equivalent (and perhaps even preferred as they
>     are deemed to be 'valid'):
>
>     https://www.google.com/?q=%__5Bz%5D <https://www.google.com/?q=%5Bz%5D>
>     https://www.google.com/?q=%__5bz%5d <https://www.google.com/?q=%5bz%5d>
>
>     equivalent.  But this is not necessarily so.  To it's credit RFC
>     3986 does describe a Comparison Ladder.  Unfortunately it does so
>     using terms like "false positives" and "false negatives" when in
>     fact, they aren't false at all.
>
> I think that this would disappear, largely, with a canonical form. RFC
> 3986 currently says that pct-encoded should use upper case - that's a
> lower case "should" - and if we take this to be a canonical form, then
> the latter canonicalizes to the former, and we win. Of course, if [] is
> made legal in the query string it'd canonicalize to the previous example.

The sweet spot that parsers seem to be converging on:

If the input isn't percent encoded and needs to be, replace the 
characters in question with the uppercase percent encoded form.

If the input is already percent encoded, leave it as is, even if it 
includes lowercase characters.

If I were designing things from scratch, I certainly wouldn't have 
proposed such a thing, but if that's what parsers do, then documenting 
that so that everybody can converge on a single and predictable behavior 
would be a big win.

> Comparison of URIs then become a case of comparing the canonicalized

Other than recognizing that there is history here and some processors 
compare URIs in different manners

> This is problematic in the case of ports - since
> http://www.google.com:80/ and http://www.google.com/ both refer to the
> same thing, but to have a single canonical form for both would require
> the parser to be aware of the default port for the scheme. But maybe
> unknown schemes cannot be canonicalized, or maybe we have two canonical
> forms. (Scheme-aware and scheme-agnostic; obviously you'd specify which
> you want and where).

The current draft does specify the default port for a precious few 
schemes, and requires the port-less string form when default ports are used.

One strategy would be to adjust the registration process to not allow 
new schemes to be registered with default ports.  Another would be to 
version the specification so that new schemes with default ports would 
require a revision of the parsing algorithm.  I would hope that this 
would be a relatively rare event.

Again, thanks for taking the time to review and comment on this!

>     simultaneously loosen up the definition of validity for I-URI's in
>     some places while tightening it up in others.  This is a work in
>     progress, so which I can't say exactly where just yet in each case,
>     I can say that now would be a good time for interested parties to
>     participate.
>
>         ht
>
>
>     - Sam Ruby
>
>     _________________________________________________
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/__listinfo/apps-discuss
>     <https://www.ietf.org/mailman/listinfo/apps-discuss>

- Sam Ruby

---

From: Larry Masinter <[email-masked]>
Date: Thu, 15 Jan 2015 23:30:40 +0000

+1 BOF at next IETF, whether or not it leads to a working group.  

Proposed Agenda: walk through draft-ruby-url-problem (I volunteer)
                                    based on that, discuss next steps.

If a working group is wanted, proposed charter milestone:
 
* finish draft-ruby-url-problem (problem statement agreed or controversies identified, proposed plans concrete or have concrete options), publish as RFC.
* Then recharter or close (within 4 months)

If you agree, it would help focus the discussion if comments were couched in terms of changes to draft-ruby-url-problem.
For example, https://tools.ietf.org/html/draft-ruby-url-problem-01#section-5.2  does note some kinds of things an update to 3986 might accomplish, but doesn't suggest changing fragment identifier repertoire, as Sam suggested in email.

If you want a mailing list other than appswg, either a new one or an old one; of the existing mailing lists, I suggest [email-masked] is the closest in scope.

I'm in favor of chartering and managing the work in IETF and not in W3C based on where the specs would get adequate review.
* URIs/URLs, although invented for the web, are now woven into many other IETF protocols and data structures than the web, and those are mainly out of scope for W3C/WHATWG priorities.
* it seems easier to get broader review in IETF.  I say this despite the past failure to get W3C/WHATWG participation in the IRI working group, because we now have real data about browser behavior  and the new possibility of getting browser participation in the discussion. 

Commitments of participation (from W3C management, URL implementors) would help.

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jan 2015 20:48:49 -0500

Thanks!

> For example, https://tools.ietf.org/html/draft-ruby-url-problem-01#section-5.2  does note some kinds of things an update to 3986 might accomplish, but doesn't suggest changing fragment identifier r…

I'll ensure that this document is updated before any BOF.

> * it seems easier to get broader review in IETF.  I say this despite the past failure to get W3C/WHATWG participation in the IRI working group, because we now have real data about browser behavior…

The following section doesn't seem like a good fit for the IETF:

   https://url.spec.whatwg.org/#api

I'd like to understand more why the previous IRI efforts stalled before 
endorsing that part of the work being done at the IETF.

I have no problem with the idea of a RFC 3986bis being done at the IETF.

> Commitments of participation (from W3C management, URL implementors) would help.

I don't think that W3C management will be an issue.

- Sam Ruby

---

From: Nico Williams <[email-masked]>
Date: Thu, 15 Jan 2015 21:30:36 -0600

> not exactly eager to do that again.

This is a very long set of threads.

Can we summarize the proposal?  By now it's lost in a sea of posts.

My first-order question is:

  Is the proposal that [some? all?] heretofore-US-ASCII-only protocol
  fields now begin accepting and carrying UTF-8 (or some other UTF)?

*That* I could see causing a freak-out.

I don't see a need for controversy if no breaking changes are proposed.
If some proposed changes are breaking, we can discuss their impact and
possibly accept them, but there will be controversy.

A change that would break things on paper might not, of course, and can
be considered even if it did, but it would be controversial even if it
turned out that the change would not be breaking in actual deployments.

I think that's a fair place to start.  I want to know just how much I
should care about this.  If there's no breaking change, then maybe I
(and probably others) don't care at all and can stop paying attention.

(APIs and protocols are different (but related) things.  RFC3986 does
not specify an API.  API changes are necessarily out of scope when
discussing RFC3986, but it'd be nice to have abstract APIs that might
help us think about URIs in general and to identify breaking changes in
particular.)

Nico
--

---

From: Larry Masinter <[email-masked]>
Date: Fri, 16 Jan 2015 07:38:36 +0000

> https://specs.webplatform.org/url/webspecs/develop/#url-code-points

My opinion:

What characters are allowed in 'a fragment' depends on the context. Fragment 
Of a "URI": My opinion: no, not a good idea, it would break ascii-only slots that now take URI only. 
Of an "IRI": Worth discussing, likely; need to review against 3987 and update where experience dictates.
                 Certainly IRI fragments can have non-ASCII characters, it's just second # and < > for example.
Of a "URL": Depends on what you mean:
-  If you mean the thing IETF now calls "IRI" was now (also) called a "URL", then see "IRI" above.
-   If you mean by "URL" some previous definition, like:
   *   "a URL is a URI that doesn't start with 'urn:'" or (alternatively)
   *  "a URL  is a URI that doesn't act like a URN".
    Then see "URI" above.

I think the nomenclature issue is the primary cause of people talking past each other or getting sarcastic.

One major change I had been trying to make in 3987bis was to change the processing model so that IRI -> URI mapping happened by parsing first, and then (only if reassembling into a URI) doing UTF-8-%xx-hex-encoding second on each of the (Unicode) parsed components. Would this possibly allow IDNA for mailto:nonascii@nonascii?subject=nonascii other slots?

> > points unrelated to what was actually proposed.  

This can happen when the same words are used with different meanings.

> > My takeaway: this mailing list is not a receptive place for people who identify specific problems and proposing specific updates.  

Your problems and updates might  not have been as specific as you thought.

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 06:12:37 -0500

> Can we summarize the proposal?  By now it's lost in a sea of posts.

I'd like to see all values that can be returned by the following 
expression to be considered valid URIs for all values of x:

   new URL(x).href

The definitions needed to evaluate the above expression can be found here:

Alternately, both Firefox and Chrome ship the URL object, so you can try 
it yourself by going into the JavaScript console.

> *That* I could see causing a freak-out.

The above proposal does currently put utf-8 into the fragment protocol 
field.

> turned out that the change would not be breaking in actual deployments.

Thank you, thank you, thank you for putting this so clearly!  I do 
believe that "break things on paper" is necessary given my data that 
valid URIs are a merely a Platonic ideal.  I do believe that "breaking 
in actual deployments" should be avoided.

> (and probably others) don't care at all and can stop paying attention.

A necessary outcome of this effort is that some things currently 
considered valid by RFC 3986 will no longer be considered such.  And 
that some things not currently considered valid by RFC 3986 will become 
valid.

> help us think about URIs in general and to identify breaking changes in

I believe that both scope and venue should be on the table.

There is a relationship between the API and the protocol.  In 
particular, for HTTP, this API is in active use for constructing data 
that goes across the wire.

If the definition of the API would have the effect of causing data to go 
across the wire to cause problems with actual deployments, then the API 
should change.  Given that versions of this API are widely deployed 
today, I would think that should be rare.

If the definition of the API would have the effect of causing data to go 
across the wire that does NOT cause problems with actual deployments, 
then that data should be considered valid.  Even if this breaks things 
"on paper".  Even if it is controversial.

Aligning the definition of validity with reality will benefit future 
producers and consumers.  In particular, it will free them from the 
necessity of reverse engineering each others implementations.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 06:34:34 -0500

> Of a "URI": My opinion: no, not a good idea, it would break ascii-only slots that now take URI only.

I would like to explore the possibility of the URL API being build on 
top of a RFC 3986bis definition.  In particular, I'd like to see all 
potential results of the following expression be considered valid URIs, 
for all values of x:

   new URL(x).href

In the process I'd like to be free to explore "breaking things on paper" 
as Niko so eloquently put it.  Of course, breaking actual deployments 
should be avoided.  But given that versions of the above are actually 
widely deployed, this should be rare.

Of course, there are people who use HTTP URIs outside of the context of 
HTTP.  It would be unfortunate if the definition of such -- including 
validity -- were context dependent.

And, of course, there are plenty of non HTTP usages of URLs.  I'd like 
the same constraint described above to apply there too.

>                   Certainly IRI fragments can have non-ASCII characters, it's just second # and < > for example.

I'd like to see the definition of IRI retired.

>     *  "a URL  is a URI that doesn't act like a URN".

None of the above.  I mean what is defined by the following:

> I think the nomenclature issue is the primary cause of people talking past each other or getting sarcastic.

I've seen sarcasm, strawmen, caricatures, being condescending, and other 
inappropriate behaviors in reaction to the very idea of breaking things 
on paper.  None of this is helpful.

> One major change I had been trying to make in 3987bis was to change the processing model so that IRI -> URI mapping happened by parsing first, and then (only if reassembling into a URI) doing UTF-8…

IDNA (Internationalizing Domain Names in Applications) is for domains 
only.  I'm not proposing applying this on anything other than a domain name.

> This can happen when the same words are used with different meanings.

None of these are an appropriate response.  Period.

> Your problems and updates might  not have been as specific as you thought.

Fair point.  That's why I have now adopted the following as the way I to 
express what I propose

   ∀x: URI.valid(new URL(x).href) = true

I'd like to explore what changes to the definitions of [IETF] URI and/or 
[WHATWG/W3C] URL would be necessary to make that statement true.

- Sam Ruby

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 16 Jan 2015 11:50:07 +0000

I'm worried we've got unnecessarily confused and/or antagonistic
because we're not carefully distinguishing names and contexts of use.

Context of use is crucial, if I understand the history and the
requirements behind the IETF specifications correctly.

3986 defines a low-level protocol element (hereafer 'I_URI'), for use
in protocols such as HTTP.  I_URIs are restricted to ASCII, and have a
relatively conservative syntax intended to make the first-order
(scheme-independent) parsing of I_URIs easy.  It is _not_ a
specification designed primarily for human authoring or reading.

In consequence, over time, a number of alternative attempts to specify
more human-friendly presentation forms for I_URIs have arisen.  IRIs
can be seen as such a form, although 3987 [1] is less than completely
clear about this.  This aspect of 3987 is made explicit:

  "IRIs can serve as presentation elements for URI protocol elements.
   An example would be an address bar in a Web user agent." [2]

But IRIs extend I_URIs in a relatively conservative way, focussed on
(to put it informally) allowing a wider range of alphabets where
I_URIs allowed only ASCII alphabetics.  For this reason IRIs might
reasonably be criticised as not really satisfying a requirement for
congenial human use.  In particular (see earlier part of this thread),
3987 did _not_ change the 'reserved' production of 3986, or add '[' or
']' to the 'iquery' production.  Nor does it allow space characters or
most punctuation characters.

The first example I'm aware of, of a specification which actually
addresses the presentation form question, is "Legacy Extended IRIs for
XML resource identification" [3], which arose from the recognition that
the XML specification [4], and subsequently the XML Schema
specification [5], did _not_ constrain the contents of language
elements used for identification of web resources to I_URI or IRI
syntax.  LEIRIs added space and a number of ASCII punctuation and UCS
non-alphabetic ranges into the 'iunreserved' production.

The WHATWG URL specification [6] is essentially targetted at the
parallel problem for browsers and HTML -- what people type in the
'address' bar, or put in 'href' or 'src' attributes, has never been
restricted in practice to I_URI syntax.  The HTML 4.01 spec [7]
already recommends error recovery behaviour for non-ASCII strings in
href attributes.

It is thus it seems to me entirely reasonable for the HTML5
specification (for which I'm happy to view [6] as a proposed
revision/additional constituent) to define a high-level protocol
element, call it W_URL, for use in browsers and HTML language
elements.

The key thing, it seems to me, is to recognise that as such the W_URL
is not an alternative to, but rather a presentation form for, IRIs or
I_URIs, with its own (obviously related) syntax, and interoperably-
defined error recovery mechanisms.

It _also_ seems entirely reasonable to me, from the perspective of the
_low_-level protocols concerned, to be extremely cautious about any
changes to I_URI syntax.

I think the only changes needed are

  1) For Sam Ruby to re-interpret the _functionality_ which needs to
     be specified wrt W_URLs as including the mapping _from_ W_URLs
     _to_ I_URIs (possibly via a mapping to IRIs, possibly not);

  2) For the IETF to entertain minor revisions to 3986 or 3987 if
     that's required to make (1) easy to achieve.

ht

[1] http://tools.ietf.org/html/rfc3987
[2] http://tools.ietf.org/html/rfc3987#section-3
[3] http://www.w3.org/TR/leiri/
[4] http://www.w3.org/TR/xml/
[5] http://www.w3.org/TR/xmlschema11-1/
[6] https://url.spec.whatwg.org/
[7] http://www.w3.org/TR/html401/appendix/notes.html#non-ascii-chars
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: t.petch <[email-masked]>
Date: Fri, 16 Jan 2015 11:52:19 +0000

----- Original Message -----
From: "Sam Ruby" <[email-masked]>
To: "Dave Cridland" <[email-masked]>
Cc: "Graham Klyne" <[email-masked]>; <[email-masked]>
Sent: Thursday, January 15, 2015 4:38 PM

Sam,

A possibility that the IETF at large discussed extensively in 2013 and
rejected.  RFC6874 records the consensus of the IETF, that the rules on
the use of the percent character should not be relaxed, while
acknowledging that this behaviour was prevalent in a number of browsers.

So the IETF has gone there and said no in the recent past (for at least
one case).

That RFC also clearly flags the issue raised here of the difference
between what is valid in a URI and what is valid in a browser's input
dialogue box.

Tom Petch

---

From: t.petch <[email-masked]>
Date: Fri, 16 Jan 2015 11:59:18 +0000

----- Original Message -----
From: "Murray S. Kucherawy" <[email-masked]>
To: "Sam Ruby" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, January 15, 2015 7:30 PM

> > I was told (perhaps bad advice?) that the first step would be to
publish a
> > Based on what I see, and discussions I have held, I'd like to
proceed as
> If you're looking at doing a BoF, I think that aligns with what I was
going
> Given the current number of participants and especially the more
passionate
> of writing up a charter to define the work?

Murray,

I am pleasantlysurprised that none of this discussion has spilled over
onto

[email-masked]

which the IETF website describes as
"This mailing list is used to discuss updates to the URI standards and
processes."
I am glad it has not since I find the administration of apps-discuss
much better, but if you wanted to offload the ietf list, then that would
be another alternative.

Tom Petch

> I'm also curious to hear, and probably see in such a charter, why this
work
> either, just don't have the history to understand that part of it.

------------------------------------------------------------------------
--------

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 16 Jan 2015 12:08:30 +0000

Sam Ruby writes:

> potential results of the following expression be considered valid
> URIs, for all values of x:
>
>   new URL(x).href
>
> In the process I'd like to be free to explore "breaking things on
> paper" as Niko so eloquently put it.  Of course, breaking actual
> deployments should be avoided.  But given that versions of the above
> are actually widely deployed, this should be rare.

[I read this after sending my recent post titled "Terminology and
 scope...", which will appear after it in this thread]

I _think_ there's an implication here that I at least had not been
aware of in your previous references to "new URL(x).href".

Is it in fact the case that the value of this expression (also I think
what you've been calling the 'stringification' of a W_URL), _is_
exactly, byte for byte, what is put on the wire in HTTP requests by
browsers?

ht

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 07:54:12 -0500

>   scope...", which will appear after it in this thread]

In the hopes of tying together the threads, I'll make some brief 
responses to that email here:

I question whether RFC 3986 succeeded in the goal of making "first-order 
... parsing of I_URIs easy".  I see many different, and incompatible, 
implementations.

The sentence ("_not_ .. designed primarily for human") is not made clear 
by RFC 3986.

The WHATWG URL specification is not limited to browsers.  It defines a 
syntax that is fit for human production and consumption that is intended 
to be consistently parsed by a variety of parsers in a variety of 
contexts.  Every modern programming language has a URI or URL parse 
mechanism.

Minor is in the eye of the beholder.  What I am gathering is that 
changing RFC 3986 -- even in ways that only break things "on paper" -- 
is controversial.

I welcome everybody to participate in the discussion as to what W_URLs 
map to.

I welcome discussion as to whether that mapping produces valid I_URIs.

As to whether the changes necessary to the definition of I_URIs turn out 
to be 'minor' or the end result being that W_URLs map to something that 
is neither a proper superset or a proper subset of valid I_URIs is only 
something we will know after those discussions take place.

> exactly, byte for byte, what is put on the wire in HTTP requests by

That is my understanding.  To put it another way: if it is not, that 
would be considered a bug.

Note that stringification produces a string of Unicode code points.  As 
all of those code points -- with the notable exception of fragments -- 
are a proper subset of US ASCII graphic characters, the mapping of such 
to bytes is straightforward.

At a minimum, the WHATWG/W3C specifications should be updated to make 
this clear.

- Sam Ruby

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 16 Jan 2015 23:05:13 +1000

Latching on to some details:

> would hope that it is uncontentious.
I know it's slightly aside, but do you have a pointer to that discussion?
Just scanning Appendix A of RFC 3986, it appears we would parse the
authority as a reg-name, except that Section 3.2.2 then says:

] A registered name intended for lookup in the DNS uses the syntax
] defined in Section 3.5 of [RFC1034] and Section 2.1 of [RFC1123].

and RFC 1123 goes on to say:

] ... a valid host name can never
] have the dotted-decimal form #.#.#.#, since at least the
] highest-level component label will be alphabetic.

I may be a bit over my head here. Maybe that's something that could do with
clarification.

> One strategy would be to adjust the registration process to not allow new
> schemes to be registered with default ports.  Another would be to version
> the specification so that new schemes with default ports would require a
> revision of the parsing algorithm.  I would hope that this would be a
> relatively rare event.
>
>
I think this is heading to a place it doesn't need to go. At what point
does something need to compare canonical forms of URIs with schemes it
doesn't recognise? I can definitely see a lot of value in a generic library
that parses any URI-type string into general 3986-defined parts (say), that
can be further analysed by scheme-aware machinery -- so generic parsing is
useful, but generic (or universal) canonicalisation is an unnecessary
complication.

Cheers

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 08:30:53 -0500

On 01/16/2015 08:05 AM, Matthew Kerwin wrote:
> On 16 January 2015 at 08:55, Sam Ruby <[email-masked]
>     Is is indeed valid (this was discussed recently on this very list).
>     I would hope that it is uncontentious.

Correction: it was on a different list.

> I know it's slightly aside, but do you have a pointer to that
> discussion? Just scanning Appendix A of RFC 3986, it appears we would
> parse the authority as a reg-name, except that Section 3.2.2 then says:
>
> I may be a bit over my head here. Maybe that's something that could do
> with clarification.

Here's where my statement that URIs like the above were valid was 
pointed out to be incorrect:

https://lists.w3.org/Archives/Public/public-ietf-w3c/2014Dec/0075.html

- Sam Ruby

---

From: Hector Santos <[email-masked]>
Date: Fri, 16 Jan 2015 08:37:06 -0500

I have not read deeply into all this, and I don't wish to go off-base 
with this, but as a small developer shop, I rely on a working 
standards understanding between all organizations.  It makes technical 
and cost engineering common sense to extract the commonality of all 
the various "standards" methods. The IETF is a "common mans" group so 
its important that it recognizes the need and works on it, if 
possible. I support such work.

Thanks

-- 
HLS

On 1/15/2015 10:56 AM, Sam Ruby wrote:
> Based on what I see, and discussions I have held, I'd like to proceed
> as outlined here:
>
>

---

From: Dave Cridland <[email-masked]>
Date: Fri, 16 Jan 2015 13:57:39 +0000

A couple of points. I shall use the terms "URI" and "IRI" as defined in the
IETF specs, and W_URL to mean the W3C/WHATWG/etc thing.

On 16 January 2015 at 12:54, Sam Ruby <[email-masked]> wrote:

> syntax that is fit for human production and consumption that is intended to
> be consistently parsed by a variety of parsers in a variety of contexts.
> Every modern programming language has a URI or URL parse mechanism.
>
>
Yes, but it is only concerned with parsers.

The IETF is mostly concerned with protocols.

Taking the example of LDAP, this relies on schemas where a URI is
transported in a IA5String, which is ASCII-only (loosely, it's actually
7-bit ASCII supersets).

Allowing non-ASCII breaks this, no matter how we wave magic wands about
parsers - the parser simply never gets to see the URI, because the BER
decoder (or whatever) throws an error.

> Minor is in the eye of the beholder.  What I am gathering is that changing
> RFC 3986 -- even in ways that only break things "on paper" -- is
> controversial.
>
>
I suspect that there are at least two different change types possible.

A) One is to change reserved "ASCII codepoints" within particular
components of an IRI. I think this is pretty reasonable to explore, and is
the kind of thing that your work with examining the behaviour of parsers is
excellent for. The output of your work essentially informs us on the actual
levels of risk involved, and allows us to consider "running code", which,
with something as widespread in deployment as this is tremendously
important.

B) The other is to consider changes to the range of codepoints allowed in a
URI (or IRI) as a whole. For a URI, this is highly controversial, and I
don't believe it is possible. That said, we clearly do have the IRI
specifications (and their bis drafts), and if there is interest from the
W3C I think we could find the energy to complete that work. Changing the
range within an IRI would, I suspect, be similarly controversial. Even if
every parser in the world copes fine, it doesn't matter.

Examples of (A) include the "[]" in query strings, "#" in fragments, and so
on. These I'm personally open to in principle.

(B) includes things like allowing non-ASCII in URI fragments (as opposed to
IRI fragments). It would also include allowing spaces, line feeds, and so
on.

> I welcome everybody to participate in the discussion as to what W_URLs map
> to.
>
> to be 'minor' or the end result being that W_URLs map to something that is
> neither a proper superset or a proper subset of valid I_URIs is only
> That is my understanding.  To put it another way: if it is not, that would
> be considered a bug.
>
> all of those code points -- with the notable exception of fragments -- are
> a proper subset of US ASCII graphic characters, the mapping of such to
> bytes is straightforward.
>
> At a minimum, the WHATWG/W3C specifications should be updated to make this
> clear.
>

How would you feel about the possibility of having an API that looked
somewhat like this:

Given:

u = new URL(S) # Accepts strictly valid, and certain invalid, IRIs.

u.href #  "cleaned" valid IRI string, with any syntax cleaned according to
rules.

u.canonical # "canonical" IRI string.

u.toASCII() # "cleaned" URI string, for use in HTTP, LDAP, etc.

So with S as http://café.im:80/tŷ/%2e <http://xn--caf-dma.im:80/t%C5%B7/%2e>

u.href => http://café.im:80/tŷ/ <http://xn--caf-dma.im:80/t%C5%B7/>.
(although variants allowed)
u.canonical => http://café.im/ty <http://xn--caf-dma.im/ty>
u.toASCII() => http://xn--caf-dma.im/t%C5%B7

How hard would it be to persuade parser implementors to do this?

Dave.

---

From: Sean Leonard <[email-masked]>
Date: Fri, 16 Jan 2015 06:48:16 -0800

>     An example would be an address bar in a Web user agent." [2]

I just wanted to add 2¢.

It is a feature of RFC 3986 that RFC 3986 plainly describes what URIs 
are. Many IETF standards (and many standards in general) make normative 
statements about what *generators* (senders) vs. *parsers* (receivers) 
SHOULD or MUST do, where the parsing activity tends to be more lax than 
the generating activity (Postel's Law). RFC 3986 does not do this. It 
does not phrase things more strictly on the output side and less 
strictly on the input side; furthermore, it doesn't even reference RFC 2119.

Perhaps this is the source of some of the frustration on this list: 
since RFC 3986 doesn't really say what happens when non-URI (e.g., 
non-ASCII, excess [ ] # ) characters are encountered; it doesn't provide 
much guidance and thus becomes an area where implementations diverge if 
they want to accept them rather than throw a parsing error. However it 
is probably more accurate to say that implementations had long diverged 
prior to RFC 3986, so RFC 3986 described what it could describe, rather 
than prescribed what would not be followed anyway.

The authors of RFC 3986 (who are on this list) can probably provide 
insight into the matter.

I dispute the characterization that "It is _not_ a specification 
designed primarily for human authoring or reading."

First of all I find RFC 3986 reasonably readable as far as standards go. 
Second of all, human authoring and reading has *always* been a design 
goal of URIs, ever since Tim Berners-Lee's original 1992 paper on UDI 
(Universal Document Identifiers on the Network).

The issue is that UDIs (URIs) were originally designed in a pre-Unicode 
era, and notably are the most compatible in the modern English range 
(i.e., US-ASCII). So, it is more appropriate to say that *English* 
authoring and reading was a design goal, at the expense of other languages.

Thus in HTML 4.01:

http://foo.org/Håkon

may be much more "readable" than

http://foo.org/H%C3%A5kon

but, to an average English speaker (who has no idea how to say "å" in an 
unambiguous way), saying or otherwise replicating

http://foo.org/Håkon

will be very difficult if not impossible. But:

http://foo.org/H%C3%A5kon

is merely tedious; it's not impossible.

And as for saying:
"http://zh.wikipedia.org /wiki/巴泰勒米·波岡達"
...for an English speaker? Fo'geddaboutit!!!

Sean

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 16 Jan 2015 15:13:53 +0000

> First of all I find RFC 3986 reasonably readable as far as standards

Oops :-).  Of course, I didn't mean that the _standard_ wasn't meant
for human readers, but that the I_URIs it was _defining_ were not
primarily so intended.

> Second of all, human authoring and reading has *always* been a
> design goal of URIs, ever since Tim Berners-Lee's original 1992 paper
> on UDI (Universal Document Identifiers on the Network).

The distinction between higher- (human-facing) contexts and
lower-level (network-traversing) protocols wasn't there in the
beginning for XXIs, that's for sure.

> The issue is that UDIs (URIs) were originally designed in a
> pre-Unicode era, and notably are the most compatible in the modern
> English range (i.e., US-ASCII). So, it is more appropriate to say that
> *English* authoring and reading was a design goal, at the expense of
> other languages.

It has become increasingly necessary to take the distinction I'm
concerned with seriously as time has gone by, not just because of the
advent of UCS, but because of the expanded requirements of the
human-facing context of use.  I'm thinking here particularly of the
status of the space character.

ht

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 10:43:27 -0500

> decoder (or whatever) throws an error.

You and I have different ideas of "magic wands".  I will also note that 
I personally dislike this style of discourse.

Whatever they are called (URI, IRI, URL), as commonly practiced they may 
contain fragments and those fragments may contain bytes with the high 
order bit on.

The exists specs that make claims that such values aren't valid.  There 
exists software components (e.g. parses) that purport to convert URIs, 
IRIs, or URLs into spec compliant strings.  As a general rule, they do 
things differently.  In particular, most leave fragments alone.  Even 
when they contain characters with a high order bit on.

People who read such specs and the high level descriptions of said 
software components may indeed be mislead into designing schema choosing 
the wrong type of string[1], and that choice will cause runtime failures 
to occur.

Given the above, and using your term ("magic wand"), it is my belief 
that the attempt to invoke "magic" occurs in the existing specification, 
and that the choice to do so does a disservice to the readers.

Note: it would be presumptuous of me to assume that my perception of 
where the "magic wand" occurs in this scenario is shared with you. 
Please permit me the same courtesy.

> That said, we clearly do have the
> IRI specifications (and their bis drafts), and if there is interest from
> the W3C I think we could find the energy to complete that work.

Finding the energy to complete the IRI work would indeed be a game 
changer.  At the moment, I'm skeptical, but suffice it to say that I 
would be very interested if that came to pass.

Until that time, my intent I view with skepticism any and all use of the 
term IRI.

> u.href #  "cleaned" valid IRI string, with any syntax cleaned according
> to rules.
>

The current (and widely deployed) definition of href would produce 
exactly what you define as toASCII below.  And, in fact, is used for HTTP.

> u.canonical => http://café.im/ty <http://xn--caf-dma.im/ty>

The use case for this would need to be provided.

> u.toASCII() => http://xn--caf-dma.im/t%C5%B7

Again, this result is indistinguishable from .href.

> How hard would it be to persuade parser implementors to do this?

The canonical case concerns me most, as it is both a lot of work and 
there doesn't seem to be agreement as to what canonicalization means.  See:

http://intertwingly.net/blog/2004/07/31/URI-Equivalence

Those results were produced over a decade ago.  I reran those tests 
recently, and saw zero change.  :-(

- Sam Ruby

[1] http://www.zytrax.com/books/ldap/apa/types.html#strings

---

From: <[email-masked]>
Date: Fri, 16 Jan 2015 16:05:44 +0000

>>Sam Ruby wrote:

>>
>>An example of something that is not a conforming URL, but is both a valid URI and a valid IRI:
>>
>>  http://257.257.257.257/
>

>Dave Cridland wrote:

>OK... I don't think that's valid, but it's probably valid according to the ABNF only (ignoring the prose). I think >specifying that in the formal syntax would be uncontentious - in fact, draft-ietf-iri-3987bis-13 already makes >this change.

It is not even valid based on the ABNF, due to a fairly creative definition of dec-octet.

https://tools.ietf.org/html/rfc3986#section-3.2.2

Darrel

From: Dave Cridland
Sent: ‎Thursday‎, ‎January‎ ‎15‎, ‎2015 ‎5‎:‎09‎ ‎PM
To: Sam Ruby
Cc: IETF Apps Discuss

Lengthy is fine.

Lets posit the existence of a W_URL parser.  Given a string, it produces an object.  One of the operations that object performs is "stringification" (an IDL term that I don't particularly care for which essentially means serialization to a string form).

 - - -

If you look at the sequence of operations: parse followed by stringification, you get a string transformation.

This can be an identity transform.  If you want, try the following in Firefox or Chrome's console:

The result will be true.

 - - -

Now if you take a look at that transformation, we can discuss the what should be desirable characteristics of the set of outputs.

The first characteristic that I would like to suggest is that the outputs round trip, and by that I mean that if you feed that output back into the transformation, you get the same result back.

OK; you're saying that for any URL string S, new URL(new URL(S).href).href == new URL(S).href

Your tests seem to imply that for any URL string S, there is one and only one correct value of (new URL(S).href); is that also the case?

To put it another way, your tests imply a canonical form for a URI, and that seems much more useful.

The second characteristic is that it actually works.  This isn't a precisely specified criteria, but it means that if you attempt to use the result it doesn't break anything.  One example of this would be placing the string 'http://ietf.org/' into the href attribute of an HTML <a> element, rendering it on the screen, and having somebody click on the link.  If this goes to the right place, then all is good.  The definition of "works" for data: W-URLs, and for javascript: W-URLs are different, but that's not an obstacle at this level of discussion.

I can readily appreciate the sentiment here; like many things, it's hard to specify. But what you're saying is that for any URL string S, where (new URL(S).href) does not produce an error, the result must be a semantically valid URL string? (i'm being delightfully wooly about "semantically valid").

 - - -

At this point, I'm troubled.  Despite the set of desirable characteristics I have specified, I have a well defined set which is neither a superset of I-URI's nor I-IRI's, nor a proper subset of either.

An example of something that is not a conforming URL, but is both a valid URI and a valid IRI:

  http://257.257.257.257/

OK... I don't think that's valid, but it's probably valid according to the ABNF only (ignoring the prose). I think specifying that in the formal syntax would be uncontentious - in fact, draft-ietf-iri-3987bis-13 already makes this change.

Similarly, http://127.0.0.1:65536/ is also not valid, but because the concept of port is not tied to a specific transport protocol, and might not be restricted to a 16-bit range, that's a harder one to capture in pure ABNF - that is, the permissable values of port are arguably scheme specific.

Both these are issues of what I called semantic validity above. Others probably exist.

Here is something that round-trips a URL parse/stringify transformation, and actually works, despite being an invalid URI and invalid IRI:

My gut feeling is that allowing this to be valid is a possible change, as I've said. I appreciate this is at odds with RFC 3986, but if URL parsers do generally ignore the error that's fine.

To further compound this, RFC 3986 claims that the following strings are to be treated as equivalent (and perhaps even preferred as they are deemed to be 'valid'):

Indeed in many, and generally most, situations, those are considered equivalent.  But this is not necessarily so.  To it's credit RFC 3986 does describe a Comparison Ladder.  Unfortunately it does so using terms like "false positives" and "false negatives" when in fact, they aren't false at all.

I think that this would disappear, largely, with a canonical form. RFC 3986 currently says that pct-encoded should use upper case - that's a lower case "should" - and if we take this to be a canonical form, then the latter canonicalizes to the former, and we win. Of course, if [] is made legal in the query string it'd canonicalize to the previous example.

Comparison of URIs then become a case of comparing the canonicalized versions of both.

This is problematic in the case of ports - since http://www.google.com:80/ and http://www.google.com/ both refer to the same thing, but to have a single canonical form for both would require the parser to be aware of the default port for the scheme. But maybe unknown schemes cannot be canonicalized, or maybe we have two canonical forms. (Scheme-aware and scheme-agnostic; obviously you'd specify which you want and where).

 - - -

So much for a deep dive.  Returning to the surface: I'd like to simultaneously loosen up the definition of validity for I-URI's in some places while tightening it up in others.  This is a work in progress, so which I can't say exactly where just yet in each case, I can say that now would be a good time for interested parties to participate.

ht

- Sam Ruby

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 16 Jan 2015 17:23:07 +0100 (CET)

On Fri, 16 Jan 2015, [email-masked] wrote:

> Here is something that round-trips a URL parse/stringify transformation, and 
> actually works, despite being an invalid URI and invalid IRI:
>
> I've said. I appreciate this is at odds with RFC 3986, but if URL parsers do 
> generally ignore the error that's fine.
̈́
$ curl 'https://www.google.com/?q=[z]'
curl: (3) [globbing] bad range in column 32

This isn't strictly the URI parser saying this, but since we know that [] are 
not legal in a URI we (curl developers) could use them for additional 
functionality...

Just suggesting that changing this will make some things break.

-- 

  / daniel.haxx.se

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 11:30:44 -0500

On 01/16/2015 11:05 AM, [email-masked] wrote:
> the ABNF only (ignoring the prose). I think >specifying that in the
> fact, draft-ietf-iri-3987bis-13 already makes >this change.
>

It is not a valid IP-literal, but it is a valid reg-name (at least 
according to the ABNF):

See also:

- Sam Ruby

> *From:* Dave Cridland <mailto:[email-masked]>
> *Sent:* ‎Thursday‎, ‎January‎ ‎15‎, ‎2015 ‎5‎:‎09‎ ‎PM
> *To:* Sam Ruby <mailto:[email-masked]>
> *Cc:* IETF Apps Discuss <mailto:[email-masked]>
>
>     <http://example.com/%27%29.href> == 'http://example.com/'
>
>     The result will be true.
>
>     <HTTP://EXAMPLE.COM/PATH%27%29.href> == 'http://example.com/PATH'
>
>     https://www.google.com/?q=[z] <https://www.google.com/?q=%5bz%5d>
>
>
>     https://www.google.com/?q=%__5Bz%5D <https://www.google.com/?q=%5bz%5d>

---

From: Sean Leonard <[email-masked]>
Date: Fri, 16 Jan 2015 08:32:05 -0800

On Jan 16, 2015, at 8:23 AM, Daniel Stenberg <[email-masked]> wrote:

> This isn't strictly the URI parser saying this, but since we know that [] are not legal in a URI we (curl developers) could use them for additional functionality...
> 

Yes that is an excellent point. In my own software I use some of those properties of URIs. Since certain characters like { } [ ] < > are not part of URIs, they can be used to develop new functionality (e.g., URI regexps). For example [z] can mean “escape the z variable string in UTF-8” vs  {z} “escape the z variable octet array as-is”, or other variations. And of course { }, < >, and the like can be used to delineate an URI in other contexts, such as in plain text <http://this.is.a.uri.com/foo>.

Sean

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jan 2015 11:41:14 -0500

> Yes that is an excellent point. In my own software I use some of those properties of URIs. Since certain characters like { } [ ] < > are not part of URIs, they can be used to develop new functional…

This is indeed valuable input.  Another example where this is done:

https://tools.ietf.org/html/rfc6570

One way to look at this is that there are tools that layer additional 
functionality on top of URIs.

As to what the breakage it, that is less clear to me.  There are 
existing parsers that don't percent encode square brackets when they 
occur at some point after a question mark is encountered in the input. 
Perhaps those parsers lose the ability to claim that they are "RFC 3986 
compliant".

- Sam Ruby

---

From: <[email-masked]>
Date: Fri, 16 Jan 2015 16:42:19 +0000

>  https://lists.w3.org/Archives/Public/public-ietf-w3c/2014Dec/0075.html

That is unfortunate.  Thanks for pointing that out, I was not aware of that potential for ambiguity.

Darrel

---

From: Julian Reschke <[email-masked]>
Date: Fri, 16 Jan 2015 17:57:26 +0100

> Yes that is an excellent point. In my own software I use some of those properties of URIs. Since certain characters like { } [ ] < > are not part of URIs, they can be used to develop new functional…

"{" and "}" are used in URI templates.

Re square brackets, see for example: 
<http://stackoverflow.com/questions/40568/square-brackets-in-urls>

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 16 Jan 2015 18:08:29 +0100

> status of the space character.

When RFC 3987 was written, adding SP to the set of allowed characters 
was considered. However, it's in conflict with cases where formats need 
a white-space separated list of identifiers, or use whitespace as 
delimiters around identifiers, and that's why RFC 3987 stayed consistent 
with 3986.

Best regards, Julian

---

From: Dave Cridland <[email-masked]>
Date: Fri, 16 Jan 2015 17:34:14 +0000

On 16 January 2015 at 15:43, Sam Ruby <[email-masked]> wrote:

> You and I have different ideas of "magic wands".  I will also note that I
> personally dislike this style of discourse.
>
>
Actually you've entirely misunderstood my use of the term. I wasn't being
flippant in this instance, and my apologies if I inadvertently sounded that
way. I do try to make it very clear when I *am* being flippant.

> contain fragments and those fragments may contain bytes with the high order
> bit on.
>
>
What they're called makes a distinct difference here. URIs as specified are
7-bit clean, IRIs as specified are a unicode sequence of codepoints.

> things differently.  In particular, most leave fragments alone.  Even when
> they contain characters with a high order bit on.
>
> the wrong type of string[1], and that choice will cause runtime failures to
> occur.
>
> Given the above, and using your term ("magic wand"), it is my belief that
> the attempt to invoke "magic" occurs in the existing specification, and
> that the choice to do so does a disservice to the readers.
>
> Note: it would be presumptuous of me to assume that my perception of where
> the "magic wand" occurs in this scenario is shared with you. Please permit
> me the same courtesy.
>
>
By use of the term "magic wand", I did not mean in the sense you seem to
imply of a religious text, I meant a device which enables the wielder warp
reality in an [otherwise] impossible manner.

The "magic wand" lies in the problems of declaring one side the winner
without considering the damage it'll have on the losing side, as it were.
To safely fix the damage in any reasonable timeframe may require such a
wand.

If we say "fragments are 7-bit in URIs", then these parsers will be fine on
valid URIs. We're in a situation of garbage in, garbage out.

If we say "fragments can be any codepoint", then X.509 breaks - given there
are X.509 certificates blown into silicon, and in every DoD CAC, for
example, this is problematic, to various degrees. There are fixes, of
course - we would have to retire the old schema, and possibly the
containing extension, and replace it. There'll be a crossover period of
decades for this, where both extensions need to be supported, and some URLs
will be unrepresentable in some X.509 systems during this period. Having
two extensions fulfilling the same function is a worrying thought in a
security protocol, and I've little doubt there'd be an excitingly named TLS
exploit with a beautifully designed logo as a result.

URIs embedded into X.509 certificates include, for example, CRL
Distribution Points and OCSP Responders, and are therefore
security-critical. So my question is why you consider that the running code
you cite should be accorded more weight than the combination of the running
code others have cited *and* the specification?

Of course, you can also say "fragments are unicode in my very own URLs,
here is a transformation to turn them into URIs", but then...

> changer.  At the moment, I'm skeptical, but suffice it to say that I would
> be very interested if that came to pass.
>
... you're set on a path of either completing the IRI work, or duplicating
it.

>> So with S as http://café.im:80/tŷ/%2e
>> <http://xn--caf-dma.im:80/t%C5%B7/%2e> <http://xn--caf-dma.im:80/t%
>> C5%B7/%2e>
>>
>> u.href => http://café.im:80/tŷ/ <http://xn--caf-dma.im:80/t%C5%B7/> <
>> http://xn--caf-dma.im:80/t%C5%B7/>.
> The current (and widely deployed) definition of href would produce exactly
> what you define as toASCII below.  And, in fact, is used for HTTP.
>
>
OK. I'd actually prefer those to stay unicode, and in particular wouldn't
have expected hostnames to be rewritten to ASCII. I particularly don't like
this because it means for computed URL objects, it'll lose the
internationalization, so it can't be used for display anywhere, which seems
unfortunate to say the least.

>  u.canonical => http://café.im/ty <http://xn--caf-dma.im/ty> <
>> http://xn--caf-dma.im/ty>
>>
>
Comparison for equivalency.

> The canonical case concerns me most, as it is both a lot of work and there
> doesn't seem to be agreement as to what canonicalization means.  See:
>
It's the part that interests me most, actually. In part, because I think
once canonicalization is properly understood, the rest just falls out. It
*is* the hardest part, I agree.

---

From: Graham Klyne <[email-masked]>
Date: Sat, 17 Jan 2015 15:25:22 +0000

On 16/01/2015 16:41, Sam Ruby wrote:
> As to what the breakage it, that is less clear to me.  There are existing
> parsers that don't percent encode square brackets when they occur at some point
> after a question mark is encountered in the input. Perhaps those parsers lose
> the ability to claim that they are "RFC 3986 compliant".

Surely, it's not the role of a *parser* to %-encode, but a *generator* of URIs?

The primary role of a URI parser is to simply decide if a given string is or is 
not a valid URI.  A parser can only be RFC3986-compliant in the extent to which 
it correctly makes this determination in accordance with RFC3986.  Of course, 
parsers may do more than this, but the detail of such behaviour is not specified 
by RFC3986.

(I would say that a *generator* of URIs that does not %-encode square brackets 
in fragments is not RFC3986 compliant.)

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Sat, 17 Jan 2015 15:31:15 +0000

> That is unfortunate.  Thanks for pointing that out, I was not aware of that potential for ambiguity.

(Aside, not central to the main discussion...)

FWIW, RFC3986 is explicit that the formal syntax is ambiguous, and describes use 
of a "first-match-wins" algorithm to obtain an unambiguous parse.  (Personally, 
I'm not a fan of this approach, but I don't see it as a great problem.)

Cf. https://tools.ietf.org/html/rfc3986#section-3.2.2

#g
--

---

From: [email-masked] (Henry S. Thompson)
Date: Sat, 17 Jan 2015 15:51:47 +0000

Julian Reschke writes:

> was considered. However, it's in conflict with cases where formats
> need a white-space separated list of identifiers, or use whitespace as
> delimiters around identifiers, and that's why RFC 3987 stayed
> consistent with 3986.

Indeed.  And that's exactly why the LEIRI spec. says "caution needed" [1].

On balance, I think the scales _just_ tip towards allowing space in
the human-facing context, but do _not_ tip for the low-level protocol
context.  That is, I think 3986 was right _not_ to allow it
and LEIRIs are right _to_ allow it.  3987 has a weakness, in my view,
which is that it hasn't fully commited to being a high-level
human-facing standard, always layered on top of/mapping onto 3986 when
used in e.g. HTTP or LDAP transactions.  Until that is clarified, it's
hard to have a sensible discussion about whether e.g. space should be
allowed in IRIs or not.

ht

[1] http://www.w3.org/TR/leiri/#charStatus

---

From: Sam Ruby <[email-masked]>
Date: Sat, 17 Jan 2015 14:00:19 -0500

On 01/17/2015 10:25 AM, Graham Klyne wrote:
> The primary role of a URI parser is to simply decide if a given string
> is or is not a valid URI.  A parser can only be RFC3986-compliant in the
> extent to which it correctly makes this determination in accordance with
> RFC3986.  Of course, parsers may do more than this, but the detail of
> such behaviour is not specified by RFC3986.
>
> (I would say that a *generator* of URIs that does not %-encode square
> brackets in fragments is not RFC3986 compliant.)

As many people have pointed out, nomenclature seems to be a big problem 
here.  I started to write a reply that spells this out, but I realized 
that I was repeating things that I've said before, and figured it made 
sense to pull it out into a separate blog post that I can point to:

http://intertwingly.net/blog/2015/01/17/RFC-3986bis

TL;DR: URL parsers consume URLs and generate URIs.  Such URIs are not 
RFC 3986 complaint.  I’d like to fix that.

- Sam Ruby

---

From: mike amundsen <[email-masked]>
Date: Sat, 17 Jan 2015 14:58:21 -0500

Sam:

i might be missing some key points but..

<snip>
What is effectively being said is that documenting how things actually work
will break things, which is clearly untrue.
</snip>

let's not confuse "changing the specification" with "documenting."
 documenting NEVER breaks things. changing a specification MAY break things.

if all this is about is writing *documentation*, have at it. if you want to
write some BCPs, no worries.

but if the work involves changing specifications, as long as those changes
are made in ways that do not break existing *spec-compliant*
implementations, i have no problem with the work.

changing shared specs to match one or more existing non-compliant
implementations is rarely the right approach.

mamund
+1.859.757.1449
skype: mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

On Sat, Jan 17, 2015 at 2:00 PM, Sam Ruby <[email-masked]> wrote:

> here.  I started to write a reply that spells this out, but I realized that
> I was repeating things that I've said before, and figured it made sense to
> pull it out into a separate blog post that I can point to:
>
> TL;DR: URL parsers consume URLs and generate URIs.  Such URIs are not RFC
> 3986 complaint.  I’d like to fix that.
>
>  #g
>> --
>>
>
> - Sam Ruby
>

---

From: Sam Ruby <[email-masked]>
Date: Sat, 17 Jan 2015 15:21:27 -0500

On 01/17/2015 02:58 PM, mike amundsen wrote:
> What is effectively being said is that documenting how things actually
> work will break things, which is clearly untrue.
> </snip>
>
> if all this is about is writing *documentation*, have at it. if you want
> to write some BCPs, no worries.
>
> but if the work involves changing specifications, as long as those
> changes are made in ways that do not break existing *spec-compliant*

I have trouble with the words "existing *spec-compliant* implementations".

Either that term is meaningless, or I have yet to find one.

LDAP schemas that make use of IA5String can be said to be spec compliant 
in that they accept all RFC 3986 compliant URIs.  But that's quite a 
different matter than saying that they implement RFC 3986 in any 
meaningful way.

I've taken a look at a lot of libraries that produce and consume URIs 
(some call them URLs, but lets not worry about that for the moment).  I 
have yet to find one that is spec compliant.

> implementations is rarely the right approach.

There are a large and growing number of non-RFC 3986 compliant 
applications today.

I am working with a small and growing number of developers of these 
libraries.  I am building a shared specification to which these 
applications will be compliant.  When done I will have a large list of 
compliant applications that I can point to.

I would like to see that there be an update to RFC 3986 so that I can 
build upon that as a base.  But that's merely a desire, not a blocker.

- Sam Ruby

> On Sat, Jan 17, 2015 at 2:00 PM, Sam Ruby <[email-masked]
>             parsers that don't percent encode square brackets when they
>             occur at
>             some point
>         Surely, it's not the role of a *parser* to %-encode, but a
>         *generator*
>         of URIs?
>
>     As many people have pointed out, nomenclature seems to be a big
>     problem here.  I started to write a reply that spells this out, but
>     I realized that I was repeating things that I've said before, and
>     figured it made sense to pull it out into a separate blog post that
>     I can point to:
>
>     http://intertwingly.net/blog/__2015/01/17/RFC-3986bis
>     <http://intertwingly.net/blog/2015/01/17/RFC-3986bis>
>
>     TL;DR: URL parsers consume URLs and generate URIs.  Such URIs are
>     not RFC 3986 complaint.  I’d like to fix that.
>
>         #g
>         --
>
>
>     - Sam Ruby
>

---

From: <[email-masked]>
Date: Sun, 18 Jan 2015 01:08:59 +0000

From: Sam Ruby

>have yet to find one that is spec compliant.

Again I can only speak for the tests that you are doing on the csharp implementation, but the test results are far from conclusive when it comes to comparing against RFC 3986.  From looking at your makefile, I’m making a guess that you are using mono to build the csharp code.  The System.Uri class is actually part of the .net framework.  Currently there are multiple versions/editions of the .net framework in addition to the mono version.  Even within the Microsoft .net framework there are configuration settings that control URI parsing behavior

You can turn on IRI compliant parsing with a global setting.  http://msdn.microsoft.com/en-us/library/bb882600(v=vs.110).aspx  The default of this global setting was changed from false to true on the move from .Net 4.0 to 4.5.

You can also configure all kinds of other behavior related to URI parsing

http://msdn.microsoft.com/en-us/library/bb882619%28v=vs.110%29.aspx

My reason for bringing up these details is because I believe that declaring RFC 3986 invalid because implementations don't match its behavior is self-defeating.  IMHO, a spec should be a goal that implementations aim for, not a reflection of the behavior of an implementation that currently has the biggest market share.

Having said all this, I will try and run your tests on various versions of Microsoft’s .net framework and see how much the results deviate.  I did try running your C# tests, but they are based on a JSON file that I believe gets created by some javascript that I didn’t find.  My js skills are less than stellar!

Darrel

---

From: mike amundsen <[email-masked]>
Date: Sat, 17 Jan 2015 21:49:16 -0500

<snip>

i can only wonder...

first, these are not the only options and are not mutually exclusive.

second, the phrase "spec-compliant implementation" *is* meaningful however,
i suspect you're trying to say something else, as in "I challenge anyone to
find a single 3986-compliant implementation anywhere, ever" or something
along those lines.

finally, asserting that there exist no compliant implementations (seems
we're in Russell's teapot territory here) still does not give license to
introduce backward-incompatible changes to the spec.

again, if you are focused on *documenting* existing practices, this is all
much-needed and non-controversial work.

mamund
+1.859.757.1449
skype: mca.amundsen

On Sat, Jan 17, 2015 at 3:21 PM, Sam Ruby <[email-masked]> wrote:

> different matter than saying that they implement RFC 3986 in any meaningful
> way.
>
>>             after a question mark is encountered in the input. Perhaps
>> those
>>             parsers lose

---

From: Sam Ruby <[email-masked]>
Date: Sat, 17 Jan 2015 21:53:08 -0500

On 01/17/2015 08:08 PM, [email-masked] wrote:
>
> *From:* Sam Ruby <mailto:[email-masked]>
>
> Again I can only speak for the tests that you are doing on the csharp
> implementation, but the test results are far from conclusive when it
> comes to comparing against RFC 3986.  From looking at your makefile, I’m
> making a guess that you are using mono to build the csharp code.  The
> System.Uri class is actually part of the .net framework.  Currently
> there are multiple versions/editions of the .net framework in addition
> to the mono version.  Even within the Microsoft .net framework there are
> configuration settings that control URI parsing behavior

I tried Mono at first, but saw very poor results.  I then removed it 
from the makefile, and reran with Microsoft .Net and saw much improved 
results.  You can see the version I ran with here:

https://github.com/webspecs/url/blob/develop/evaluate/useragent-results/csharp

> You can turn on IRI compliant parsing with a global setting.
> http://msdn.microsoft.com/en-us/library/bb882600(v=vs.110).aspx  The
> default of this global setting was changed from false to true on the
> move from .Net 4.0 to 4.5.
>

If you have new results or updates to the code to suggest, I'm 
interested.  My preference is GitHub pull requests, but whatever works 
for you is fine with me.

> My reason for bringing up these details is because I believe that
> declaring RFC 3986 invalid because implementations don't match its
> behavior is self-defeating.  IMHO, a spec should be a goal that
> implementations aim for, not a reflection of the behavior of an
> implementation that currently has the biggest market share.

I, too, believe that a spec should be a goal that implementations aim 
for.  I also believe that spec writers should listed to implementers.  I 
am NOT saying that that wasn't done here, but I am saying that there is 
clear evidence that a decade after RFC 3986 was published that 
implementors (with the possible exception of Microsoft?) have not 
adopted RFC 3986 as a goal.  This includes quite a number of 
implementations that make claims of compatibility.

I will say that in the case of Microsoft, they have identified the URL 
specification as one that they are considering:

https://status.modern.ie/urlapi

They have a developer who is working on this, and he has posted feedback 
to the W3C mailing list.

> Having said all this, I will try and run your tests on various versions
> of Microsoft’s .net framework and see how much the results deviate.  I
> did try running your C# tests, but they are based on a JSON file that I
> believe gets created by some javascript that I didn’t find.  My js
> skills are less than stellar!

I'll make it easier for you.  You can find the json file you need here:

- Sam Ruby

---

From: Larry Masinter <[email-masked]>
Date: Sun, 18 Jan 2015 21:09:32 +0000

>         The primary role of a URI parser is to simply decide if a given
>         string is or is not a valid URI.  A parser can only be
>         RFC3986-compliant in the extent to which it correctly makes 
>         this determination in accordance with RFC3986.

I disagree with both these sentences, at least as I understand the
terms "primary role", "parser", "valid", "compliant".

The primary role of a parser http://en.wikipedia.org/wiki/Parsing#Parser
is to take input data and give a structural representation of the input,
while checking for correct syntax in the process.

Deciding validity is the role of a http://en.wikipedia.org/wiki/Validator.

RFC 3986 mainly defines conformance for URIs, and I don't
find any 2119-MUST/MAY/SHOULD for parsers, so 
any parser that accepts valid-per-3986 URI input and parses
it successfully could be "conforming",  even if also accepts invalid
input. 

> different matter than saying that they implement RFC 3986 in any 
 
Pass-through applications that embed URIs, and implementations
that only do minimal validation (but don't reject all invalid input)
should be counted as meaningful use cases and implementations
when considering whether changes are compatible.

RFC 3986 is an Internet Standard, see "Revising a Standard"
https://tools.ietf.org/html/rfc2026#section-6.3 
 
Changes would be best accomplished by making a new
standard rather than update the old, unless it's clear
that the overwhelming majority of the installed base
wouldn't be affected.  I don't think your examples
meet that criterion. 
It's possible that other updates to 3986 might. We
should discuss:

* Changes that make previously invalid strings valid URIs
 (e.g., allowing additional reserved ASCII characters like #
and <>  in the fragment?)

* Changes that make previously valid strings invalid
 (e.g., disallowing 258.258.258.258 as a host?) 

* Changes that affect how the structure of URI strings
  are described, the output of parsing. These might be
 OK but so many other specs make normative
  reference into 3986 that it's worthwhile being cautious.

Updating or replacing 3987 I_IRI to align with W_URLWW
is a different story because it's a different status.

Larry

---

From: Graham Klyne <[email-masked]>
Date: Mon, 19 Jan 2015 14:10:14 +0000

On 17/01/2015 19:00, Sam Ruby wrote:
> TL;DR: URL parsers consume URLs and generate URIs.  Such URIs are not RFC 3986
> complaint.  I’d like to fix that.

That's a usage which is at odds with (or at least extends) established computer 
science meaning of "parser".

#g

---

From: Graham Klyne <[email-masked]>
Date: Mon, 19 Jan 2015 14:16:44 +0000

> while checking for correct syntax in the process.

At risk or going down an unhelpful rathole...

A key phrase in your quote is "while checking for correct syntax".

In the absence of a definition of a "structural representation" in RFC3986, I 
don't see how RFC3986 compliance can extend beyond the determination of correct 
syntax, which is what I meant by "valid".

> Deciding validity is the role of a http://en.wikipedia.org/wiki/Validator.

Per that definition, I would say a parser is (or incorporates) a particular kind 
of validator.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Mon, 19 Jan 2015 14:35:57 +0000

On 16/01/2015 11:12, Sam Ruby wrote:
> Thank you, thank you, thank you for putting this so clearly!  I do believe that
> "break things on paper" is necessary given my data that valid URIs are a merely
> a Platonic ideal.  I do believe that "breaking in actual deployments" should be
> avoided.

I've been mulling this comment, as I think it goes to the heart of the debate. 
In particular, I wholly agree that "breaking in actual deployments should be 
avoided" is the crux.

The main point I want to make is that in considering actual deployments, the URI 
analyzers built into web application frameworks (such as ruby/rails, Django, 
Tomcat, Node, etc.) are an important part of the deployment landscape.  I hazard 
that these may be likely areas of breakage if incompatible changes are made to 
the URI spec as defined by RFC3986.  (If we lived in a world where all Web 
applications were built and designed along REST principles, including HATEOAS, 
then this would not be the case, but we are a long way from that, even among 
applications that claim to be RESTful.)

A second, lesser point is that while there are many parsers that are lax in what 
they accept, as shown by your data, they do not of themselves cause breakage by 
virtue of being lax.

#g
--

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 20 Jan 2015 20:03:53 +0900

>> The use case for this would need to be provided.

Like Sam, I'm strongly doubting there is a use case here.

> Comparison for equivalency.

Definitely, 'ty' and 'tŷ' cannot be assumed to be equivalent.

What's more, as explained in
http://tools.ietf.org/html/rfc3986#section-6
comparison/equivalence isn't a single operation, because different 
contexts need and use different equivalences. At one end are XML 
namespaces and RDF, where even 'http://www.example.org' and 
'Http://www.example.org' are different. At the other end are crawlers, 
which in general don't go to 'http://www.example.org/A' if they have 
been to 'http://www.example.org/a', because the chance that they get the 
same thing once again is too big for them to waste time, even if there's 
a chance that the two addresses will return totally different stuff.

So any kind of 'equivalence' or 'canonicalization' or 'normalization' or 
whatever we want to call it has to be qualified: What kind of e.g. 
equivalence, for what.

Regards,    Martin.

---

From: t.petch <[email-masked]>
Date: Thu, 22 Jan 2015 10:19:31 +0000

Larry

I notice that this (expired) I-D updates RFC3986 which seems germane to
recent discussions on this list.  Reading it, I can see why it might
update RFC3987 or 3987bis but cannot see where it updates RFC3986.  What
am I missing?

Tom Petch

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 22 Jan 2015 19:57:38 +0900

Hello Tom,

> update RFC3987 or 3987bis but cannot see where it updates RFC3986.  What

First, it says "3986 (if approved)", so this was conditional (of course).

Second, there's the following sentence in the introduction, which may be 
enough of an explanation:

    As every URI is also an IRI, the comparison and canonicalization
    methods also apply to URIs.

Third, I think that for this topic, 10 different people would write 10 
different stories, but the conclusion would be the same: *it depends*. 
Larry hoped that we could write a better story than RFC 3986 (and RFC 
3987, which is mostly the same story, just with some twists added). 
Also, on occasion, we had the dream to get rid of the URI vs. IRI 
distinction.

Based on the experience with that document, I'd recommend to mostly 
leave the equivalence/canonicalization/comparison story alone to save 
cycles for more immediately relevant work. Of course that doesn't 
exclude the possibility that somebody write a better story, but that can 
be done as a paper or a blog entry or an informal RFC independent of 
standards work.

Just my 2¢.

Regards,   Martin.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 23 Jan 2015 05:23:14 +0000

Tom Petch wrote, re draft-ietf-iri-comparison

> update RFC3987 or 3987bis but cannot see where it updates RFC3986.
> What am I missing?

I'm afraid

" As URIs are a subset of IRIs, the guidelines apply to URI comparison as well."

should have said the id ("Comparison, Equivalence and Canonicalization
 of Internationalized Resource Identifiers")
was intended to replace RFC 3986 section 6 "Normalization and Comparison"
in its entirety.

At this point I'd suggest comments on open in URL issues
https://github.com/webspecs/url/issues/18
https://www.w3.org/Bugs/Public/show_bug.cgi?id=27640

Should equivalence of URLs should mandate that if you parse URL  A and
get back URI B, then A must be equivalent to B?
Then, if you have a different URL C that turns into B, C must also be equivalent
to A and B. Even for URLs that are valid URIs. 
This requirement in 3986 is just a MAY.

Larry

---

From: Larry Masinter <[email-masked]>
Date: Fri, 23 Jan 2015 06:58:44 +0000

> cycles for more immediately relevant work.

I think separating it out into a separate document that can proceed at its own pace is a good idea, but there are important specific use cases to standardize.

> Of course that doesn't

I think there are requirements on comparison/equivalence/canonicalization that need to be normative, and what we have, now, is normative but misleading or incorrect. A URL needs to be equivalent to the URI it parses into.

Larry

---

From: t.petch <[email-masked]>
Date: Fri, 23 Jan 2015 13:21:47 +0000

---- Original Message -----
From: "Larry Masinter" <[email-masked]>
To: "Martin J. Dürst" <[email-masked]>; "t.petch"
<[email-masked]>
Cc: <[email-masked]>
Sent: Friday, January 23, 2015 6:58 AM

> I think separating it out into a separate document that can proceed at
its own pace is a good idea, but there are important specific use cases
to standardize.
> I think there are requirements on
comparison/equivalence/canonicalization that need to be normative, and
what we have, now, is normative but misleading or incorrect. A URL needs
to be equivalent to the URI it parses into.

Larry, Martin

Thank you for the clarification.  I see now how the I-D would have
updated RFC3986 had it advanced to completion.

I was not intending to pursue the subject of comparison further at this
time, rather the I-D is part of a Bibliography I hark back to when
issues of IRIs come up (as  recently they did).  Expired Draft it may
be, but I still find it a useful source of ideas.

Tom Petch

---

From: Graham Klyne <[email-masked]>
Date: Mon, 26 Jan 2015 14:56:00 +0000

IF this issue gets further discussion, I'd be very concerned that a comparison 
of URI beyond simple character comparison would not be universally implemented. 
  E.g., I do from time-to-time use URIs as index values for dictionary lookups - 
that depends implicitly on character-based equality.

Further I don't believe it is possible to completely define URI equivalence of 
different URI strings in a meaningful way, because the notion of equivalence 
depends on context of use.  When working with, say, RDF, when it is important to 
be sure that two references are denoting the same thing, the easy way is to just 
use the same referring string;  alternatively, use some extrinsic assertion that 
two different URIs are denoting the same thing.

In other words, why bother?  Just keep it simple.

#g
--

---

From: David Singer <[email-masked]>
Date: Mon, 26 Jan 2015 19:21:37 +0100

> On Jan 26, 2015, at 15:56 , Graham Klyne <[email-masked]> wrote:
> 
> IF this issue gets further discussion, I'd be very concerned that a comparison of URI beyond simple character comparison would not be universally implemented.  E.g., I do from time-to-time use URIs as index values for dictionary lookups - that depends implicitly on character-based equality.
> 
> Further I don't believe it is possible to completely define URI equivalence of different URI strings in a meaningful way, because the notion of equivalence depends on context of use.  When working with, say, RDF, when it is important to be sure that two references are denoting the same thing, the easy way is to just use the same referring string;  alternatively, use some extrinsic assertion that two different URIs are denoting the same thing.
> 

I guess it depends on what you mean by simple. 

The simplest for the receiver is to say it’s byte-by-byte equality.  The simplest for each environment that the URI might pass through is to insist it’s in the normal form for that environment.

These are in conflict.

So, if at the terminal the URIs arrive, hypothetically one by a route that wants UTF-8 and escaping, and the other by a route that handles everything in UTF-16, the terminal won’t find byte equality when the user expects it. The URIs need to be put in a canonical form.  Which means that that needs defining, alas.

What am I missing?

David Singer

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 26 Jan 2015 18:05:40 -0800

> So, if at the terminal the URIs arrive, hypothetically one by a route that wants UTF-8 and escaping, and the other by a route that handles everything in UTF-16, the terminal won’t find byte equalit…

Technically speaking, what namespaces use is character-by-character
equality.  Hence, that is not a problem.

> What am I missing?

I think people assume there exists some canonical form, and that it
makes some difference to the application that it knows that form (or at
least is able to derive it from a non-canonical URI).

The reality is that a canonical form (if any) is only known by the person
who configures the software to provide a service that is associated with
a given URI.  All other forms are aliases. Some of those aliases might
be mechanically expected (like hostname case), but others could be
just as canonical as the first.

In some cases, the preponderance of use in one form over another causes
the original owner to change their mind about which URI is canonical.

Regardless, what Graham noted is true: the definition of canonical for
caching/reusing a GET is very different from the definition of canonical
for something like namespaces.  That was a deliberate decision, whether
we agree with it or not, so it is fair to say that there is no single
standard for canonicalizing a URI (or URL).

This is, of course, why we have a link relation for canonical.

It is not an algorithm that can be written, in code or in spec,
so it is not a bug that needs fixing in RFC3986 (or anything that
pretends to supplant it).  The URI spec defines normalization of
URIs in ways that are not scheme-specific.  The scheme specs are
supposed to define what can be normalized within each scheme.

....Roy

p.s.  Am I the only one that finds it incredibly annoying that the
      APPS area uses this umbrella list for technical discussions about
      topics for which we still have the original working group mailing
      list active, archived, and populated with folks still capable of
      answering these questions?  Ya know,

         "http://lists.w3.org/Archives/Public/uri/"
         "http://lists.w3.org/Archives/Public/public-iri/"

---

From: Larry Masinter <[email-masked]>
Date: Tue, 27 Jan 2015 17:19:16 +0000

> IF this issue gets further discussion, I'd be very concerned that a
> comparison of URI beyond simple character comparison would 
> not be universally implemented.

I don't understand your concern. RFC 3986 (Standard) and RFC 3987
(Proposed Standard) already talk about comparison, in a way that
is incomplete, and at times confusing (e.g., by not being clear
about comparison vs. equivalence vs. canonicalization or
normalization as concepts.)

And some kinds of comparison need standardization, like for
'origin'.

> Further I don't believe it is possible to completely define URI
> equivalence of different URI strings in a meaningful way, 

I don't think the goal is to "completely define" it, but rather
put some constraints on conforming comparison methods
such that URI/URL/IRI producers can make assumptions about
the IRI/URL -> URI conversion (and URI -> IRI/URL too),
Namely that the conversion result will be equivalent under
any conforming comparison method (other than code-point
by code-point.)

Larry

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Jan 2015 10:01:19 +1100

>         "http://lists.w3.org/Archives/Public/public-iri/"

Nope - you are far from alone. However, appsawg is an active WG, whereas URI/IRI are not, so perhaps that's why people tend to direct attention here.

I'd far prefer that appsawg were shut down and replaced by "mail" and "Web" WGs, the latter taking responsibility for URIs. They'd fight over format stuff, but that mirrors real life...

/me notes arc of grenade and makes hasty exit

--
Mark Nottingham   https://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 27 Jan 2015 15:59:23 -0800

Silent +1 thrown in from somewhere in the back
On Jan 27, 2015 3:01 PM, "Mark Nottingham" <[email-masked]> wrote:

> Nope - you are far from alone. However, appsawg is an active WG, whereas
> URI/IRI are not, so perhaps that's why people tend to direct attention here.
>
> I'd far prefer that appsawg were shut down and replaced by "mail" and
> "Web" WGs, the latter taking responsibility for URIs. They'd fight over
> format stuff, but that mirrors real life...
>

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jan 2015 08:20:32 +0000

Going all meta for a moment...

Some of the most interesting discussions have arisen because the audience
here is broad enough to observe the uses of URIs outside the traditional
web space (ie, HTML). At the risk of sounding a little high and mighty, the
web world has a lot to learn from the more conservative mail world about
stability and compatibility - and the reverse is undoubtedly true as well.

Still, things like DMARC show that the mail world can be screwed over by a
few big players just as badly. Luckily those players have self censored
themselves from discussions...

In any case, unless this list were shockingly busy, I'd rather not hive off
discussions into groupthink collectives. Dissension is a fine product of
diversity.

Dave.

---

From: t.petch <[email-masked]>
Date: Wed, 28 Jan 2015 10:52:13 +0000

----- Original Message -----
From: "Roy T. Fielding" <[email-masked]>
To: "David Singer" <[email-masked]>
Cc: "Graham Klyne" <[email-masked]>; <[email-masked]>
Sent: Tuesday, January 27, 2015 2:05 AM

....Roy

Roy

Over the years, almost all the lists I deal with have come under the
umbrella of the IETF administration, giving them a consistent look and
feel, an adequate set of features, an easy way of reporting when the
archiving has collapsed (again), advance notice of many of the enforced
changes to my way of working, etc.

The result is less time spent on useless admin, more on real work.

So separate list, eg arcmedia, fine, under some third party which does
things differently or not at all .. thanks but no thanks.

Tom Petch

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Jan 2015 12:19:46 +0100

> things differently or not at all .. thanks but no thanks.

-1

The W3C lists work just fine, and having things (incl historical 
discussions) in a single place makes a lot of sense; that's why HTTPbis 
sticks with that as well.

Best regards, Julian

---

From: Sean Leonard <[email-masked]>
Date: Wed, 28 Jan 2015 06:43:02 -0800

On 1/28/2015 12:20 AM, Dave Cridland wrote:
> Some of the most interesting discussions have arisen because the 
> audience here is broad enough to observe the uses of URIs outside the 
> traditional web space (ie, HTML).
+1

Sean

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 29 Jan 2015 16:18:22 +1100

> On 28 Jan 2015, at 7:20 pm, Dave Cridland <[email-masked]> wrote:
> 
> Some of the most interesting discussions have arisen because the audience here is broad enough to observe the uses of URIs outside the traditional web space (ie, HTML). At the risk of sounding a little high and mighty, the web world has a lot to learn from the more conservative mail world about stability and compatibility - and the reverse is undoubtedly true as well.
> 
> Still, things like DMARC show that the mail world can be screwed over by a few big players just as badly. Luckily those players have self censored themselves from discussions...
> 
> In any case, unless this list were shockingly busy

I think we get into that state pretty often on this list... YMMV.

> , I'd rather not hive off discussions into groupthink collectives. Dissension is a fine product of diversity.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 31 Jan 2015 08:36:44 +0100

>it successfully could be "conforming",  even if also accepts invalid

Parsers are based on a language and "parse" input given to them, i.e.

  p = new Parser(language);
  result = p.parse(input);

If it is not possible to tell from `result` whether `input` is a member
of `language` then `p` is not a "`language` parser". To illustrate, the
`language` might be `XML`, and `p` could be any of these functions:

  * while (true) {}
  * return true;
  * return true if is_xml(input) or true;
  * return true if is_xml(input) or is_json(input) or true;

In none of these cases would `p` be an "XML parser". And it would not
suddenly become one if, in the last example, `or true` were removed.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Larry Masinter <[email-masked]>
Date: Sun, 1 Feb 2015 00:31:16 +0000

I don't think there's much point in arguing about the meaning of a word like "parser". Clearly there are different meanings in use.  You say

> of `language` then `p` is not a "`language` parser".

But this definition has many counter-examples, languages where conforming parsers for that language do not usually, or ever, signal validity errors. Start with HTML5. In fact, for most mime types, most parsers are more forgiving than strict.

But whether or not a parser checks validity, it is rare that "the PRIMARY role of a URI parser is to SIMPLY  DECIDE  parser is to simply decide if a given string is or is not a valid URI" (emphasis mine). The primary role of a parser is to be the first stage of "Parse and interpret". Validity checking (so you don't stuff invalid data to the interpreter) is a nice-to-have. 

"A parser can only be RFC3986-compliant in the extent to which it correctly makes this determination in accordance with RFC3986"

Just to make sure we're using the same vocabulary:

We usually talk about "conformance" and not "compliance", I think. A specification has a scope and defines roles, and conformance requirements for agents that fill those roles. With a language or protocol element kind of definition, the roles usually include "writer", "reader", "validator". A reader does parsing (the parser part) and interpreting (the part that does something with the parsed components that are the result of parsing.

If you agree with those definitions, then it is reasonable to not require conforming readers to not be "liberal in what they accept", and RFC 3986 makes no explicit conformance requirement for parsers used by readers of the URI protocol element.

Larry

> -----Original Message-----
> From: Bjoern Hoehrmann [mailto:[email-masked]]
> Sent: Friday, January 30, 2015 11:37 PM
> To: Larry Masinter
> Cc: Sam Ruby; [email-masked]
> Subject: Re: [apps-discuss] Scope of RFC3986 and successor - what is a
> URI?
> 
> * Larry Masinter wrote:
> >The primary role of a parser
> http://en.wikipedia.org/wiki/Parsing#Parser