Subject: [apps-discuss] Fun with URLs and regex

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Jan 2015 16:35:58 -0500

I’ve updated my Python script that serves as a translation of ABNF for URIs into regex.

https://gist.github.com/mnot/138549

It now validates the following URI schemes according to their respective specifications:
  - http
  - https
  - file
  - data
  - gopher
  - ws
  - wss
  - mailto

I didn’t finish mailto or data, because they allow quoted-string inside of URLs, and that makes my head hurt.

Would the respective communities review the regex to make sure they’re faithful (except for the caveat around quoted strings)?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Sam Ruby <[email-masked]>
Date: Wed, 07 Jan 2015 17:45:21 -0500

> It now validates the following URI schemes according to their respective specifications:

My test data:

http://intertwingly.net/stories/2014/10/05/urltestdata.json

A program to test each input:

import uri_validate
import json
import re

f = open('urltestdata.json')
tests = json.load(f)
f.close()

valid = {}
for test in tests:
   instr = test['input']
   valid[instr] = False
   if re.match("^%s$" % uri_validate.URI_reference, instr, re.VERBOSE):
     try:
       scheme_validator = "%s_URI" % instr.split(":", 1)[0].lower()
       validator = getattr(uri_validate, scheme_validator)
       if re.match("^%s#" % validator, instr, re.VERBOSE):
         valid[instr] = True
     except AttributeError:
       valid[instr] = True

print json.dumps(valid, indent=2)

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 8 Jan 2015 09:09:51 +1000

On 8 January 2015 at 07:35, Mark Nottingham <[email-masked]> wrote:

> ​[...]
> ​
> ​
> ​
> ​
> ​

This is very cool. I'm definitely going to use it to help me retool the
file ABNF. Thanks.

> ​​
> Would the respective communities review the regex to make sure they’re
> faithful (except for the caveat around quoted strings)?
>
>
​Having a look at file:

#456-458:

​| ​
# f-auth         = [ userinfo "@" ] host
​
| ​

| file_f_auth = r"(?: %(userinfo)s @ %(host)s )" % locals()

Shouldn't there be a `()?` around userinfo@​ ?

​Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 08 Jan 2015 00:23:45 +0100

>I’ve updated my Python script that serves as a translation of ABNF for 

I note that there are a number of automated tools for this; I have not
tried any of them recently though, but perhaps other can suggest one.

>of URLs, and that makes my head hurt.

The bigger problem might be that URI scheme grammars typically do not
account for %xx-encoding. It should be fine to write `dAtA:;B%41se64,`,
so you cannot use literals like `;base64` directly, unless you apply
some syntax-preserving pre-processing. RFC 6068 also re-writes the RFC
5322 productions it uses in prose, and considering how complex the RFC
5322 grammar is, it is probably not wise to attempt to do this manually.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 7 Jan 2015 17:18:12 -0800

> I’ve updated my Python script that serves as a translation of ABNF for URIs into regex.

If I don't get back to you, remind me, I have a tool that does the same:

https://github.com/martinthomson/abnf2regex

It might be old.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 08 Jan 2015 12:00:24 +0900

>> of URLs, and that makes my head hurt.

It'd probably make my head hurt, too, but I'll try to do it 
(automatically or manually) to help verify or update 
http://tools.ietf.org/html/rfc6068.

> account for %xx-encoding. It should be fine to write `dAtA:;B%41se64,`,

By RFC 3986 at least, it is.

> 5322 grammar is, it is probably not wise to attempt to do this manually.

Did you mean 'automatically'? I would agree in that case.

Regards,   Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Jan 2015 10:05:59 -0500

> 5322 grammar is, it is probably not wise to attempt to do this manually.

This is exactly the problem I ran into.

I think that this really needs to be clearer in the documents; while we all understand that the ABNF needs to be read within the context of the prose, this is stretching credibility IMO.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Jan 2015 10:07:48 -0500

Fixed, thanks.

I’m hoping to put this into a proper github project soon and refactor it, to make reading it and making contributions easier.

Cheers,

> On 7 Jan 2015, at 6:09 pm, Matthew Kerwin <[email-masked]> wrote:
> 
> This is very cool. I'm definitely going to use it to help me retool the file ABNF. Thanks.
> 
> 
> ​​Would the respective communities review the regex to make sure they’re faithful (except for the caveat around quoted strings)?
> 
> 
> ​Having a look at file:
> 
> #456-458:
> 
> ​| ​# f-auth         = [ userinfo "@" ] host
> ​| ​
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Sam Ruby <[email-masked]>
Date: Thu, 08 Jan 2015 11:54:56 -0500

> I’m hoping to put this into a proper github project soon and refactor it, to make reading it and making contributions easier.

First, Mark: thanks for doing this!

Would you consider putting it in https://github.com/webspecs/url, 
perhaps in the evaluate directory?

Wherever it is placed, I plan to write scripts that make use of this 
information.

For context (and in full disclosure, I discussed much of this F2F with 
Mark yesterday at the TAG meeting, but am including the background here 
for everybody's benefit):

Different people care about different things, and that's all right.  As 
an example, I was able to surprise Alex Russell and Dominic Denicola 
(both Google employees) by showing that the following expression 
produced different results on Chrome on Windows as compared to Chrome on 
OS/X:

   new URL("file:c|foo").pathname

Why does this matter?  Well in the case of web browsers, developers of 
those web browsers want content that shows up on the web to behave 
interoperabily, both across platforms, and across competing 
implementations.   More specifically, if you can construct a web page 
(including javascript) that produces different results on Internet 
Explorer on Windows vs Apple's Safari on OS/X, then that's a problem.

In the case of libraries like Node.js, there is a desire to parse URLs 
just like web browsers do.  Given that Dominic cares about node.js and 
Chrome, I can point him to date such as the following

https://url.spec.whatwg.org/interop/test-results/?select=nodejs&baseline=chrome

Looks like there is quite a bit of yellow on that page.

Since I care about the URL Standard, I created converted the prose into 
executable form so that it could be compared.  This allows me to very 
easily produce a similar page comparing nodejs and (by proxy) the URL 
standard:

https://url.spec.whatwg.org/interop/test-results/?select=nodejs

With this data, we can triangulate: and have a discussion with real data 
as to whether node.js should change or whether the URL standard should 
change.

Mark cares about valid URIs.  He's certainly not alone in that.  What he 
has done is express his interests not merely in high level prose, but in 
concrete, executable form.  Given that he has done that, I can pose some 
interesting questions.  For example, if you consider the process of 
canonicalizing a href value on an <a> element and stringifying the 
result, an implementation like Chrome will produce something that will 
be sent across the wire.  I've captured the results here:

https://raw.githubusercontent.com/webspecs/url/develop/evaluate/useragent-results/chrome

Given that data and Mark's script, I can produce a list of outputs that 
Mark doesn't consider valid:

   "http://f:21/%20b%20?%20d%20# e": false,
   "http://example.com/foo%": false,
   "http://2001::1]/": false,
   "http://[www.google.com]/": false,
   "http://f:fifty-two/c": false,
   "http://foo:-80/": false,
   "http://example.com/foo/.%2": false,
   "http://2001::1/": false,
   "http://[google.com]/": false,
   "http://example.org/foo/bar#\\": false,
   "http://example.org/foo/[61:24:74]:98": false,
   "gopher://example.com/": false,
   "http://example.org/foo/bar#\u03b2": false,
   "http://example.com/foo%2%C3%82%C2%A9zbar": false,
   "gopher://foo/": false,
   "http://example.com/foo%2zbar": false,
   "http://f:b/c": false,
   "a: foo.com": false,
   "http://[1::2]:3:4/": false,
   "http://example.org/foo/[61:27]/:foo": false,
   "http://f:%2021%20/%20b%20?%20d%20# e": false,
   "http://example.com/foo%2": false,
   "http://foo/path;a??e#f#g": false,
   "data:example.com/": false,
   "http://www.google.com/foo?bar=baz# \u00bb": false,
   "http://f:%20/c": false,
   "http://:www.example.com/": false,
   "data:/example.com/": false

With this data, we can have a discussion as to whether Mark's script 
should be updated, or Chrome should change, or some spec should change.

I have taken this a step further.  I wrote a script that will convert 
Mark's script from Python to JavaScript.  This means that he can 
maintain his script in Python and I can include an equivalent script on 
web pages and use that information to filter out things that aren't 
interesting or highlight things that are.

Some people here may have different interests, and that is OK too.  I 
encourage everybody to find a way to express their interests in the form 
of code.  While my preference is JavaScript, I'm otherwise pretty 
agnostic.  XSLT, Perl, PHP, C#: doesn't matter to me.

If you do so, I'll find a way to include that as a filter or as a base 
for comparison.  Doing so means that as the test suite grows, or 
implementation results change(*), you will be able to get instant 
results to the questions that interest you.

- Sam Ruby

(*) As an example, I'm trying to get updated results for IE.  Current 
status:

http://intertwingly.net/blog/2015/01/08/Ununzippable-Modern-IE

---

From: Sam Ruby <[email-masked]>
Date: Sun, 18 Jan 2015 17:52:06 -0500

> https://gist.github.com/mnot/138549

I've attempted to convert this to JavaScript:

https://url.spec.whatwg.org/reference-implementation/uri-validate.js

I've built a web page that makes use of it:

https://url.spec.whatwg.org/reference-implementation/uri-validate.html

- - -

Issues I encountered the process:

1) file_auth_path is defined with four instead of three double quotes

2) the last re.match rejects inputs that do not have a hash sign

3) extra, and potentially misleading, output is provided if instr starts 
with the characters "absolute:".

- Sam Ruby

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Jan 2015 12:53:48 +1100

Hi Sam,

> On 9 Jan 2015, at 3:54 am, Sam Ruby <[email-masked]> wrote:
> 
> Mark cares about valid URIs.  He's certainly not alone in that.  What he has done is express his interests not merely in high level prose, but in concrete, executable form.  Given that he has done that, I can pose some interesting questions.  For example, if you consider the process of canonicalizing a href value on an <a> element and stringifying the result, an implementation like Chrome will produce something that will be sent across the wire.  I've captured the results here:
> 
> Given that data and Mark's script, I can produce a list of outputs that Mark doesn't consider valid:

[...]

> With this data, we can have a discussion as to whether Mark's script should be updated, or Chrome should change, or some spec should change.

What I was hoping for was an update of the "valid URI" filter to take this into account at <https://url.spec.whatwg.org/interop/test-results/?filter=valid>.

E.g., that list currently includes test case 63, "https:/example.com/", which is valid according to the generic syntax in RFC3986, but not when you consider the scheme-specific constraints for HTTPS in RFC7230.

By filtering out these cases, we can see the places we potentially need to pay attention to in the RFCs.

Is that possible?

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Jan 2015 12:56:52 +1100

Fixed, thanks!

> On 19 Jan 2015, at 9:52 am, Sam Ruby <[email-masked]> wrote:
> 
> 3) extra, and potentially misleading, output is provided if instr starts with the characters "absolute:".
> 
> - Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Tue, 27 Jan 2015 21:24:50 -0500

> By filtering out these cases, we can see the places we potentially need to pay attention to in the RFCs.

Since that code runs filters on the browser, it would be easiest for me 
integrate code written in JavaScript.

Can you review the generated JavaScript?

Also, I would like to discuss where this code should live:

http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13635.html

- Sam Ruby

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Jan 2015 16:36:20 +1100

> On 28 Jan 2015, at 1:24 pm, Sam Ruby <[email-masked]> wrote:
> 
> Since that code runs filters on the browser, it would be easiest for me integrate code written in JavaScript.
> 

It looks like a reasonable transcription. I do notice you copied my error:

return new RegExp("^" + known[scheme] + "(#|$)").test(string)

I think it should just be:

return new RegExp("^" + known[scheme] + "$").test(string)

... which brings about another interesting observation -- only http and https define fragments in their syntax; the other schemes do not.

> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg13635.html

I want to keep it going in Python, so I'll probably create a separate repo at some point; is it bothersome to just keep the JS in your repo?

If bugs are found in the regex themselves, I'll make sure to notify you (and would appreciate the same).

Cheers,

---

From: Sam Ruby <[email-masked]>
Date: Wed, 28 Jan 2015 07:32:44 -0500

> return new RegExp("^" + known[scheme] + "(#|$)").test(string)

Not a copy: it's actually different.  My version checks for a match on 
the regular expression followed by either a hash mark or the end of the 
string.

> ... which brings about another interesting observation -- only http and https define fragments in their syntax; the other schemes do not.

I can make that change.

> I want to keep it going in Python, so I'll probably create a separate repo at some point; is it bothersome to just keep the JS in your repo?

I'm not proposing changing it from Python.

What I have is a script that does the conversion:

https://gist.github.com/rubys/b9ccaf304f06cf3e2e88

> If bugs are found in the regex themselves, I'll make sure to notify you (and would appreciate the same).

I'd suggest putting them both in the same repository.

In any case, later today I'll update my filter to replace the validation 
check with yours.

- Sam Ruby

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Jan 2015 15:05:17 -0600

On Wed, Jan 28, 2015 at 04:36:20PM +1100, Mark Nottingham wrote:
> ... which brings about another interesting observation -- only http
> and https define fragments in their syntax; the other schemes do not.

But STD66 describes fragments.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Jan 2015 22:14:26 +0100

On 2015-01-28 06:36, Mark Nottingham wrote:
> ...

It's because you asked for that in 
<https://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/0187.html>, and 
apparently were successful in convincing Roy.

(I still disagree with this outcome :-)

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 28 Jan 2015 13:40:27 -0800

On Jan 28, 2015, at 1:14 PM, Julian Reschke wrote:
> It's because you asked for that in <https://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/0187.html>, and apparently were successful in convincing Roy.

It is a very very very old debate regarding whether the fragment is part
of a URI or something attached to the end of a URI, but that was resolved
in RFC3986 (since the only thing that really matters here is that a fragment
is going to be parsed as such regardless of the scheme).

HTTP was merely updated to reflect what STD66 calls a URI.

....Roy

---

From: Sam Ruby <[email-masked]>
Date: Wed, 28 Jan 2015 16:51:05 -0500

> HTTP was merely updated to reflect what STD66 calls a URI.

Based on this discussion, I am gathering that the correct way to 
validate a URI with a known scheme is as follows:

   return new RegExp("^" + known[scheme] + "($|#" + fragment + 
")").test(string)

Anybody care to confirm or deny?

The full script is available here:

The function in question is at the bottom of this script.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Jan 2015 22:56:07 +0100

> HTTP was merely updated to reflect what STD66 calls a URI.

I agree that the fragment is part of the URI; the question, as far as I 
understand, is whether the *scheme* definition should include the 
fragment, given the fact that you can attach a fragment to any URI anyway.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 28 Jan 2015 14:06:50 -0800

On Jan 28, 2015, at 1:56 PM, Julian Reschke wrote:
> I agree that the fragment is part of the URI; the question, as far as I understand, is whether the *scheme* definition should include the fragment, given the fact that you can attach a fragment to any URI anyway.

The answer is "no".  http://tools.ietf.org/html/rfc3986#section-4.3

I don't know how I got beaten into submission on that for RFC7230.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 29 Jan 2015 09:54:28 +1100

> I don't know how I got beaten into submission on that for RFC7230.

I followed the e-mail discussion, resulting issue to the ultimate commit:
   <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2279>

I wouldn't call it a beating -- you didn't seem to need much convincing at all :)

That aside, it looks like we need an errata here...

It does look like draft-ietf-appsawg-uri-scheme-reg clarifies this, which is gratifying.

Cheers,

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Jan 2015 17:06:47 -0600

> That aside, it looks like we need an errata here...

an erratum.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Jan 2015 17:08:32 -0600

> Anybody care to confirm or deny?

For dereferenceable URI schemes that looks right to me.

It's not really right for mailto: URIs (which aren't dereferenceable).

I suppose data: URIs could have fragments.

Nico
--

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 29 Jan 2015 09:59:17 +1000

> fragment, given the fact that you can attach a fragment to any URI anyway.

The answer to this affects what I write in the 'file' scheme draft. I
was advised early on to not mention fragments (which I took as
"disallow by omission") because, while it's easy to define syntax, the
scheme also has to define semantics, and fragment semantics are tied
to content type, and dereferenced 'file' URIs don't have a
well-defined content type.

Whether or not I mention it comes back to the definition and intended
use-case of RFC 3986; if it defines an 'abstract' syntax - in the POO
sense - then there's no such thing as a universal parser (i.e. It's
impossible to parse a URI with an unknown scheme). If it defines a
low-level structure, then any URI can be parsed, but the individual
components can't be validated without deferring to scheme-specific
machinery.

If the former and I don't include the fragment in 'file', it isn't
allowed. If the latter, I just leave a hole in the spec.

Cheers

---

From: Larry Masinter <[email-masked]>
Date: Thu, 29 Jan 2015 00:23:10 +0000

> well-defined content type.

This leads to the (currently abandoned) work on MIME sniffing
which attempts to well-define a content-type for 'file' URIs:

https://mimesniff.spec.whatwg.org/ 

(bug list 
https://www.w3.org/Bugs/Public/buglist.cgi?component=MIME&list_id=51221&product=WHATWG&resolution=---)

based on (the abandoned)
https://tools.ietf.org/html/draft-ietf-websec-mime-sniff-03

Note https://www.w3.org/Bugs/Public/show_bug.cgi?id=20003
"Clarify scope of MIME sniffing" from IETF tracker #15

Based on
http://trac.tools.ietf.org/wg/websec/trac/ticket/15

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 28 Jan 2015 17:24:55 -0800

> allowed. If the latter, I just leave a hole in the spec.

It isn't that black and white.  The grammar for the scheme is what
excludes a fragment.  That doesn't prevent the scheme docs from
talking about fragments (in reference to RFC3986) and using them
within examples.

That part of the URI spec was written specifically to address
issues created by folks who thought they could redefine the meaning
of fragments within individual schemes, or forbid them entirely,
when in fact the meaning and use of fragments are independent of
scheme.

What makes you think that dereferenced files don't have a well-defined
content type?  The client might not know what it is, but that doesn't
mean the content type doesn't exist, and any decision to process the
file is basically an assumption of some content type (and its rules
for processing fragments).

....Roy

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 29 Jan 2015 15:14:11 +1000

> when in fact the meaning and use of fragments are independent of

Poking around again, I just saw the line "Fragment identifier semantics are
independent of the URI scheme and thus cannot be redefined by scheme
specifications." So yes, I think I now understand where everyone is coming
from in the discussion, and I also think I understand RFC 3986 less than I
did before. Time for more reading...

RFC 3986 "defines a grammar that is a superset of all valid URIs, allowing
an implementation to parse the common components of a URI reference without
knowing the scheme-specific requirements of every possible identifier",
that's clear enough - it's not an abstract grammar. The twist that I'd
missed was that, of the resulting components, not all of them are inputs to
the individual schemes; the 'fragment' component feeds into the content
handler directly.

I'm still suffering a misalignment: RFC 3986 defines the whole generic URI
syntax as:

    URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

and my draft that references it essentially defines (or will soon define)
the whole file-URI syntax as:

    file-URI = subset-of-scheme ":" subset-of-hier-part

By leaving off the query part, I've either said that a URI with a query
part cannot be a 'file' URI, or that a URI that starts with "file:" and has
a query part is invalid. Potayto potahto.

I don't know what I've said by leaving off the fragment part. According to
RFC 3986 I'm not allowed to touch the it, but now I have a collected ABNF
for 'file' URIs that doesn't have fragments. I think my way forward is to
include the fragment part in the grammar, and then deflect it the way 3986,
7230, etc. do.  "The fragment's format and resolution is ... dependent on
the media type of a potentially retrieved representation, even though such
a retrieval is only performed if the URI is dereferenced. ... The [client]
MAY either assume a media type of "application/octet-stream" or examine the
data to determine its type." Or something to similar effect. Does that seem
right to you? How do other representation-retrieving schemes deal with it?

​​
> for processing fragments).
Perhaps I s​hould have said "well known" instead of "well-defined," or that
the *means of determining* the content type is not well defined.

Back to Sam's library:

> Based on this discussion, I am gathering that the correct way to validate
")").test(string)
> Anybody care to confirm or deny?

What with all the reading and thinking I've just done, I suppose that's
right. As long as all the known schemes don't have their own #fragment bits
(http_URI does).

Aesthetically it's strange way to denote an optional fragment, but it gets
there in the end.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 29 Jan 2015 08:30:23 +0100

> For dereferenceable URI schemes that looks right to me.

Any non-dereferencable scheme can later become referencable by 
installing a resolver.

> I suppose data: URIs could have fragments.

They do.

Best regards, Julian

---

From: Sean Leonard <[email-masked]>
Date: Thu, 29 Jan 2015 01:10:44 -0800

On 1/28/2015 9:14 PM, Matthew Kerwin wrote:
>
> I don't know what I've said by leaving off the fragment part. 
> According to RFC 3986 I'm not allowed to touch the it, but now I have 
> a collected ABNF for 'file' URIs that doesn't have fragments. I think 
> my way forward is to include the fragment part in the grammar, and 
> then deflect it the way 3986, 7230, etc. do.

More than anything else, this is probably an artifact of the way that 
ABNF is written. It is not possible to "subclass" ABNF expressions; 
however, that is what URI scheme registrations should do. I.e., they 
should say "this URI scheme subclasses/profiles the following ABNF 
productions from RFC 3986...scheme = "file", hier-part = such-and-such 
restrictions". Something like that. (In the case of file: , there is the 
other problem that some things that are not valid per RFC 3986, like the 
use of |, you want to say something about anyway, so can be both 
restrictive and permissive in different ways.)

>     ​ ​
> Perhaps I s​hould have said "well known" instead of "well-defined," or 
> that the *means of determining* the content type is not well defined.
>

Sounds about right.

Sean

---

From: t.petch <[email-masked]>
Date: Thu, 29 Jan 2015 10:46:09 +0000

----- Original Message -----
From: "Matthew Kerwin" <[email-masked]>
To: "Roy T. Fielding" <[email-masked]>
Cc: "IETF Apps Discuss" <[email-masked]>
Sent: Thursday, January 29, 2015 5:14 AM

> when in fact the meaning and use of fragments are independent of

Poking around again, I just saw the line "Fragment identifier semantics
are
independent of the URI scheme and thus cannot be redefined by scheme
specifications." So yes, I think I now understand where everyone is
coming
from in the discussion, and I also think I understand RFC 3986 less than
I
did before. Time for more reading...

RFC 3986 "defines a grammar that is a superset of all valid URIs,
allowing
an implementation to parse the common components of a URI reference
without
knowing the scheme-specific requirements of every possible identifier",
that's clear enough - it's not an abstract grammar. The twist that I'd
missed was that, of the resulting components, not all of them are inputs
to
the individual schemes; the 'fragment' component feeds into the content
handler directly.

I'm still suffering a misalignment: RFC 3986 defines the whole generic
URI
syntax as:

and my draft that references it essentially defines (or will soon
define)
the whole file-URI syntax as:

By leaving off the query part, I've either said that a URI with a query
part cannot be a 'file' URI, or that a URI that starts with "file:" and
has
a query part is invalid. Potayto potahto.

<tp>

When I read your I-D, I assumed that you knew what you were doing:-)  I
can see a use for 'query' but if noone implements it, then better the
I-D left it out.  But I did have it as a point to pursue, once the
bigger issue, to me, of splitting what is valid according to RFC3986
from what is not and seeing that reflected in the I-D (and of course,
there is the mini-charter to agree:-(

Tom Petch

</tp>

I don't know what I've said by leaving off the fragment part. According
to
RFC 3986 I'm not allowed to touch the it, but now I have a collected
ABNF
for 'file' URIs that doesn't have fragments. I think my way forward is
to
include the fragment part in the grammar, and then deflect it the way
3986,
7230, etc. do.  "The fragment's format and resolution is ... dependent
on
the media type of a potentially retrieved representation, even though
such
a retrieval is only performed if the URI is dereferenced. ... The
[client]
MAY either assume a media type of "application/octet-stream" or examine
the
data to determine its type." Or something to similar effect. Does that
seem
right to you? How do other representation-retrieving schemes deal with
it?

that

> Based on this discussion, I am gathering that the correct way to
validate
")").test(string)

What with all the reading and thinking I've just done, I suppose that's
right. As long as all the known schemes don't have their own #fragment
bits
(http_URI does).

Aesthetically it's strange way to denote an optional fragment, but it
gets
there in the end.

  Matthew Kerwin

---

From: Sam Ruby <[email-masked]>
Date: Thu, 29 Jan 2015 10:25:17 -0500

On 01/27/2015 08:53 PM, Mark Nottingham wrote:
>
> What I was hoping for was an update of the "valid URI" filter to take
> this into account at
> <https://url.spec.whatwg.org/interop/test-results/?filter=valid>.
>
> E.g., that list currently includes test case 63,
> "https:/example.com/", which is valid according to the generic syntax
> in RFC3986, but not when you consider the scheme-specific constraints
> for HTTPS in RFC7230.
>

Done and deployed.  Let me know if you see any problems.

For now, I decided to match your Python's script's behavior with respect
to fragments and known but non-HTTP schemes.  This doesn't affect any of
the existing items in the current urltestdata set.

- Sam Ruby

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 31 Jan 2015 02:03:23 +0100

>    file-URI = subset-of-scheme ":" subset-of-hier-part

Think of this as

  URI = scheme ":" scheme-specific-part [ "#" fragment ]

And the logic is: if $scheme is 'file' then whatever is matched by
`scheme-specific-part` must also match such and such.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/