Subject: [apps-discuss] I-D Action: draft-ietf-appsawg-file-scheme-09.txt

From: [email-masked]
Date: Sat, 14 May 2016 22:15:08 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the ART Area General Applications Working Group of the IETF.

        Title           : The file URI Scheme
        Author          : Matthew Kerwin
	Filename        : draft-ietf-appsawg-file-scheme-09.txt
	Pages           : 18
	Date            : 2016-05-14

Abstract:
   This document specifies the "file" Uniform Resource Identifier (URI)
   scheme, obsoleting the definition in RFC 1738.

   It defines a common syntax which is intended to interoperate across
   the broad spectrum of existing usages.  At the same time it notes
   some other current practices around the use of file URIs.

Note to Readers (To be removed by the RFC Editor)

   This draft should be discussed on the IETF Applications Area Working
   Group discussion list <[email-masked]>.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-appsawg-file-scheme/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme-09

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-appsawg-file-scheme-09

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Sean Leonard <[email-masked]>
Date: Sun, 15 May 2016 03:15:52 -0700

> 	Filename        : draft-ietf-appsawg-file-scheme-09.txt

Did a full review; looks very good. Nice work!

The References section lists several references to MSDN articles.

Shorter MSDN article URLs are of the form:
https://msdn.microsoft.com/library/{ARTICLEID}

Such as:
https://msdn.microsoft.com/library/gg465305
https://msdn.microsoft.com/library/aa365247

I suggest simplifying the links, and updating the dates. Note that some 
library articles have dates; others do not. For example, [MS-DTYP] is 
most recently dated October 16, 2015, revision 30.0, according to its 
changelog at article ID cc230273.

In Appendix C, a discussion of Win32 Namespaces is given, but then it 
says "This specification does not define a mechanism for translating 
namespaced paths to or from file URIs." Readers would appreciate an 
informative reference for a mechanism to translate namespaced paths to 
and from file URIs, especially since there is a lot of technical content 
about Win32 processing in the rest of the Appendices.

I believe that the (American) English capitalization of Appendix D 
should be:
Appendix D. System-Specific Operations

Just curious: what happens if you try to access 
file://USERNAME@HOST/foo/bar.txt on a Windows UNC/network share, using a 
Windows machine and suitable API operations (e.g., Internet Explorer)? 
Will Windows try to access the network path with the given username, or 
something else? And does USER:PASS work?

I would be remiss if the term "alternate data streams" (ADS) were not 
mentioned in the context of Windows naming conventions. E.g.:
file:///G:/somebody.txt:other.txt

The syntax is already supported by the normative Syntax section, and is 
acknowledged without being named in the Security Considerations section. 
Therefore, no additional text is needed (but others may find it 
desirable--I leave that decision to this WG). At least it's now a part 
of the discussion record on this mailing list.

I just did a cursory check of some browsers on a Windows 7 machine. 
Ironically, Windows Explorer and Internet Explorer claim that they 
cannot find the ADS path, but Firefox and Chrome/Chromium-based browsers 
will load the ADS path all right. (I did not test what happens with 
relative references in, e.g., ./foo.txt:composite.html.)

Not sure if other file: URI processors handle such things, e.g., 
extended attributes / getxattr in Linux.

Sean

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 15 May 2016 22:24:08 +1000

On 15 May 2016 at 20:15, Sean Leonard <[email-masked]> wrote:

> library articles have dates; others do not. For example, [MS-DTYP] is most
> recently dated October 16, 2015, revision 30.0, according to its changelog
> at article ID cc230273.
>
>
​Well, I did access the en-us versions. When I go to the short URL you
linked above, I get the en-au versions ;)

Nevertheless I can refresh the references and update the dates. I'm
confused, though; MS-DTYP now includes an ABNF (which is cool) but it's
broken (which is not cool.) I think I know what they meant to say, but
that's not what they've actually said. I don't feel great referencing that,
even informatively.
​

> In Appendix C, a discussion of Win32 Namespaces is given, but then it says
> "This specification does not define a mechanism for translating namespaced
> paths to or from file URIs." Readers would appreciate an informative
> reference for a mechanism to translate namespaced paths to and from file
> URIs, especially since there is a lot of technical content about Win32
> processing in the rest of the Appendices.
>
>
​They might, but I don't have one. What would you suggest? What is
currently done?
​

> I believe that the (American) English capitalization of Appendix D should
> be:
​Sure, I can do that.
​

Just curious: what happens if you try to access
file://USERNAME@HOST/foo/bar.txt
> on a Windows UNC/network share, using a Windows machine and suitable API
> operations (e.g., Internet Explorer)? Will Windows try to access the
> network path with the given username, or something else? And does USER:PASS
> work?
>
>
​Didn't work for me in IE11, with any combination of @, user@, user:pass@.
Incidentally, it was also happy to accept both "c$" and "c%24" to access
the magic C:\ share.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Sean Leonard <[email-masked]>
Date: Sun, 15 May 2016 09:13:24 -0700

On 5/15/2016 5:24 AM, Matthew Kerwin wrote:
>
>
> On 15 May 2016 at 20:15, Sean Leonard <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On 5/14/2016 10:15 PM, [email-masked]
>     <mailto:[email-masked]> wrote:
>
>     <https://msdn.microsoft.com/library/%7BARTICLEID%7D>
>
>     Such as:
>     I suggest simplifying the links, and updating the dates. Note that
>     some library articles have dates; others do not. For example,
>     [MS-DTYP] is most recently dated October 16, 2015, revision 30.0,
>     according to its changelog at article ID cc230273.
>
>

Yep. Short URL is language/locale-neutral.

> confused, though; MS-DTYP now includes an ABNF (which is cool) but 
> it's broken (which is not cool.) I think I know what they meant to 
> say, but that's not what they've actually said. I don't feel great 
> referencing that, even informatively.

Well, the best solution is to report it to Microsoft and get them to fix 
it. You can reference this thread, among others. I see the following 
problems with that ABNF in gg465305:

host-name          = "[" IPv6address ‘]" / IPv4address / reg-name

has a turned comma; it should be:

host-name          = "[" IPv6address "]" / IPv4address / reg-name

pchar, fchar, and schar are defined in the range 00h-FFh. However, the 
definition says that "The filespace selector is a null-terminatedUnicode 
character 
<https://msdn.microsoft.com/en-us/library/cc230275#gt_fd33af2e-e1ce-4f8e-a706-f9fb8123f9b0>string". 
The reference to "Unicode character" says "Unless otherwise specified, a 
16-bit UTF-16 code unit." Also: "Unless otherwise specified, allUnicode 
strings 
<https://msdn.microsoft.com/en-us/library/cc230275#gt_b069acb4-e364-453e-ac83-42d469bb339e>follow 
the UTF-16LE encoding scheme with no Byte Order Mark (BOM)." (Unicode 
string) This would mean that the domain of the ABNF is 0000-FFFF.

I am fairly certain that Windows does not enforce well-formedness 
Unicode surrogate sequences, which makes the ABNF easier: you can blow 
through %x80-FFFF without needing special handling of high and low 
surrogate points.

/However/, the RPC type is defined as STRING, which is UCHAR*, which is 
"a null-terminated string of 8-bit characters". Therefore, we might be 
in UTF-8 land. If in UTF-8 land, is the UTF8-non-ascii production 
appropriate? (RFC 6532 & RFC 3629)

I haven't groked pchar, fchar, or schar. Overall, however, the authors 
of that MS spec have to determine and fix:

  * Is the ABNF in 0000-FFFF (16-bit units), or 00-FF (8-bit units)?
  * Is UTF-8 or UTF-16 intended?
  * What's actually going on in real life? I know that the Windows API
    is UTF-16LE. What about SMB on the wire?

> ​
>
>     In Appendix C, a discussion of Win32 Namespaces is given, but then
>     it says "This specification does not define a mechanism for
>     translating namespaced paths to or from file URIs." Readers would
>     appreciate an informative reference for a mechanism to translate
>     namespaced paths to and from file URIs, especially since there is
>     a lot of technical content about Win32 processing in the rest of
>     the Appendices.
>
>

Need someone else to chime in on this (and the other stuff).

Sean

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 16 May 2016 17:06:59 +1000

Hi Matthew,

I think the new Section 4 is an improvement -- much more precise than it was. However, a few questions still linger for me:

> 4.  File Name Encoding
> 
>    File systems use various encoding schemes to store file and directory
>    names.  Many modern file systems encode file and directory names as
>    arbitrary sequences of octets,

This is a bit confusing -- perhaps s/encode/store/ ? Or remove this part of the sentence altogether?

>    in which case the representation as an
>    encoded string often depends on the user's localization settings, or
>    defaults to UTF-8 [STD63].
> 
>    Without other encoding information, percent-encoded octets in a file
>    URI ([RFC3986], Section 2.1) MAY be interpreted according to the
>    preferred or configured encoding of the system on which the URI is
>    being interpreted.

Do the current implementations of file:// do this -- i.e., use the filesystem's encoding for the URI? 

Doing it that way seems unfortunate; two different users on the same machine (or network) won't see good interop in this approach.

If it's this way intentionally, or if we think it can't change, that's understandable, but if not we should have a good hard look at changing it IMO.

Has there been any discussion of this to date (sorry if I missed it)?

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 20 May 2016 16:37:02 +1000

Hi Mark,

On 16 May 2016 at 17:06, Mark Nottingham <[email-masked]> wrote:

> Hi Matthew,
>
> I think the new Section 4 is an improvement -- much more precise than it
> was. However, a few questions still linger for me:
>
> > 4.  File Name Encoding
> >
​I'll think about what I'm trying to say here, and how it could be said
better. Your suggestion is probably right.

> filesystem's encoding for the URI?
​Apparently. I don't have a spare drive lying around where I can reformat a
partition to test it for myself, though. A discussion I had with Dave
Thaler back at the very start of this draft revolved around the fact that
percent-encoded URIs are ambiguous (apparently a real issue for Windows),
which was why for a very long time the draft contained advice to use an
IRI​ instead, or at the least normalize.

> Doing it that way seems unfortunate; two different users on the same
> machine (or network) won't see good interop in this approach.
>
> If it's this way intentionally, or if we think it can't change, that's
> understandable, but if not we should have a good hard look at changing it
> IMO.
>
A fair bit of discussion on what the draft should say (for example
http://www.ietf.org/mail-archive/web/apps-discuss/current/msg14943.html ),
not much discussion on whether we should advise URI libraries to update.

Cheers

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 20 May 2016 18:55:15 +1000

> On 20 May 2016, at 4:37 PM, Matthew Kerwin <[email-masked]> wrote:
> 
> ​Apparently. I don't have a spare drive lying around where I can reformat a partition to test it for myself, though. A discussion I had with Dave Thaler back at the very start of this draft revolved around the fact that percent-encoded URIs are ambiguous (apparently a real issue for Windows), which was why for a very long time the draft contained advice to use an IRI​ instead, or at the least normalize.

VMs are good for testing.

It appears that both Windows and OSX have used UTF-8 for file name encoding for some time (since NT for the former, 10.0 for the latter, AIUI). See: <https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding>

Linux uses whatever locale is set. However, it appears that both Gnome and Firefox (at least) try to be 'smart' and will recognise UTF-8 even if ISO-8859-1 is set as the locale. Having said that, it's not too smart; if I try to open a file with a UTF-8 encoded name, Firefox can't find it when the locale isn't UTF-8 (although the file chooser *does* see it).

This is an important point; the advice above that they "MAY be interpreted according to the preferred or configured encoding of the system on which the URI is being interpreted" doesn't account for the fact that a single filesystem might have several users who have different encodings set*.

Other encodings seem to just be percent-encoded straight into the file URI, and Firefox doesn't make any attempt to display them as IRIs. 

(This seems to mirror how most browsers handle non-ascii characters in HTTP headers, e.g., Location; they just percent-encode them, since that's an encoding of bytes, not characters).

Can we say something more like this?

---%<---
When a file URI is produced, characters not allowed by the ABNF MUST be percent-encoded as characters using UTF-8 encoding, as per RFC3986 Section 2.5. 

However, encoding information for file and/or directory names might not be available. In these cases, implementations MAY use heuristics to determine the encoding. If that fails, they SHOULD percent-encode the raw bytes of the label directly.
--->%---

Cheers,

* Possible but unlikely, since most people are going to be using UTF-8. Still...

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 May 2016 08:05:44 +1000

Hi Mark, sorry for sitting on this for another week. I've been called up
for jury duty.

On 20 May 2016 at 18:55, Mark Nottingham <[email-masked]> wrote:

> > ​Apparently. I don't have a spare drive lying around where I can
> reformat a partition to test it for myself, though. A discussion I had with
> Dave Thaler back at the very start of this draft revolved around the fact
> that percent-encoded URIs are ambiguous (apparently a real issue for
> Windows), which was why for a very long time the draft contained advice to
> use an IRI​ instead, or at the least normalize.
>
​But some operating systems don't come cheap, if you don't happen to
already have a Windows VM to hand. ;)​

> It appears that both Windows and OSX have used UTF-8 for file name
> encoding for some time (since NT for the former, 10.0 for the latter,
> AIUI). See: <
> https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding>
>

​I thought NTFS used UCS2 for storing file and directory names; and the
Windows API functions I've seen (like PathCreateFromURL) return either UCS2
or Windows-1252 strings, depending on how you use them. I can't speak to
HFS+.​

>
> Linux uses whatever locale is set. However, it appears that both Gnome and
> Firefox (at least) try to be 'smart' and will recognise UTF-8 even if
> ISO-8859-1 is set as the locale. Having said that, it's not too smart; if I
> try to open a file with a UTF-8 encoded name, Firefox can't find it when
> the locale isn't UTF-8 (although the file chooser *does* see it).
>
> This is an important point; the advice above that they "MAY be interpreted
> according to the preferred or configured encoding of the system on which
> the URI is being interpreted" doesn't account for the fact that a single
> filesystem might have several users who have different encodings set*.
>
>
​Not explicitly, although I had it in mind when I wrote it. Maybe your
"heuristics" bit below covers it well enough.​

> Other encodings seem to just be percent-encoded straight into the file
> URI, and Firefox doesn't make any attempt to display them as IRIs.
>
> (This seems to mirror how most browsers handle non-ascii characters in
> HTTP headers, e.g., Location; they just percent-encode them, since that's
> an encoding of bytes, not characters).
>
> When a file URI is produced, characters not allowed by the ABNF MUST be
> percent-encoded as characters using UTF-8 encoding, as per
> ​​
> RFC3986 Section 2.5.
>
>
​That's strengthening the requirement. RFC3986 §2.5 only says that: if
characters are part of UCS they "should first be encoded" as UTF-8 before
being percent-encoded. The MUST-level requirement is just that octets not
allowed by the ABNF must be percent-encoded.

> However, encoding information for file and/or directory names might not be
> available. In these cases, implementations MAY use heuristics to determine
> the encoding. If that fails, they SHOULD percent-encode the raw bytes of
> the label directly.
>

​This says this part well.​

> * Possible but unlikely, since most people are going to be using UTF-8.
​Cheers