Subject: Re: [apps-discuss] Fwd: FW: New Version Notification for draft-kerwin-file-scheme-13.txt

From: Larry Masinter <[email-masked]>
Date: Tue, 9 Dec 2014 19:54:41 +0000

I applaud the move to document file:.

Three comments on http://tools.ietf.org/html/draft-kerwin-file-scheme

First comment:

Please align with the special processing for “file:” in:
https://specs.webplatform.org/url/webspecs/develop/

That spec (developed in sync with WHATWG URL) focuses on
parsing and processing of base + relative, when the scheme
(or base scheme) is  “file:”, while the kerwin-file-scheme
talks more about mapping between local file paths and “file:”
IRIs (URLs).

But the overlap is significant, and I hope could be eliminated,
if the direction of moving the webplatform.org spec to
obsolete 3987 is accepted.

Second comment:

You describe the relationship between local file names
and ‘file:’ URLS for “UNIX-like”, “DOS- or Windows-based”
and “VMS Files-11”. These categories aren’t precise or
exhaustive, and file system processing of file names can
vary depending on version, language pack installed in OS,
as well as network protocol (Nfs?).

I would rather see a more extensible organization of
the specification by establishing a template, and then
separately documenting, on a file-system by file-system
basis, how to translate and match, based not on the
URI but on the parsed components resulting from
URL-parsing. This would allow new or different file
systems to be documented.

Third comment:

The translation between file names and URLs and things
like them happen in several specs:

content-disposition for file download, multipart/form-data
for file upload, by common HTTP servers, when packaging files
together into archive types as being discussed by those proposing
a new ‘archive’ top level media type.

It would be beneficial if we converge these, as it reduces interoperability
to have incompatible methods of translation for local files names depending
on the categorization.

Larry
--
http://larry.masinter.net

From: apps-discuss [mailto:[email-masked]] On Behalf Of Matthew Kerwin
Sent: Thursday, September 25, 2014 6:13 PM
To: IETF Apps Discuss; [email-masked]
Subject: [apps-discuss] Fwd: FW: New Version Notification for draft-kerwin-file-scheme-13.txt

FYI: this is my file URI draft, now listing apps-discuss as the official place for discussion.

A new version of I-D, draft-kerwin-file-scheme-13.txt
has been successfully submitted by Matthew Kerwin and posted to the
IETF repository.

Name:           draft-kerwin-file-scheme
Revision:       13
Title:          The file URI Scheme
Document date:  2014-09-26
Group:          Individual Submission
Pages:          16
URL:            http://www.ietf.org/internet-drafts/draft-kerwin-file-scheme-13.txt
Status:         https://datatracker.ietf.org/doc/draft-kerwin-file-scheme/
Htmlized:       http://tools.ietf.org/html/draft-kerwin-file-scheme-13
Diff:           http://www.ietf.org/rfcdiff?url2=draft-kerwin-file-scheme-13

Abstract:
   This document specifies the "file" Uniform Resource Identifier (URI)
   scheme, replacing the definition in RFC 1738.

   It attempts to document current practices, while at the same time
   defining a common core which is intended to interoperate across the
   broad spectrum of existing implementations.

Note to Readers (To be removed by the RFC Editor)

   This draft should be discussed on the IETF Applications Area Working
   Group discussion list <[email-masked]<mailto:[email-masked]>>.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

--
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 15:13:23 -0600

On Mon, Sep 29, 2014 at 11:35:57AM -0700, Sean Leonard wrote:
> Colleagues:
> 
> We ought to step back for a moment and ask ourselves what exactly
> the goals are of getting a common file URI scheme through the IETF
> process.
>
> [...]

For one thing, there are contexts where a URI is needed (e.g., browser
location bar, the DOM location, ...).

For another, file: is undeniably in use.

For a third, this is not unlike PKCS#11 URIs: there are cases where
local interoperability between applications would be helped by having
this.

IMO,

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 16:10:47 -0600

On Tue, Sep 30, 2014 at 09:37:05AM +1000, Matthew Kerwin wrote:
> On 30 September 2014 03:17, Sean Leonard <[email-masked]> wrote:
> > Specifying the Path to a File or Directory
> > <https://developer.apple.com/library/mac/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/AccessingFilesandDirectories/AccessingFilesandDirectories.html#//apple_ref/doc/uid/TP40010672-CH3-SW6>
> 
> The biggest reason for that omission is that, quite simply, I'd not seen
> it. (I figured, right from the start, that people would be ready and
> willing to point out all my shortcomings and direct me to their favoured
> implementations' documentation and justifications.)

Hmmm, it might be worth discussing because a file: URI using st_dev +
st_ino might be useful for matching purposes.  (There really shouldn't
be much in the way of an openi() outside very special cases where file:
URIs probably aren't getting used.)  But, "meh", besides, this
convention kinda camps on the traditional three-slashes file: URIs.

> Regarding the amount of text dedicated to Windows, all I said was: "they
> have drive letters, which you map this way," and: "some people actually map
> drive letters this way instead, but that's not good." [...]

You missed /cygdrive/<drive>/... (Cygwin), though Cygwin falls into the
POSIX "mostly no need to mention it" informational category.

> Is there somewhere else I've disproportionately represented legacy
> DOS/Windows cases to the exclusion of other systems?

I think it's fine.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 16:21:51 -0600

On Fri, Sep 26, 2014 at 09:19:14AM +0200, Daniel Stenberg wrote:
> On Fri, 26 Sep 2014, Matthew Kerwin wrote:
> >http://www.ietf.org/internet-drafts/draft-kerwin-file-scheme-13.txt
> 
> [...]
> I would rather have a new spec straigten up and tighten the language
> somewhat so that we can get a stricter interpretation of how a
> file:// is supposed to work. Sure, there will then possibly be a few
> non-compliant parsers out there that accept a few non-standard
> formats but it should be clear to those who generates or writes
> file:// URIs which the correct way is. That way we could potentially
> move into a future with less file:// URI variations rather than
> more. The rest of my comments are based on this basic stand.

My biggest concern is that ambiguities, if any, between various
historical forms should be documented, and the normative "modern" form
should be given.

Switching from traditional to modern forms may not be easy.  Text about
how to get there from here is needed.

> I also think that the 'host' part of the file: URI was a mistake to
> begin with. It was always a mystery to readers of RFC1738 and I
> think it should be better clarified here as well that it is useless
> and not interoperable.

At any rate, we need only one way to represent UNC-like paths.

> The query part. It didn't exist in 1738, so introducing that now
> will break a lot of old parsers (like mine) even if I recognize that
> 3986 says that it should be supported globally. This, for just a
> vague extension model without any clear purpose or use case.

About the only things I could think of stuffing there would be st_dev
and st_ino like things.

Are there any uses of queries that we know of for file: URIs?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 16:27:19 -0600

FWIW, HFS+ normalizes to something close to NFD.  Not adhering to the
recommendation to normalize might cause problems here.

Also, if file: is intended for local use only, then there may not be any
point to normalizing.  Hmmm, that will vary, no?  It will generally
depend on what the filesystem does.  Most just-use-8, some do better.
ZFS does normalization-insensitive matching; HFS+ normalizes to NFD.

Anyways, normalizing to NFC is probably the best thing to do, but
mentioning the above might give implementors the sense of urgency they
might need to actually implement this recommendation (or a clue as to
bug reports they run into).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 16:42:12 -0600

On Tue, Dec 09, 2014 at 10:37:19PM +0000, Phillips, Addison wrote:
> Although normalization is often a good idea... normalization might be
> a problem if the local filesystem allows normalized and non-normalized
> representations both to appear. You wouldn't be able to specify a
> non-normalized representation.

At least for Latin scripts most input methods produce NFC-ish output.
This is true even on OS X even though its own HFS+ decomposes file/
directory names (!).  What can I say other than "add that to the pile of
mistakes we need a time machine to correct"?

Anyways, in general normalizing to NFC does less damage than any other
normalization, and not normalizing can cause weird things, but then, so
can normalizing.

My preference (for those who don't already know) is for all [new]
filesystems to do normalization-insensitive matching, as that's the
least problematic option.  But this isn't going to happen anytime soon,
apps can't tell if the filesystem is going to do this, and so on and so
forth.

Nico
--

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 10 Dec 2014 08:59:35 +1000

On 10 December 2014 at 08:37, Phillips, Addison <[email-masked]> wrote:

> Although normalization is often a good idea... normalization might be a
> problem if the local filesystem allows normalized and non-normalized
Do you have an example? I'm trying to think it through, but I keep going in
circles. The one I think of is ext[2-4] where the filesystem stores octet
sequences, and shell/applications/etc. use things like the user's locale
environment when representing those octets as text strings. Are you saying
that if we mandate NFC normalisation of URIs, you can't distinguish between
a files whose filename octets are {0xE4} vs {0xC3, 0xA4} (i.e. U+00E4 "ä"
in WIndows-1252 / UTF-8)?

Wouldn't "file://%E4" would cover that?

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 9 Dec 2014 15:26:21 -0800

Hi Matthew,

Are you looking to process this through APPSAWG or as an individual
submission?

-MSK

On Thu, Sep 25, 2014 at 6:13 PM, Matthew Kerwin <[email-masked]>
wrote:

> URL:
> Diff:
> http://www.ietf.org/rfcdiff?url2=draft-kerwin-file-scheme-13
>
> Abstract:
>    Group discussion list <[email-masked]>.
>
>
>
>
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>
>
>
>
> --
>   Matthew Kerwin
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>
>

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 10 Dec 2014 09:30:35 +1000

> Are you looking to process this through APPSAWG or as an individual
I'd prefer it to be through the WG.

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 17:43:07 -0600

On Tue, Dec 09, 2014 at 10:57:17PM +0000, Phillips, Addison wrote:
> Yes, I completely agree that NFC is the right choice and does the
> least harm. For wire-format exchange the ideal would be for NFC to be
> the rule.
> 
> file:// is a little different, since it is mainly trying to represent

It may not be for a wire-format, but it exists nonetheless to
interoperate, even if locally-only, between different apps, possibly
portable apps.  Therefore the issue comes up.

> the characters/byte codes the file system is using while still trying
> to be recognizable to the user.  Maybe it doesn't matter, though: most

Even on a typical Unix system that doesn't help: the filesystem
typically treats filenames as octet strings not including '/'.

> user-agents provide a dialog box, auto-complete, or some other
> mechanism. Perhaps the rule is "NFC for the URI, file system/user
> agent provides re-normalization or normalization-independent selection
> for file systems that use a different normalization form,
> de-normalized files use escapes (== don't do that)"?

This comes up in many places (NFSv4, WebDAV, ...).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 17:54:26 -0600

> Wouldn't "file://%E4" would cover that?

Suppose one app doesn't normalize.  And another does.  The user might be
unable to type in the name of a file they want to open that exists.
This is a bit contrived because they'll be able to pick the file in a
file selection combo box, but still.

A classic example many years ago was a git repo that had such characters
in some filenames and which then broke on OS X.  I don't have a link
handy.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 20:23:13 -0600

On Tue, Dec 09, 2014 at 09:07:59PM -0500, John Cowan wrote:
> Phillips, Addison scripsit:
> > These are both in UTF-8, are visually indistinguishable, and are
> > identical under NFC, but fopen() cares which bag of bytes you grab.
> 
> The same is true on Windows, where filenames are 16-bit code units rather
> than 8-bit code units.  In general, we simply cannot normalize file names,
> because both Windows and Unix filesystems distinguish between names that
> are equivalent under canonical equivalence.

The preferable thing to do is to have form-preserve-on-create and form-
insensitive lookups in the filesystem, which creates some unlikely
aliases, but mostly avoids real problems.

In practice few filesystems do this, so applications layered above them
have to do something.  "Nothing" works most of the time.  Sometimes it
doesn't work, and when it doesn't, it hurts.  The example that comes to
mind is git on HFS+.  git has a configuration option to normalize:

core.precomposeunicode::
        This option is only used by Mac OS implementation of Git.
        When core.precomposeunicode=true, Git reverts the unicode decomposition
        of filenames done by Mac OS. This is useful when sharing a repository
        between Mac OS and Linux or Windows.
        (Git for Windows 1.7.10 or higher is needed, or Git under cygwin 1.7).
        When false, file names are handled fully transparent by Git,
        which is backward compatible with older versions of Git.

Also, the ideal is that the filesystem stores Unicode filenames, and
consumers in any non-Unicode locales convert.

I think I have an I-D lying about discussing this.  This keeps coming
up.  Maybe we should publish it?  Though the file: URI scheme makes a
poor trigger for publishing it: it'll be easier to ignore normalization
in the file: scheme.

Nico
--

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 9 Dec 2014 22:37:19 +0000

Although normalization is often a good idea... normalization might be a problem if the local filesystem allows normalized and non-normalized representations both to appear. You wouldn't be able to specify a non-normalized representation.

Addison

> -----Original Message-----
> From: Nico Williams [mailto:[email-masked]]
> Sent: Tuesday, December 09, 2014 2:27 PM
> To: Matthew Kerwin
> Cc: IETF Apps Discuss; [email-masked]
> Subject: Re: [apps-discuss] Fwd: FW: New Version Notification for draft-
> kerwin-file-scheme-13.txt
> 
> 
> Also, if file: is intended for local use only, then there may not be any point to
> normalizing.  Hmmm, that will vary, no?  It will generally depend on what the
> filesystem does.  Most just-use-8, some do better.
> Anyways, normalizing to NFC is probably the best thing to do, but mentioning
> the above might give implementors the sense of urgency they might need to
> actually implement this recommendation (or a clue as to bug reports they run
> into).
> 
> Nico
> --

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 9 Dec 2014 22:57:17 +0000

> normalization, and not normalizing can cause weird things, but then, so can
> normalizing.

Yes, I completely agree that NFC is the right choice and does the least harm. For wire-format exchange the ideal would be for NFC to be the rule.

file:// is a little different, since it is mainly trying to represent the characters/byte codes the file system is using while still trying to be recognizable to the user.  Maybe it doesn't matter, though: most user-agents provide a dialog box, auto-complete, or some other mechanism. Perhaps the rule is "NFC for the URI, file system/user agent provides re-normalization or normalization-independent selection for file systems that use a different normalization form, de-normalized files use escapes (== don't do that)"?

Addison

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 9 Dec 2014 23:05:28 +0000

Yeah, that’s exactly the example filesystem I had in mind.

Actually, my thought was that U+00E4 and U+0061.0308 would be:

{ 0xC3.A4 } vs. { 0x61.CC.88 }

These are both in UTF-8, are visually indistinguishable, and are identical under NFC, but fopen() cares which bag of bytes you grab.

Addison

From: [email-masked] [mailto:[email-masked]] On Behalf Of Matthew Kerwin
Sent: Tuesday, December 09, 2014 3:00 PM
To: Phillips, Addison
Cc: Nico Williams; IETF Apps Discuss; [email-masked]
Subject: Re: [apps-discuss] Fwd: FW: New Version Notification for draft-kerwin-file-scheme-13.txt

On 10 December 2014 at 08:37, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:
Although normalization is often a good idea... normalization might be a problem if the local filesystem allows normalized and non-normalized representations both to appear. You wouldn't be able to specify a non-normalized representation.

Do you have an example? I'm trying to think it through, but I keep going in circles. The one I think of is ext[2-4] where the filesystem stores octet sequences, and shell/applications/etc. use things like the user's locale environment when representing those octets as text strings. Are you saying that if we mandate NFC normalisation of URIs, you can't distinguish between a files whose filename octets are {0xE4} vs {0xC3, 0xA4} (i.e. U+00E4 "ä" in WIndows-1252 / UTF-8)?

Wouldn't "file://%E4<file:///\\%E4>" would cover that?

---

From: John Cowan <[email-masked]>
Date: Tue, 9 Dec 2014 21:07:59 -0500

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
The whole of Gaul is quartered into three halves.
        --Julius Caesar

---

From: Doug Royer <[email-masked]>
Date: Wed, 10 Dec 2014 13:32:31 -0700

The fopen() API on windows uses 8-bit or 16-bit depending on how you
compile your code with non-.NET. However the disk storage is a sequence of
8-bit bytes that could care less what character set you want to view the
characters in.

The .NET framework defaults to 16-bit UTF-16-like, and that also has
nothing to do with the actual file name on the disk that is stored as 8-bit
sequences.

fopen() does not care how the name looks to you. It only compares the byte
sequences to the byte sequences you asked to open. If you show the sequence
asUTF-8 it will display a name. If you use a non-UTF-8 compatible character
set, you get gibberish on you screen, but you can still open  the same file.

-
Doug Royer
[email-masked]
(714)989-6135

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 11 Dec 2014 12:58:57 +0900

On 2014/12/11 05:32, Doug Royer wrote:
> compile your code with non-.NET.

Yes.

However the disk storage is a sequence of
> 8-bit bytes that could care less what character set you want to view the

Mostly wrong.

Windows (since Windows NT as far as I remember) uses 16-bit UTF-16LE 
internally, and also on newer file systems (but not necessarily e.g. on 
floppy disks or CDs).
Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 11 Dec 2014 07:34:10 +0100

There is also here and there normalization that is done in the file system stack, which you discover when you want to move files/disks between systems that uses NFD and NFC as normalization forms (Apple and Linux for example).

See switch --iconv for rsync.

So, please decide one normalization form (pick NFC please) and say that that is what is used. Or, if you are working on a higher layer in the stack (sorry for not having had the time to know) do not touch the bits because there is normalization done elsewhere and maybe user A want to have her NFC normalized string go to the file system while someone else want her NFD normalized string to be passed on.

   Patrik

> On 11 dec 2014, at 04:58, Martin J. Dürst <[email-masked]> wrote:
> 
> Windows (since Windows NT as far as I remember) uses 16-bit UTF-16LE internally, and also on newer file systems (but not necessarily e.g. on floppy disks or CDs).
> Regards,   Martin.
> 
>