Subject: Re: [apps-discuss] Review of draft-ietf-appsawg-file-scheme

From: Matthew Kerwin <[email-masked]>
Date: Sat, 7 May 2016 17:54:05 +1000

Hi again everyone, sorry for the dead air. I've been trying to work on
these last open comments but am not really sure how tp address them.

On 15/04/2016 5:17 PM, "Julian Reschke" <[email-masked]> wrote:
>
> On 2016-04-15 09:10, Matthew Kerwin wrote:
>>
>>
>> Regarding security considerations, I've added some tentative hand-waving:
>>
>> ""
>> Some file systems have case-sensitive file naming and some do not.
>> Care must (?) be taken to avoid issues resulting from possibly
>> unexpected aliasing from case-only differences between file paths or
>> URIs.
>> ""
>>
>> I'm open to suggestions for improvement (or deletion.)
>
>
> case-insensitive (DOS?) < case-preserving (NTFS) < case-sensitive (others)
>
> ...so maybe mention all the three cases.
>

A case-based typo in a case-sensitive system gives an unwanted 404; an
intentional difference in a case-insensitive system gives an unwanted 200.
Case preservation doesn't really add anything to the issue, does it?

> Best regards, Julian
>
> PS: it might also be good to touch Unicode normalization forms...
>

How's this?

""
Care must be taken to avoid issues resulting from aliasing from mismatched
encodings or Unicode equivalences.
""

Cheers

---

From: Julian Reschke <[email-masked]>
Date: Sat, 7 May 2016 11:31:38 +0200

On 2016-05-07 09:54, Matthew Kerwin wrote:
> On 15/04/2016 5:17 PM, "Julian Reschke" <[email-masked]
> <mailto:[email-masked]>> wrote:
>>
> intentional difference in a case-insensitive system gives an unwanted
> 200. Case preservation doesn't really add anything to the issue, does it?

That's true, but maybe we can rephrase it in a way that it doesn't seem 
to neglect this case?

""
File systems vary in the way they handle case.
""

> Care must be taken to avoid issues resulting from aliasing from
> mismatched encodings or Unicode equivalences.
> ""

That can't hurt.

I wonder whether we can say something useful about how to map Unicode 
code points into the file URI (that is, before percent-encoding the 
UTF-8 octets). Is it a good idea to apply unicode normalization at this 
step? (I don't know but maybe others do?)

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 7 May 2016 20:03:19 +1000

On 7 May 2016 at 19:31, Julian Reschke <[email-masked]> wrote:

>> ​​
> That's true, but maybe we can rephrase it in a way that it doesn't seem to
> neglect this case?
>
> ""

That works nicely, thanks.

> ​​
> I wonder whether we can say something useful about how to map Unicode code
> points into the file URI (that is, before percent-encoding the UTF-8
> octets). Is it a good idea to apply unicode normalization at this step? (I
> don't know but maybe others do?)
>
>
That's kind of what Section 4 (Encoding) does; though that's the part that
Dave Crocker suggested I cut right back. In any case I am working on
rewriting that section since it doesn't quite feel right, but the intention
will remain the same -- use UCS + NFC + UTF-8 when possible.

Best regards, Julian
Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: John C Klensin <[email-masked]>
Date: Sun, 08 May 2016 12:26:41 -0400

--On Saturday, May 07, 2016 11:31 +0200 Julian Reschke
<[email-masked]> wrote:

>> Care must be taken to avoid issues resulting from aliasing
>> from mismatched encodings or Unicode equivalences.
>> ""
> 
> That can't hurt.
> 
> I wonder whether we can say something useful about how to map
> Unicode code points into the file URI (that is, before
> percent-encoding the UTF-8 octets). Is it a good idea to apply
> unicode normalization at this step? (I don't know but maybe
> others do?)

The developing conventional wisdom outside the IETF is to not
bother (and run the risk of normalization losing information or
causing other problems) but to save normalization for the actual
point at which the URI or other identifier is interpreted.  In
the particular case of "file" that might be particularly
important since some systems prefer different normalization
forms and have different restrictions.

Note that is not necessarily advice, just a report.

   john

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 10 May 2016 19:19:43 +0900

>> unicode normalization at this step? (I don't know but maybe

In general, NFC gives you a higher chance for a match that NFD. The Mac 
filesystem uses (mostly) NFD internally, but is able to handle NFC. On 
the other hand, Windows and Linux don't do normalization inside the file 
system, but the chances that files were created in NFC is higher than 
for NFC.

But milage(s) may vary.

Regards,   Martin.

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
> .
>

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 May 2016 12:06:28 -0400

--On Tuesday, May 10, 2016 19:19 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

> In general, NFC gives you a higher chance for a match that
> NFD. The Mac filesystem uses (mostly) NFD internally, but is
> able to handle NFC. On the other hand, Windows and Linux don't
> do normalization inside the file system, but the chances that
> files were created in NFC is higher than for NFC.

Agreed.  But note that this is partially an artifact that
illustrates why the i18n / "multilingual" versus localization
issues are important.    NFC gives a higher chance for a match,
especially with strings that are not systematically normalized
because, if one is using a keyboard designed for a particular
language or location, that keyboard is likely to support
locally-used characters and hence far more likely to product
precomposed characters than combining sequences.  The same is
generally true when people select characters from some sort of
online character-picker, assuming the precomposed forms exist at
all.   On the other hand, if I'm an experience user of one
script trying to use a keyboard designed for a wildly different
script or one with too many distinct character forms
("graphemes" or "grapheme clusters") to allow single-stroke
arrangements to work well, all bets are off.

For some scripts, there are also what look from the outside like
internal consistency problems with Unicode: for example, NFD is
more internally consistent then NDC because many recently-added
precomposed characters decompose under NFC rather than
composing.  And some don't, leading to some of the problems that
led to the "non-decomposable character" mess that led to the
LUCID BOF and the IETF's apparent paralysis about Unicode 7.x.

It is hard to say something in cases like this that will always
deliver the best, or even the most-expected, results.

   john

---

From: Ned Freed <[email-masked]>
Date: Wed, 11 May 2016 07:27:54 -0700 (PDT)

> deliver the best, or even the most-expected, results.

I'm trying, but I find it difficult to have much sympathy here. The underlying
problem is that repeated applications of the "this is a tiny bit better for
this constituency so we must have it or at least allow for it" rule in this
space has led to a situation where no single best practice exists.

If this sounds like we've reached a distinctly suboptimal pareto optimum, it's
because that's exactly what has happened.

I therefore think the best thing is to document the situation as best
we can and be done with it. This is supposed to be a specification
for a URL scheme; there's no reqruiement that such documents also serve
as a BCP.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 May 2016 16:35:54 -0400

--On Wednesday, May 11, 2016 07:27 -0700 Ned Freed
<[email-masked]> wrote:

>...
>> For some scripts, there are also what look from the outside
>> like internal consistency problems with Unicode: for example,
>> NFD is more internally consistent then NDC because many
>> recently-added precomposed characters decompose under NFC
>> rather than composing.  And some don't, leading to some of
>> the problems that led to the "non-decomposable character"
>> mess that led to the LUCID BOF and the IETF's apparent
>> paralysis about Unicode 7.x.
> 
>> It is hard to say something in cases like this that will
>> always deliver the best, or even the most-expected, results.
> 
> I'm trying, but I find it difficult to have much sympathy
> here. The underlying problem is that repeated applications of
> the "this is a tiny bit better for this constituency so we
> must have it or at least allow for it" rule in this space has
> led to a situation where no single best practice exists.
> 
> If this sounds like we've reached a distinctly suboptimal
> pareto optimum, it's because that's exactly what has happened.
> 
> I therefore think the best thing is to document the situation
> as best we can and be done with it. This is supposed to be a
> specification for a URL scheme; there's no reqruiement that
> such documents also serve as a BCP.

I hope it doesn't surprise you, but I agree.  Less is probably
more here, it is only the (IMO unnecessary) effort to be
exhaustive without either getting it right to the best of our
present knowledge or being clear that the guidance is general
only that I'm concerned about.

    john

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 12 May 2016 06:45:22 +1000

> for a URL scheme; there's no reqruiement that such documents also serve

I agree. I think I've almost worked out how to cut the Encoding section
back appropriately; I'll send some text to the group once I hace it, to see
if it covers it.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 13 May 2016 17:05:45 +1000

Okay

> back appropriately; I'll send some text to the group once I hace it, to see

How does this sound?

~~~
4. Encoding

File systems use various encoding schemes to store file and directory
names. Many modern file systems encode file and directory names as
arbitrary sequences of octets, in which case the representation as an
encoded string often depends on the user’s localization settings, or
defaults to UTF-8 [STD63].

Without other encoding information, percent-encoded octets in a file URI
([RFC3986], Section 2.1) MAY be interpreted according to the preferred or
configured encoding of the system on which the URI is being interpreted.
~~~

I don't know what advice to offer, if any.

Cheers

---

From: Dave Crocker <[email-masked]>
Date: Fri, 13 May 2016 06:05:16 -0700

On 5/13/2016 12:05 AM, Matthew Kerwin wrote:
> names.

Just to be particularly picky, since the text is about name encoding and 
not, for example, content encoding, I suggest the title be

    4. File Name Encoding

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 15 May 2016 15:26:30 +1000

> not, for example, content encoding, I suggest the title be
​Fair enough. I've published this version, with the cut down text and the
new section name, as -09 so people can view it in the whole or using the
diff tools.

Cheers

---

From: Sean Leonard <[email-masked]>
Date: Sun, 15 May 2016 02:38:18 -0700

On 5/14/2016 10:26 PM, Matthew Kerwin wrote:
>
>
> On 13 May 2016 at 23:05, Dave Crocker <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Just to be particularly picky, since the text is about name
>     encoding and not, for example, content encoding, I suggest the
>     title be
>
>        4. File Name Encoding
>
>     d/
>
>
> ​Fair enough. I've published this version, with the cut down text and 
> the new section name, as -09 so people can view it in the whole or 
> using the diff tools.

In the picky vein, Section 4 should be called
4. Path Encoding

Since we are talking about the whole path (including the host), not just 
file names; "path" is a well-defined computing term and the draft itself 
uses it. (But not really the fragment, which is not part of the path.) 
The text of Section 4 does not need to be changed.

Best regards,

Sean

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 15 May 2016 19:53:21 +1000

On 15 May 2016 at 19:38, Sean Leonard <[email-masked]> wrote:

> uses it. (But not really the fragment, which is not part of the path.) The
> text of Section 4 does not need to be changed.
>
>
​Actually, the host isn't part of the path, and this doesn't cover the host
anyway (that's handled by the term "fully qualified domain name" and the
ABNF for `host`.) The text actually talks about the file system encoding
(i.e. file path/name) rather than URI encoding. I was wondering about "File
Path Encoding" as a title, but I stuck with the suggested "File Name
Encoding" because I think people get it (it's only a title, after all, not
an interop-dependent normative requirement), and we've previously stated
that directories can be treated as files, so dirname encoding should also
be covered by filename encoding.​

Best regards,
> Sean
​Cheers

---

From: Sean Leonard <[email-masked]>
Date: Sun, 15 May 2016 03:06:31 -0700

On 5/15/2016 2:53 AM, Matthew Kerwin wrote:
>
>
> On 15 May 2016 at 19:38, Sean Leonard <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>>         encoding and not, for example, content encoding, I suggest
>>         the title be
>>
>>            4. File Name Encoding
>>
>>         d/
>>
>>
>>     ​Fair enough. I've published this version, with the cut down text
>>     and the new section name, as -09 so people can view it in the
>>     whole or using the diff tools.
>
>     Since we are talking about the whole path (including the host),
>     not just file names; "path" is a well-defined computing term and
>     the draft itself uses it. (But not really the fragment, which is
>     not part of the path.) The text of Section 4 does not need to be
>     changed.
>
>
> ​Actually, the host isn't part of the path, and this doesn't cover 
> the host anyway (that's handled by the term "fully qualified domain 
> name" and the ABNF for `host`.) The text actually talks about the file 
> system encoding (i.e. file path/name) rather than URI encoding. I was 
> wondering about "File Path Encoding" as a title, but I stuck with the 
> suggested "File Name Encoding" because I think people get it (it's 
> only a title, after all, not an interop-dependent normative 
> requirement), and we've previously stated that directories can be 
> treated as files, so dirname encoding should also be covered by 
> filename encoding.​
>

Compromise: how about "File Path Encoding"? ;-) Since "file name" 
implies file name (as distinct from directory name). Otherwise, we can 
go back to "Encoding"...

Sean