Subject: Re: [apps-discuss] Unicode normalization and Content-Disposition filename (RFC6266)

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Oct 2013 16:22:22 +0200

On 2013-09-30 15:01, Julian Reschke wrote:
> ...
> Funny enough, this just turned into a product issue for me (hey, paid
> work!).
>
> Interesting links:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=695995 and
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=14526
> ...

I just did some testing, and, indeed, Firefox sends the filename as 
obtained from the OS (and specified in HTML5), while Chrome normalizes 
to NFC.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Fri, 4 Oct 2013 10:14:42 -0500

> I just did some testing, and, indeed, Firefox sends the filename as
obtained from the OS (and specified in HTML5), while Chrome normalizes to
NFC.

Interesting.  Because most filesystems are just-don't-normalize-anywhere or
just-use-8 and most input modes produce something close to NFC, it helps to
normalize to NFC in the clients for some protocols or some filesystems
even.  I think there's no ideal, but the best combo is probably
form-insensitive/preserving on the server, with fallback on normalization
to NFC on the client.

For NFSv4, for example, knowing what a given filesystem is just-use-8 then
the client should normalize to NFC, but if the fs is normalize-to-NFDish or
form-insensitive/preserving then the client should not bother normalizing.
Is three any reason not to apply this logic to all fileserver protocols?

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Oct 2013 17:28:06 +0200

On 2013-10-04 17:14, Nico Williams wrote:
>
> On Oct 4, 2013 11:26 AM, "Julian Reschke" <[email-masked]
> <mailto:[email-masked]>> wrote:
>  >
> Interesting.  Because most filesystems are just-don't-normalize-anywhere
> or just-use-8 and most input modes produce something close to NFC, it
> helps to normalize to NFC in the clients for some protocols or some
> filesystems even.  I think there's no ideal, but the best combo is
> probably form-insensitive/preserving on the server, with fallback on
> normalization to NFC on the client.
> ...

In WebDAV, if a server preserves "everyting" (== allows both NFC and NFD 
in the same folder), clients get very confused (at least Mac OSX). 
Wouldn't it help if the server always normalized to NFC?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Fri, 4 Oct 2013 15:24:05 -0500

On Fri, Oct 4, 2013 at 10:28 AM, Julian Reschke <[email-masked]> wrote:
> In WebDAV, if a server preserves "everyting" (== allows both NFC and NFD in
> the same folder), clients get very confused (at least Mac OSX). Wouldn't it
> help if the server always normalized to NFC?

We thought about it long and very carefully for ZFS and we settled on
form-insensitive/preserving as the most interoperable behavior
("f-i/p").  We did that specifically because of HFS+'s impedance
mismatch with input modes and the fact that clients in general did
not, do not, and really, still cannot be counted on to normalize or
otherwise implement f-i/p behavior with decent performance (of
course).

Of course, the server could normalize to NFC, as you suggest.  Both
options, I think, can be said to create some surprise (i.e., to
violate POSIX), but ZFS has had this for many years now and no one
ever complained (except about the Apple port, which was changed to
match HFS+'s behavior).  Whereas people do notice problems caused by
HFS+ -- i.e., we know that normalize on CREATE, at least if it's to
any form other than that produced by input modes, is surprising.

F-i/p behavior happens only violates POSIX in the same sort of way as
case-insensitive/preserving behavior would: by creating apparent
aliases that are neither symlinks nor hardlinks.  Normalizing on
CREATE has a different surprise: that names you create don't appear
when you list a directory back.  I think f-i/p is less surprising than
normalization on CREATE is, and I think the HFS+/ZFS experience backs
this up.

Of course, it might be that normalization to NFC on CREATE is also not
surprising, and we don't have enough experience to back that up.  But
if it works it's only because it matches input modes, and therein lies
a problem: input modes don't produce NFC, they produce something close
to NFC.  Worse, different OSes/input modes may produce different forms
for the same names.  Normalizing on CREATE to *any* form might well
have all of the problems that HFS+'s normalization to NFD(ish) has.

My conclusion (biased by our experience with ZFS and my then-advocacy
for f-i/p behavior) is that f-i/p is the best solution.  It can only
be implemented at the filesystem though -- not the fileserver, not the
"VFS" (where there is one) on the server, and certainly not the
client.  IF f-i/p is simply not available then the next best choice is
to normalize on CREATE to either NFC or a form derived from common
input mode output forms, and then you get to figure out which of
client and/or server should do *that*.

BTW, note that f-i/p at the filesystem and normalization on CREATE at
the client and/or server are not mutually exclusive.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 4 Oct 2013 15:27:46 -0500

> Is three any reason not to apply this logic to all fileserver protocols?

Right, I forgot, there is a reason not to apply this willy-nilly: the
client doesn't always know its local inputs' codeset/encoding.  In
particular most NFS/AFS/Lustre/... clients have no clue what locale
user-land is running.  However, client-side normalization on CREATE
could be recommended when: a) the filesystem (note: not the server)
advertises that it wants only Unicode, and b) the filesystem
advertises that it does nothing about normalization.

Nico
--