Subject: [apps-discuss] Unicode normalization and Content-Disposition filename (RFC6266)

From: Larry Masinter <[email-masked]>
Date: Sat, 21 Sep 2013 14:22:33 -0700

RFC 6266 defines a "filename*" parameter for content-disposition, as well as "filename". 
We weren't going to do that for form-data, as it seemed like overkill and no one
implements it.

While I was looking into this, though, I noticed that RFC6266 doesn't
mention the fact that different file systems handle Unicode normalization
differently.  So an a with an accent grave is represented as two Unicode
characters in some file systems and as one Unicode character in another.

When sending a file name from one system to the other, these representations
need to be mapped, but there is no completely reliable indication of
the file system defaults.

How is this handled in practice? If I upload a file with an unnormalized
name to a server which uses normalized names for files, who does
the normalization? 

Larry
--
http://larry.masinter.net

---

From: Nico Williams <[email-masked]>
Date: Sat, 21 Sep 2013 19:14:42 -0500

On Sat, Sep 21, 2013 at 02:22:33PM -0700, Larry Masinter wrote:
> How is this handled in practice? If I upload a file with an
> unnormalized name to a server which uses normalized names for files,
> who does the normalization? 

How timely.  We're discussing this very subject in NFSv4 WG.  NFSv4 was
supposed to have gotten this right back in RFC3530.  Reality intruded
however.

I've written up an I-D (just submitted) discussing the those realities,
I18N lessons learned in the NFSv4 space, boundary analysis for
filesystem protocols / implementations in general, and some advice:

http://tools.ietf.org/html/draft-williams-i18n-boundary-analysis-00

Some things to note:

 - I bet most implementations of RFC6266 use plain local filesystems, so
   what those filesystems do must be of interest to you :)

 - HFS+ normalizes to NFD on CREATE (and on LOOKUP IIRC);

   Arguably this violates POSIX.

 - ZFS has an option to normalize on LOOKUP (but NOT on CREATE)
   (i.e., normalization-insensitive but form-preserving, like
   case-insensitivity, which it also has an option for);

   Arguably this violates POSIX.

 - no other filesystem (to my knowledge) does anything at all as far as
   Unicode equivalence goes;

 - Unicode input methods generally produce NFC or close to it (even OS X
   does), at least for Latin scripts;

 - legacy filesystem metadata (file/directory names) in codesets other
   than Unicode abounds

IMO the right thing to do is for filesystems (note: not file *servers*)
to normalize on LOOKUP.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Sep 2013 10:51:33 +0200

> name to a server which uses normalized names for files, who does

For C-D, I have this test case:

   <http://greenbytes.de/tech/tc2231/#attwithfn2231utf8comp>

On Windows, the mapping *does* happen; not sure whether it's the 
filesystem or the browser.

Note that these kind of problems bug any system that maps HTTP directly 
to file system resources; such as WebDAV and whatever Subversion uses 
today...

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 22 Sep 2013 12:40:28 +0200

>filesystem or the browser.

I tested Opera 12.x and IE 9.x on Windows 7 and in both the "Save As"
dialog contains the decomposed form and the saved file is named with
the decomposed form. I figured perhaps some standard user interface
might normalise the file name to explain the discrepancy, but neither
Explorer nor the file properties dialog seems to. Could you re-run
this test?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Sep 2013 12:55:01 +0200

> Explorer nor the file properties dialog seems to. Could you re-run

That's what I see as well. (so the UA displays the lowercase a umlaut).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Sep 2013 13:16:02 +0200

> That's what I see as well. (so the UA displays the lowercase a umlaut).

Ok, so yes, the UI (both browser and Windows Explorer) show the umlaut, 
but what ends up in the filesystem is the composed form (checked with 
cygwin).

I'll have to clarify the test results.

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 22 Sep 2013 13:31:05 +0200

>I'll have to clarify the test results.

To be perfectly clear, the test case suggests `foo-a<U+0308>.html` and
whatever I did, I never saw that turned into `foo-<U+00E4>.html`, so the
browsers and operating system used the file name literally as suggested,
with no normalisation or other modification of any kind.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Sep 2013 13:35:33 +0200

> IMO the right thing to do is for filesystems (note: not file *servers*)

That would mean you won't be able to get back at some of the files that all have the same name after normalization.

If you normalize on LOOKUP, you have to normalize on CREATE.  (You also MAY want to keep the unnormalized form around for directory listings, just as you would do this in a case-insensitive filesystem.  Or maybe better not...:)

Unless you normalize to NFC on CREATE, the principle of least surprise more or less requires you to normalize to NFC on directory listing.

(I don't agree with the backward facing direction of your section 3.6 in
http://tools.ietf.org/html/draft-williams-i18n-boundary-analysis-00 -- file servers and file access protocols SHOULD do almost nothing of what you ask them to do.
You may HAVE TO do ugly backwards stuff for legacy, but SHOULD restrict this to the presence of an appropriate mount option etc.  Non-legacy code should never come into contact with it.)

Grüße, Carsten

---

From: Hector Santos <[email-masked]>
Date: Sun, 22 Sep 2013 10:17:07 -0400

I think I can offer some legacy considerations. I would like to run 
some test on our backend server that supports all of the original and 
traditional methods of evolving file transfer methods such as:

   - Ascii, XYZ, Kermit over RS232
   - Ascii, XYZ, Kermit over X.25, CIS
   - Ascii, XYZ, Kermit over TCP/IP (Telnet)
   - Java wire for native clients (PPP-like framing, very close to 
WebSockets)
   - FTP with extended QUOTE system for optional file info
   - HTTP
   - HTTP REST/automation

and see what are the current issues are for the TCP/IP methods.  Of 
course, escaping was already the main thing but I have been seeing 
more of the I8LN, UTF8 stuff not only with files but mostly currently 
with messages.  So the mail part is now doing the transformation thru 
ejected blackbox functions.  That would be one of the key issues of 
where in the backend does any transformation need to be done:

   - Client upload handler (hosting script),
   - Backend Database Server (ISAM/BTREE API, SQL, etc).

For some, the backend can offer the single source entry point where it 
will support all incoming clients and APIs as well.  Despite where its 
done, the main thing I recall was basically detecting the type of 
transformation needed, if any.  With files, escaping was the main 
thing and I believe it is mostly done for the HTTP clients.

Anyway, I think a good solution to the multi-functionality issues;

   - data transfer,
   - data storage,
   - data displaying, and
   - data lookup (searching)

is to provide guidelines for the backend server of what sort of 
functionality is possible (and desirable for the future).

So for example, the last big file transfer file name transformation 
industry revamping done was related to 8.3 file name vs Long File 
names.   For backend API and compiled clients backward capability,  we 
had to keep two file names on record for all files uploaded:

     Backend File Info Fields:
     ...
     ShortFileName  (was FileName)
     FileName       (new field, supports LFN)

This allows us to support the structured binary API as well.  So with 
the more modern clients now issuing LFNs, the backend would call a 
special (Windows) API that would reflect the file system naming 
nomenclature to obtain and reserve the *would be* 8.3 file name.  I 
recall one case where this was not possible with a particular brand 
NFS.   This required the NFS's file system driver to be supportive of 
this Win32/64 File I/O functionality.

So one additional question for the backend server is whether to keep 
multiple forms of a normalized or unnormalized filename for backend 
and clients compatibility.

--
HLS

On 9/21/2013 5:22 PM, Larry Masinter wrote:
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>
>

-- 
HLS

---

From: Nico Williams <[email-masked]>
Date: Sun, 22 Sep 2013 15:33:24 -0500

> That would mean you won't be able to get back at some of the files that all have the same name after normalization.

Not so, and I have an existence proof (ZFS).  ZFS with this enabled
will NOT allow you to have two files with equivalent names in the same
directory.  ZFS will also not allow you to enable this feature except
at dataset (filesystem) creation time, so you can't create such
conflicts ex-post.  Given that there may not be multiple equivalent
names in the same namespace then normalization on LOOKUP is indeed
sufficient, and it provides the least surprise: the names you CREATE
are the names you get back on READDIR.

> If you normalize on LOOKUP, you have to normalize on CREATE.  (You also MAY want to keep the unnormalized form around for directory listings, just as you would do this in a case-insensitive filesys…

The analogy of normalization-[in]sensitivity/preservation to
case-[in]sensitivity/preservation is correct.  A filesystem need not
fold case on CREATE in order to be case-insensitive, folding case on
LOOKUP will do.

> Unless you normalize to NFC on CREATE, the principle of least surprise more or less requires you to normalize to NFC on directory listing.

You're mixing layers.  If your filesystem does not normalize on LOOKUP
then your only workaround is to normalize on CREATE, LOOKUP, and
READDIR at the next layer up (the fileserver, the application,
whatever).

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Sep 2013 22:37:31 +0200

On Sep 22, 2013, at 22:33, Nico Williams <[email-masked]> wrote:

> ZFS with this enabled
> directory. 

Yeah, that's exactly what I'd call normalize on CREATE.
(But you are right, these terms aren't really well-defined.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Sun, 22 Sep 2013 17:25:07 -0500

> (But you are right, these terms aren't really well-defined.)

Ah, perhaps we have a terminology problem, and we agree after all.
Maybe we should come up with terms we can agree on widely?

The three behaviors implemented in the wild are:

 - form-insensitive and -preserving (ZFS)
 - form-insensitive but not form-preserving (HFS+)
 - no normalization anywhere at any time (all other filesystems, to my
knowledge)

What should we call these?

Creation of named objects in filesystems is generally expected to be
atomic, and involves an implied lookup in the relevant namespace -- to
me this went without saying, but perhaps it really should be stated
explicitly in any document discussing this subject.  That's made it
convenient for me to speak of "normalization on CREATE (and LOOKUP)"
as "create objects with normalized names" and "normalization on
LOOKUP" as "preserve name forms, but be form-insensitive".

I've seen complaints about HFS+'s normalization of object names to NFD
at CREATE time (i.e., they are normalized then stored normalized).
I've never seen a complaint about form-insensitive/preserving
behavior, but, of course, it helps that most input modes reliably
produce names in the same form (something close to NFC), so that
usually one would not notice any problems.  HFS+'s behavior causes
problems more than anything because its choice of NF results in
different forms on-disk than what users can input with usual input
modes.  This means we don't have very solid evidence for deciding what
is the best behavior empirically, but I think f-i/f-p behavior
produces the least surprising and most interoperable results.

Nico
--

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 23 Sep 2013 16:48:08 +0900

>   - HFS+ normalizes to NFD on CREATE (and on LOOKUP IIRC);

Please be careful. As far as I'm aware, this is a form closer to NFD 
than to NFC, but not identical to NFD.

Regards,   Martin.

---

From: Nico Williams <[email-masked]>
Date: Mon, 23 Sep 2013 13:37:29 -0500

On Mon, Sep 23, 2013 at 2:48 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> Please be careful. As far as I'm aware, this is a form closer to NFD than to
> NFC, but not identical to NFD.

You're right, and the differences are described here:
https://developer.apple.com/legacy/library/technotes/tn/tn1150.html#UnicodeSubtleties

The key point, however, is that there is a noticeable difference
between what OS X (and Windows, and *nix, and probably every) input
modes produce and what HFS+ produces.  If you run "touch foó" in a
shell and then move/copy that to a filesystem that doesn't normalize
on lookup then a subsequent "ls foó" will fail.  I know because I
tried and blogged about this back in 2006[*].  This is, in fact, what
led us to add form-insensitive/form-preserving functionality to ZFS
(or at least it's what caused me to push for it).

I find the explanation given for why HFS+ decomposes characters in
filenames confusing:

"To reduce complexity in the B-tree key comparison routines (which
have to compare Unicode strings), HFS Plus defines that Unicode
strings will be stored in fully decomposed form, with composing
characters stored in canonical order."

The use of *any* canonical form would permit the use of memcmp()
semantics in the B-tre key comparator.  Clearly using a form that more
closely matches the input modes' output form would have been better
for interoperability.  There must be subtleties that are not
described, such as trade-offs in string size vs normalization
performance (NFC is specified as a two stage process the first of
which is decomposition, therefore NFD requires less computation than
NFC), and/or memcmp() collation (which will be different for NFC and
NFD).

In the case of ZFS we optimized for mostly-ASCII filenames, so that
the comparator compares by sliding two-byte windows one byte down in
its fast path, branching to a slow path when a difference involves at
least one byte value outside the ASCII range.  The ZFS slow path
normalizes one character at a time, so as to avoid allocation.  This
is used both, where ZFS compares strings and where it hashes them.

[8] http://cryptonector.com/2006/12/filesystem-i18n/

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Mon, 30 Sep 2013 15:01:34 +0200

> name to a server which uses normalized names for files, who does

Funny enough, this just turned into a product issue for me (hey, paid 
work!).

Interesting links:

https://bugzilla.mozilla.org/show_bug.cgi?id=695995 and 
https://www.w3.org/Bugs/Public/show_bug.cgi?id=14526

Best regards, Julian