Subject: Ftp-WG: MLST draft - NUL CR LF again

From: Pat LaVarre <[email-masked]>
Date: Wed, 24 Apr 2002 16:09:03 -0500

[ BC [email-masked] ]

> From: Gregory A Lundberg [mailto:[email-masked]] 
> Sent: Wed 4/24/2002 1:29 PM 
> Attached: mlst.txt (17,657 bytes)
> ... crumudeon ...

Thank you, I enjoyed all I read ... but could you (or
anyone) explain in more detail how you reached the
conclusions you did re the proper treatment of the
bytes x 00 .. 1F appearing in pathnames, in particular
the bytes x 00 0D 0A NUL CR LF?

I thought not-shortest form UTF-8 was broadly accepted
as an evil to be discouraged?

Particularly evil are implementations that pick and
choose when to use not-shortest form?  My own personal
history of pain includes how differently the de facto
standards for Java .class files define the treatment
of shortest-form UTF-8.  The de jure standard there
flatly forbids not-shortest form except for 0, which
is required to appear in the minimal not-shortest form
x C0 80.

> QuotingFrom:
> ... mlst.txt (17,657 bytes)

2.2 Pathnames

In the third paragraph, discussing encoding of
pathnames, it may be informative to explicitly state
that the CR character MUST NOT appear in pathnames but
that, when using UTF-8, a CR MAY be sub-optimally
encoded using a two (or more) byte UTF-8
representation instead of the normal one-byte
encoding.  Sub-optimal encoding SHOULD NOT be used
except to represent the CR character as a data
character rather than a line-termination character.

The fourth paragraph should read:

Implementations should also be aware that the control
connection uses a subset of the Telnet NVT
conventions.  The Telnet IAC character, if part of a
pathname sent over the control connection, MUST be
correctly escaped as defined by the Telnet protocol
prior to transmission and after conversion from to the
transport representation from the host's internal
representation.

> Quoting from:
> http://www.unicode.org/versions/corrigendum1.html

Corrigendum #1: UTF-8 Shortest Form

The conformance clause C12 in The Unicode Standard,
Version 3.0 forbids the generation of "non-shortest
form" UTF-8, and forbids the interpretation of illegal
sequences, but not the interpretation of "non-shortest
form".  Where software does interpret the non-shortest
forms, security issues can arise.  For example:

Process A performs security checks, but does not check
for non-shortest forms.  Process B accepts the byte
sequence from process A, and transforms it into UTF-16
while interpreting non-shortest forms.  The UTF-16
text may then contain characters that should have been
filtered out by process A. 

To address this issue, the Unicode Technical Committee
has modified the definition of UTF-8 to forbid
conformant implementations from interpreting
non-shortest forms for BMP characters, and clarified
some of the conformance clauses.

> ParaphrasingFrom past Ftp reflector traffic ...

I think I understand that we're hoping for
shortest-form UTF-8 for Unicode x0020..xFFFF to just
plain work in pathnames.

I'm not at all clear on what we mean to say about the
bytes x00..1F that include the x 0D 0A CR LF that
appear in pathnames created by best of breed Gui's
such as the classic MacOS.

Yours in breathtaking, hopefully refreshing,
ignorance,
Pat LaVarre

-----Original Message----- 
From: Gregory A Lundberg [mailto:[email-masked]] 
Sent: Wed 4/24/2002 1:29 PM 
To: FTPEXT Working Group 
Cc: 
Subject: Ftp-WG: Re: comments on the MLST draft

Preface: To be honest, the MLST draft as it stands is
as good (read: bad) as
any other FTP RFC.

I've been spending a lot of time recently thinking
about the entire FTP, and
what I've attached is a mixure of my notes to myself
on implementing the FTP
applied to the MLST draft, and some specific issues
which I happened across
when reading/thinking about other issues.

That said, turn on "he's being a crumudeon" mode and
peruse the attached ...

__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Wed, 24 Apr 2002 16:28:55 -0500

> as an evil to be discouraged?

It is.  But there is no other way to unambiguously separate Telnet
end-of-line (CR LF and CR NUL) from data characters with the same encoding.

It just bugs me that we can support any file, directory or path name EXCEPT
those which include the Telnet EOL.

> Where software does interpret the non-shortest
> forms, security issues can arise.

An FTP conforming implementation SHOULD NOT transmit non-shortest form
UTF-8, but it MUST properly interpret it.  That was already spelled out in
both 959 and 1123 long before UTF-8 came along.

> some of the conformance clauses.

Since they did not specify alternate encodings for legacy character sets,
there will be times their mandate MUST be ignored.

This is a SHOULD NOT requirement and we have a case where their requirement
can not be applied; unless they provided some OTHER form of representation
for the CR LF and NUL characters elsewhere in the character sets.

> such as the classic MacOS.

It is specifically those instances which are why I'm loath to have to turn
around and clearly state that a file, directory or pathname MUST NOT include
UTF-8 encodings 0x00 through 0x20.