Subject: Ftp-WG: Re: MLST draft - NUL CR LF again

From: Pat LaVarre <[email-masked]>
Date: Wed, 24 Apr 2002 17:04:42 -0500

[BC [email-masked]]

> From: Gregory A Lundberg [[email-masked]]
> Sent: Wed 4/24/2002 3:28 PM 

> It just bugs me
> that we can support any file, directory
> or path name EXCEPT those
> which include the Telnet EOL.

Bugs me too.

> > > http://www.ietf.org/rfc/rfc854.txt
...
> > I thought not-shortest form UTF-8
> > ... an evil to be discouraged?
...
> It is.  But there is no other way
> to unambiguously separate Telnet end-of-line
> (CR LF and CR NUL) from data characters
> with the same encoding.

Sorry, too fast for me.

I thought Telnet by design Can exchange lines of
completely arbitrary content and length?  Yes CR LF
ends a line.  To send a line with CR inside, to escape
the CR, send CR NUL instead.  Always on receipt of CR
NUL, substitute CR.

This is unambiguous on paper ... no?

For example, we could send the six byte command "STOR
\r" as the nine bytes "STOR \r\0\r\n" to create a file
named by the one byte string containing only x0D CR.

Somehow in real life Telnet is less robust?

You're saying in some sense CR NUL is also a Telnet
Eol?  And I've seen actual implementations accept NUL
and LF as Eol?

Somehow it is not properly rfc761 conservative to let
real live human people type the three traditional
platform-specific end-of-line encodings (LF, CR, and
CR LF) in their path names?

Maybe we're trying to say Eol in an Ftp path name is
x0A LF only, and systems will vary over whether they
will report more than one file as having precisely
this same path name?

Thanks again in advance,    Pat LaVarre

-----Original Message----- 
From: Gregory A Lundberg [mailto:[email-masked]] 
Sent: Wed 4/24/2002 3:28 PM 
To: [email-masked]; [email-masked] 
Cc: 
Subject: Ftp-WG: Re: MLST draft - NUL CR LF again

> I thought not-shortest form UTF-8 was broadly
> accepted as an evil to be discouraged? 

It is.  But there is no other way to unambiguously
separate Telnet end-of-line (CR LF and CR NUL) from
data characters with the same encoding. 

It just bugs me that we can support any file,
directory or path name EXCEPT those which include the
Telnet EOL. 

> Where software does interpret the non-shortest 
> forms, security issues can arise. 

An FTP conforming implementation SHOULD NOT transmit
non-shortest form UTF-8, but it MUST properly
interpret it.  That was already spelled out in both
959 and 1123 long before UTF-8 came along. 

> To address this issue, the Unicode Technical
> Committee has modified the definition of UTF-8 to
> forbid conformant implementations from interpreting 
> non-shortest forms for BMP characters, and clarified

> some of the conformance clauses. 

Since they did not specify alternate encodings for
legacy character sets, there will be times their
mandate MUST be ignored. 

This is a SHOULD NOT requirement and we have a case
where their requirement can not be applied; unless
they provided some OTHER form of representation for
the CR LF and NUL characters elsewhere in the
character sets. 

> I'm not at all clear on what we mean to say about
> the bytes x00..1F that include the x 0D 0A CR LF
> that appear in pathnames created by best of breed
> Gui's such as the classic MacOS. 

It is specifically those instances which are why I'm
loath to have to turn around and clearly state that a
file, directory or pathname MUST NOT include UTF-8
encodings 0x00 through 0x20. 

...

__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Wed, 24 Apr 2002 22:34:33 -0500

> named by the one byte string containing only x0D CR.

RFC 1123 *REQUIRES* the TELNET implementation to accept CR NUL as equivalent
to CR LF.  The the proper intepretation of your example is:

  STOR <SP> <CRLF>
  <CRLF>

And a conforming server SHOULD issue the replies

  501 Syntax error in parameters or arguments
  500 Syntax error, unrecognizable command

in response to the sequence.

---

From: Robert Elz <[email-masked]>
Date: Thu, 25 Apr 2002 00:33:29 -0500

Date:        Wed, 24 Apr 2002 22:34:33 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | RFC 1123 *REQUIRES* the TELNET implementation to accept CR NUL as equivalent
  | to CR LF.

You are misinterpreting 1123, that would make no sense at all.

What 1123 is trying to say (perhaps not as well in this instance as it
does in most others), is that the telnet server must treat a received
CR over the telnet connection the same as it treats a CR from a local
keyboard.

That is, if CR ends a command, then CR over a telnet connection ends a
command - as does end of line.   So sending CR NULL or CR LF should be
treated the same way - but that applies only if CR is normally treated
as an "end of line" at the telnet server.

Note, the wording ...

	On server hosts that use ASCII, in particular,
        receipt of the Telnet sequence CR LF must cause the same effect
        as a local user pressing the CR key on a local terminal.

That is a reference to ...

	For terminal input, this corresponds to a command-
        terminal; on an ASCII terminal, this is the CR key, but it may
        also be labelled "Return" or "Enter".

That is, the "ASCII terminal" or host, being assumed, is one where CR
means "end of line".    Note this isn't true of unix hosts, on unix, CR
is just another character (the fact that the keyboard driver takes the
button labelled "CR" (or similar) and generates from it the character that
is the end of line indicator is irrelevant - that can easily be turned off
if you want to investigate what really happens).

For FTP none of this is relevant.   CR by itself is never an end of line
terminator.   Ever.   (Buggy implementations ignored).   If you want to
put a CR in a file name, you so it exactly as Pat LaVarre suggested.
If you happen to be talking to a broken implementation, then you might not
be able to put a CR in a file name - the answer is to fix the implementation,
not to change the way FTP is defined.   And it is irrelevant how common such
broken implementations might be (creating some new method to encode CR isn't
going to work on any existing implementation anyway - it is just as easy, if
not easier, to simply fix the bug as it is to support something new).

I will look at the rest of the comments later, but I rather suspect (or
perhaps even hope) that they're really too late now for this draft.

kre

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Thu, 25 Apr 2002 01:19:52 -0500

At issue are the following rules governing the existing FTP
specification:

   The server-FTP MUST indicate end-of-line using the sequence CR LF.
   [per RFC 1123]

   The user-FTP SHOULD indicate end-of-line using the sequence CR LF,
   but it MAY use the alternate sequence CR NUL or the character LF; and
   it SHOULD provide the user the ability to chose the form to use for a
   given session.  [per RFC 1123]

   The CR character MUST NOT be appear except as the first character in
   one of the two-character sequences CR LF and CR NUL. [per RFC 854].

   The FTP SHOULD NOT place constraints upon file, directory or path
   names unless failing to do so would violate a constraint of
   underlying host system.  [implied, but unstated, in RFC 959, RFC
   2640 and the MLST Draft]

   The characters CR and LF MUST NOT appear in file, directory or path
   names.  [per RFC 1123]

And the following observations:

   The constraint against CR and LF in file, directory or path names
   violates the fourth rule given above, and is well-known to cause
   inter-operation problems with the FTP on some host systems.

   The TVFS constrains the use of "/" in file and directory names, in
   violation of the fourth rule given above, and could cause inter-
   operation problems.

   The number of hosts with contraints upon the NUL character is
   sufficiently diverse that this character may warrant special
   attention in the FTP specifications.

The problems to be addressed are

   How to remove the constraint against CR and LF in file, directory and
   path names; moving the problem from the FTP and its implementation to
   the host system, where it more properly belongs.

   How to eliminate the constraint against "/" in file and directory
   names; moving the problem from the FTP and its implementation to the
   host system, where it more properly belongs.

   Whether, and how, to advertize or negotiate a host system's
   constraint against NUL, CR, LF, "/", or any other character, when
   used in a file, directory or path name.

Let's take the easiest problem first.

The FTP already provides a means to advertize a host's contraints
against certain characters in file, directory and path names: the 553
response code.  We need only clearly state the rules:

   In response to any command taking a <pathname> argument (such as
   STOR, RETR and STAT), the server-FTP MUST issue a 553 reply if the
   <pathname> argument contains characters violating a host contraint.

   The server-FTP SHOULD provide a configurable means to constrain the
   use of certain characters or patterns; but MUST NOT allow
   configurations which would violate a constraint imposed by the
   underlying host system.  For example, on a host which uses the NUL
   character to terminate file name parameters, the server-FTP MUST NOT
   allow its configuration to create a condition where the it would
   attempt to violate the constraint against using NUL in a file name.

   The free-form text of the 553 reply SHOULD clearly indicate the
   source of the contraint (host, implementation or configuration) and,
   if possible, the specific character(s) or pattern(s) violating the
   constraint.

   Examples, on a host which constrains the use of NUL (represented as
   <NUL>) and has been configured to constrain the use of hyphen (-) as
   the first character:

      C> RETR Embedded<NUL>.Data
      S> 553 Host constraints prevent NUL in a pathname
      C> STOR -exec rm -fR .
      S> 553 Configuration constraints prevent that pathname

It MAY be beneficial for the FTP to provide a FEAT response which
indicates the current set of constrained characters or patterns, but
this does not appear to be required, since it would not enhance inter-
operation.

Now, to the real problems: we must devise some means to escape the
character codes CR and LF when the user-FTP intends them to be treated
as "normal" characters (i.e., as part of a file, directory or path
name), as opposed to "control" characters indicating end-of-line.

The character NUL is not at issue here.  It can be transmitted without
special consideration; parsing it to the point of issuing a 553
response code (if required) is a solvable implementation issue.

The solution chosen for CR and LF should be applicable to "/" as well,
when using TVFS and "/" is intended as a "normal" character in a file
or directory name as opposed to a path name component separator.

We've already identified two cases where special consideration should be
given to specific characters.  In systems analysis, there are generally
only three numbers of import: 0, 1 and infinity.  By that I mean
something (in this case, a protocol-level constraint) does not occur,
it occurs exactly once or not at all, or it can occur many, many times.
Applying that principal here, the solution chosen for CR, LF and "/"
should be applicable to any character a future FTP specification might
need to constrain, so that a mechanism already exists to bypass the
protocol's constraint.

Possible solutions are:

1) Ignore the issue.  This implies problems with inter-operation will
   occur when an implementation attempts to send CR or LF as "normal"
   characters.

   In this case, the FTP would appear to operate normally in all other
   respects, yet at times the protocol would fail.  This way lies CHAOS.
   Without a specification, implementations will choose whatever means
   suits them, inter-operation problems grow, and user discontent rises.

2) Specifically state that, while all other characters which may be
   represented by UTF-8 (or the 7-bit ASCII subset of UTF-8, for older
   implementations) MAY appear in a file, directory or path name, the
   characters CR and LF MUST NOT be appear, and the "/" character must
   not appear in a TVFS path name.

   In this case, we intentionally create problems with inter-operation
   where the host supports CR and/or LF in file, directory or path
   names, or allow "/" in file or directory names.  The FTP would
   appear to operate normally in all other respects, yet at times the
   user-FTP would refuse to perform certain operations because to do so
   would require violating the constraint against CR and LF.

   This is the choice taken by RFC 1123, with respect to the 7-bit ASCII
   character set (and prior to the advent of UTF-8); and presently taken
   in the MLST Draft with respect to "/".

   While we may not envision any future specifications placing protocol-
   level constraints upon characters in file and directory names, the
   existance of the MLST Draft proves that such cases may arise.  Taken
   to a ridiculous limit, it is POSSIBLE to specify enough such cases
   that NO character is left un-constrained.

   This practice MUST cease.

3) Choose some means to "escape" the CR or LF.

   To inter-operate properly, the chosen means MUST:

   a) have the value 128 or greater, to avoid conflict with any value 7-
      bit ASCII representation, and

   b) not create a conflict with any valid UTF-8 encoding.

   One such value which comes to mind is the TELNET IAC command lead-in
   code.  We COULD specify that IAC be used to escape protocol-
   constrained characters.  In fact, the TELNET protocol has already
   specified exactly that.  Unfortunately, the TELNET protocol HAS
   already specified exactly that.  Since the FTP MAY use an external
   TELNET implementation, should we chose the IAC code, if we have an
   FTP-level protocol constraint concerning the IAC code, we would
   escape it as IAC IAC; passing that through the external TELNET would
   mean the sequence actually transmitted is IAC IAC IAC IAC.  I'm
   sorry, but THAT is just disgusting.  Besides, it makes life much
   harder for implementors; when they see IAC will it be a TELNET IAC
   escape code or an FTP IAC escape code.  Considering the chances of
   misundertanding and misinterpetation, let's just not go there.

   That leaves us with the "holes" in the UTF-8 encoding scheme.  So
   which hole do we pick?

   My suggestion is to state the escaped character code SHOULD be
   encoded using the next-larger non-shortest (what I call sub-optimal)
   UTF-8 encoding.  The implementation SHOULD accept any non-shortest
   encoding to represent the escaped character, but SHOULD only transmit
   the shortest of the non-shortest encodings.  (Muhaha!  Now, do you
   see why I call it sub-optimal? .. the shortest of the sub-optimal
   encodings .. it just reads better.)

   The advantage of this suggestion is that the UTF-8 schemes presented
   in RFC 2640 handle the case with only minor modification; and, as
   has already been pointed out, sub-optimal encodings MUST be
   supported on the receiving side; but SHOULD NOT be transmitted.  All
   we're doing is "borrowing" some of the holes created by the UTF-8
   governing body, using them for our purposes, and .. oh yeah .. we
   JUST HAPPEN to be putting codes in those places which look an awful
   lot like their un-encoded, "normal" (yet constrained)
   representations.

   They made the road .. they don't have any right to complain if we
   actually find a use for their potholes!

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Thu, 25 Apr 2002 01:26:56 -0500

Back at ya ..

The FTP specification states that the control connection operates using
normal Telnet clients.

A normal Telnet client sends CR LF, or CR NUL, or LF, at the end-of-line.

We don't know which, so we must be prepared for all.

Now, if you want to DROP the requirement that FTP work with a normal Telnet
client, that's fine.  But, if you do, expect inter-operation problems with
existing implementations.

---

From: Robert Elz <[email-masked]>
Date: Thu, 25 Apr 2002 01:54:23 -0500

Date:        Thu, 25 Apr 2002 01:26:56 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | The FTP specification states that the control connection operates using
  | normal Telnet clients.

yes.

  | A normal Telnet client sends CR LF, or CR NUL, or LF, at the end-of-line.

If it is working correctly, it sends CRLF - nothing else is a telnet end of
line.

A telnet client on the other hand needs to be able to (somehow) generate
all 3 (1123 does say that) in case the user wants to send them - but neither
CR NUL nor LF are end of line in telnet.

  | We don't know which, so we must be prepared for all.

not in FTP we don't.   The telnet client sends CRLF or it isn't sending
end of line.

A telnet server needs to treat both CRLF and whatever sequence would be
its normal direct connect end of line marker (which might be CR (transmitted
as CR NUL) or LF, or conceivably ETX or something) as "end of command" and
pass that to the command interpreter (with CRLF replaced by whatever is the
appropriate local end of command indicator).   But FTP servers aren't
telnet servers, and they don't pass commands to command interpreters, and
so don't have to deal with that.

  | Now, if you want to DROP the requirement that FTP work with a normal Telnet
  | client, that's fine.  But, if you do, expect inter-operation problems with
  | existing implementations.

No, I don't want to drop that at all.   FTP servers should work fine with
any correct telnet client.   A telnet client that sends anything other than
CRLF when it receives the local user's "end of line" indicator is horribly
broken though, we don't need to support those.

kre

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Thu, 25 Apr 2002 07:10:51 -0500

OK.  Obviously we're reading different versions of RFC 1123.
I got mine from rfc-editor.org and here's what is says on page 21:

   3.3.1  Telnet End-of-Line Convention

      The Telnet protocol defines the sequence CR LF to mean "end-
      of-line".  For terminal input, this corresponds to a command-
      completion or "end-of-line" key being pressed on a user
      terminal; on an ASCII terminal, this is the CR key, but it may
      also be labelled "Return" or "Enter".

      When a Server Telnet receives the Telnet end-of-line sequence
      CR LF as input from a remote terminal, the effect MUST be the
      same as if the user had pressed the "end-of-line" key on a
      local terminal.  On server hosts that use ASCII, in particular,
      receipt of the Telnet sequence CR LF must cause the same effect
      as a local user pressing the CR key on a local terminal.  Thus,
      CR LF and CR NUL MUST have the same effect on an ASCII server
      host when received as input over a Telnet connection.

      A User Telnet MUST be able to send any of the forms: CR LF, CR
      NUL, and LF.  A User Telnet on an ASCII host SHOULD have a
      user-controllable mode to send either CR LF or CR NUL when the
      user presses the "end-of-line" key, and CR LF SHOULD be the
      default.

      The Telnet end-of-line sequence CR LF MUST be used to send
      Telnet data that is not terminal-to-computer (e.g., for Server
      Telnet sending output, or the Telnet protocol incorporated
      another application protocol).

Nowhere in there does it make ANY requirement that end-of-line may
only be represented by CR LF.

The closest I see is the first sentance of the first paragraph,
which tells us that CR LF is one possible representation for end-of-
line.  To arrive at the requirement you infer, the sentance would
have been worded as `The Telnet protocol defines "end-of-line" as
being the sequence CR LF.'

The second paragraph goes on to inform us that, when received via a
Telnet connection, the sequences CR LF and CR NUL are interchangable.

The third tells us that when sent over a Telnet connection from a
user-FTP (the only user-Telnet in the FTP), either CR LF or CR NUL
can be sent for the end-of-line, the choice is up to the user, and
the default is CR LF.

The fourth tells us that when sent over a Telnet connection from a
server-FTP, only CR LF may be sent.

In RFC 2119 terms:

   FTP implementations MUST interpret CR LF as marking the end-
   of-line.  [First paragraph, explicitly.]

   FTP implementations MAY interpret sequences other than CR LF
   as marking the end-of-line.  [First paragraph, by conversion.]

   FTP implementations MUST interpret CR LF and CR NUL
   interchangably. [Second paragraph, explicitly.]

   FTP implementations MUST therefore interpret CR NUL as marking
   the end-of-line. [Second paragraph, by deduction.]

   The user-FTP, when implemented as a user-Telnet, MUST offer a
   configuration option allowing transmission of CR NUL to mark
   the end-of-line.  The default for this option MUST be to
   transmit CR LF to mark the end-of-line. [Third paragraph,
   explicitly.]

   The user-FTP, when not implemented as a user-Telnet, MAY offer
   a configuration option allowing transmission CR NUL to mark
   the end-of-line.  The default for this option MUST be to 
   transmit CR LF to mark the end-of-line.  [Third paragraph,
   by conversion.]

   The user-FTP, when not implemented as a user-Telnet and which
   does not offer a configuration option allowing transmission of
   CR NUL to mark the end-of-line, MUST transmit CR LF to mark
   the end-of-line.  [Third paragraph, also by conversion.]

   The server-FTP MUST transmit the sequence CR LF to mark the end-
   of-line.  [Fourth paragraph, explicitly.]

While the server-FTP MAY interpret sequences other than CR LF and
CR NUL to mark the end-of-line, the user-FTP cannot send any others.

So, in answer to the points raised so far about my comments on the
MLST draft:

1) You are incorrect.  CR NUL quite definitely MUST be interpreted
   as the Telnet end-of-line.  Therefore we have no alternate form
   for the "bare" CR to be used in a file, directory or path name.

2) You are correct, LF is not to be interpreted as the Telnet end-
   of line and does not require special escaping when intended as
   a character in a file, directory or path name.

---

From: Alun Jones <[email-masked]>
Date: Thu, 25 Apr 2002 07:44:38 -0500

>We don't know which, so we must be prepared for all.

RFC 959 also specifically states:
   The argument field consists of a variable length character string
   ending with the character sequence <CRLF> (Carriage Return, Line
   Feed) for NVT-ASCII representation; for other negotiated languages
   a different end of line character might be used.  It should be
   noted that the server is to take no action until the end of line
   code is received.

In the absence of any other negotiated language (and, per RFC 1123, the 
client MUST NOT negotiate a different language), this seems to be specific 
in that <CRLF> is required, not <CR><NUL> or <LF>.

>existing implementations.

I'm trying hard to find that requirement in the RFCs.  RFC 959 specifies in 
several places that the control connection follows the Telnet protocol, but 
it (and RFC 1123) further defines several places where behaviour is 
different.  Where there is such a contradiction, it would seem that the 
obvious conclusion is that RFC 959, and the FTP sections of RFC 1123, win 
out over Telnet.  Perhaps this deserves being stated somewhere, but I don't 
think this is ambiguous.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Alun Jones <[email-masked]>
Date: Thu, 25 Apr 2002 08:06:49 -0500

...
...
...
>    of-line.  [Fourth paragraph, explicitly.]

That's not how I read the fourth paragraph of RFC 1123, section 3.3.1.  I 
read it as saying that when "the Telnet protocol is incorporated in another 
protocol", as we have here, "the Telnet end-of-line sequence CR LF MUST be 
used".

Alun.
~~~~

---

From: Robert Elz <[email-masked]>
Date: Thu, 25 Apr 2002 08:12:51 -0500

Date:        Thu, 25 Apr 2002 07:10:51 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | OK.  Obviously we're reading different versions of RFC 1123.

No, same thing, just interpreting it differently...

  | The closest I see is the first sentance of the first paragraph,
  | which tells us that CR LF is one possible representation for end-of-
  | line.

no, it says "the sequence CR LF to mean "end-of-line"."

That is, if you mean end of line, CR LF is how it is represented.
Not that this is really very important, for us, as it is about telnet,
not FTP.

But note it also says: "The Telnet protocol defines" - it is clearly
just restating what RFC854 said - and if you look there you'll see
quite reasonably defined what CR, LF and CRLF achieve.  What wasn't
defined well, was what was supposed to happen when a telnet client
sends just CR (CR NUL of course) or LF to the server, whereas that was
defined from server to client (moving the carriage and all that).

That is what 1123 is fixing - making it clear that if the client happens
to send what the server system would treat as the end of command sequence,
then the that's how it should be treated.  Otherwise you get absurd
situations where I type

	DIR <cr><nl>

on a unix system, where I have mapping from CR to end of line disabled
(ie: the CR is just a character) - the telnet client correctly sends

	D I R SP CR NUL CR LF

to the server.  Assume that's a DOS type server, where EOL is <CR>.
Now the telnet server there would have (under some readings of 854)
been required to transfer the command "DIR <CR>" to the DOS "shell"
where the <CR> isn't supposed to represent end of line - which is simply
impossible (or impossible to do in any rational way).

The text in 1123 is all there to make it clear that the telnet server in
that case is allowed to (in fact, must) treat that CR as an end of line
(and the following CR LF as another).

See also, in 854:

      Therefore, the sequence "CR LF" must be treated as a single "new
      line" character and used whenever their combined action is
      intended; the sequence "CR NUL" must be used where a carriage
      return alone is actually desired;

That's the definition of the telnet "newline" character, which is
also called "end of line" in other places (in 854 only when discussing the
GA (go ahead) command), but FTP assumes it...).

Once again however, we're not concerned with Telnet here, but with FTP,
and in rfc959, what it says is ...

	    In accordance with the NVT standard, the <CRLF> sequence
            should be used where necessary to denote the end of a line
            of text.

(see page 11).   Not CR NUL, not LF, but CRLF.   For FTP, that is it.
Period.

See also ...

   The File Transfer Protocol follows the specifications of the Telnet
   protocol for all communications over the control connection.  Since
   the language used for Telnet communication may be a negotiated
   option, all references in the next two sections will be to the
   "Telnet language" and the corresponding "Telnet end-of-line code".
   Currently, one may take these to mean NVT-ASCII and <CRLF>.

Once again, CRLF, nothing else.

  | The second paragraph goes on to inform us that, when received via a
  | Telnet connection, the sequences CR LF and CR NUL are interchangable.

No, that's not what it says at all.   It says that "On server hosts that
use ASCII" (which seems to be some peculiar way to subset servers),
"receipt of the Telnet sequence CR LF must cause the same effect
as a local user pressing the CR key on a local terminal."

Now if that meant what it seems to say, then receiving CR LF on a unix
host with cr -> lf mapping turned off (which it would be if you were
using one of the terminals that actually had a "newline" key which
generated "lf" - which is the kind of terminal that gave unix its
LF==end of line characteristic) would simply cause a CR to be inserted
into the data stream, and wouldn't be an end of line at all.

But since everyone agrees that CR LF is end of line, that would be
nonsense.

So, instead read this so it makes sense - and assume that by "hosts thst
use ASCII" what they really mean is as defined in the previous paragraph...

That is, this there's an assumption that on an "ascii host" CR is the
normal terminal end of line character.

And there, we have exactly the situation described above - whether we
send the telnet end of line, or the host's native end of line, they both
have to have the same effect (for telnet) because anything else is absurd.

Note, this isn't a modification of NVT, where CR LF is end of line, but
a specific direction to the server to (in that circumstance) treat CR NUL
identically to end of line (they're different, but they both achieve the
same thing).

  | The third tells us that when sent over a Telnet connection from a
  | user-FTP (the only user-Telnet in the FTP), either CR LF or CR NUL
  | can be sent for the end-of-line, the choice is up to the user, and
  | the default is CR LF.

no, that paragraph doesn't say that either.   What it says is that users
need to be able to send all 3 forms, and the telnet client must make a
mechanism available (to the end user) to send any of the three when the
user desires.   That's to allow for interoperability with broken servers,
or more likely, with systems that need to be able to receive any arbitrary
one of the three.   But CRLF is what gets sent unless the user specifically
changes things.

A user using telnet to talk to a FTP server would have no reason to alter
the default, as FTP servers only treat CRLF as end of line (and always
treat CRLF as end of line).   We don't need to do anything to cope with
users who decide to deliberately send something non-conformant.

  | The fourth tells us that when sent over a Telnet connection from a
  | server-FTP, only CR LF may be sent.

You didn't read that one either ...   I'll quote it again...

Note the final clause in the parenthesised sentence.

	or the Telnet protocol incorporated another application protocol

That is exactly what we have here (and in SMTP, and others) - the Telnet
protocol incorporated in another application protocol.   And see what
1123 is telling you "The Telnet end-of-line sequence CR LF MUST be used".

Can it be any clearer?

The text that Alun Jones just sent is also directly on point.

So ...

  | In RFC 2119 terms:
  | 
  |    FTP implementations MUST interpret CR LF as marking the end-
  |    of-line.  [First paragraph, explicitly.]

yes.

  |    FTP implementations MAY interpret sequences other than CR LF
  |    as marking the end-of-line.  [First paragraph, by conversion.]

no.

  |    FTP implementations MUST interpret CR LF and CR NUL
  |    interchangably. [Second paragraph, explicitly.]

no.

  |    FTP implementations MUST therefore interpret CR NUL as marking
  |    the end-of-line. [Second paragraph, by deduction.]

no.

  |    The user-FTP, when implemented as a user-Telnet, MUST offer a
  |    configuration option allowing transmission of CR NUL to mark
  |    the end-of-line.  The default for this option MUST be to
  |    transmit CR LF to mark the end-of-line. [Third paragraph,
  |    explicitly.]

no.   If you happen to be using a telnet implementation to talk to
a ftp server, then the implementation must offer that option, but to
correctly interoperate with ftp, the user must not use it.   Just as
the user must not cause telnet option negotiation to be attempted,
though a telnet client would normally allow that as well.

  |    The user-FTP, when not implemented as a user-Telnet, MAY offer
  |    a configuration option allowing transmission CR NUL to mark
  |    the end-of-line.  The default for this option MUST be to 
  |    transmit CR LF to mark the end-of-line.  [Third paragraph,
  |    by conversion.]

no.

  |    The user-FTP, when not implemented as a user-Telnet and which
  |    does not offer a configuration option allowing transmission of
  |    CR NUL to mark the end-of-line, MUST transmit CR LF to mark
  |    the end-of-line.  [Third paragraph, also by conversion.]

just always send CRLF, no options, no choices.

  |    The server-FTP MUST transmit the sequence CR LF to mark the end-
  |    of-line.  [Fourth paragraph, explicitly.]

yes, though the part you're getting that from isn't the part of the
paragraph that is most on point.

  | While the server-FTP MAY interpret sequences other than CR LF and
  | CR NUL to mark the end-of-line, the user-FTP cannot send any others.

user FTp has to send CRLF, nothing else.

  | So, in answer to the points raised so far about my comments on the
  | MLST draft:
  | 
  | 1) You are incorrect.  CR NUL quite definitely MUST be interpreted
  |    as the Telnet end-of-line.  Therefore we have no alternate form
  |    for the "bare" CR to be used in a file, directory or path name.

No, that is simply wrong.    The fact that it takes such a convoluted
argument to arrive at this position should be a pretty good clue that
it isn't correct.   The early RFCs were not big on convoluted arguments.
Everything is meant to be interpreted in the obvious way.

kre

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Thu, 25 Apr 2002 18:05:14 -0500

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Thu, 25 Apr 2002 20:12:28 -0500

> out over Telnet.  Perhaps this deserves being stated somewhere, but I
don't

RFC 959, page 1, paragraph 1 "INTRODUCTION" ...

   FTP, though usable directly by a user at a terminal, is
   designed mainly for use by programs.

The "terminal" being referred to here is the Telnet NVT.  This is
a "may" statement: "A Telnet user MAY directly use the FTP."

RFC 959, page 7, "TERMINOLOGY" ...

   user

   A person or a process on behalf of a person wishing to obtain
   file transfer service.  The human user may interact directly
   with a server-FTP process, but use of a user-FTP process is
   preferred since the protocol design is weighted towards
   automata.

This already is a "may" statement, followed by a "should": "A
human user MAY interact directly with a server-FTP process, but
SHOULD use a user-FTP process since the protocol design is
weighted towards automata."

RFC 959, page 8, paragraph 2, "THE FTP MODEL" ...

   The user may establish a direct control connection to the
   server-FTP, from a TAC terminal for example, and generate
   standard FTP commands independently, bypassing the user-
   FTP process.

The term "TAC terminal" here is historical and unfortunate.  It
owes to copying this section from older versions.  With the
advent of Telnet, the physical "TAC terminal" was replaced with
the NVT.

We cannot get must more clear than this.  The clear intention
is that the user-FTP process is OPTIONAL.

Interaction with the server-FTP is through the control
connection.  The control connection follows the Telnet
protocol.  Therefore, the tool to use to bypass the
user-FTP process is a user Telnet implementation.

RFC 959, page 9 ...

   The Relationship between FTP and Telnet:

   The FTP uses the Telnet protocol on the control connection.
   This can be achieved in two ways: first, the user-PI or the
   server-PI may implement the rules of the Telnet Protocol
   directly in their own procedures; or, second, the user-PI or
   the server-PI may make use of the existing Telnet module in the
   system.

   Ease of implementaion, sharing code, and modular programming
   argue for the second approach.  Efficiency and independence
   argue for the first approach.  In practice, FTP relies on very
   little of the Telnet Protocol, so the first approach does not
   necessarily involve a large amount of code.

This informs us that NOT ONLY can we bypass the user-FTP process,
but an FTP implementation (server- OR user-) MAY its own internal
implementation of the Telnet protocol, or it MAY use an existing
Telnet module provided by the host system.

While some hosts may provide a link-time module which an FTP
implementation can use, the most widely available Telnet module
is the user Telnet implementation.  On Unix, for example, this
module is generally called the 'telnet command'.

Given the relationship described here, a perfectly good (albeit
slow) server-FTP could be implemented using nothing more than
'expect' scripts and the 'telnet' command.  While doing a
server-FTP in 'expect' is probably something few have ever
considered, implementing a user-FTP in 'expect' and 'telnet' is
uncommon but has been done.  Implementing a user-FTP with 'shell'
scripts using the 'telnet' command is common enough that I get
several questions a year pertaining to it.

There are probably several more citations I could make, but I think these
are enough.  Besides, it's time to put the kids to bed, and I have a
contract out-of-town for the next few days so I need to get to bed early and
hit the road tomorrow ...

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Fri, 26 Apr 2002 23:12:26 -0500

Bitch all you want, but we ALL support CR NUL as end-of-line.

$ telnet
telnet>unset crlf
Will send carriage returns as telnet <CR><NUL>.
telnet>open munnari.oz.au 21
Trying 128.250.22.2...
Connected to munnari.oz.au.
Escape character is '^]'.
220 munnari.OZ.AU FTP server (Version 5.95 2000/07/08 21:15:53) ready.
user ftp
331 Guest login ok, send ident as password.
pass lundberg@
230 Guest login ok, access restrictions apply.
quit
221 Goodbye.
Connection closed by foreign host.
$ telnet
telnet>unset crlf
Will send carriage returns as telnet <CR><NUL>.
telnet>open ftp.ipswitch.com 21
Trying 156.21.4.254...
Connected to ftp.ipswitch.com.
Escape character is '^]'.
220-ftp1.ipswitch.com X2 WS_FTP Server 3.0.1 (1609819023)
220-Welcome to ftp1.ipswitch.com
220-This server is located in Massachusetts, USA
220 ftp1.ipswitch.com X2 WS_FTP Server 3.0.1 (1609819023)
user ftp
221 Password required
pass lundberg@
230 user logged in
quit
221 Good-Bye
Connection closed by foreign host.
$ telnet
telnet>unset crlf
Will send cariage returns as telnet <CR><NUL>.
telnet>open ftp.wu-ftpd.org 21
Trying 205.133.13.60...
Connected to ftp.wu-ftpd.org.
Escape character is '^]'.
220 ftp.wu-ftpd.org FTP server ready.
user ftp
331 Guest login ok, send your complete e-mail address as password.
pass [email-masked]
230-Welcome to the FTP server for the WU-FTPD Development Group
230-
230-This server is the primary distribution site for the WU-FTPD daemon.
230-
230-The pub directory contains the distribution and supporting files.
230-
230-If you are uploading contributions, please place them in the incoming
230-directory and email [email-masked] announcing your upload.
230-
230 Guest login ok, access restrictions apply.
quit
221 Goodbye.
Connection closed by foreign host.
$

---

From: Robert Elz <[email-masked]>
Date: Sat, 27 Apr 2002 02:23:22 -0500

Date:        Fri, 26 Apr 2002 23:12:26 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Bitch all you want, but we ALL support CR NUL as end-of-line.

I know, my server is full of bugs in that area, no question about it.

kre

---

From: Alun Jones <[email-masked]>
Date: Sat, 27 Apr 2002 08:24:16 -0500

>Bitch all you want, but we ALL support CR NUL as end-of-line.

Does this mean that you feel you have lost the argument over standards, and 
now you want to try and win it by pointing to implementations?

Alun.
~~~~

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Sat, 27 Apr 2002 15:18:46 -0500

> now you want to try and win it by pointing to implementations?

No, actually I think it's just more evidence that no matter how much you
don't like it, CR NUL does mean end-of-file.

As for the argument, no .. but

Here's my opinion in the argument: What the TRUE definition is does not
matter.

- The standards CAN be analyzed as written, without opinion, and if so, they
WILL show CR NUL to end-of-line.

- The real-world implementations show CR NUL to be end-of-line.

- But those who don't like the fact that CR NUL is end-of-line are so well
entrenched in their wrongheadedness that it appears fruitless to try to
change their minds.

Here's my take on it:

1) FTP is based on Telnet.

   This has not (yet) been disputed.

2) A server-FTP MUST interoperate with a user-Telnet.

   This is plain from the wording of RFC 959.  A user-FTP is OPTIONAL.

3) A user-Telnet SHOULD send CR LF for end-of-line;
   A user-Telnet SHOULD have an option to send other codes for end-of-line;
   That option MUST provide CR NUL as one of the other codes for
end-of-line.
   That option MAY allow other codes beyond CR NUL as alternatives.

   This is plain from the wording of RFC 1123.  In addition, real-world
tests show the authors of user-Telnet understand this and provide at least
the minimum requirement of CR NUL as an alternative for CR LF.

Therefore a server-FTP

 a) MUST accept CR LF as meaning end-of-line

    This is plain from the wording of RFC 959.

 b) SHOULD accept CR NUL as meaning end-of-line

 c) SHOULD provide an option allowing additional codes to be taken as
meaning end-of-line.

As a result of (b), there is no means to accept a bare CR (which MUST be
followed by a NUL, as is plain from the RFCs) as a "data" character.

As a result of (c), there MAY be no means to accept other characters from
the 7-bit ASCII character set (EM was given as an example) as a "data"
character.

Therefore, the FTP needs a method to "escape" any character from the 7-bit
ASCII character set so that the user-Telnet's "local convention" of using
some other character to mean end-of-line may be circumvented.

The arguments I've seen so far are:

1) FTP is not required to interoperate with user-Telnet.  Manifestly false.
The clear intention of the original FTP authors, from as far back as you
care to go, is that FTP interoperate with Telnet.

2) A user has no reason to change the default, therefore we don't need to
worry about those who did.  Violates the principles of robustness (see the
top of RFC 1123, just to name one place where this principle is stated.  I
believe you'll find it in RFC 959 as well).

3) Early RFCs should were intended to be taken as read and should not
require long discussions analyzing the intended meaning.  Obviously false.
Read RFC 1123 and it becomes clear that the oversights and inclear wording
in early RFCs caused a great deal of problems.

4) In matters where FTP and Telnet cover the same ground, the FTP should win
out.  False.  By extension, this faultly thinking would have the FTP rule
TCP and IP in matters where the requirements of those protocols differ from
the requirements of FTP.  Telnet defines Telnet.  If the FTP is at odds with
Telnet, either the FTP should make it clear it no longer users Telnet, or
the Telnet specification wins.

5) The Telnet implementation in the FTP is somehow "different" than the
Telnet implementation in Telnet.  Well, this _is_ true.  But not for
end-of-line.

The ONLY difference between the Telnet in Telnet and the Telnet in FTP is
the Telnet Go-Ahead function: A server-FTP MUST NOT send Telnet Go-Ahead
while a server-Telnet MUST send it until negotiated otherwise.  (Note this
has implications on HOW a server-FTP should "refuse to negotiate" concerning
the Telnet Suppress Go-Ahead Option; which I'm still trying to work out the
effects of.)

---

From: Alun Jones <[email-masked]>
Date: Sat, 27 Apr 2002 22:03:16 -0500

>don't like it, CR NUL does mean end-of-file.

As the developer of an FTP server that accepts CR NUL as an end of line 
sequence, I can tell you that I consciously made the decision to allow CR, 
LF, or CR LF to act as end-of-line sequences, not because of any reading of 
the RFCs, but because I saw requests coming from clients that contained 
those sequences.  Originally, I had set the program up to only accept CR LF 
as an end-of-line sequence, but had received a few reports of 
incompatibilities (often with user-written clients).

Hang the RFC compliance, I thought at the time, nobody's going to actually 
_want_ a CR or LF in a file name, and if they do, then I can wave the 
compliance flag at the broken clients, because by that time I'll be more 
established.

>Here's my opinion in the argument: What the TRUE definition is does not

So in other words, it doesn't matter whether anyone else agrees with you or 
not, your opinion is absolute.

>WILL show CR NUL to end-of-line.

Analysing the standards as written, without opinion, demonstrates that 
there is contradiction and confusion.

>- The real-world implementations show CR NUL to be end-of-line.

The real-world implementations _treat_ CR NUL as if it were end-of-line.

>entrenched in their wrongheadedness that it appears fruitless to try to

Those of us who persist in our "wrongheadedness" are convinced that the 
RFCs are confused and contradictory, and that the FTP RFC outranks the 
Telnet RFC in describing what behaviour FTP implementations should adhere 
to.  You appear to be of the opinion that Telnet outranks FTP in describing 
FTP.

>    This is plain from the wording of RFC 959.  A user-FTP is OPTIONAL.

And yet... you point so frequently to "real implementations" - how many 
real implementations of FTP clients rely on Telnet client code as their 
base?  How many real implementations of FTP servers rely on Telnet server 
code as their base?

>the minimum requirement of CR NUL as an alternative for CR LF.

I believe that it is a mistake to apply the Telnet updates in RFC 1123 as 
if they were an update to the FTP specifications in RFC 959.  If the 
authors of the FTP section of RFC1123 had meant to include such additions, 
they surely would have - after all, they even went so far as to note that 
TCP is a stream-based protocol, and that you shouldn't make assumptions 
otherwise; surely if they had anything to say about including Telnet 
updates since RFC 854, they would have.

>     This is plain from the wording of RFC 959.

Nobody is disputing that.

>  b) SHOULD accept CR NUL as meaning end-of-line

Most implementations, if not all, do that at present, but not because of 
any reading of the RFC, so much as out of a desire to cope with 
poorly-ported FTP clients on the Macintosh (where CR _is_ the line terminator).

Today, when file names with CR in them are more common, and Macintosh 
client code better written, the desire to support such file names appears 
stronger than the desire to support poor FTP clients, and thus server 
implementors seem willing, perhaps eager, to drop the use of CR NUL as EOL.

>  c) SHOULD provide an option allowing additional codes to be taken as

Good grief, I'm not sure where I could imagine reading that in to 
being.  Such an option would have to be negotiated with a connecting 
client, surely, using the forbidden Telnet option negotiation.

>followed by a NUL, as is plain from the RFCs) as a "data" character.

So why not make it explicit that CR NUL is not acceptable as an EOL?

>the 7-bit ASCII character set (EM was given as an example) as a "data"

Hell, why not decide on a client-server combination that uses 'q' as an 
end-of-line sequence?  By that argument, surely any character should be 
forbidden from file names?

>some other character to mean end-of-line may be circumvented.

You're heading towards arguing that FTP can't use any 7-bit ASCII character 
to represent anything other than EOL.

>1) FTP is not required to interoperate with user-Telnet.  Manifestly false.

And yet, holding FTP back to its "every user is TCP savvy, and is 
comfortable typing bare protocol" roots would seem to be stifling its 
future development.

>care to go, is that FTP interoperate with Telnet.

The clear intention of the original FTP authors is that it support EBCDIC 
to ASCII translation, and that it handle mail and Instant Messaging as 
well.  Times change, people change, hair styles change.

>believe you'll find it in RFC 959 as well).

However, new features often kill off old and unused functionality.  Yes, 
accepting CR NUL as indicative of a CR in a file name would lead to people 
using poorly written FTP clients, or poorly configured Telnet clients 
(really, how many people are you talking about) being unable to continue to 
use such an FTP server.  But it would benefit by granting the ability to 
put a CR into a file name transferred by FTP.

Are you so in awe of the original authors of RFC 959 that you believe that 
it is written in stone tablets by the hand of ghod?  Did you join this 
working group mailing list to suggest that nothing ever be changed?

>in early RFCs caused a great deal of problems.

And they still do.  But I didn't see anyone arguing that they don't.

>the Telnet specification wins.

FTP uses almost nothing of Telnet.  Modern FTP implementations, at least as 
far as I have seen, have no use of Telnet code base.  The original reasons 
for using Telnet as an under-layer for FTP are almost entirely gone.

>the Telnet Suppress Go-Ahead Option; which I'm still trying to work out the

The Telnet in FTP should also not try any option negotiation.  The Telnet 
in FTP is also not designed to be character-based or provide any 
line-editing facilities.  In fact, there's so little of Telnet in FTP that 
it's strange that it was required to be included.

Alun.
~~~~