Subject: Ftp-WG: Fwd: to mlst-15 from mlst-14

From: "Paul Hethmon" <[email-masked]>
Date: Mon, 8 Apr 2002 21:33:41 -0500

==================BEGIN FORWARDED MESSAGE==================
>Return-Path: [email-masked]
>Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com
>    (Hethmon Smtpd Perseus) id 20020408180705-12780-8 ; Mon, 08 Apr 2002 18:07:05 -0500
>Message-Id: <[email-masked]>
>Date: Mon, 8 Apr 2002 18:07:05 -0500
>Sender: Steward-owner <[email-masked]>
>From: Steward <[email-masked]>
>Reply-To: [email-masked]
>Subject: Approval Request for ftp-wg
>To: [email-masked]
>

----- Steward Marker Line -- Do Not Modify -------
Original-From:  "Pat LaVarre" <[email-masked]>
Original-To:  <[email-masked]>
Original-Subject:  to mlst-15 from mlst-14
----- Steward Marker Line -- Do Not Modify -------
Remember in January we here reviewed how we got to mlst-14 from mlst-13? =
 Here
now ...

> - ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-14.txt.Z
> - [50360 bytes]
> -Expiration Date: July 2002
> -                                                            January =
2002
...
> + ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-15.txt.Z
> + [52106 bytes]
> +Expiration Date: October 2002
> +                                                              April =
2002

The substantive diff's I see to April mlst-15 from January mlst-14 are =
as
follows ...

>   2.1. Basic Tokens
...
> +        SCHAR          =3D RCHAR / "=3D" ;
...
>     ALPHA, in particular, is case sensitive.
>     That implies that a "token" is a case sensitive
>     value.
> -   That implication is correct.
> +   That implication is correct, except where explicitly stated
> +   to the contrary in this document, or in some other specification
> +   which defines the values this document specifies be used in a
> +   particular context.

>   2.2. Pathnames
...
>     Implementations are advised against converting a UTF-8 pathname
> -   to a local encoding, and
> -   then attempting to invert the encoding later.
> +   to a local charset that
> +   isn't capable of representing the full Unicode character =
repertoire,
> +   and then attempting to invert the charset translation later.

>   7.1. Format of MLSx Requests
...
>     If no argument is given then MLSD must return a listing of the
>     contents of the current working directory, and MLST must return a
>     listing giving information about the current working directory
>     itself.
>     For these purposes, the contents of a directory are whatever
> -   file names (not pathnames)
> +   file or directory names (not pathnames)
>     the server-PI will allow to be referenced when the current
>     working directory is the directory named, and which the
>     server-PI desires to reveal to the user-PI.

>   7.2. Format of MLSx Response
...
> -        value            =3D *RCHAR
> +        value            =3D *SCHAR
...
>     Facts
>     should be provided in each output line only if they both provide
>     relevant information about the file named on the same line, and =
they
>     are in the set requested by the user-PI.
> +   See section 7.9 (page 51).

>  7.5.1. The type Fact
> +   The value of the type fact (the "type-val") is a case independent
> +   string.

>  Acknowledgments
> +        Bill Fenner (and the rest of the IESG)

The less substantive diff's includes fuller and more perfectly =
consistent
examples:

>  7.7.9. Example from another server
...
>     Finally notice that the NVFS supported
>     by this server, in contract to the earlier ones, implements its
>     pathnames in a case independent manner.
> +   The server seems to return
> +   files using the case in which they were requested, when the name =
was
> +   sent by the client, and otherwise uses an algorithm known only to
> +   itself to select the case of the names it returns.

> +7.7.11. A server with a difference
> +
> +C> ... S> ... D> ...
> +
> +   The server shown here returns its directory listings in seemingly
> +   random order, and even seems to modify the order of the directory =
as
> +   its contents change -- perhaps the underlying directory structure =
is
> +   based upon hashing of some kind.  Note that the "pdir" and "cdir"
> +   entries are interspersed with other entries in the directory.  =
Note
> +   also that this server does not show a "pdir" entry when listing =
the
> +   contents of the root directory of the virtual filestore, it does
> +   however include multiple "cdir" and "pdir" entries when it fees
> +   inclined.  The server also uses obnoxiously "cute" messages.

>   3.4. MDTM Examples
...
> -   If we assume the existence of three files, A B and C,
> -   and a directory D, and no other files at all,
> -   then the MDTM command may behave as indicated.
> +   If we assume the existence of three files, A B and C,
> +   a directory D, two files with names that end with the string
> +   "ile6", and no other files at all,
> +   then the MDTM command may behave as indicated.
...
> -   C was modified 21 days and several hours later.
> +   C was modified 20 days and several hours later.

>  7.5.1.5. System defined types
> -        os-type   =3D "OS." os-name "=3D" os-type
> -        os-type   =3D token
> +        os-type   =3D "OS." os-name "=3D" os-kind
> +        os-kind   =3D token

>     OS specific types are registered by the IANA
> -   using the procedures specified in section 10.  The "os-type" =
provides
> -   the system dependent information as to the type of the file =
listed.
> -   The os-name and os-type strings in an os-type are case =
independent.
> +   using the procedures specified in section 10.  The "os-kind" =
provides
> +   the system dependent information as to the type of the file =
listed.
> +   The os-name and os-kind strings in an os-type are case =
independent.

> 7.8.1. Examples
> -   All of the facts supported by
> -   this server are enabled by default.
> +   All but one of the facts
> +   supported by this server are enabled by default.

The "to me not clearly substantive" diff's include:

> -   directory has a file name relative to the directory that contains =
it,
> +   directory has a name relative to the directory that contains it,

> -   storage structure,
> +   storage structure.
=20
> -   The VCHAR (from [5]), TCHAR, and RCHAR types
> +   The VCHAR (from [5]), RCHAR, SCHAR, and TCHAR types

> -   is returned over the control connection,)
> +   is returned over the control connection)

Also I see we're again swapping "modelled" with "modeled" and "labelled" =
with
"labeled".

As in January, before comparing the texts, ... gleefully I stripped page
numbers, headers, trailers and I compressed runs of blank lines to =
single
blank lines.  And this time I also ended by ignoring all whitespace in =
the
comparison.

x4402 Pat LaVarre   [email-masked]
http://members.aol.com/plscsi/

===================END FORWARDED MESSAGE===================

Paul Hethmon
[email-masked]
http://www.hethmon.com

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Wed, 24 Apr 2002 14:29:47 -0500

Preface: To be honest, the MLST draft as it stands is as good (read: bad) as
any other FTP RFC.

I've been spending a lot of time recently thinking about the entire FTP, and
what I've attached is a mixure of my notes to myself on implementing the FTP
applied to the MLST draft, and some specific issues which I happened across
when reading/thinking about other issues.

That said, turn on "he's being a crumudeon" mode and peruse the attached ...

---

From: Stephen Tihor <[email-masked]>
Date: Wed, 24 Apr 2002 17:12:10 -0500

bravo

---

From: Robert Elz <[email-masked]>
Date: Sun, 28 Apr 2002 08:56:57 -0500

Date:        Wed, 24 Apr 2002 14:29:47 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  |    Second paragraph SHOULD read
  |
  |       These commands allow a client to obtain directory listings in a
  |       machine-friendly, predictable format; and MAY allow a client to
  |       restart an interrupted transfer in transfer modes not previously
  |       supported in any documented manner."

I have no idea what that trailing quote is about (there's no opening one)
but no it shouldn't.   If the commands are implemented as described, they
do allow a client to restart.   That isn't going to always work, because
the file might have changed, etc - but the introduction doesn't have get
into that kind of detail.   (And MAY in upper case would be completely
inappropriate anyway).

  | 2.2 Pathnames
  |
  |    In the third paragraph, discussing encoding of pathnames, it may be
  |    informative to explicitly state that the CR character MUST NOT
  |    appear in pathnames

That would be wrong.  If the server supports CR in filenames, and the
client desires to use it, then it can do so.   There's no requirement on
the server to actually allow the CR character to appear in a filename of
course.

  |    but that, when using UTF-8, a CR MAY be sub-
  |    optimally encoded using a two (or more) byte UTF-8 representation
  |    instead of the normal one-byte encoding.  Sub-optimal encoding
  |    SHOULD NOT be used except to represent the CR character as a data
  |    character rather than a line-termination character.

We've just been through this argument, the CR character is not a 
line-termination character, unless it appears as CRLF, so this is
irrelevant, if needed, CR can be sent as CR NUL.   If the server is
going to support CR in file names, then it had better support that
method of transferring it.   If it has no CRs in file names, then
it really doesn't matter very much (the error messages will be nicer
if it does the right thing, but that's about it).

  |    The fourth paragraph should read:
  |
  |       Implementations should also be aware that the control connection
  |       uses a subset of the Telnet NVT conventions.

Actually, it uses all of them.

  |       The Telnet IAC
  |       character, if part of a pathname sent over the control connection,
  |       MUST be correctly escaped as defined by the Telnet protocol prior
  |       to transmission and after conversion from to the transport
  |       representation from the host's internal representation.

Those look to be extra words that don't actually say anything new.

  | 2.2.1 Pathname syntax.
  |
  |    Last sentence should read:
  |
  |       The server-FTP implementation SHOULD NOT restrict the syntax of
  |       valid file and directory names unless failure to do so would
  |       violate the contraints placed upon file and directory names by the
  |       host system.

The last sentence of that section is currently ...

	Similarly, a server-PI may parse the pathname, and assign meaning to
	the components detected.

I don't see that your replacement is in any way related.   The penultimate
sentence would be closer, but I can't think of any reason that we need to
attempt to tell FTP server implementors what to do there - that looks more like
a quality of implementation issue.   If I want to supply a server for
a unix system which only allows file names made up of a's and b's, why should
any spec attempt to prohibit that?   If all the files that some user needs
to deal with are named a aa ab ba baa aba (etc) then that server would work
fine for that installation.

  | 2.3 Times
  |
  |    It may be informative to explicitly state that the reason for the UTC
  |    requirement is to make it possible for a user to compare time stamps
  |    across servers.

Perhaps, but I don't think I really want to issue a revision of this doc
for just that.  And in any case, it also allows the user/client to actually
know (reasonably closely) what the time represents.   The real reason to
pick UTC is because we don't want to have to try and send time zone information
(ie: we either pick a zone, and everyone knows what it is, or we have to
include the zone with the time ... because part of the use of this is
existing code from 16-17 years ago now, we're not going to alter the choice
that was made back then).

  |    The user SHOULD be aware that servers MAY not have
  |    synchronized clocks and some may use GMT or one of the other time
  |    standards.

I think that's clear enough in there already.

  |    Thus, appropriate use of the time stamps produced by
  |    different hosts would be relative order, not absolute marks,

As is that - and if the user can't figure out that attempting to
make sense out of unsynchronised clocks, then I'm not sure that anything
that is said here will help.

  |    with
  |    differences of less than a few hours being considered equality.

If we were to start a discussion on that, I don't think that would be the
approach that would make sense.   Basically times from different servers are
just different - they should be used for two purposes, first for comparing
against other times from the same server (where equality is equality, and
nothing else is), or for using to modify timestamps of local files to match
those of remote files (for mirroring, etc).

  |    It
  |    might also be a good idea to explicitly caution against reliance upon
  |    time stamps; not only are clocks often not synchonized, they are
  |    often set by hand (if set at all) and may be wildly incorrect.

I think that's at least implied well enough.

  | 3.1 (MDTM) Syntax
  |
  |    This command is OPTIONAL

No, I don't think we need to go adding that - it is already obvious
in the text, eg, in ...

	When replying to the FEAT command [6], an FTP server process that
	supports the MDTM command MUST

that makes it pretty clear that there can also be FTP server processes
that don't support the FEAT command.

The same response applies to the other times you have suggested adding
this line, I won't repeat my replies (or any others where the same reply
would apply to multiple comments).

  |    Server-FTP implementors are cautioned to carefully consider the
  |    implications of supporting MDTM command. The existance of certain
  |    files MAY be used by attackers to "fingerprint" host systems.

Be reasonable!   The LIST command is much better at that kind of thing
that MDTM will ever be.   As is MLSD of course.

  |    In
  |    particular, where the existance of certain non-retrievable files is
  |    to be hidden from the user, the MDTM command SHOULD be implemented
  |    in such a manner that its reply does not inadvertently disclose the
  |    existance of a file. Thus, the choice of response for  non-
  |    retrievable files SHOULD be governed by the server-FTP's response
  |    elicited if the RETR were attempted using the same pathname argument.

There's nothing here that applies to MDTM any more than to any other file.
If you want to write a treatise on how to implement FTP servers that hide
things from various subsets of users, please do so, but elsewhere.

  | 3.3 FEAT response for MDTM
  | =0A=
  |    The user-FTP SHOULD NOT rely upon support of the FEAT command, or
  |    the response to the FEAT command, to indicate support of the MDTM
  |    command.  While the inclusion of MDTM in the FEAT response is
  |    positive indication of support for the command, its absence MUST NOT
  |    be taken as indicating lack of support.

There's no need for a MUST NOT there, if a client wants to use the
absense of MDTM in FEAT output as indicating lack of support, then
let it do so.    Since it is clear from the doc that these commands
(MDTM/REST/SIZE) have been around for ages, long before FEAT appeared,
it should be pretty obvious that servers that implement them aren't
necessarily going to implement FEAT.   I don't really think we need to
go out of the way to say so (and one presumes, that over time, the
number or servers that don't support FEAT will dwindle, at which time
it might be entirely reasonable to ignore the few that do support the
commands but not FEAT - that is, for clients to assume that no FEAT output
means no support).   Other clients will ignore FEAT and just try the
commands anyway - either strategy is acceptable.

  | 3.4 MDTM Examples
  |
  |    User-FTP implementors are cautioned that some existing server-FTP
  |    implementations MAY misinterpret the pathname "19990929043300 File6"
  |    as a command to change the modification time of "File6".

That might not be bad advice, if the doc were to be re-opened, but again,
I don't think we need to issue a new version, just for this.

  |    For maximum inter-operation, the user-FTP SHOULD use other means to
  |    determine the existance of the pathname prior to attempting the MDTM
  |    command.

That's not bad advice, but would just be document bloat.   It doesn't
even help here, as if File6 exists, and "19990929043300 File6" also
exists, a broken server will treat "MDTM 19990929043300 File6" as an
attempt to modify the time of File6 - simply ignoring the other file
(the decision gets made in the command parser, long before the filesystem
is consulted).

  | 4. File SIZE
  |
  |    This command is OPTIONAL.
  |
  |    Server-FTP implementors are cautioned to carefully consider the
  |    implications of supporting the SIZE command.  The requirement is
  |    that the value returned be precise, when coupled with the current
  |    data transfer options, can consume significant resource.

All this is already pretty clear...

  |    To arrive
  |    at a precise value for the SIZE response, the server-FTP=0A=
  |    implementation MAY need to process the entire file as if it were
  |    being transmitted, but without the delays involved with transmission.
  |    An attacker could use the SIZE command to reduce or exhaust
  |    resources, potentially reducing the host's ability to deliver file
  |    transfer or other services offered by the host.

Documenting every possible DoS attack that might ever be made would be
a nice thing to do, but is way out of scope of this effort.

  | 4.2 (SIZE) Error responses
  |
  |    When the current parameters would require conversion or modification
  |    of the file during the data transfer process, the SIZE command
  |    SHOULD return a 550 reply.

Huh?  You mean that unix servers shouldn't support SIZE in ASCII mode.

We're documenting a command that already exists, and the servers *do*
support it...

  |    Thus, a positive completion reply to the
  |    SIZE command indicates both the precise size of the file, and that
  |    the data transfer process for the file involves no conversion or
  |    modification operations on the server-FTP.

But that isn't what it means in current implementations, and there's no
way that is every going to be what it means.

  | 4.4 Size examples
  |
  |    In the example give, the correct response to the second (TYPE A)
  |    example should read:
  |       S> 550 Size can not be determined.

No, as the doc quite clearly says, all the examples come from actual
FTP dialogs (even when I needed to create a bizarre server to make
that be true...)   No server I have sever seen says that, certainly
none that I tested against, hence that response cannot appear in the
doc.

  | 5.1 Restarting in STREAM mode.
  |
  |    First paragraph, last sentence should read:
  |
  |       However, there MAY not really be a need to have explicit restart
  |       markers in this case, since restart markers MAY by implied by the
  |       octet offset into the data stream.

The upper case MAYs here are totally inappropriate.  The paragraph as it
stands is completely correct - in stream mode (where there's only one way
to ever transfer a particular file), a restart position *can* (always) be
determined by counting octets transferred.   Sometimes it might be expensive
to do, but it can *always* be done.  There's no "may" about it.  (Compare
this with other transfer modes, where several different octet streams can
transfer the identical file, so counting octets would be useless).

  |    Second paragraph, second sentence should read:=0A=
  |
  |       Thus, given the same transfer options, an octet offset SHOULD
  |       always represent the same position within a given file.

Again, upper case is totally inapproriate, and it isn't should, it is.
If that isn't true, then it is a different file.

  |    Implementors are cautioned to remember that, at the time the REST
  |    command is interpreted, the specific file and direction of transfer
  |    are unknown.

That's true, but I have no idea why it is relevant.   All the REST
command ever does is same the arg to be used on a later STOR/RETR
command (assuming it can be parsed correctly of course).

  |    Server-FTP implementors are cautioned to consider the implications of
  |    supporting the REST command for stream mode transfers.  When
  |    positioning to the desired octet offset, significant resources can
  |    be consumed when the current transfer options would require the
  |    server-FTP to perform conversion or modification on the data stream.

I think this is all already clear.

  |    The server-FTP MUST ensure that the REST command does not, itself,
  |    imlicity extend the size of the file, or allow access to areas of
  |    the file system allocated to, but not actually used by, the file.

The server FTP doesn't need to do anything of the kind.   It will often
do that, but if it chooses to allow REST to extend a file, or write
anywhere at all, then that's it's business.

  |    When the octet offset given with the REST command would indicate a
  |    position outside the current range of possible values (i.e., less
  |    than zero or greater than the file's size), the server-FTP MUST
  |    reply 550.

To what?  It can't be the REST command, as the file's size isn't known
then.   If a server is unable to restart a transfer, there is already
test telling it to issue an error on the following transfer command.
I really don't think we should be attempting to enumerate all the possible
errors that can occur - and I certainly don't think we should be picking
some of them for special treatment.

  |    On some hosts, specifying a file position past the end
  |    of file implicitly extends the file (possibly filling the new area
  |    with uninitialized data).  This could be used by an attacker to
  |    exhaust file system resources.

You mean as distinct from actually sending lots of data ???   there's
nothing new here.

  |    In addition, should the file
  |    subsequently be retrieved, the uninitialized areas of the file could
  |    contain potentially sensitive information which would otherwise not
  |    be available via file transfer.

I think I'll leave that bizarre scenario out...

  |    For TYPE I, the acceptable range of values for the octet offset is
  |    (0 <= REST <= SIZE); where SIZE is the size of the file.  If the
  |    restart marker is positioned at (or, if allowed, past) SIZE, a
  |    subsequent RETR transfer SHOULD transfer no data, immediately
  |    indicating end-of-file.  If the restart marker is allowed to be
  |    positioned past SIZE, a subsequent STOR transfer SHOULD fail; if the
  |    restart marker is positioned at SIZE, a subsequent STOR transfer
  |    SHOULD append to the file (as if REST had not been issued and the
  |    APPE command had been used instead).

All that is fine, but I don't think we actually need to say any of it.
This is not an implementor's guide.

  | 5.3 (REST) Syntax
  |
  |    Support for restart in stream mode is RECOMMENDED.

Beats me why this is recommended and the others are just optional...
They're all recommended, that's why there being documented.  None of
them is required.

  |  |    Explicitly state the offset zero is the first octet of the file.

Not needed, the offset is the count of bytes transferred, if zero have
been transferred, then we're at the start of the file.  No need to treat
the reader like an idiot.

  |    The existing [RFC959] requirement for REST is that the next command
  |    be a transfer command.  Many existing server-FTP implementations
  |    ignore this, and many existing user-FTP implementations depend upon
  |    it being ignored.  The following additional specification are needed:
  |
  |       The restart marker MUST be reset to zero following any data
  |       transfer (successful or otherwise).  The ABOR and REIN commands
  |       MUST discard any restart marker, resetting it to zero.

I think that is already clear enough.

  |       The SIZE command MUST ignore the restart marker and leave it
  |       unchanged.

Ignore it, yes, that's fine - leave it unchanged, no, I can think of no
reason for that at all, and if existing clients assume that, then they
are broken, and should be fixed.

  |       The REST command MAY occur at any time; it SHOULD, however, be
  |       followed by a RETR or STOR command.

no, like 959, we want REST as near as practical before REST/STOR, not
at any time.

  |       The server-FTP MAY require=0A=
  |       that the following a REST command with a non-zero argument to be
  |       RETR or STOR by issuing a 350 reply; in which case, if the user-
  |       FTP does not intend to issue a RETR or STOR, it MUST issue either
  |       an ABOR command, or a REST 0 command, before proceeding with the
  |       session.

This is inventing some entirely new protocol, what we have done here is
to document the existing (16 year old, or whatever) protocol, no invention.

  |       If the restart marker has been set to a non-zero value, the
  |       following commands SHOULD return negative completion replies:
  |       LIST, MLST, NLST and STOU.

But not MLSD ?   But no, none of that.   They're just undefined.

  |       Following a STOR (or APPE or STOU), the file SHOULD be truncated
  |       at the point the data transfer stream just completed indicated
  |       end-of-file.  For example, the effect of a STOR REST STOR
  |       sequence SHOULD be the same as if the entire file had transferred=0A=
  |       with a single STOR command.

STOR commands end the same way regardless of whether a REST occurred or not.
Nothing anywhere says any different.   Whether the truncation occurs after
the STOR has finished, or before it starts, is none of our business.

  |    User-FTP implementors are cautioned ensure the user is well aware
  |    that, although stream mode restart is supported, the results actually
  |    obtained MAY differ from those which would occur if restart were not
  |    used.  While the MDTM, SIZE and MLSx commands MAY be used to indicate
  |    when restarting a stream transfer will produce differing results,
  |    they can not be reliably used to indicate the sequences would produce
  |    identical results.  For this reason, the user-FTP implementation
  |    SHOULD NOT automatically attempt a restart following an interrupted
  |    transfer, but MAY offer it as a option to the well-informed user.

This is all quality of implementation stuff, and doesn't belong.

  |    The server-FTP implementor is cautioned to carefully consider the
  |    implications of allowing restart, especially in stream mode.

Just above you said it was recommended...

  |    Many "download accelerators" exploit some TCP features and attempt to
  |    explicitly break up transfers by use of multiple sessions, the REST
  |    command, and other features of the FTP.  Such applications MAY
  |    consume excessive host and network resources which MAY adversely
  |    effect file transfer and other services offered by the host.

Again, documenting every way that resources can be consumed, or DoS
possibilities, is just beyond the scope.

  | 5.4 FEAT response for REST
  |
  |    The user-FTP SHOULD rely upon support of the FEAT command, and
  |    the response to the FEAT command, to indicate the REST command is
  |    supported for stream transfers.

Beats me why this one is different than the others - stream mode restart
is supported in almost exactly the same set of servers that support MDTM
and SIZE now, with or without FEAT.

I'll reply to the rest another day...

kre

---

From: "Fastream Technologies" <[email-masked]>
Date: Sun, 28 Apr 2002 09:36:57 -0500

Hello,

In our application (Fastream FTP++ P2P) we have multiple connection
downloads (similar to Download Accelerator/GoZilla) with the help of REST
command. I simply divide the file into segments and then merge the
downloaded parts on the local-side.

I want to implement the same feature for remote part also. I mean multiple
connection uploads. However, on the remote size, there is nothing like a
merge command. There is the command APPE but it only appends a local file to
a remote file. Wouldn't it be nice if we add such a command to FTP?

Also, in our application, we have a command called SITE MD5 similar to ones
found in UNIX systems. But in our case, the MD5's are cached with a
background thread. This enables KaZaA/iMesh like P2P functionality with
multiple source-IP/port downloads. However lack of support from other
applications limit the use to within FTP++-to-FTP++ only. As you may have
understood, I propose this command also.

Best Regards,

G. Ates

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Mon, 29 Apr 2002 05:39:17 -0500

Granted, a lot of that "implementor be aware" stuff is because the doc
started life as my notes to myself to remember to check things.

On SIZE and MDTM being optional: I don't think they add anything.  MLSx has
the info; newer implementations should use MLSx and forget SIZE and MDTM.
Older servers which don't have them shouldn't need to add them.  Making the
commands optional is a strong hint to client authors to switch to MLSx.

On the problem with MDTM: You're right, there's really no good workaround if
the server tries to change the time as in the example.

On whether SIZE should error if the current mode would require conversion:
To be honest, having supported the darned thing for years now, I'm just
plain tired of explaining to admins that the reason their server got shut
down was not an error in the server, but a consequence of them allowing such
large files to begin with.  I tried adding a sleep call occassionally, to
let other processes have a chance, and all that did was gather complaints
that it was taking WAY too long to get the size.  Can anyone come up with a
zero-growth cost means of determining the size?  The only solution I can
come up with is to marry something like MySQL and store such 'facts' there;
and I don't EVEN want to think about the problems keeping that synchronized
would cause.

I didn't look too closely at the facts for MLSx, but I assume the size fact
there is intended to be the actual, stored, size and not the on-the-fly
size.  If not, it'll suffer the same problems as SIZE.

To be BRUTALLY honest, I wish SIZE had never been added.  The "error on
conversion" solution is the only way I've come up with which maintains the
command as a cheap way to get the file's size without having to read through
LIST output, and stop the DoS complaints.

On why I recommend REST and not SIZE and MDTM: unlike SIZE and MDTM, REST
for streams actually adds something useful to the protocol.  I don't think
it's the panacea the users think it is, and personally dis-recommend its use
for most transfers.

On why I say REST can appear anywhere: What I was trying to describe was the
way existing clients misuse the command in an attempt to see if it _might_
work today.  Yes, 959 says it must be immediately followed with a transfer
command.  But clients send it at the top of the session, when they're doing
SYST and such, and it's typically a command or two away from the data
transfer.

On truncating the file to the STOR: You're right it does not matter that the
truncation occur before or after the STOR, but it must occur; and I don't
recall ever seeing it stated, though, that STOR following a REST truncates.

On letting REST exceed the size: Sure, on a lot of systems, all you get is a
huge file.  But on others you get memory-image garbage.  And on others you
get whatever was recorded on that area of the drive.

Allowing the creation of holes is much the same as allowing a short-length
STOR which does not truncate.  With REST we have given the users lseek().
They already have read() and write().  Suddenly we have a random-access file
system implemented over FTP.

As to whether implementor advice should be part of an RFC; security starts
with design.  On of the most common complaints about the FTP is it's total
lack of regard for security.  Much of that owes to the protocol's desire to
make it easy to share files.  But a lot of it comes from the fact that the
common refrain is "Security issues are not discused in this memo."  The
intention of adding that section to the RFCs was to encourage authors to
consider and discuss security issues.  To date, it's main purpose, though,
has been for the author to admit that, if he did the considering, he didn't
do the discussing.

On the stuff you're leaving for later: in the Security section, I suggest
saying something like some of the issues have been corrected in the memo and
others may be handled by implementation and configuration choices.  If you
don't remove the DoS caused by SIZE (and the size fact, if it's the same
there) you should read that as saying "none of the issues reported have been
corrected, although some may be handled by implementation or configuration
choices."

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Mon, 29 Apr 2002 05:57:43 -0500

> downloaded parts on the local-side.

Yeah.  Those are some of the applications causing the DoS complaints.  Sure,
they're not malicious DoS, but the network operators at your vict..er
download site are certainly noticing that you're choking their routers.

> a remote file. Wouldn't it be nice if we add such a command to FTP?

There's some interesting research going on with that.  Most of them start
with a hacked up FTP daemon.  If you want I can root around my bookmarks,
but you'll probably find some of it faster if you Google for 'file transfer'
and 'satelite'.  The US Gov't has a spec which is basically 959 with
parallel transfers like you're wanting.

> understood, I propose this command also.

I put MD5 (and Unix checksums) into WU-FTPD some time ago, simply because I
thought it would be nice to be able to remotely verify signatures.  It met
with total and complete silence; noone asking how to enable it, no problems
with it, nothing.  I assume the demand is a very good approximation of zero.
I still like the idea, but I expect the users are satisfied enough with the
present reliability of downloads that they're not interested in the added
step.

In specific answer to your question, though, there's no reason the add a new
command to support MD5.  Adopt MLSx and simply add an MD5 fact to your
servers.  If you want, write a memo describing the fact.  Sorta like the way
new Telnet Options get done.

---

From: "Fastream Technologies" <[email-masked]>
Date: Mon, 29 Apr 2002 06:26:17 -0500

> download site are certainly noticing that you're choking their routers.

I look into this aspect from the end-users point of view. And we should not
forget the fact that router capacity and bandwidth is growing exponentially.
So today's DoS, probably will be a small percentage increase in tomorrow's
traffic. And standards are designed for future, am I wrong?

> > merge command. There is the command APPE but it only appends a local
file
> to

Actually a NASA official demanded such feature from me, but since there were
a lack of standards, I replied that only for download it is possible.

> new Telnet Options get done.

Sorry but I am new to MLSx. Which RFC should I be looking for? (I am
currently using ICS under BCB for FTP engine)

Thanks.

---

From: Robert Elz <[email-masked]>
Date: Mon, 29 Apr 2002 07:53:19 -0500

Date:        Mon, 29 Apr 2002 05:39:17 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | On SIZE and MDTM being optional: I don't think they add anything.  MLSx has
  | the info; newer implementations should use MLSx and forget SIZE and MDTM.

MDTM doesn't add much, though it can be easier to fetch and parse, but the
doc does already recommend using MLST to get the same value MDTM returns.
The SIZE cmd and fact return different things though, they're not 
interchangable.

  | I didn't look too closely at the facts for MLSx, but I assume the size fact
  | there is intended to be the actual, stored, size and not the on-the-fly
  | size.  If not, it'll suffer the same problems as SIZE.

The size fact is "an approximation to the size of the file" (or words like
that), and yes, it's intended that whatever the filesystem reports as the
file size can be used (if that's a number in blocks that you then multiply
by the block size to get an approximation, and then find out the last block
wasn't completely filled later, that's fine -- the unix st_size is definitely
fine).  transfer modes are irrelevant.   This is the one that clients should
use when they want the size to display a progress bar, or similar.  But it
is useless for restart.

  | To be BRUTALLY honest, I wish SIZE had never been added.  The "error on
  | conversion" solution is the only way I've come up with which maintains the
  | command as a cheap way to get the file's size without having to read through
  | LIST output, and stop the DoS complaints.

You can certainly do that, I think the spec makes it OK to refuse to
calculate the size of a file that you don't like - set a conf file option,
files bigger than N and SIZE doesn't work (perhaps except in image mode).

  | On why I say REST can appear anywhere: What I was trying to describe was the
  | way existing clients misuse the command in an attempt to see if it _might_
  | work today.  Yes, 959 says it must be immediately followed with a transfer
  | command.  But clients send it at the top of the session, when they're doing
  | SYST and such, and it's typically a command or two away from the data
  | transfer.

Yes, it often is, but I don't think we ought to encourage that.
In practice, lots of commands are pretty obviously going to have no
effect, but I wouldn't like to constrain implementations too much.
The doc is (I think) fairly clear that it is pretty undefined to do
REST other than just before a transfer.

  | On the stuff you're leaving for later:

Yes, of course.

kre

---

From: Robert Elz <[email-masked]>
Date: Mon, 29 Apr 2002 07:55:12 -0500

Date:        Mon, 29 Apr 2002 06:26:17 -0500
    From:        "Fastream Technologies" <[email-masked]>
    Message-ID:  <000f01c1ef6f$73cba9e0$2b38aed4@loser>

  | Sorry but I am new to MLSx. Which RFC should I be looking for? (I am
  | currently using ICS under BCB for FTP engine)

It isn't yet an RFC, we're patiently waiting for the IESG to tell us that
it will be published.

See draft-ietf-ftpext-mlst-15.txt in the I-D directories.  It is (aside from
the different boilerplates etc) possibly identical to what will appear in an
RFC (unless some discussion here causes a new one to emerge).

kre

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Mon, 29 Apr 2002 23:27:31 -0500

> a lack of standards, I replied that only for download it is possible.

NASA should have given you the spec.  It's THEIR birds it's being used on.
:P

> currently using ICS under BCB for FTP engine)

I see Elz has answered this one.

If you want to come up to speed, faqs.org has all the RFCs organized, and
www.wu-ftpd.org has them as well, including the drafts and some other stuff,
but I'm biased since I designed that page.

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Mon, 29 Apr 2002 23:30:57 -0500

> REST other than just before a transfer.

I guess my only real point there, then, is the 300 reply.  that indicates
the next command MUST be a transfer command.  959 already says must, so the
300 is appropriate, but actually specifying 300 breaks all those existing
implementations and .. if you can't tell .. I'm real big on trying to live
with the mess created by existing implementations.

---

From: Robert Elz <[email-masked]>
Date: Tue, 30 Apr 2002 02:24:33 -0500

Date:        Mon, 29 Apr 2002 23:30:57 -0500
    From:        "Gregory A Lundberg" <[email-masked]>
    Message-ID:  <[email-masked]>

  | 300 is appropriate, but actually specifying 300 breaks all those existing
  | implementations and ..

Yes, that was what I thought - for this, the intent was just to (finally)
document what actually exists, and has done for a long time.  The WG spent
essentially no time trying to think of ways that restart could be "fixed"
or improved, or altered at all - at this late stage, I doubt that there
is much real interest in doing that, here anyway.

kre

---

From: Alun Jones <[email-masked]>
Date: Tue, 30 Apr 2002 07:52:53 -0500

>is much real interest in doing that, here anyway.

There's so many working implementations out there, in client and in server, 
that to define a new protocol element for resuming that was fundamentally 
incompatible with the existing implementations would have likely led to the 
same situation that we have with FTP URLs - the RFCs say one thing, and the 
implementations completely ignore it.

RFCs have a long and cherished history of documenting that which has been 
implemented, and found to work.  Occasionally they document only that which 
has been found to work "well enough".  Changing the behaviour of REST at 
this point would be a waste of time - we'd end up with an RFC that muddied 
the water, rather than cleared it.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.