Subject: Ftp-WG: LIST vs. NLST

From: Trevin Beattie <[email-masked]>
Date: Tue, 30 Jan 2001 21:09:17 -0500

I have recently run into a problem with the interpretation of RFC-959.
The authors of wu-ftpd claim that NLST should restrict its list to
regular files and exclude directories and symbolic links.  However,
the only place I could find in the RFC's which distinguish directories
from regular files is in RFC-1127, which indirectly claims that
including directories in NLST output is a "bug".  And this statement
is listed under "Proposed FTP extensions"; it is not part of an actual
standard.

In RFC-959, the ONLY distinction made between NLST and LIST is that
NLST must return a machine-readable list of names, while LIST may
return a human-readable list.  This follows BSD's implementation of
the ftp client, where the "dir" command is documented to print a
listing of the contents of a directory including "any system-dependent
information that the server chooses to include".

Apparently the wu-ftp folks have interpreted "names of files" under
NLST to mean names of regular files.  But if you read the paragraph
under LIST, it uses similar wording, "list of files".  Neither
paragraph mentions "regular files" or "symbolic links".  And remember
than on unix-like systems, directories and symbolic links ARE files,
just files of a special type.

Many RFC's prior to 959 have urged the network community to clarify
the distinction between LIST and NLST.  Following is the complete list
of additions to the description of these two commands:

    Under LIST, change "If the pathname specifies a directory" to "If
    the pathname specifies a directory or other group of files", and
    add the following sentence: "Since the information on a file may
    vary widely from system to system, this information may be hard to
    use automatically in a program, but may be quite useful to a human
    user."

    Under NLST, add the following sentence: "This command is intended
    to return information that can be used by a program to further
    process the files automatically.  For example, in the
    implementation of a "multiple get" function."

As a side note to the last sentence, there are other ways an ftp
client can use both regular file and directory names in NLST, for
example tab-completion of filenames and expanding "mget" functionality
to recursively get files in subdirectories.  (I have seen the latter
already implemented in one ftp client.)

-----------------------
Trevin Beattie          "Do not meddle in the affairs of wizards,
[email-masked]     for you are crunchy and good with ketchup."
      {:->                                     --unknown

Paul Hethmon
[email-masked]
http://www.hethmon.com

---

From: Gregory A Lundberg <[email-masked]>
Date: Wed, 31 Jan 2001 01:10:35 -0500

We made the change to allow commands in widely deployed clients to work.
Specifically, there's been a long-standing complaint that mget will fail
because NLST returns non-gettable objects (directories).  959 specifically
states NLST is for use by automata, mentioning mget specifically.

It turns out that some vendors consider humans automata, and use NLST as
the default for human-readable output.  Most notable of these is Sun, which
uses NLST in an attempt to add pathname completion to an FTP session much
like you normally see with command-line shell access.

If it's a problem for your site, there is a patch available (which will be
a config option in the next version) which will allow you to break some
clients in favor of others.

-- 

Gregory A Lundberg              Senior Partner, VRnet Company
1441 Elmdale Drive              [email-masked]
Kettering, OH 45409-1615 USA    1-800-809-2195

---

From: Robert Elz <[email-masked]>
Date: Wed, 31 Jan 2001 01:51:54 -0500

Date:        Tue, 30 Jan 2001 21:09:17 -0500
    From:        Trevin Beattie <[email-masked]>
    Message-ID:  <[email-masked]>

  | I have recently run into a problem with the interpretation of RFC-959.

This was also asked on the NetBSD list, here's a copy of the reply
I sent there...

    Date:        Tue, 30 Jan 2001 12:48:28 -0800
    From:        Trevin Beattie <[email-masked]>
    Message-ID:  <[email-masked]>

  | I have recently run into a problem with the interpretation of RFC-959.
  | The authors of wu-ftpd claim that NLST should restrict its list to
  | regular files and exclude directories and symbolic links.

That's wrong.   Or more correctly, an FTP server can return whatever it
likes for any of the listing commands - FTP provides a virtual filestore,
it doesn't have to represent anything that actually exists if it doesn't
want to.  So the "should" is incorrect, there's no such requirement.
But restricting it that way isn't necessarily wrong either.

  | In the official standard, RFC-959, the ONLY distinction made between
  | NLST and LIST is that NLST must return a machine-readable list of
  | names, while LIST may return a human-readable list.

Yes.

  | NetBSD Problem Report #8937 has suggested changing the implementation
  | of ftp to make "dir" and "ls" both use LIST.  I believe this is a
  | mistake, as it removes the distinction between the commands.

NLST output is intended for machine processing, returning it (raw) to
the user is a dumb idea.

The "ls" and "dir" came about because unix users naturally like to use
"ls", whereas most ftp clients pre unix had used "dir" as their command
for the similar function (those other OS's...).   So, both were included.
They're not supposed to be different (that some version of the unix client
somewhere along the way changed "ls" to use "NLST" was a truly brain
dead idea).

Things got worse as servers tended to just run /bin/ls to implement these
commands, passing whatever arg the user passed, so running "ls -lrt" from
the client would cause the server to run "/bin/ls -lrt" the output of
which was just dumped to the user's screen, which made things look very
clean to your average unix user.   Try that on a non-unix server though
and it wouldn't do at all what you expect (except those faked to pretend
to be unix, which actually look to see if you're passing unix style flags
as the arg ... the lengths to which some implementors will go...)

The only defined arg to the LIST/NLST commands is the file to list, there
are, or should be, no "flags".

In the future, clients will be able to use MLST and then generate
different output formats suitable for the client environment, based
upon the command they were given, and the information returned from
the server   (There are already servers, including NetBSD's, that
support it).

kre

---

From: Robert Elz <[email-masked]>
Date: Wed, 31 Jan 2001 01:55:49 -0500

Date:        Wed, 31 Jan 2001 01:10:35 -0500
    From:        Gregory A Lundberg <[email-masked]>
    Message-ID:  <[email-masked]>

  | It turns out that some vendors consider humans automata, and use NLST as
  | the default for human-readable output.

That's dumb.

  | Most notable of these is Sun, which
  | uses NLST in an attempt to add pathname completion to an FTP session much
  | like you normally see with command-line shell access.

But that's not, pathname completion is "automaton" work, not human
work.   Still, it is a little hard to see how it can work sanely, as
from NLST (even when directories are included, which I would generally
think they should be) you have no idea what is a directory, and what isn't.

All this can be solved with MLST, which gives machine parseable listings
with additional information (so the client can tell if the file is fetchable,
and so use, or not use it with mget, and can tell if it is a directory
or not, and so do intelligent path name completion).

Does wu-ftpd support MLST yet?   If not, could it soon?   Getting that
in most of the major servers on the net would certainly provide lots more
motivation for clients to start using the command.

kre

---

From: Gregory A Lundberg <[email-masked]>
Date: Wed, 31 Jan 2001 02:26:55 -0500

> Does wu-ftpd support MLST yet?   If not, could it soon?   Getting that in
> most of the major servers on the net would certainly provide lots more

My personal opinion: we're lucky it even supports 959 .. and that's only
because it came from the old BSD sources.  I'm sure someone could come up
with some way to get MLST into the utter mess which is there.  I, though,
have put it on my list of things to do when I get back to doing a complete
rewrite.

--