Subject: [ftpext] New Version of FTP HASH, RANG, LOCK, and HOST

From: Anthony Bryan <[email-masked]>
Date: Sun, 20 Jan 2013 12:00:56 -0500

we updated the FTP HASH, RANG, and LOCK drafts, while the authors of
HOST have updated their (hopefully final, get any comments or reviews
in ASAP) draft.

we are mostly interested in finishing up HASH & RANG, but also feel
free to comment on LOCK. fancy diffs are available. HASH, RANG, & LOCK
changes are all editorial or just updating the draft.

HASH FTP command to be used by clients to request cryptographic hashes of files.
http://tools.ietf.org/html/draft-bryan-ftpext-hash

RANG command to be used by clients to designate a start and end point
to permit restarts and repairs of interrupted data transfers in STREAM
mode.
http://tools.ietf.org/html/draft-bryan-ftp-range

LOCK command to be used by clients to request the server to use the
control connection for data transfers, using a single port instead of
two.
http://tools.ietf.org/html/draft-bryan-ftp-lock

also, HOST has been updated:

FTP command that provides a mechanism for FTP clients and servers to
identify individual virtual hosts on an FTP server.

http://datatracker.ietf.org/doc/draft-hethmon-mcmurray-ftpext-ftp-hosts/
http://tools.ietf.org/html/draft-hethmon-mcmurray-ftpext-ftp-hosts (diffs)

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Bruce Blackshaw <[email-masked]>
Date: Thu, 24 Jan 2013 11:45:39 +1000

Re LOCK, I think it would be useful to have an alternative to "boundary" 
called "size", that could provide the size of the raw data for binary 
transfers.  That might provide quite a performance advantage compared to 
having to search for the boundary.

regards

Bruce Blackshaw
EnterpriseDT
http://www.enterprisedt.com

On 21/01/2013 3:00 AM, Anthony Bryan wrote:
> _______________________________________________
> ftpext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ftpext
>

---

From: Ángel González <[email-masked]>
Date: Sun, 27 Jan 2013 18:27:45 +0100

Bruce Blackshaw wrote:
> Re LOCK, I think it would be useful to have an alternative to
> "boundary" called "size", that could provide the size of the raw data
> for binary transfers.  That might provide quite a performance
> advantage compared to having to search for the boundary.
>
> regards

Good point. I think it would be much more useful than the boundary method.
The value of such size parameter should be the amount of data sent on
the wire.

---

From: "Alun Jones" <[email-masked]>
Date: Sun, 27 Jan 2013 10:32:39 -0800

> Of Ángel González
> The value of such size parameter should be the amount of data sent on the
> wire.

This has an obvious problem - what if the size of the file changes during
the transfer?

There are many uses of FTP (webcam being the one that comes immediately to
mind) where a "file" is retrieved, and is handled as if it is a
potentially-infinite stream. We have a webcam at work that encodes video as
an animated GIF that never ends. Doesn't display in Internet Explorer,
because IE is paranoid, but it's a use-case that should be considered. The
boundary method works for this use, where the size method doesn't.

Alun.
~~~~

---

From: Ángel González <[email-masked]>
Date: Sun, 27 Jan 2013 22:20:14 +0100

> boundary method works for this use, where the size method doesn't.
Probably due to the reference to MIME, I was thinking in multiple chunks
with
a size attached, not in one big chunk with the size. In the later case,
it would
be possible (assuming it doesn't shrink below what you have already
committed
to send).

---

From: "Alun Jones" <[email-masked]>
Date: Sun, 27 Jan 2013 15:46:56 -0800

After reading the LOCK draft, I have a couple of comments:

0. RFC 959 already defines a mechanism - Block Mode over the default port -
which doesn't require PORT or PASV to operate, and doesn't close connections
between files. This means that under such a scheme, a firewall's life is
made easier - you open 21 inbound, and allow outbound connections from port
20 on the server. Document this in your draft, and people may start using
that, just because it already works on many servers.

1. You need an example showing how an upload works, just to make that clear.
"When uploading files, the client MUST use end-of-marker solution." You
don't explain this anywhere, or use the term "end-of-marker" anywhere else
in the draft.

2. You need to answer at least the security questions of how a client is
protected from a malicious server, and how a server is protected from a
malicious client, as well as how the connection (client and server) is
protected against a malicious file. Consider what happens if a file contains
a number of predicted boundary values, and the server or client is not good
at picking random numbers for those boundaries. Note, for instance, how SMTP
does this in the DATA command - the DATA ends with a line containing only
".". Any line beginning with a "." in the data has to be prefixed by another
".". The receiver then simply strips out the first dot on each line. Don't
give in to the idea that "random numbers can't be guessed", or that
accidental occurrences or malicious action won't ever successfully reproduce
the boundary condition. Make it IMPOSSIBLE.

3. At least three commands are documented to work while a transfer is in
progress. STAT, which reports the status of the transfer, NOOP, which does
nothing (but many clients use to keep the control connection open when a
firewall might close it over a long data transfer), and ABOR, which stops an
existing transfer. My suggestion would be that STAT will return a response
after the data transfer has finished (after the response from the transfer
completion); NOOP the same; ABOR will interrupt the transfer, the transfer's
end response will be signaled as interrupted, and the ABOR command will send
a 426 and 226.

4. Answering your question of "how to unlock" - the obvious way is to use a
PORT or PASV (or similar) command to re-establish such behavior, or REIN if
you want to go back to default port operation (and blow away your login,
other settings, etc).

5. You should indicate in your examples whether the boundary and separator
lines are sent by client or server, just to avoid confusion.

Alun.
~~~~

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Anthony Bryan
> Sent: Sunday, January 20, 2013 9:01 AM
> To: [email-masked]
> Subject: [ftpext] New Version of FTP HASH, RANG, LOCK, and HOST
> 
> HOST have updated their (hopefully final, get any comments or reviews in
> ASAP) draft.
> 
> we are mostly interested in finishing up HASH & RANG, but also feel free
to
> comment on LOCK. fancy diffs are available. HASH, RANG, & LOCK changes
> are all editorial or just updating the draft.
> 
> 
> RANG command to be used by clients to designate a start and end point to
> permit restarts and repairs of interrupted data transfers in STREAM mode.
control
> connection for data transfers, using a single port instead of two.

---

From: Bruce Blackshaw <[email-masked]>
Date: Mon, 28 Jan 2013 09:52:08 +1000

It's true that you don't always know the size at the start.

Nonetheless, it could be be specified that iff the size of the raw data 
is known, then "size" could be used. In the many cases that this is 
possible, performance should be faster (although I can't quantify by how 
much).

regards

Bruce Blackshaw

---

From: "Alun Jones" <[email-masked]>
Date: Sun, 27 Jan 2013 16:01:25 -0800

After reading the RANG spec, I note that there isn't anything to state what
errors are given by RETR if the range specified is incorrect, or by STOR /
APPE if the range specified starts more than one byte after the end of the
existing file on the server (and therefore, presumably, incorrect?).

I also think that a simple "RANG" on its own should serve to clear out the
range on a request. "RANG 1 0" seems an arbitrary choice of numbers (why not
"RANG 0 -1" or "RANG 0 Inf"?).

Alun.
~~~~

to
control

---

From: "Alun Jones" <[email-masked]>
Date: Sun, 27 Jan 2013 16:17:36 -0800

Reviewing the HASH draft, I think it would be good to say that server
implementors SHOULD use the name for the hash algorithm as defined in "Hash
Function Textual Names" registry, if one exists - this is implied, but not
stated.

to
control

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Tue, 29 Jan 2013 01:45:08 +0900

Hi,

> "RANG 0 -1" or "RANG 0 Inf"?).

We discussed internally how to reset ranges set by RANG command.
First we said that "REST 0" resets ranges for RANG too. But, later we decided
not to use REST because in the future we can completely replace REST with RANG.
And we chose invalid range pattern, like "RANG 1 0"
to reset range since it is the obvious parser pattern and checking
range is very easy.
Yes, the numbers are arbitrary. We just chose them. But we avoided negative
numbers or non-digits because the range is inherently non-negative
numbers, we don't want
to make parser accept other than digits just for this.

Best regards,

Tatsuhiro Tsujikawa

---

From: Ángel González <[email-masked]>
Date: Mon, 28 Jan 2013 22:04:20 +0100

> existing file on the server (and therefore, presumably, incorrect?).
I think it should be documented that the server must fill the gap with
nul bytes,
with a note that it may use it as a hint to do that using a sparse file.

---

From: Anthony Bryan <[email-masked]>
Date: Tue, 29 Jan 2013 12:14:10 -0500

thanks for the review!

the registry is here:
http://www.iana.org/assignments/hash-function-text-names/hash-function-text-names.xml

how does this work for you?

The server-FTP process MUST use the "Hash Function Name" as defined in
the IANA registry named "Hash Function Textual Names" for possible
supported hash algorithms.

in context:

When replying to the FEAT command [RFC2389], a server-FTP process that
supports the HASH command MUST include a feature line indicating that
the HASH command is supported, along with a list of all supported hash
algorithms, including the currently selected hash algorithm, in a
semicolon separated list. The server-FTP process MUST use the "Hash
Function Name" as defined in the IANA registry named "Hash Function
Textual Names" for possible supported hash algorithms.