Subject: [ftpext] draft-ietf-ftpext2-hash - partial hashes

From: Robert Oslin <[email-masked]>
Date: Fri, 14 Jan 2011 17:32:58 -0600

draft-ietf-ftpext2-hash open issues indicates: "Current version of the draft defines full file hashes, but not partial file hashes."

Partial hashes are valuable and applicable to re-world scenarios and should be considered for inclusion in the HASH specification.

Use Case: 

	User initiates download of multi-gigabyte file, such as ISO or similar.
	Transaction is interrupted
	[Later] User re-initiates download of same (supposedly) file

At this point the client software must make a decision as to whether to resume the transfer or not. Filename is the first criteria observed, followed by size. However, these do not take into consideration a corrupted partial file (even with identical byte count), or that the remote file could have changed, especially given the difficulty in assessing time differences between client and host systems.

By requesting the hash for the portion of the remote file matching the bytes for the partial local file, the client could determine whether the local file is indeed valid and partial and subsequently resume the transfer from the appropriate byte offset.

A similar need occurs when the client has no prior knowledge of the transfer (no queue or cache mechanism) and a same name file is identified and the client must determine whether the local file is just a segment/part of the larger file located on the remote. 

Below is an example of overwrite logic performed today by our FTP client using hashes and size comparisons: 

If a user requests to download a file and a file with the same name exists locally, the client will determine if the file sizes are the same or if the destination (local) size is smaller (indicating a possible partially transferred file). If file sizes are the same then the client will compute the hash for the entire file and ask the server for to provide the hash for the corresponding remote file. The client will then skip the transfer if the hashes are identical or overwrite the file if the hashes do not match. If  the remote (source) file is larger the client will ask the server for a partial hash, up to the bytes that match the local (destination) file size. If the partial hash matches then the client will resume the transfer from the byte offset. If the hashes are different then the client will overwrite the file. (e.g. local partial file was corrupted or is not same file).

Robert Oslin
Director of Product Management
Tel: 1 (210) 293-7902
Fax: 1 (210) 690-8824
Send me large files securely
www.globalscape.com

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 17 Jan 2011 03:59:23 -0500

> If a user requests to download a file and a file with the same name exists locally, the client will determine if the file sizes are the same or if the destination (local) size is smaller (indicatin…

Robert, thanks for joining us & for posting.

a very quick introduction for Robert would mention that he works on
CuteFTP and is the originator of the XCRC command.

I think everyone's been unanimous in that we want HASH to support
partial file hashes.

I think there are a few things to iron out.

1) how to (optionally) select the byte range to be hashed.

we proposed a new byte RANGe selection command,
http://tools.ietf.org/html/draft-bryan-ftp-range , which needed to be
fleshed out of course but would look something like this:

   C> RANG 802816 1000000
   S> 350 Byte range starting at 802816, ending at 1000000.

2) how to show that partial hashes are supported, or if that's even
needed? add a "p" to the FEAT? or just use an error code if someone
tries to do a partial file hash and & it's not allowed or unsupported
on the server?

      C> FEAT
      S> 211-Extensions supported:
      S>  ...
      S>  HASH SHA-256p;SHA-512p;SHA-1p*;MD5p
      S>  ...
      S> 211 END

3) the server response which shows it's a partial file hash and not a
full file hash. it would probably be good to have the range in there,
and it could be mandatory, where if it was a full file hash it would
list the start & end of the file

   C> HASH filename.ext
   S> 213 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext
-802816 1000000

from Lothar:

S> 226 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext\
 802816 1000000 ASCII transfer complete

from Sob:

Rather than inventing new custom reply format, wouldn't it be better
to adopt MLSx style? It's simple, readable, extensible, ...

E.g.:

  S> 213 Hash.SHA-256=f0ad929cd...;Range=802816-1000000; filename.ext

(I intend to reply to the other hash messages backlog shortly)
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Mat Berchtold <[email-masked]>
Date: Mon, 17 Jan 2011 09:31:40 +0000

Hello Anthony ..

>RANG for HASH
I understand that RANG is a replacement for REST. Using the newly introduced command now to extend other commands as HASH might not be what RANG has been originally intended for.  My concerns:

- FEAT Reply
One concern is how to announce the feature in the FEAT reply. Right now the RANG draft proposed to use RANG STREAM. Assuming that the server now supports RANG for HASH as well, how does the FEAT line look like? RANG STREAM;HASH? What if another command that might be introduced in the future wants to use the RANG concept as well? 

- Multiple commands for one function 
The use of multiple commands for one function makes it more complex than it needs to be. 

>Anthony's proposed FEAT reply for partial hashes
I think partial hashes should be a MUST and not optional. Hence there is no need for a special FEAT response.

>MLST style reply
I'm also in favor of the MLST style reply for commands with more than one return value. For the HASH output, everything beside the "Value" fact should be optional. Maybe MFF style arguments should be considered as well:
C>HASH Range=0-1000; File.ext
S>213 Value=f0ad929cd...;Algorithm=sha-256;Range=0-1000;

And my favorite would be to pass the algorithm the same way to the HASH command:
C>HASH Algorithm=sha-256;Range=0-1000; File.ext
S>213 Value=f0ad929cd...;Algorithm=sha-256;Range=0-1000;

Regards,
Mat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anthony Bryan
Sent: Monday, 17 January, 2011 09:59
To: Robert Oslin
Cc: [email-masked]
Subject: Re: [ftpext] draft-ietf-ftpext2-hash - partial hashes

a very quick introduction for Robert would mention that he works on CuteFTP and is the originator of the XCRC command.

I think everyone's been unanimous in that we want HASH to support partial file hashes.

we proposed a new byte RANGe selection command, http://tools.ietf.org/html/draft-bryan-ftp-range , which needed to be fleshed out of course but would look something like this:

   C> RANG 802816 1000000

2) how to show that partial hashes are supported, or if that's even needed? add a "p" to the FEAT? or just use an error code if someone tries to do a partial file hash and & it's not allowed or unsupported on the server?

      C> FEAT

3) the server response which shows it's a partial file hash and not a full file hash. it would probably be good to have the range in there, and it could be mandatory, where if it was a full file hash it would list the start & end of the file

   C> HASH filename.ext

from Lothar:

from Sob:

Rather than inventing new custom reply format, wouldn't it be better to adopt MLSx style? It's simple, readable, extensible, ...

E.g.:

(I intend to reply to the other hash messages backlog shortly)
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads _______________________________________________
ftpext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ftpext

---

From: Robert Oslin <[email-masked]>
Date: Mon, 17 Jan 2011 09:58:12 -0600

>>[Anthony]- Multiple commands for one function 
>>[Mat] The use of multiple commands for one function makes it more complex than it needs to be.

I agree with Mat regarding unnecessary complexity. I agree with both of you regarding proposed MLSx style request/reply. If the optional parameters were left out then defaults would apply (i.e. default algorithm, all bytes, etc.). 

Robert Oslin

www.globalscape.com

   (NYSE Amex:GSB)
*This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 25 Jan 2011 22:05:40 +0100 (CET)

On Mon, 17 Jan 2011, Mat Berchtold wrote:

> I understand that RANG is a replacement for REST. Using the newly introduced 
> command now to extend other commands as HASH might not be what RANG has been 
> originally intended for.

It actually is. We just wanted to start slowly and gently and specified it for 
RETR only to start with, as that is certainly a major use case.

> One concern is how to announce the feature in the FEAT reply. Right now the 
> RANG draft proposed to use RANG STREAM. Assuming that the server now 
> supports RANG for HASH as well, how does the FEAT line look like? RANG 
> STREAM;HASH? What if another command that might be introduced in the future 
> wants to use the RANG concept as well?

I think we shouldn't try to look too far ahead and guess what the future might 
and might not bring for this. As RANG and HASH aren't even yet very far taken, 
I suggest we simply say that FEAT resports RANG when the server supports it, 
and then it expect that to mean for the commands we include in the RANG spec: 
RETR and HASH so far.

> The use of multiple commands for one function makes it more complex than it 

(I assume you here talk about having to do both RANG and HASH to get a ranged 
hash output.)

Perhaps, but in my view RANG goes very well in hand with how RETR works 
already and that's a long established concept in FTP so I don't think this is 
a stretch.

Also, it could be seen as odd to have multiple commands allow a range instead 
of having a single command that sets a range...

> need for a special FEAT response.

I quite agree. Also, since HASH allows a server to reject to respond with a 
hash at will, it can select to do that on ranges if it thinks that is cool.

-- 

  / daniel.haxx.se

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 25 Jan 2011 22:08:23 +0100 (CET)

> already and that's a long established concept in FTP so I don't think this is 

Ouch, I meant that to be s/RETR/REST ...

-- 

  / daniel.haxx.se