Subject: Re: [ftpext] I-D Action:draft-ietf-ftpext2-hash-00.txt

From: Anthony Bryan <[email-masked]>
Date: Mon, 3 Jan 2011 17:16:42 -0500

2010/11/29 Anthony Bryan <[email-masked]>:
>
> so, people seem to be interested in partial file hashing. I think it
> is worth considering, even though it makes things more complicated and
> there are potential security issues.
>
> any suggestions how to add this to the current draft?

so, we have a new RANGe command [1] ID for specifying byte ranges.

used with RETR:

   C> RANG 802816 1000000
   S> 350 Restarting at 802816. End Byte range at 1000000. Send RETRIEVE
   C> RETR cap60.pl198.tar

what does everyone think of using RANGe with HASH to specify the byte
range for the optional partial file hashing?

   C> RANG 802816 1000000
   S> 350 Byte range starting at 802816, ending at 1000000.
   C> HASH filename.ext
   S> 213 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext
802816 1000000

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

[1] http://tools.ietf.org/html/draft-bryan-ftp-range

---

From: Lothar Kimmeringer <[email-masked]>
Date: Wed, 05 Jan 2011 15:59:41 +0100

Am 03.01.2011 23:16, schrieb Anthony Bryan:
> 2010/11/29 Anthony Bryan<[email-masked]>:
>>

I don't see security issues that might come up with that. Can you
be a bit more specific with that?

> range for the optional partial file hashing?

IMHO a good thing since it's addressing the same issue.

>     S>  213 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext

Do you need an additional RANG 802816 1000000 to finally retrieve that
part of data or is that setting after the call of HASH still "active"
and the client can continue with RETR immedately? That should be speci-
fied as well.

In addition to that I think adding hashing to the transfer of files
using APPE, RETR and STOR would be a good thing as well and allows
to get immediate feedback for a transfer of data if it has been
succeeded without transfer error. The capability of the server to
do so should be announced as additional FEAT-element:

C> FEAT
S> 211-Extensions supported:
S>  ...
S>  HASH SHA-256;SHA-512;SHA-1*;MD5
S>  THSH NONE*;SHA-256;SHA-512;SHA-1;MD5
S>  ...
S> 211 END

Rules are the same as for HASH with the additional option "NONE"
which will result into the old behavior i.e. no hash digest cal-
culation during transfers. NONE MUST be the default value being
set when initiating an FTP-session to ensure backward compatibility.

If a hash-calculation is active the return-message of RETR, STOR
and APPE will contain the output similar to HASH in addition to the
transfer complete message:

S> 226 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext\
  802816 1000000 ASCII transfer complete

The data being used for calculating the hash MUST be the original
data before any transformations take place (e.g. when transfering
in ASCII mode)

I see a couple of advantages for this approach:

  - The caluclation of a hash while already handling the data-
    transfer can easily be done in most implementing languages I'm
    aware of. E.g. in Java it's simply a DigestInputStream/
    DigestOutputStream being added to the already existing chain
    of streams being used. The calculation happens "on the fly".
  - Returning the hash as response to the transfer itself allows
    the client to check the resulting hash with the hash being
    calculated locally (e.g. by using above chain of streams as well)
    No need to call RANG and HASH specifically reducing the number
    of commands being exchanged between server and client. As well
    the file IO on the server-side is reduced since the data doesn't
    need to be processed twice (transfer and hash-calculation).
  - The correct transfer of data using other modes than "I" can be
    checked that way since it's operated on the data as it is
    received during transfer before conversions take place. A later
    call of RANG/HASH on a file being transfered in ASCII-mode most
    likely will produce a different hash than the one being
    calculated on the client side.

I hope this will not complicate things too much.

Best regards, Lothar
-- 
Lothar Kimmeringer                E-Mail: [email-masked]
                PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                  questions!

---

From: Richard Koenning <[email-masked]>
Date: Wed, 05 Jan 2011 16:26:53 +0100

> data before any transformations take place (e.g. when transfering

The data on the server before the transformation isn't necessarily the 
same as the data on the client after undoing the transformation. E.g. 
regard systems with different EOL conventions (LF on Unix systems, CR on 
Mac OS (afaik), CR LF on Windows systems and even more complicated 
conventions on mainframe systems), the common denominator of the FTP 
partners is the data on the wire, therefore the data being used for 
calculating the hash MUST be the data after the transformations take 
place on the sending side and before the transformations take place on 
the receiving side (with the SIZE command we have already a similar 
situation with the same rule).
Best regards,
Richard
-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH

---

From: Sob <[email-masked]>
Date: Thu, 06 Jan 2011 02:01:58 +0100

>    S> 213 SHA-256 f0ad929cd259957e160ea442eb80986b5f... filename.ext

Range information after filename doesn't seem to be the best idea. If 
optional, then it's wrong, because it can't be told if it's there or 
if it's part of filename. If mandatory, then it would work and only 
doesn't look nice.

Rather than inventing new custom reply format, wouldn't it be better 
to adopt MLSx style? It's simple, readable, extensible, ...

E.g.:

   S> 213 Hash.SHA-256=f0ad929cd...;Range=802816-1000000; filename.ext

And the relation between HASH and MLSx doesn't necessarily have to 
end with this. Hash as optional MLSx fact would be nice answer to 
already requested hashes for multiple files at the same time.

--

---

From: Lothar Kimmeringer <[email-masked]>
Date: Sat, 08 Jan 2011 18:29:41 +0100

Am 05.01.2011 16:26, schrieb Richard Koenning:
> The data on the server before the transformation isn't necessarily
 > the same as the data on the client after undoing the transformation.
 > E.g. regard systems with different EOL conventions (LF on Unix
 > systems, CR on Mac OS (afaik), CR LF on Windows systems and even more
 > complicated conventions on mainframe systems), the common denominator
 > of the FTP partners is the data on the wire, therefore the data being
 > used for calculating the hash MUST be the data after the
 > transformations take place on the sending side and before the
 > transformations take place on the receiving side (with the SIZE
 > command we have already a similar situation with the same rule).

We thought of the same, your description is more common than mine,
where I only regarded the server side when data is being sent from
client to sever.

Best regards, Lothar