Subject: [ftpext] draft-ietf-ftpext2-hash command name and default algorithm

From: Robert Oslin <[email-masked]>
Date: Mon, 17 Jan 2011 10:04:45 -0600

draft-ietf-ftpext2-hash documents over 40 (and I'm certain this not a comprehensive list) implementations of XCRC, the non-formalized/ratified file integrity command that has been the de facto standard over the past decade. As some on this list know well, it can be extremely time consuming and costly for vendors to implement new standards in their products (I could write pages on the soup-to-nuts process involved). I believe it would make life easier on developers and speed the adoption of your RFC if you considered changing the following:

1. Keep the command XCRC vs. HASH. XCRC standards for "Transfer (xfer) Cyclical Integrity Check", not "eXperimental Cyclical Integrity Check" as some have supposed. Personally I like "HASH" better, but keeping the command the same would mean that possibly hundreds of vendors wouldn't have to change anything to start supporting it today (exception is #2 below).

2. Keep the default hash CRC32 (maybe if no opts shown in FEAT response), but recommend MD5, SHA1, or higher. Now, I understand this is a big red flag: "What about potential for collisions!" and "What about malicious users?". Yes, those are both [remote] possibilities; I grant you that. I would not advocate for CRC32 unless it were already in use by hundreds of FTP clients/servers across the world, with great success, and zero reports (that I know of) of problems. Now before you reply, consider that it would be the default, for legacy support sakes, but the RFC would strongly urge/recommend that new implementations or existing ones use SHA-1 or higher. By defaulting to CRC32 and XCRC you would avoid disenfranchising hundreds of vendors from the new standard, while encouraging new and existing vendors/developers to switch to stronger algorithms to mitigate data integrity security risks.

There is one more reason to consider using CRC32 as the default aside from grandfathering in existing XCRC vendors. CRC32 is fast! It is

2-4X faster than geekdom approved algorithms such as SHA-256. This is a big deal when it comes to FTP sessions, with the typical 30-60 second client imposed timeout between client request and server replies. According to a crypto++ API benchmark, CRC32 can process 253 MiB/Second vs. SHA-256's 111MiB/Second. That is a difference between 15 seconds for CRC32 vs. 36 seconds for SHA-256 to hash a 4 GB file (not counting disk i/o time!). Therefore you run a greater risk of timing out the socket connection when using stronger hashing algorithms.

We should also consider the problem we are trying to solve with HASH/XCRC. Are we simply trying to detect modifications due to random transmission errors, as was the original intent of XCRC? Or are we trying to detect malicious tampering or mitigate risk of collisions? If the former, then I would argue that CRC32 as the default (for legacy support and performance) is the preferred approach. If the latter, then perhaps we should enforce SHA-1 or higher as the default and force vendors to bite the bullet.

Final thoughts. Changing the command name to XCRC but not changing the default algorithm to CRC32 would make no sense. Its neither or or both. I do understand if everyone votes for a stronger (default) hashing algorithm, and I do see the benefits going forward; I just want everyone to consider the impact to existing vendors/developers and impact to performance, which could lead to more timeouts - hence failed transfers, which could adversely impact business processes.

Robert Oslin
Director of Product Management
Tel: 1 (210) 293-7902
Fax: 1 (210) 690-8824
Send me large files securely<https://robertoslin.cutesendit.com/>

www.globalscape.com<http://www.globalscape.com/>

   (NYSE Amex:GSB)<http://www.globalscape.com/cgi-bin/links.cgi?page=gsb>

*This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.

---

From: Sob <[email-masked]>
Date: Mon, 17 Jan 2011 18:58:12 +0100

At 17:04 17.1.2011, Robert Oslin wrote:
>I believe it would make life easier on developers and speed the 
>adoption of your RFC if you considered changing the following:
>
>1. Keep the command XCRC ...
>2. Keep the default hash CRC32 ...

You would have to keep one more thing, "250 <crc>" as default (or 
only) reply format, otherwise old clients with knowledge of only the 
old XCRC wouldn't know what to do with it.

So you can either:

a) scratch all new reply format ideas and stick with the old one; 
after all, it shouldn't be that hard for client to remember what it 
asked for (filename, range, hash type)

b) force new clients to support two completely different reply 
formats, otherwise they wouldn't understand old servers and the 
"instant success" would not happen; again, it's probably doable, 
developers are clever people and the two formats should be different 
enough to get properly recognized

But essentially it would still be two different commands, only with 
the same name, no real advantage. Old client wouldn't benefit from 
new server and the other way around.

--

---

From: Robert Oslin <[email-masked]>
Date: Tue, 18 Jan 2011 08:40:05 -0600

I don't see why we need a reply that includes information the client already knows (hash used, filename, etc.). Only the hash is needed, hence option (a) below. 

Again, I'm not insisting on XCRC over HASH and CRC32 over SHA-1 as the default algorithm; I'm simply make the case that doing so will ease the path for adopting the new standard for most FTP server client/server vendors.

Robert Oslin

-----Original Message-----
From: Sob [mailto:[email-masked]] 
Sent: Monday, January 17, 2011 11:58 AM
To: [email-masked]
Subject: Re: [ftpext] draft-ietf-ftpext2-hash command name and default algorithm

So you can either:

--

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 18 Jan 2011 15:46:19 +0100 (CET)

On Tue, 18 Jan 2011, Robert Oslin wrote:

> Again, I'm not insisting on XCRC over HASH and CRC32 over SHA-1 as the 
> default algorithm; I'm simply make the case that doing so will ease the path 
> for adopting the new standard for most FTP server client/server vendors.

Sure, if we want to standardize XCRC using CRC32

I personally think CRC32 is too weak for multi-gigabyte files.

-- 

  / daniel.haxx.se

---

From: Robert Oslin <[email-masked]>
Date: Tue, 18 Jan 2011 09:52:06 -0600

>> I personally think CRC32 is too weak for multi-gigabyte files.

Do you mean that CRC32 is weak period? I don't see how file size would make a difference other than the time needed to calculate the hash. One may argue that the "weaker" the hash the faster it will compute, making it ideal for use with larger files (in order to mitigate the chance of client timeouts).

I know that I'm arguing from a losing position. If academics frown on MD5 then how much more will they frown on CRC32? For enterprise level mission critical transfers you want as close to zero chance of collisions, whether by malicious intent or otherwise, and CRC32 is relatively weak in this regard.

However the current draft states: 

"Implementations SHOULD NOT make any algorithm the default that is known to be weaker than SHA-1. Support for any additional algorithms is OPTIONAL"

SHA-1 at 153 MiB/Second is only marginally faster than SHA-256 at 111 MiB/Second while still being susceptible to collisions (http://en.wikipedia.org/wiki/SHA-1). If performance is the goal then changing from SHA-1 to CRC32 as the default algorithm will result in 2X performance gain while remaining just as susceptible to collisions. If security is the overriding goal then SHA-256 should be the minimum recommended, with less (and more) secure algorithms optional.

I.e. "Implementations SHOULD NOT make any algorithm the default that is known to be weaker than SHA-256. Support for any additional algorithms is OPTIONAL"

I don't know how else to balance the need for performance (avoiding timeouts) while minimizing the chance of collisions. CRC32 and MD5 are fast but collision prone. SHA-1 is slow and still collision prone. SHA-256 is slower but almost zero chance for collisions. Note that performance will become less of a problem in future as CPU speeds increase.

Sorry for the rant, but even the smallest decision can often result in huge repercussions for vendors and users later on. 

Robert Oslin

-----Original Message-----
From: Daniel Stenberg [mailto:[email-masked]] 
Sent: Tuesday, January 18, 2011 8:46 AM
To: Robert Oslin
Cc: [email-masked]
Subject: Re: [ftpext] draft-ietf-ftpext2-hash command name and default algorithm

-- 

  / daniel.haxx.se

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 18 Jan 2011 17:01:14 +0100 (CET)

On Tue, 18 Jan 2011, Robert Oslin wrote:

> Do you mean that CRC32 is weak period?

CRC32 uses 32 bits to indicate a checksum for the data. I claim it gets weaker 
the more data to try to check with it. Having a CRC32 field for a sufficiently 
small amount of data will be fine IMO - but that's not what we design for 
here.

> One may argue that the "weaker" the hash the faster it will compute, making 
> it ideal for use with larger files (in order to mitigate the chance of 
> client timeouts).

Clients don't have to timeout. The server can easily send "213-" lines every 
5-10 seconds or so while calculating the hash. Or is there a reason why not?

-- 

  / daniel.haxx.se

---

From: Robert Oslin <[email-masked]>
Date: Tue, 18 Jan 2011 10:16:51 -0600

I like the idea of an intermittent 213 reply that would effectively ask the client to hang in there (maybe reset its timeout counter, perhaps with a client defined hard limit to the number of resets in order to prevent malicious use).

This seems in accord with RFC959 section 5.4: Certain commands require a second reply for which the user should also wait.  These replies may, for example, report on the progress or completion of file transfer or the closing of the data connection.  They are secondary replies to file transfer commands.

Anthony, could a provision be made such that a server MAY return a 213 reply every N seconds (5, 15, other) for long running hash operations? I think that would be especially useful if SHA-256 ends up being the default recommended and implemented hash algorithm.

Alternatively client logic could be made to auto-extend the internal timeout when making a hash request for a large file. The draft could recommend either/or approach.

Robert Oslin

-----Original Message-----
From: Daniel Stenberg [mailto:[email-masked]] 
Sent: Tuesday, January 18, 2011 10:01 AM
To: Robert Oslin
Cc: [email-masked]
Subject: Re: [ftpext] draft-ietf-ftpext2-hash command name and default algorithm

-- 

  / daniel.haxx.se

---

From: Sob <[email-masked]>
Date: Tue, 18 Jan 2011 17:21:01 +0100

At 15:40 18.1.2011, Robert Oslin wrote:
>I don't see why we need a reply that includes information the client 
>already knows (hash used, filename, etc.). Only the hash is needed, 
>hence option (a) below.

I have no strong opinion on this. While bare hash should be enough, 
including additional info might be good for something.

>Again, I'm not insisting on XCRC over HASH and CRC32 over SHA-1 as 
>the default algorithm; I'm simply make the case that doing so will 
>ease the path for adopting the new standard for most FTP server 
>client/server vendors.

Sure, "new, improved and backward compatible version of good old 
trusted XCRC command" sounds better than "unknown new HASH command". 
On the other hand, if someone advertises support for XCRC and it 
turns out that it means only very small subset of the specification 
(i.e. old XCRC), it wouldn't be so great.

And it shows another problem on the horizon, what about FEAT? What's 
the correct line for current XCRC anyway? I saw:

XCRC filename;start;end
XCRC "filename" SP EP

Even others maybe? Whichever we choose, we instantly get millions of 
servers violation the specification. Unless the XCRC line for FEAT is 
defined as "XCRC<anything>".

When you think about it, new and independent HASH command is not such bad idea.

--

---

From: Richard Koenning <[email-masked]>
Date: Tue, 18 Jan 2011 20:14:54 +0100

> small amount of data will be fine IMO - but that's not what we design for 

 From the draft it is still unclear to me whether the hash shall be used 
only for detecting "normal" transmission problems or also for detecting 
active attacks on the data channel.

For the latter CRC32 is out of discussion not only for the size of the 
hash value but for its linear properties: it is even easy to change the 
data while keeping the CRC unchanged.
Regarding MD5: Since long time cryptologists warn against using MD5 for 
new protocols/applications. As an application of this policy there are 
no newer TLS cipher suites using MD5.
Regarding SHA1: Despite showing some weakness too it is imho still in 
wide usage e.g. in the TLS area. On the other hand the search for new 
long-term hash functions is still ongoing 
(http://csrc.nist.gov/groups/ST/hash/timeline.html). Therefore it seems 
to me still acceptable to use SHA1 as default hash function.

When the hash function is only used for detecting transmission errors 
than the collision probability is one of the main points, this 
probability is approximately the inverse of the square root of the hash 
value space. So with CRC32 one has to expect collisions already with a 
set of some ten thousand files. With MD5 the corresponding number has 19 
figures, which probably suffices already for most applications.

-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH

---

From: Anthony Bryan <[email-masked]>
Date: Wed, 19 Jan 2011 04:44:57 -0500

> Anthony, could a provision be made such that a server MAY return a 213 reply every N seconds (5, 15, other) for long running hash operations? I think that would be especially useful if SHA-256 ends…

I think we can do anything that is internally consistent within the
FTP universe and seems logical for the published FTP RFCs. :)

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 19 Jan 2011 11:16:16 +0100 (CET)

On Wed, 19 Jan 2011, Anthony Bryan wrote:

>> Anthony, could a provision be made such that a server MAY return a 213 
>> reply every N seconds (5, 15, other) for long running hash operations? I 
>> think that would be especially useful if SHA-256 ends up being the default 
>> recommended and implemented hash algorithm.
>
> I think we can do anything that is internally consistent within the FTP 
> universe and seems logical for the published FTP RFCs. :)

Right. I was only suggesting a "213-" response line, which isn't the final 
line of a 213 so I don't think we'd have to write anything in particular to 
allow a server to do that. It would be following standard FTP behavior.

We could of course still add something in the spec to make it more obvious 
that this might be a good idea for servers to do.

-- 

  / daniel.haxx.se

---

From: Robert Oslin <[email-masked]>
Date: Wed, 19 Jan 2011 08:19:54 -0600

Correct. I meant "SHOULD" return an intermediate (213) reply for long-running...

-----Original Message-----
From: Daniel Stenberg [mailto:[email-masked]] 
Sent: Wednesday, January 19, 2011 4:16 AM
To: Anthony Bryan
Cc: Robert Oslin; [email-masked]
Subject: Re: [ftpext] draft-ietf-ftpext2-hash command name and default algorithm

-- 

  / daniel.haxx.se