Subject: RE: Additional thoughts on text transfers

From: Jeffrey Altman <[email-masked]>
Date: Mon, 1 Apr 2002 10:32:51 EST

Richard:

Many of us in the community have raised these issues before.  The vast
majority of the SSH development community is on Unix.  The vast
majority of the rest is on Windows.  Between the two groups they do
not see this as an issue because they are not affected by the scale of
the M x N problem which is alleviated by defining a standard on the
wire representation for End of Line.

The way things are now you are expected to solve this problem in the
client by knowing based upon the type of the host you are
communicating with how the End of Line is represented.  Some people
suggest that you should download the file first in binary and then
perform the conversion with external tools.  Of course, that assumes
the receiver is aware of the End of Line representation for both the
server and client operating system.

The Kermit Project has decided to address this problem by not
recommending the use of SFTP.  Instead we have modified C-Kermit 8.0
to allow it to be installed as a SSH Subsystem for the purpose of file
transfer.  Of course, you can still use it to just send files across
the SSH terminal session as well.

Kermit protocol provides you with recursive directory tree transfers;
conversion of pathname formats if necessary; automatic recognition of
text or binary mode based upon the content of the file (not just
extensions which mean different things to different operating
systems); as well as many other features.  For details see

  http://www.kermit-project.org/skermit.html

- Jeff

> Text files are still in high usage, even with new (often proprietary)
> formats that may not be converted.
> 
> Source code for programs are generally stored as text files, and the input
> for programs may be text files as well.
> 
> 
> > -----Original Message-----
> > From: Markus Friedl [mailto:[email-masked]]
> > Sent: Saturday, March 30, 2002 11:39 AM
> > To: Richard Whalen
> > Cc: [email-masked]'
> > Subject: Re: Additional thoughts on text transfers
> > 
> > 
> > On Fri, Mar 29, 2002 at 08:50:05AM -0500, Richard Whalen wrote:
> > > Since modern day files may be encoded in a variety of 
> > textual formats the
> > > client and server need to come to an agreement on a format 
> > when exchanging
> > > text files.
> > 
> > Modern day files are binary documents (pdf, doc, viruses), so no
> > conversions is necessary.  SFTP is slim and simple, you can do text
> > conversion in you client application based on suffix-heuristics,
> > for example.
> > 
> 

 Jeffrey Altman * Sr.Software Designer      C-Kermit 8.0 available now!!!
 The Kermit Project @ Columbia University   includes Telnet, FTP and HTTP
 http://www.kermit-project.org/             secured with Kerberos, SRP, and 
 [email-masked]                OpenSSL. Interfaces with OpenSSH

---

From: Darren Moffat <[email-masked]>
Date: Mon, 1 Apr 2002 17:26:31 -0800 (PST)

>On Mon, 1 Apr 2002, Richard Whalen wrote:
>
>How/why can filexfer be expected to know about all these?

The only person that knows what they want for a given file is the
end user of that file.

I really don't want the protocol doing stuff like that on mybehalf,
particularly if I'm going to subsequently check anything about the file
hasn't changed - doing a EOL conversion changes the size and certainly
the MD5 digest of the file.

I think that implemenations that care about adding a conversion process
for EOL can do that outside of the filexfer protocol.  One proposal I
have heard was that the server could use extensions to say what the
EOL mechanism was on that given server, however this might not always work
well for MacOS X because it uses both Mac and UNIX style depending on
who created the file.

It is issues like this that continually make me think that the security
area is the wrong place in IETF to try and solve this problem.

--
Darren J Moffat

---

From: Richard Whalen <[email-masked]>
Date: Tue, 2 Apr 2002 09:23:18 -0500

> Sent: Tuesday, April 02, 2002 9:13 AM
> To: Richard Whalen
> On Tue, Apr 02, 2002 at 09:00:25AM -0500, Richard Whalen wrote:
> > The OPEN/READ/WRITE mechanism is where the problem lies.  
> It specifies
> > particular bytes from the file, and on some systems those 
> bytes may contain
> > a mixture of actual data and system specific details.
> 
> But when the server sends different data in READ replies (depending on
> the client system) then the data get be corrupted, i.e. the MD5 of the
> data would not match among different clients.
> 

With a defined text transfer mechanism the server would send the same data
to all requests for the file in text mode (as long as the file has not been
changed).  The MD5 of the data would match with proper handling of the
carriage control information.

---

From: Jeffrey Altman <[email-masked]>
Date: Tue, 2 Apr 2002 10:28:26 EST

> On Tue, Apr 02, 2002 at 12:16:19AM -0500, Jeffrey Altman wrote:
> > > That kinda proves the point that I was trying to make - that the format
> > > decision and processing can be built into the clients (or servers) and not
> > > into the protocol itself.
> > 
> > No it doesn't.  It provies that it must be built into the protocol.
> 
> Then why does everybody write lines and lines of email instead of
> writing a draft using the mechanisms provided by
> 	draft-ietf-secsh-filexfer-02.txt
> 

For a very simple reason:

  up until the last six months or so the only significant deployment
  of SSH v2 and SFTP was on Unix or Windows.  Therefore, the only 
  developers working on the protocol were people that didn't care about
  this functionality.  

I must assume that Process Software is now implementing SSHv2 on
OpenVMS.  Therefore, there is now someone that might care enough to
take the responsibility of authorship.  However, as Richard points
out the IESG is perfectly capable and often willing to impose design 
requirements on a working group if they believe that the protocol that
has been designed does not provide for operating system independence.

Another reason is that people do not like to step on other people's
toes.  I'm sure Richard would rather work with the I-D authors to come
up with a design that try to figure it out himself.  

As for myself, this has been a very bad year and a half financially.
I have been unable to attend IETF meetings or even participate on
mailing lists at any significant level due to the economic realities
of keeping my co-workers employed.  That means I'm not spending my
time writing I-Ds for protocols I haven't implemented (yet).  That
doesn't mean I won't point out periodicly when a protocol design is
flawed.

Perhaps in two months I will have time to implement and then extend
SFTP.  In the meantime, the protocol should define alternate modes
other than Binary.  To keep things simply you probably want to have
two variations for sending text files.  The first (TEXT) simply uses a
standard EOL indicator (CR-LF).  The second (UTF8-TEXT) uses the
standard EOL indicator (CR-LF) and indicates that the data is being
sent as UTF8 allowing the receiver to translate it to a local
character-set if that is possible.  

The choice of which mode is to be used is always made by the sender.
If the receiver does not understand the mode, the incoming stream
should be treated as a binary stream to allow for later translation by
external tools.

- Jeff

---

From: Richard Whalen <[email-masked]>
Date: Tue, 2 Apr 2002 11:15:35 -0500

> From: Jeffrey Altman [mailto:[email-masked]]
> Sent: Tuesday, April 02, 2002 10:28 AM
> To: Markus Friedl
> Cc: Damien Miller; Richard Whalen; [email-masked]'

Process Software recently shipped SSH2 & SCP2 with MultiNet 4.4, based on
the F-Secure 2.4.0 build 15 code.
We are currently working on integrating the F-Secure 3.1.0 code with TCPware
(the version number will probably be 5.6), which is expected to ship this
summer (Northern Hemisphere).

We found the extended attribute mechanism in SFTP to be very flexible for
maintaining the VMS file attributes when transferring files between VMS
systems.

We determined before we went to Beta test that a binary only transfer
mechanism would not meet our customer needs for transferring text files.  We
can determine which files may be translatable, and have provided a
translation mechanism.  These mechanisms work well when the file is accessed
sequentially, but are ad-hoc (environment variables).

I have not started prototyping any protocols due to time constraints.
Exchanging ideas via email on how to deal with the text transfer problem has
been useful to help figure out what might have to be added to the protocol.

I don't know whether or not I will be able to travel to IETF, I do have time
to work on this via email.

----------------------
Richard Whalen
Process Software

---

From: Richard Whalen <[email-masked]>
Date: Thu, 4 Apr 2002 09:26:54 -0500

> From: nico [mailto:[email-masked]]
> Sent: Wednesday, April 03, 2002 10:05 PM
> To: Richard Whalen
> Cc: 'Markus Friedl'; [email-masked]'
> > The mechanisms provided by draft-ietf-secsh-filexfer-02.txt 
> fall short of
> > what is needed to provide a text transfer mechanism.
> > 
> > The file attributes mechanism has plenty of flexibilty to present
> > information about the type of information in the file, and 
> whether or not
> > any translations can be provided.  Right now it says 
> nothing about the type
> > and that no translations are available.
> 
> I don't see why the attributes wouldn't suffice for the case of ASCII
> text files:
> 
> Unix client: open(some file, write|create|trunc, attrs: {..., 
> content-type:
> 			text, encoding: ASCII, eol: LF, ...}, ...)
> 
> VMS server: OK, handle
> 
> client: write(handle, ...)
> client: write(handle, ...)
> ..
> server: OK
> server: OK
> ..
> client: close(handle)
> server: OK
> server: <create/truncate the file, write text sans LF and marking the
> 	 record boundaries where the LF characters were in the original>
> 
> client: open(some other file, read)
> server: OK, handle
> client: stat(handle)
> server: OK, attrs: ... content-type: test, encoding: ASCII, 
> eol: X, ...
> client: read(handle,...)
> ..
> client: close(handle)
> server: OK
> 

> 
> A canonical file format (preferably as a stream of octets) is 
> needed for
> checksumming.
> 
> Perhaps SFTP should move to a different working group :)
> 
> > ----------------------
> > Richard Whalen
> > Process Software
> 
> 
> Cheers,
> 
> Nico
> 

Your example for writing a file is reasonable, though the READ & WRITE
operations have not be defined to allow conversion of data upon storage.

Writing files in the local format when there is a defined text format is the
easy case though.

How do you read a file that is stored in a local format and needs to be
delivered in a defined text format?

The idea of exchanging a checksum to verify that a file has been completely
transferred is good; it could eliminate the need to process the file in the
defined transfer method to get the exact size to check to see if the
complete file has been received.

----------------------
Richard Whalen
Process Software