Subject: RE: Text file type hint proposal for filexfer

From: Richard Whalen <[email-masked]>
Date: Mon, 4 Oct 2004 09:31:25 -0400

As one who has argued for text transfers in the past, I support this, or an
equivalent method for the server to communicate this file characteristic to
the client.

-----Original Message-----
From: Jacob Nevins [mailto:[email-masked]]
Sent: Saturday, October 02, 2004 9:53 PM
To: [email-masked]
Subject: Text file type hint proposal for filexfer

SFTP v4 (draft-ietf-secsh-filexfer-04) added the FXF_TEXT flag to OPEN
to allow the client to request that the server open a file in text
mode.

However, no means is provided for the server to advise the client on
whether this transfer mode is appropriate. In all cases it is likely
up to the user to manually specify which mode to use.

In FTP, this has tended to lead to corrupted file transfers when
unintentional translation took place, etc. Some FTP implementations
have performed "automatic" conversions, e.g., using heuristics based
on file contents; this is not reliable, and anyway rather defeats the
point of having the server translate to a known representation.

The following proposal allows the server to optionally communicate
this information to the client via attributes. (Since it's an
attribute, the client can also manipulate it on the server as far as
is specified here if it corresponds to state there.)

(This proposal is against draft-ietf-secsh-filexfer-05.)

To section 5 "File Attributes", 2nd para, add between `attrib-bits' and
`extended_count':

       byte     content_type         present only if flag CONTENT_TYPE

To section 5.1 "Flags", add:

       #define SSH_FILEXFER_ATTR_CONTENT_TYPE      0x00000400

Add new section after section 5.8 "attrib-bits":

5.x Content type

   The `content_type' field, if present, indicates whether the file is
   known to be a text file, known _not_ to be a text file, or is of
   unknown content.  When sent from server to client, it acts as a
   hint to the client as to whether a file should be opened in
   SSH_FXF_TEXT mode.  The following values are defined:

        #define SSH_FILEXFER_CTYPE_UNKNOWN         0
        #define SSH_FILEXFER_CTYPE_BINARY          1
        #define SSH_FILEXFER_CTYPE_TEXT            2

To the end of the description of SSH_FXF_TEXT in section 6.3.1
"Opening a File", add:

      Clients MAY decide whether to use SSH_FXF_TEXT based on a
      previously seen `content_type' attribute; see Section 5.x.

(Of course, this could all be easily reformulated as an extension
attribute if desired.)

A version of filexfer-05 marked up with this proposal can be found at
<http://www.chiark.greenend.org.uk/ucgi/~jacobn/cvsweb/ssh-filexfer-filetype
/draft-ietf-secsh-filexfer-05-plus-filetype.txt.diff?r1=1.3&r2=1.4&f=H>
or <http://tinyurl.com/4cbfz>.

Warts with this proposal:

It's unfortunate that this can't be made atomic within the existing
protocol design; it requires participating clients to do a STAT or
similar before OPEN.
(In fact, perhaps there should be some guidance on which of STAT or
LSTAT to do in this case.)
I don't know whether the non-atomicity would be a problem in practice.

It's perhaps tempting to use the IETF's usual content-type notation,
"MIME types" (RFCs 2045-9 and related standards). However:
 - I don't know of any real filesystems which use it.
 - Consider "multipart/mixed" and friends.
 - Does everything in "text/*" want to be opened FXF_TEXT?
 - Does everything outside "text/*" _not_ want to be opened FXF_TEXT?
   (Excluding composite types like multipart.)
 - It risks adding further to SFTP's bloat if not specified carefully.
   We don't want to end up accidentally requiring implementations to
   contain entire MIME implementations (or more likely, ill-defined
   subsets thereof with poor interoperability).

---

From: Richard Whalen <[email-masked]>
Date: Tue, 5 Oct 2004 13:33:04 -0400

If the conversion is being performed, then the file can not be opened
in binary mode, as a request to open a file for read access that does
not specify text mode will automatically be checked to see if it can
be converted.

Which files are converted is controlled by an environment variable that
can be set on the server (outside of the SFTP server), or set by the client
with an SSH_FXP_EXTENDED command with a command name of
[email-masked]
and an integer parameter that is a bit mask of the types files to be
converted.
0 (zero) is none. Bit 0 (little endian) is fixed length files, bit 1 is
variable length record files, and bit 2 is files with variable length
records and fixed carriage control information.  Legal values are between
0 and 7.

Richard Whalen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 11:32 AM
To: Richard Whalen
Cc: [email-masked]'
Subject: Re: Text file type hint proposal for filexfer

> Our current SFTP implementation uses the file header information to do
> on the fly conversion of the various VMS text file formats to a stream
> of characters format when transferring text file to a non-VMS
> system.

Out of curiosity, what happens if the sftp client tries to open these
files in binary mode?

/Niels

---

From: Richard Whalen <[email-masked]>
Date: Tue, 5 Oct 2004 16:01:22 -0400

Yes, the file is opened differently depending upon whether it is a binary or
text transfer.  I might be able to change on the fly if I avoid the C run
time library and go directly to the system calls, but that would mean more
programming work in the end too.

I don't understand the demand for atomicity, but if some feel that it will
improve the suability of implementations, then I'm not going to fight the
idea.

-----Original Message-----
From: Joseph Galbraith [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:53 PM
To: Richard Whalen
Cc: [email-masked]
Subject: Re: Text file type hint proposal for filexfer

Ahhh... do you actually open the file differently at
the OS level for text mode?  That hadn't occured to
me.

Still, for VMS, this proposal isn't any worse than the
previous one, and this proposal is a little more compact.
VMS would not set the TEXT_HINT bit in it's
supported-attrib field, and clients would know they can't
switch into text-mode.

I'm not sure atomicity is a really that big deal...
maybe I should just bag the switch to text mode
anyway.

I do still prefer adding a couple of bits to the
attrib-bits field to adding a whole new field.

Thanks,

Joseph

Richard Whalen wrote:
> I don't think that this scheme will work for VMS.
> Though we can report on the format of the file, the idea of opening it in
an
> unspecified mode, then specifying that you really want text mode may not
be
> possible without sacrificing the atomicity that some want.
> 
> From: Joseph Galbraith [mailto:[email-masked]]
> Sent: Tuesday, October 05, 2004 3:21 PM
> To: [email-masked]
> How about this alternative-- sftpv5 has (and naturally v6 will have)
> an 'attrib-bits' field in the attrib structure which describes
> various bit attributes, such as case-sensitivity, if the file
> is encrypted on disk, compressed on disk, advisory readonly,
> a hidden file, a system file, etc.
> 
> What if we add two bits there:
> 
>      #define SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT      0x00000800
> 
>         The server is reasonably certain that the file
>         contains textual data, and therefore the file
>         should be opened in text mode.
> 
>         This flag MUST NOT be present during a setstat operation.
>         If this flag is present during an fsetstat operation,
>         the file handle is converted to a text-mode handle, as
>         if it had been opened with SSH_FXF_ACCESS_TEXT_MODE.
> 
>         The server MUST NOT set this bit unless it is reasonably
>         certain the file should be transfered in text-mode because
>         many clients will use this flag to initiate automatic
>         text-mode translation.  If the server sets this flag
>         in error, data corruption will result.
> 
>      #define SSH_FILEXFER_ATTR_FLAGS_BIN_HINT       0x00001000
>         The server is reasonably certain that the file
>         contents are binary and should not be opened
>         with SSH_FXF_ACCESS_TEXT_MODE.
> 
> This gets around your atomicity issue (I think.)  The client
> can open sans SSH_FXF_ACCESS_TEXT_MODE, do an fstat, and then,
> if the server comes back with SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT
> in the attrib.attrib-bits, it can do a fsetstat to put the
> handle into text mode.
> 
> mime types have come up enough times that I think I'm
> going to add a field for them; anybody that doesn't
> want shouldn't need to implement them though.
> 
> (I know of at least one filesystem that does know the
> mime-type information though.)
> 
> - Joseph
> 
> Jacob Nevins wrote:
>

---

From: Richard Whalen <[email-masked]>
Date: Tue, 5 Oct 2004 16:02:35 -0400

"suability" should have been "usability"  - I clicked too fast to the spell
checker!

-----Original Message-----
From: Joseph Galbraith [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:53 PM
To: Richard Whalen
Cc: [email-masked]
Subject: Re: Text file type hint proposal for filexfer

Thanks,

Joseph

an
be