Subject: Text file type hint proposal for filexfer

From: Jacob Nevins <[email-masked]>
Date: Sun, 3 Oct 2004 02:53:14 +0100

SFTP v4 (draft-ietf-secsh-filexfer-04) added the FXF_TEXT flag to OPEN
to allow the client to request that the server open a file in text
mode.

However, no means is provided for the server to advise the client on
whether this transfer mode is appropriate. In all cases it is likely
up to the user to manually specify which mode to use.

In FTP, this has tended to lead to corrupted file transfers when
unintentional translation took place, etc. Some FTP implementations
have performed "automatic" conversions, e.g., using heuristics based
on file contents; this is not reliable, and anyway rather defeats the
point of having the server translate to a known representation.

The following proposal allows the server to optionally communicate
this information to the client via attributes. (Since it's an
attribute, the client can also manipulate it on the server as far as
is specified here if it corresponds to state there.)

(This proposal is against draft-ietf-secsh-filexfer-05.)

To section 5 "File Attributes", 2nd para, add between `attrib-bits' and
`extended_count':

       byte     content_type         present only if flag CONTENT_TYPE

To section 5.1 "Flags", add:

       #define SSH_FILEXFER_ATTR_CONTENT_TYPE      0x00000400

Add new section after section 5.8 "attrib-bits":

5.x Content type

   The `content_type' field, if present, indicates whether the file is
   known to be a text file, known _not_ to be a text file, or is of
   unknown content.  When sent from server to client, it acts as a
   hint to the client as to whether a file should be opened in
   SSH_FXF_TEXT mode.  The following values are defined:

        #define SSH_FILEXFER_CTYPE_UNKNOWN         0
        #define SSH_FILEXFER_CTYPE_BINARY          1
        #define SSH_FILEXFER_CTYPE_TEXT            2

To the end of the description of SSH_FXF_TEXT in section 6.3.1
"Opening a File", add:

      Clients MAY decide whether to use SSH_FXF_TEXT based on a
      previously seen `content_type' attribute; see Section 5.x.

(Of course, this could all be easily reformulated as an extension
attribute if desired.)

A version of filexfer-05 marked up with this proposal can be found at
<http://www.chiark.greenend.org.uk/ucgi/~jacobn/cvsweb/ssh-filexfer-filetype/draft-ietf-secsh-filexfer-05-plus-filetype.txt.diff?r1=1.3&r2=1.4&f=H>
or <http://tinyurl.com/4cbfz>.

Warts with this proposal:

It's unfortunate that this can't be made atomic within the existing
protocol design; it requires participating clients to do a STAT or
similar before OPEN.
(In fact, perhaps there should be some guidance on which of STAT or
LSTAT to do in this case.)
I don't know whether the non-atomicity would be a problem in practice.

It's perhaps tempting to use the IETF's usual content-type notation,
"MIME types" (RFCs 2045-9 and related standards). However:
 - I don't know of any real filesystems which use it.
 - Consider "multipart/mixed" and friends.
 - Does everything in "text/*" want to be opened FXF_TEXT?
 - Does everything outside "text/*" _not_ want to be opened FXF_TEXT?
   (Excluding composite types like multipart.)
 - It risks adding further to SFTP's bloat if not specified carefully.
   We don't want to end up accidentally requiring implementations to
   contain entire MIME implementations (or more likely, ill-defined
   subsets thereof with poor interoperability).

---

From: [email-masked] (Niels Möller)
Date: 04 Oct 2004 16:43:31 +0200

> "MIME types" (RFCs 2045-9 and related standards). However:

I think a MIME content-type makes perfect sense as an extension, but I
agree it shouldn't be in the sftp spec proper. Therefore, I think it
is a little confusing to use the name "CONTENT_TYPE" for the attribute
you propose. Perhaps something like FILE_MODE, OPEN_MODE, FILE_TYPE
might work?

One other observation:

If you have this attribute, *and* there is some way of knowing the
server's line end convention, then it would be possible for a client
to use the following strategy:

Always SFTP_OPEN files in binary mode. Use FSTAT to ask the server if
it's a text or binary file. Next, figure out what are the server's and
the client's native line ending convention. Then convert the file as
it is read.

As far as I can see, this simple scheme will work just right for
servers where your new attribute is fully supported (i.e it is always
correct, and never returns "unknown"). It will of course always fail
on servers on operating systems that

 * differentiate between text and binary files

 * doesn't provide a reliable way for an application, e.g. the sftp
   server, to find out if a given file name refers to a text or binary
   file.

However, I don't see any way to get things right automatically in this
case; there's no reliable information anywhere that says definitely if
the file is text or binary, so it must be guessed, either by the user,
or by the client or by the server. And whenever that guess is wrong,
file corruption can be expected.

Is this problematic case common? To me, it seems so fundamentally
broken that I don't think we should work very hard to try to fix it.

Best regards,
/Niels

---

From: Jacob Nevins <[email-masked]>
Date: Mon, 4 Oct 2004 22:35:57 +0100

Thanks for the feedback.

> you propose. Perhaps something like FILE_MODE, OPEN_MODE, FILE_TYPE

Fair point. How about `textmode_hint'? 

(Updated proposal:
<http://www.chiark.greenend.org.uk/ucgi/~jacobn/cvsweb/ssh-filexfer-filetype/draft-ietf-secsh-filexfer-05-plus-filetype.txt.diff?r1=1.3&r2=1.6&f=H>
or <http://tinyurl.com/53sbd>.)

> server's line end convention,

Well, there's the rub, really. How do you do that? You can't rely on the
"newline" extension being present, or if it is, corresponding with what
the server will send in binary mode.

Also: I'm not familiar with VMS, but the impression I have is that line
breaks there are not represented by a byte sequence at all, but rather
by something that it's not possible to squirt down an SFTP "binary"
connection. Hence our having an explicit text mode in SFTP in the first
place.

My intended semantics for the flag are purely "it's a good idea to open
this file in FXF_TEXT mode", not "if you open this file in binary mode
you can assume a certain line ending delimiter".

> then it would be possible for a client to use the following strategy:
> 
> the client's native line ending convention. Then convert the file as it
> is read.

This proposal sounds like an entirely different approach to that
supported by SFTP currently. If my understanding of the VMS issue is
correct, it runs into trouble with that. Anyway, I think we've argued
this one enough on this WG already, and we should stick with the current
mechanism.

(Are you proposing this to get round the atomicity issue I mentioned?)

> file corruption can be expected.

Of course. I don't think there's anything the SFTP protocol can do to
mitigate the latter possibility. The former (which could be said to
include VMS, if my understanding's correct), we can and should support.

(An SFTP server in the latter situation is of course at liberty to
populate this flag by guessing based on file contents, for instance; I'd
say that was a quality-of-implementation issue and outside the scope of
the spec.)

> Is this problematic case common?

Endemic, I think; probably more the rule than the exception. No OS I use
frequently has the ability to reliably distinguish text and non-text
files (Unix, DOS/Windows). That shouldn't stop us providing the facility
in the protocol.

> To me, it seems so fundamentally broken that I don't think we should
> work very hard to try to fix it.

Agree that we can't magic the information out of nowhere, which is why I
provide UNKNOWN.

Personally, I wasn't too sure about having SFTP deal with text file
semantics in the first place; but given that the WG has decided to do so
(and it is a frequent feature request from users), we should provide a
complete set of operations for our chosen abstraction, so that we are
not the link that prevents reliable transfer of text files. Let's learn
something from FTP...

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 05 Oct 2004 13:20:59 -0600

How about this alternative-- sftpv5 has (and naturally v6 will have)
an 'attrib-bits' field in the attrib structure which describes
various bit attributes, such as case-sensitivity, if the file
is encrypted on disk, compressed on disk, advisory readonly,
a hidden file, a system file, etc.

What if we add two bits there:

     #define SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT      0x00000800

        The server is reasonably certain that the file
        contains textual data, and therefore the file
        should be opened in text mode.

        This flag MUST NOT be present during a setstat operation.
        If this flag is present during an fsetstat operation,
        the file handle is converted to a text-mode handle, as
        if it had been opened with SSH_FXF_ACCESS_TEXT_MODE.

        The server MUST NOT set this bit unless it is reasonably
        certain the file should be transfered in text-mode because
        many clients will use this flag to initiate automatic
        text-mode translation.  If the server sets this flag
        in error, data corruption will result.

     #define SSH_FILEXFER_ATTR_FLAGS_BIN_HINT       0x00001000
        The server is reasonably certain that the file
        contents are binary and should not be opened
        with SSH_FXF_ACCESS_TEXT_MODE.

This gets around your atomicity issue (I think.)  The client
can open sans SSH_FXF_ACCESS_TEXT_MODE, do an fstat, and then,
if the server comes back with SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT
in the attrib.attrib-bits, it can do a fsetstat to put the
handle into text mode.

mime types have come up enough times that I think I'm
going to add a field for them; anybody that doesn't
want shouldn't need to implement them though.

(I know of at least one filesystem that does know the
mime-type information though.)

- Joseph

---

From: Jacob Nevins <[email-masked]>
Date: Tue, 5 Oct 2004 22:53:14 +0100

Joseph Galbraith <[email-masked]> writes:
> an 'attrib-bits' field in the attrib structure which describes various
> bit attributes, such as case-sensitivity, if the file is encrypted on
> disk, compressed on disk, advisory readonly, a hidden file, a system
> file, etc.

Yes, that's fair enough. I just recycled my proposal from SFTP v4
without taking account the new attrib-bits field. This information is
probably better there.

It might be worth spelling out that in the absence of either of these
bits being set, the client SHOULD NOT automatically assume anything
about the textiness or otherwise of the file. (The user can tell it, of
course.)

>         if it had been opened with SSH_FXF_ACCESS_TEXT_MODE.

(Hmm. I deliberately hadn't ruled out the possibility that the "text
flag" corresponded to some state on the server that you might want to be
able to set with setstat. However, I'm not strongly wedded to that idea;
it's probably a bit kludgy when you consider a filesystem that can
represent a range of file types but only has this single distinction
exposed over SFTP.)

It seems a bit kludgy to override fsetstat's semantics in this way, when
all other attributes correspond to state on disc. I think I'd prefer
some new freopen()-like operation to change the file open mode.

This however doesn't get round Richard Whalen's VMS concerns, except
that perhaps being able to say SSH_FX_OP_UNSUPPORTED without ambiguity
as to _what_ precisely is unsupported might be helpful (unlike setstat,
where any of the attributes could have upset the server)?

Perhaps the new `reopen' operation could be an extension, so that its
lack on servers like VMS will be obvious up front via the "supported"
extension, rather than the client not finding out until halfway through
an operation?

(I don't feel strongly about the atomicity issue, but I thought there
might be people who did, perhaps including my future self.)

  [snip]
> mime types have come up enough times that I think I'm going to add a
> field for them; anybody that doesn't want shouldn't need to implement
> them though.
> 
> (I know of at least one filesystem that does know the mime-type
> information though.)

Which one is that, out of interest?

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 05 Oct 2004 16:16:02 -0600

> about the textiness or otherwise of the file. (The user can tell it, of

All right; I'll do that.

[snip]

> where any of the attributes could have upset the server)?

Well, in theory, a VMS server would not set 
SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT and hence the client
would know that it can't send the attribute.

However, unless someone speaks up at least moderately
in favor of the atomicity issue, I think I'm going to
drop that and forbid both flags during any kind of
setstat operation.

(Which I'd prefer to do rather than have some kind of poorly
defined result when setting the flags.)

> might be people who did, perhaps including my future self.)

I think if we decide to try and address atomicity issues that
is a fine idea.  I will probably go that route.

> Which one is that, out of interest?

I believe BeFS (from BeOS) supported mime types as it's
native typing mechanism.

There is at least a small movement creating an open source clone,
so it isn't completely dead.

A MS Windows based server that wanted to provide the information
could query the registry from the file extension as well.

(I doubt we will, but it is possible, and someone else might decide
to do it.)

- Joseph

---

From: "denis bider" <[email-masked]>
Date: Fri, 8 Oct 2004 04:32:03 +0200

> > It might be worth spelling out that in the absence of 
> > either of these bits being set, the client SHOULD NOT
> > automatically assume anything about the textiness or
> > otherwise of the file. (The user can tell it, of
> > course.)
> 
> All right; I'll do that.

I would like to note that checking whether the file is textual or not is a
cheap operation only on VMS.

Defining the text/binary hint as a file attribute makes it less likely that
Unix and Windows servers will implement this.

I think we would benefit if it were possible for a client to query the likely
textual or binary state of a file independently of anything else. In other
words, without obliging the server to either show this possibly expensive
information as part of regular easily-obtainable attrib-bits, or not support
this feature at all.

I think the attribute hints are a nice idea and will work for VMS. However for
Unix and Windows servers, we might also benefit from a way for the client to
prompt the server to determine on the fly whether the file is textual or
binary, if the server has not indicated the file type in attrib-bits. This
query would be done right before downloading the file.

If the spec does not provide for this, and only defines the text hints in
attrib-bits, then a complete client implementation wishing to implement
automatic file type conversion will look as follows:
- if server gives text hint, open file in text mode and download;
- if server gives binary hint, open file in binary mode and download;
- if server gives no hint, open in binary mode, then scan bytes received, see
if the file can be recognized as textual, and do conversion on the fly.

Granted, this is much better than what we have now.

However if there was widespread server-side support for a query to determine
file type, the detection and conversion algorithms could be entirely removed
from the client.

The spec already requires servers to implement text file -> canonical format
conversion (the SSH_FXF_TEXT flag). But this feature is useless if the client
cannot query the server for the file type. The attrib-bits hint does not solve
this for Unix and Windows servers because they are unlikely to be able to give
this information as part of attrib-bits. It requires a dedicated request.

My thinking is, either we require Unix and Windows servers to implement both
(a) conversion - SSH_FXF_TEXT flag, AND (b) file type detection; or we should
require them to do neither.

The corresponding solutions are:

(a) Specify attrib-bits as you propose, but also a dedicated file type query
request in case this determination is more expensive.

(b) Say in the spec that SSH_FXF_TEXT need not be supported by servers that
don't send the text hint. The binary hint in this case is also not necessary.
No text hint -> open the file in binary, detect and convert if you please.

The idea is, let's be consistent, and let's not specify things halfway through.

denis

---

From: Jacob Nevins <[email-masked]>
Date: Sun, 10 Oct 2004 23:56:14 +0100

denis bider <[email-masked]> writes:
> I would like to note that checking whether the file is textual or not
> is a cheap operation only on VMS.
> 
> Defining the text/binary hint as a file attribute makes it less likely
> that Unix and Windows servers will implement this.

My intention in defining the attribute extension was to support those
servers which have a reasonably authoritative text/binary distinction,
for example in the filesystem, rather than those which have to use a
heuristic approach, such as Unix/Windows, where I would much prefer that
UNKNOWN is returned rather than a guess made.

> I think we would benefit if it were possible for a client to query the
> likely textual or binary state of a file independently of anything
> else. In other words, without obliging the server to either show this
> possibly expensive information as part of regular easily-obtainable
> attrib-bits, or not support this feature at all.
> 
> I think the attribute hints are a nice idea and will work for VMS.
> However for Unix and Windows servers, we might also benefit from a way
> for the client to prompt the server to determine on the fly whether
> the file is textual or binary, if the server has not indicated the
> file type in attrib-bits. This query would be done right before
> downloading the file.

I'd prefer any such extension to be optional, and to have defined
semantics of being heuristic rather than authoritative if that's how
it's implemented; as a client (or rather client user), I would like to
be able to distinguish heuristic from authoritative information so that
I can choose not to use the former while still getting the benefit of
the latter. (Perhaps the extension could have a flag indicating whether
it contains "authoritative" information?)

I should come out and say that my personal view is that I'd rather not
trust the integrity of my files to text/binary distinction heuristics,
which is why I'm pushing for this distinction. I realise this isn't a
universal view, but I don't think I'm alone in this; it's a popular
feature but I'd rather it was possible for the end user to choose
whether to use it.

> However if there was widespread server-side support for a query to
> determine file type, the detection and conversion algorithms could be
> entirely removed from the client.

Except that the client would need to convert from canonical form.
I agree that if you're going to do file type detection on server->client
transfers, it's better to do it on the server.

(Clients will also need to have some means of file type detection for
client->server transfers if they don't have that information to hand.)

> My thinking is, either we require Unix and Windows servers to
> implement both (a) conversion - SSH_FXF_TEXT flag, AND (b) file type
> detection; or we should require them to do neither.
  [...]
> The idea is, let's be consistent, and let's not specify things halfway

I don't think there's anything inconsistent here.

The server may not know _which_ files are text files, but it could well
be in a position to know what the likely line ending convention is if
the user asserts that given file _is_ a text file, which is a better
position than the client is in; in such a situation the server should
convert to canonical form iff requested. This is the position I see
simple (non-heuristic) Unix/Windows servers as being in.

Given the which, having no text-hint-attrib-bits support and yet having
SSH_FXF_TEXT support does make sense; it leaves you in the similar
situation to plain FTP (or SFTP before this extension) of the user
having to decide which are text files, which is IMO where you should be
(or at least have the option of being) with such servers.

That said, I wouldn't object to making SSH_FXF_TEXT support optional in
general.

---

From: "denis bider" <[email-masked]>
Date: Mon, 11 Oct 2004 04:47:56 +0200

Hi Jacob, all,

I think you are right on all counts.

I agree that attrib flags for text files are OK as proposed.

About server-side heuristics, I agree they would need to be optional, and if
they're optional the client cannot rely on their presence, so they wouldn't be
very useful in a general case. Also, a client supporting auto-mode transfers
would need to implement heuristics and conversion internally anyway for
interoperability with currently existing servers.

(Note - when I say a client supporting auto-mode transfers, I do not mean a
client not giving the user a choice. Auto mode is usually accompanied with text
and binary choices. However, I believe that a user will mostly prefer the
program to make the right decision for him if it can, and will choose auto-mode
most of the time.)

If the attrib flags go through though - and I hope they do - I would like to
propose this addition:

  Files not advertised by the server as textual in the
  attrib flags MUST NOT be opened with the SSH_FXF_TEXT flag.
  A server MAY fail textual open requests for files which
  were not advertised as textual in attrib-flags.

This simplifies the server significantly, and does not affect the client.

The assumption for this proposal is that on platforms where file type
(text/bin) is not determined (Unix, Windows), line endings are LF/CR or a
combination (CRLF, LFCR), and the client can convert from any of these formats
into its native format with a simple filter (interpret the first LF/CR as line
end, discard complementary CR/LF if follows - this algorithm could be mentioned
as a good solution in the spec BTW).

My above proposal is ill-conceived only if there is a platform where:
* file type (text/bin) is not determined, and
* line endings are not one of LF/CR or a combination (CRLF, LFCR).

If anyone is aware of such a platform, please let me know. If there isn't one
though, I suggest the server shouldn't have to support SSH_FXF_TEXT if file
type isn't determined. The client can read binary and convert line endings if
it so pleases.

denis

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Jacob Nevins
> Sent: Monday, October 11, 2004 00:56
> To: [email-masked]
> Subject: Re: Text file type hint proposal for filexfer
> 
> 
> > The idea is, let's be consistent, and let's not specify 
> things halfway
> > through.
>

---

From: Albert Lunde <[email-masked]>
Date: Mon, 11 Oct 2004 09:02:07 -0400

> type isn't determined. The client can read binary and convert line endings if

If your are looking for cases that are unlike Unix semantics, I'd
suggest looking at the IBM mainframe operating systems: CMS,
TSO, MVS, etc.  One of the cases that appears there is fixed-length
records with no delimiters in the binary representation.
This may be blocked (though that's more common on tape devices).

I think there are also variable-length records with a none-of-
the-above line representation.

There are various file attributes, but I wouldn't promise
that there are no cases where the file format is ambiguous.
(And of course, the default encoding was EBCDIC.)
(I, thankfully, haven't had to work on those systems for several
years, so I can't provide too many details.)

---

From: "denis bider" <[email-masked]>
Date: Mon, 11 Oct 2004 23:18:59 +0200

I guess then it would be wise to shorten my proposed addition and remove this
part:

And leave only this:

I.e. make it clear that the server only NEEDS to support SSH_FXF_TEXT for files
it advertises as textual in attrib-flags. For files of unknown text/bin
disposition, the server MAY support SSH_FXF_TEXT, but does not need to...

> [mailto:[email-masked]] On Behalf Of Albert Lunde
> Sent: Monday, October 11, 2004 15:02
> To: [email-masked]