Subject: SFTP and unicode file names...

From: Joseph Galbraith <[email-masked]>
Date: Tue, 05 Oct 2004 10:48:02 -0600

Okay, I've been forced to face up to the unfortunate truth:

Unix directories can contain files encoded in multiple
different char-sets and the server has no way to tell
what these multiple char-sets are and translate them
to UTF-8.  Because the transformation is one way, once
the server has mistranslated the filename, there
is no way for the client to get back to the original
data.

So, for these file-systems, the best possible thing
to do is send the filename raw and let the client
(with help from the user decode it.)

On the other hand, maximum possible interoperability
between different language and regions is obtained
through use of UTF-8 where available.

I haven't been able to come up with a solution I
really like.

Here are some possibilities:

1. Let the server say what it is going to use,
    UTF-8 or 'undefined-raw' at the beginning
    of the sftp session.

    pro: simple.  really simple.
    con: Doesn't address cases where the server
         might be able to do utf-8 for some file
         systems (ext-3 under fedora, I think is
         one example) but not others.

2. Allow the filename to be prefixed by a flag
    byte saying it is raw.  For example, the
    byte 0xFF is an invalid UTF-8 lead byte.
    If the first byte of the filename is 0xFF,
    then the 0xFF is discarded, and the rest
    of string is the 'raw, undefined' filename
    data.

    pro: It handles the real life complexity of
         being able to tell sometimes, but not
         others.
    con: It is a little more complex, and a bit
         icky.

3. Give the filename structure.  Filenames are
    always specified in the following structure:

    uint32 length of the structure
    boolean utf-8
    byte   filename[length-1]

    pro: This also handles being able to give
         UTF-8 sometimes, but not all the time.
    pro: This isn't icky.
    con: This is a bit more complex.

4. Use the high order bit of the length field
    to flag raw more.  In practice, no file name
    will ever be more than 2 gig long :-)  We
    can safely borrow that bit for other purposes.

    pro: This also handles being able to give
         UTF-8 sometimes, but not all the time.
    con: Slightly icky, a little more complex.

Can anyone think of a better solution?

I think I prefer solution three, I could live
with 2 or 4; I'd really rather not go with 1.

What do others think?

Thanks,

Joseph

---

From: Jacob Nevins <[email-masked]>
Date: Thu, 7 Oct 2004 00:20:35 +0100

Joseph Galbraith <[email-masked]> writes:
> Unix directories can contain files encoded in multiple different
> char-sets and the server has no way to tell what these multiple
> char-sets are and translate them to UTF-8.  Because the transformation
> is one way, once the server has mistranslated the filename, there is
> no way for the client to get back to the original data.
> 
> So, for these file-systems, the best possible thing to do is send the
> filename raw and let the client (with help from the user decode it.)

I support this. While it would be nice for all filenames to be in UTF-8
or reliably convertible to/from it, it is beyond the power of this
working group to cause this to happen, so this requirement simply won't
be implemented.

(We've pinched stuff from NFS before. Do they have any views on this
issue?)

> On the other hand, maximum possible interoperability between different
> language and regions is obtained through use of UTF-8 where available.
> 
> I haven't been able to come up with a solution I really like.
> 
> 1. Let the server say what it is going to use, UTF-8 or
>     'undefined-raw' at the beginning of the sftp session.
> 
>     con: Doesn't address cases where the server might be able to do
>          utf-8 for some file systems (ext-3 under fedora, I think is

It's true that this is rather limiting. But if your directory tree has
heterogenous UTF-8-ness, don't you run into problems with having some
path components "raw" and others UTF-8? Given our approach of paths
being rather opaque things, I can't think of a way round this.

Therefore, I suspect that something like this simple approach may have
to be taken...

> 2. Allow the filename to be prefixed by a flag byte saying it is raw.
>     For example, the byte 0xFF is an invalid UTF-8 lead byte. If the
>     first byte of the filename is 0xFF, then the 0xFF is discarded,
>     and the rest of string is the 'raw, undefined' filename data.
> 
>     pro: It handles the real life complexity of being able to tell
>          sometimes, but not others.
>     con: It is a little more complex, and a bit icky.
> 
> 3. Give the filename structure.  Filenames are always specified in the
>     following structure:
> 
>     pro: This also handles being able to give UTF-8 sometimes, but not
>          all the time.
>     pro: This isn't icky.

In particular, backwards-compatible implementations now need two sets of
filename-handling code.

> 4. Use the high order bit of the length field to flag raw more.  In
>     practice, no file name will ever be more than 2 gig long :-)  We
>     pro: This also handles being able to give UTF-8 sometimes, but not
>          all the time.
> I think I prefer solution three, I could live with 2 or 4; I'd really
> rather not go with 1.

I think I slightly prefer 2 to 4, if the issue I mention above can be
got round.

---

From: [email-masked] (Niels Möller)
Date: 08 Oct 2004 13:28:45 +0200

>          one example) but not others.

I'd much prefer the simple way. Just let the server say what character
set it uses, and do no conversions at the server end (a server that
really knows what it is doing MAY handle a file system with mixed
character sets by converting them to and from utf-8, and use utf-8
exclusively on the wire, but I don't think that is very practical).
Some possible values are "unknown", "usascii", "iso-8859-1", "utf-8",
and "utf-8-with-normalization-form-c".

(Note that utf-8 with undefined normalization is suboptimal for
filenames and for identifiers in general).

This approach should work fine for systems that uses utf-8
consistently in the filesystem (I think windows-ce does that, not sure
about more regular windows versions).

I think the typical unix filesystem uses usascii for all "system
files", and then has some home directories with iso-8859-x names, some
with utf-8 names, which all include usascii. (I admit that I don't
have any experience with asian unix installations). Then it will work
reasonably to set the advertised charset from the user's $LC_CTYPE.

There will naturally be some difficulties if different parts of the
filesystem tree uses different character sets, or if there are single
pathnames that use components (directory names) with different
character sets. However, those difficulties are precisely the same
with *local* access to the file system, so it makes absolutely no
sense to try to solve them in the sftp spec.

Best regards,
/Niels

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 08 Oct 2004 12:05:30 -0400

> and "utf-8-with-normalization-form-c".

So far, so good.  Note that the appropriate set of possible values is that 
contained in the IANA Character Set registry 
(http://www.iana.org/assignments/character-sets).  When a character set in 
that registry has more than one name or alias, the alias designated as 
"preferred MIME name" SHOULD be used.

But see below...

> filenames and for identifiers in general).

Yes.  I'd normally argue that in any case where UTF-8 will be used on the 
wire, it probably ought to be normalized.  However, the key issue here is 
not really normalization of names sent by the server but of those sent by 
the client, which refer to the names in the filesystem.  Unfortunately, 
there are filesystems that use unnormalized Unicode, so the server would be 
required to do path processing itself, normalizing all the names in each 
directory until it found one matching what the user requested.  That's 
going to get unwieldy real fast, I think.

> reasonably to set the advertised charset from the user's $LC_CTYPE.

I would expect this to be more or less the case.  In fact, I'd expect that 
on most systems, most users will be using the _same_ character set.

> sense to try to solve them in the sftp spec.

I agree -- solving that problem is beyond our ability, and I do not think 
it is worth the effort to try.

There is one thing I'd do differently.  Whenever possible, I'd prefer that 
the server not just advertise the character set but actually do the 
conversions.  The theory here is that the server is likely to have at its 
disposal the tools needed to convert between Unicode and any other 
character set in use on the server.  The _client_, on the other hand, can 
only reasonably be expected to have those tools for character sets in use 
on the client.  When these sets are disjoint, interoperability demands the 
use on the wire of a common character set (i.e. Unicode).

Let the server advertise the character set it thinks is in use.
Let the client decide whether it wants UTF-8 or raw bytes.

-- Jeff

---

From: [email-masked] (Niels Möller)
Date: 08 Oct 2004 19:46:30 +0200

Jeffrey Hutzelman <[email-masked]> writes:

> Yes.  I'd normally argue that in any case where UTF-8 will be used on
> the wire, it probably ought to be normalized.  However, the key issue
> here is not really normalization of names sent by the server but of
> those sent by the client, which refer to the names in the
> filesystem.

There's actually one important use case where normalization doesn't
matter. That is if the client first requests a directory listing, and
then presents those names to the user, via some graphical file dialog,
or via TAB-completion. In this case, the only important thing is that
the client doesn't modify the coding when a name is copied from the
directory listing to the open request.

> Unfortunately, there are filesystems that use unnormalized Unicode,

That's going to be painful. Examples?

> There is one thing I'd do differently.  Whenever possible, I'd prefer
> that the server not just advertise the character set but actually do
> the conversions. [...]

> Let the client decide whether it wants UTF-8 or raw bytes.

Good idea. I agree it makes sense to give clients that choice.

I'm not so sure I share your preference, though; I'm afraid subtle
problems may occur when file names are converted to utf-8, processed
by client ui code, and back.

One interesting case: When converting from the local charset to utf-8,
it is natural to require that the server uses normalized utf-8, right?
(With canonical normalization, not compatibility normalization).

However, if the server's local character set happens to be utf-8, and
some file names are unnormalized, then having the server normalize the
data for transmission is likely to break things.

Regards,
/Niels

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 08 Oct 2004 14:21:37 -0400

> directory listing to the open request.

Correct.

> That's going to be painful. Examples?

I don't believe Windows normalizes anything.

> data for transmission is likely to break things.

Yes.  If the server's local character set is UTF-8, it SHOULD NOT 
normalize.  It it's something else, then the server should convert to UTF-8 
using whatever rules it deems appropriate, and should accept any reasonable 
representation from the client (particularly, it SHOULD NOT require 
normalized input).

---

From: Joseph Galbraith <[email-masked]>
Date: Sat, 09 Oct 2004 16:48:22 -0600

Jeffrey Hutzelman wrote:
> So far, so good.  Note that the appropriate set of possible values is 
> that contained in the IANA Character Set registry 
> (http://www.iana.org/assignments/character-sets).  When a character set 
> in that registry has more than one name or alias, the alias designated 
> as "preferred MIME name" SHOULD be used.
> 
> But see below...
> 
> 
> 
> those sent by the client, which refer to the names in the filesystem.  
> Unfortunately, there are filesystems that use unnormalized Unicode, so 
> the server would be required to do path processing itself, normalizing 
> all the names in each directory until it found one matching what the 
> user requested.  That's going to get unwieldy real fast, I think.
> 
> 
> I would expect this to be more or less the case.  In fact, I'd expect 
> that on most systems, most users will be using the _same_ character set.
> 
> 
> I agree -- solving that problem is beyond our ability, and I do not 
> think it is worth the effort to try.
> 
> 
> 
> that the server not just advertise the character set but actually do the 
> conversions.  The theory here is that the server is likely to have at 
> its disposal the tools needed to convert between Unicode and any other 
> character set in use on the server.  The _client_, on the other hand, 
> can only reasonably be expected to have those tools for character sets 
> in use on the client.  When these sets are disjoint, interoperability 
> demands the use on the wire of a common character set (i.e. Unicode).

I would definitely prefer to see the server do the translation
when it can... that's why we went to UTF-8 in the first place.
If the server knows the charset, it should just do the conversion.
For example, Windows systems don't know about EUC-JIS (for Japanese
encoding), they use an alternate encoding call Shift-JIS.  So the server
can easily translate from EUC-JIS to UTF-8, using built-in os support
(if it is using EUC-JIS anyway.)  There has to be custom code written
on the client side though to do the conversion.  That is yucky.  So
it really is better for the server to do the translation.

The problem is if the server doesn't (really) know the charset,
and tries to do the conversion anyway (as would happen, for
example, using the users LC_TYPE).  In this case, the conversion
loses data, and can not be reversed.  This means that the client
not only can't display a meaningful filename, but it can't even
open the file by sending back the garbage name (because the server
can't reverse the conversion.)

So, what I'd like is a way for the server to convert what it
can, and to give the client something that can be used to open
files even if it can't convert them.

So... how about the following--

1. A extension the server sends to advise of the charset
    it thinks is most likely in use (probably from the
    users LC_CTYPE under unix, windows will probably just
    always send UTF-8.)

    By the way, how does one get a charset out of the iana
    registry from the users NLS environment under most
    unixes?  Is it a simple thing to do?

2. An extension the client can use to control the translation
    on the server; something like translation-mask:

     SSH_TRANSLATE_KNOWN         0x00000001
                Translate files with a known charset.  If the
                translation fails, use the rawname instead.

     SSH_TRANSLATE_GUESSED       0x00000002
                Translate files if the server has a reasonable
                idea what the charset is, but doesn't know.

                (For example, using the users LC_CTYPE.)

                If the translation fails (an invalid character
                for example) the server should use the rawname
                instead.

     SSH_TRANSLATE_INCLUDE_KNOWN_RAW    0x0000004
                Include the rawname as an a part of the attrib
                if charset was known, and translation was successful.
                (If translation was unsuccessful, the rawname is
                sent as the name.)

                (New attribute field SSH_FILEXFER_ATTR_RAW_NAME.)

     SSH_TRANSLATE_INCLUDE_GUESSED_RAW    0x0000004
                Include the rawname as an a part of the attrib
                if charset was guessed, and translation was successful.

     The initial mode is SSH_TRANSLATE_KNOWN|SSH_TRANSLATE_GUESSED.

3. Allow the server and the client to prepend a 0xFF to any
    filename to indicate it should be passed down to the system
    APIs without further processing.

The thing this does is allow most clients to continue
to operate as they do under sftp v4 & 5, except that
if the translation fails, behavior is well defined
and there is a way to open the file, even if the
filename can not be displayed correctly.

If a user runs into trouble, they can set INCLUDE_GUESSED_RAW,
or turn off TRANSLATE_GUESSED.

Okay... after all that, I've just realized that the per-file
tag for rawmode is more complicated than I thought-- it has
to be per path component.  Rats.

I really wanted to be able to rely on the server as much
as possible and only punt the exceptions to the client
for processing... and the client wouldn't really process
them, because it would know it didn't know the charset
(cause if it was the one the server thought it was,
it would have worked on the server side.)  It would just
use them as opaque handles to at the file contents--
or the user would have to manually intervene to display
the filename.

Argh... well, I'm going to go off an think now.

- Joseph

---

From: der Mouse <[email-masked]>
Date: Sun, 10 Oct 2004 01:41:06 -0400 (EDT)

>>> [...filenames...UTF-8...character set...conversions...]

I see a philosophical problem here that I suspect is at the root of
many of these difficulties.

What are file names?

On most Unix variants, file names are opaque octet sequences (with 0x00
octets not permitted and 0x2f octets permitted only as component
separators).  (Some very old variants also forbid octets 0x80-0xff.)

Note that they are not character sequences.  Interpreting those octets
as characters is something done only for human-interaction purposes and
is done using whatever character set the display device is using.
Thus, even _contemplating_ doing character set conversions anywhere is,
from a Unix perspective, a grave mistake.

However, other systems complicate matters.  I have seen it said (with
what truth I know not - I am no Windows geek) that Windows filenames
are sequences of Unicode characters, or more precisely sequences of
Unicode codepoints, from which perspective character set conversions
are perfectly reasonable.

I see no good way to cater to both philosophies other than either to
have a bit indicating whether a given filename is an octet sequence or
(an encoding of) a character sequence, or perhaps to have a character
set indicator but with a special value reserved to indicate "raw octet
sequence".

Thoughts?  Agree?  Disagree?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: [email-masked]
Date: Mon, 11 Oct 2004 11:33:02 +0300 (EEST)

> through use of UTF-8 where available.

User can use cat/type/ls/dir <FILENAME> to access a local file and it
should be able to do same with SFTP.
As example "echo 'get <FILENAME>' | sftp localhost" should get same file.
In all cases <FILENAME> should be same and is encoded in user
charmap(charset/codeset/etc.).

> [SNIP]

I'm not sure that server is responsible to do decision for encoding:
'UTF-8' or 'RAW'.
Since in a SFTP session client can request more that one file negotiation
of file name encoding should be at begining of session.
Client should request encoding from server.

I guest that a new extension "encoding" will solve problem:
1.) Client send to server list of accepted encodings and server return
prefered one or "RAW" or "UTF-8".

To do this sftp implementation MUST implement extension "encoding".
Extension should be defined in draft as "newline" is defined.

I not sure that sftp can use names like "ascii", "usascii", "C", "POSIX",
"ANSI_X3.4....", since ascii define only 7 bit charset. When SFTP server
support 7-bit encoding is should(must?) reject file names containing
symbols with code greater that 127.

When encoding is not set server should treat filenames in "raw" or "utf-8"
format.
This must be annonced in "Server Initialization".
Empty "encoding" is alias to "RAW".
When encoding is set server must convert "local filename in encoding" <->
"wire filename in utf-8".
Client may convert "wire filename in utf-8" <-> "local filename in
encoding". Note that client know name conversion on the server.

I guess that this solution is interoperable with SFTP clients version 1, 2
or 3.
For version 4(four) clients, when server support encodung it should
announce 3(three) as maximum version.
For client version 1,2,3 server must use "RAW".
Server version N(N>=5) must support "UTF-8", "RAW" and "ISO8859-1" encodings.
Server version N(N>=5) may support "ISO8859-N" encodings, where N is in
range 2-15.
Client version N(N>=5) must support the "UTF-8" and "RAW" "encoding"
extension.

P.S.:
As esample cyrillic use many encodings. Most popular are IS08859-5,
KOI8-R, CP1251.
In case of cyrillic one utf-8 file name in cyrillic can address different
files on file system and this depend of encoding.
In this case SFTP client with help from the user is responsible to select
correct encoding.
This is same case as access to local file system.

I don't have problem to adress correct file name on remote host.
My system is properly setup and I can use UTF-8, IS08859-5, KOI8-R and
CP1251 in file/directory names and in GUI termininals.
For the test in directory $HOME/tmp/cyr I have four files with name in
format f.<ENCODING>.<NAME>,
where <ENCODING> is one of mentioned above and
<NAME> is first three leter from cyrillic alphabet in uppercase followed
by same leters in lovercase in same encoding.
Content of each file is "data.<ENCODING>:<NAME>" where <ENCODING> and
<NAME> match the file name.
In four xterm for every encoding I run same command sequence.
Results are attached images in "ssh_session.UTF-8.png",
"ssh_session.ISO8859-5.png", "ssh_session.KOI8-R.png"
and "ssh_session.CP1251.png".
With command "echo get tmp/cyr/f.*.<NAME> | sftp localhost", where sftp is
openssh SFTP version 3 client, file that I get depend of my locale
charmap.

Regards,
Roumen

---

From: [email-masked] (Niels Möller)
Date: 11 Oct 2004 16:54:06 +0200

> when it can... that's why we went to UTF-8 in the first place.

I think there's one more use case that you need to consider, which I
expect is quite common:

The remote filesystem using the foo charset. The local system using
the same foo charset. Why do I think this is common? Because on both
sides, it's the same user's files, and the user is likely to use his
or her favourite charset (iso-8859-1, utf-8, euc-jis, whatever) on
most or all systems where he or she has an account.

What you call "raw mode" will work fine in this case, no matter if the
sftp implementation on server or client side knows about the foo
charset.

I like Jeffrey Hutzelman's proposal: Have two modes of operation, and
let the client select which mode it prefers,

 1. Server tells client the server's best guess as to what character
    set is used for filenames, and doesn't convert filenames in any
    way.

 2. All filenames on the wire are utf-8. Server converts filenames to
    and from utf-8 on a best effort basis, according to it's best
    guess of the actual charset. (What's the right thing to do if/when
    conversion fails, I don't know yet).

In both cases, server can tell client what charset is used on the
server side (or "unknown") at startup. Client can select mode of
operation either as a global protocol state, or a per request-flag. I
don't think I care very much if the mode selection is global or per
request; I'd expect most clients to choose one mode and stick to that.

The reasons I like this are:

 * It is fairly simple.

 * The client has the final say on whether or not the server should do
   any conversion.

 * The use case above is easily supported, without requiring any
   special server (or client) support for the obscure foo charset.

 * I don't see any use cases where the above scheme fails, and a more
   complicated scheme could do better.

Regards,
/Niels

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 11 Oct 2004 11:52:41 -0400

>     conversion fails, I don't know yet).

IMHO it is important that the server identify its character set up front, 
so that the client is able to use that information in deciding which mode 
to use.  Also, the server needs to be able to indicate that it is incapable 
of performing Unicode conversion.  I'd like to be able to say that the 
server MUST be capable of performing the conversion, but I don't think 
that's realistic.

If the server is capable of doing conversion, then conversion from the 
local character set to unicode should not be able to fail.  If it does, 
something is very wrong.

I can think of only two cases in which the conversion from UTF-8 to the 
local character set can fail.  The first is when the input is somehow 
invalid (bad UTF-8, contains illegal Unicode code points, etc).  We could 
handle this in a number of ways, up to and including terminating the 
connection. :-)

The second case is when the input is valid, but contains characters not 
present in the local character set.  The server's mapping should be good 
enough to handle cases where the same local character can be represented in 
multiple ways in unicode (for example, there are at least two ways to write 
Å in unicode, but only one way in iso-8859-1).  For characters which are 
genuinely not in the local character set, the server can return an 
appropriate error depending on context.  For example, trying to open a file 
whose name contains an untranslateable character will always fail with 
something ENOENT-like.  Trying to create such a file should fail with an 
illegal filename.

-- Jeff

---

From: der Mouse <[email-masked]>
Date: Mon, 11 Oct 2004 12:39:41 -0400 (EDT)

> IMHO it is important that the server identify its character set up
> front, [...]

What about cases where there is no well-defined character set, such as
most Unix variants?  (Suppose I have a file whose name consists of the
octets 0xc3 0xa1 0xd0 0xbf.  Is that a file named LATIN SMALL LETTER A
WITH ACUTE - CYRILLIC SMALL LETTER PE (UTF-8), a file named LATIN
CAPTIAL LETTER A WITH TILDE - INVERTED EXCLAMATION MARK - LATIN CAPITAL
LETTER EDH - INVERTED QUESTION MARK (8859-1[%]), a file named GREEK
CAPITAL LETTER GAMMA - some character whose name I'm not sure of -
GREEK CAPITAL LETTER PI - GREEK CAPITAL LETTER OMEGA WITH TONOS
(8859-7[%]), a file whose name represents the integer 3173013909 rather
than any character sequence (base 254 with digits 0x01-0x2e and
0x30-0xff), or what?  (Given how nonsensical the others are, the last
of those may actually be the most plausible.)

Nobody but the entities (software and/or humans) the file was created
for and used by can tell.  Certainly ssh/sshd can't, not without being
told somehow.

Anything predicated on the assumption that every filename consists of
characters will run into this problem.  I strongly believe there needs
to be some provision for filenames to be treated as opaque octet
sequences rather than insisting that they are character sequences.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

[%] Names taken from Unicode 0080-00FF and 0370-03FF; I don't have a
    handy reference to the names 8859 uses.

---

From: [email-masked] (Niels Möller)
Date: 11 Oct 2004 22:23:17 +0200

> front, so that the client is able to use that information in deciding
> which mode to use.

Agree.

> Also, the server needs to be able to indicate that it is incapable
> of performing Unicode conversion. I'd like to be able to say that
> the server MUST be capable of performing the conversion, but I don't
> think that's realistic.

The server can be incapable of doing a proper conversion in at least
two ways: Either the server has no idea whatsoever which charset to
use to interpret its local filenames. Or it knows somehow that it
should use a prticular characterset (say, euc-jis), but it hasn't been
compiled with support for that conversion.

I don't think it is possible to forbid the first failure mode; we must
allow the server to say charset "unknown" or "" in this case.

We could forbid the second case, in effect forcing the server to say
"unknown" when it in fact knows the character set, but isn't able to
convert it. But I don't think it is a good idea to do that; it is
better to at least tell the client what the charset is.

> I can think of only two cases in which the conversion from UTF-8 to
> the local character set can fail.

I don't think conversion utf-8 -> local is a big problem. Some
filenames can't be represented, and they must simply be treated as
non-existing files. And invalid utf-8 should be a protocol error
(under no circumstances should an implementation be allowed to say
that it's using utf-8, and then send invalid utf-8 filenames).

Conversion the other way is more difficult. The problem is, when the
server thinks it should convert from A to utf-8 (based, for example,
on examining the user's $LC_CTYPE), but in fact some file names are
stored using a different charset B, where B sequences are invalid when
interpreted as A-characters. Only example I can come up with off the
top of my head is A = utf-8 and B = latin1, but I'm pretty sure there
are some other examples where neither A nor B is utf-8.

I could think of some ways to deal with this: (i) pretend the files
don't exist, or (ii) replace unexpected characters or character
sequences with uniquely chosen private use characters. I think (i)
will be a very confusing failure mode for users, and (ii) seems quite
ugly.

Telling the client "sorry, here's some strange filename I can't
convert to utf-8, you can try again with raw filenames, if you like"
seems simpler to udnerstand. Note that this can happen even (or
perhaps it's even more likely to happen) when the local character set
is believed to be utf-8, and some file names violate this assumption.

Regards,
/Niels

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 11 Oct 2004 17:01:55 -0400

Agree.

Agree.

> is believed to be utf-8, and some file names violate this assumption.

I think it is more likely that the local character set is believed to be 
UTF-8 and actually is not, than that we will believe the local character 
set to be something "flat" like iso-8859-1 and then find characters which 
are invalid in that set.

In any case, I agree the behaviour you describe seems to be the most sane 
thing to do in this situation.

FWIW, it is worth noting that it is possible to tell with fairly high 
probability whether a particular string is UTF-8 or some flat 8-bit 
character set, because valid UTF-8 is fairly well structured and the 
sequences which represent non-7-bit characters are fairly unlikely to occur 
in flat character sets.  So when conversion to UTF-8 is being done by the 
server, it is possible for an implementation to apply a heuristic to allow 
support of filesystems containing both UTF-8 and an 8-bit character set, 
even in separate components of the same pathname.

-- Jeff

---

From: [email-masked]
Date: Tue, 12 Oct 2004 09:21:41 +0300 (EEST)

> server, it is possible for an implementation to apply a heuristic to allow

heuristic ???
This is inpossible.
As example a valid cyrillic name in UTF-8 can be converted into valid name
in more that one charset (ISO-8859-5, KOI8-R, CP1251).
Client with help from user should tell server how to convert "utf-8" <->
8-bit character set, i.e. prefered charset for filename on server(remote
host).

---

From: [email-masked] (Niels Möller)
Date: 12 Oct 2004 15:03:52 +0200

Jeffrey Hutzelman <[email-masked]> writes:

> I think it is more likely that the local character set is believed to
> be UTF-8 and actually is not, than that we will believe the local
> character set to be something "flat" like iso-8859-1 and then find
> characters which are invalid in that set.

I agree, with the reservation that I don't have much experience with
character sets that use eight-bit characters, and lots of different
shift states.

> character set,

That may be correct (at least that's an advertised property of utf-8),
but as far as I can see, that doesn't help much unless you have some
clue about *which* flat 8-bit character set was used. If you know that
a file system contains mixed latin-1 and utf-8 filenames (and no other
character sets), you can probably make the heuristics work most of the
time, but if you have a filesystem with mixed utf-8, iso-8859-1,
iso-8859-5 and koi8, it gets rather difficult.

I don't think heuristic decoding is a good idea. One problem is that
you really need to be able to convert both ways:

Say you have a directory containing filenames in utf-8 and iso-8859-1.
The client asks for a directory listing, and you give it a utf-8
listing, using some working heuristic for converting the latin-1 names
to utf-8 on the fly. Next, the user selects one file using the
client's ui, and the client tries to open it. Now the server has to
figure out if the filename to be opened should be converted to latin-1
or not (in the worst case, there exists files with both the utf-8 name
and the corresponding latin1 name). I think this gets ugly.

The hack I mentioned, to replace the sequences that are invalid utf8
with uniquely chosen private use characters, have a better chance of
surviving the roundtrip to the client, but I'm sure that has other
problems.

Regards,
/Niels

---

From: "denis bider" <[email-masked]>
Date: Sat, 23 Oct 2004 04:00:58 +0200

Hello Joseph,

any information on when the SFTP v6 draft will be available?

I will be starting implementation of a new module soon and my design is
pending availability of the SFTP v6 specification.

What is the current progress? When can the v6 draft be anticipated?

Thanks!

denis

---

From: Joseph Galbraith <[email-masked]>
Date: Mon, 25 Oct 2004 13:02:20 -0600

I made the deadline (sent to IETF at 3:00 AM this
morning my time.)

I put a big ** DO NOT IMPLEMENT ** in it though.  My
plan is to spend some time @ IETF and to publish a
revision on the monday after we meet.

Here are the changes (a rough list anyway):

- Added NOFOLLOW option.
- Made TEXT_HINT an attrib of it's own.  Servers that 'know'
   might include this information as part of the directory
   read; if the server is going to have to do lots of work to
   figure it out, it should wait for a stat request expressing
   interest in the data.
- Added 'filename-charset' and 'filename-translation-control'
   extensions (which still need some polishing in the language.)

   Also added the ability for the server to set an attrib-bits flag
   if the filename translation to UTF-8 failed, and to include the
   untranslated name in the attributes.
- Add 'versions' extension to allow server to express non-contigous
   set of supported versions to client, and 'version' extension
   from client-to-server for server to pick one.  (Because the
   client to server request isn't allowed to fail w/o closing
   the channel, it doesn't require a round trip.  The client
   can fire it and move on without waiting.)
- Added link-count attribute and mime-type attribute.
- Added space available extension.
- Added DIR_NOT_EMPTY, NOT_A_DIRECTORY, INVALID_FILENAME and
   LINK_LOOP error codes.  Please, everyone, look at the list
   of error codes and suggest errors codes that we should have
   but aren't there.  If something can go wrong on your OS and
   we can't represent it, now is the time to have a dialog about
   it.

So... draft should appear shortly, and we WILL HAVE another draft
on Monday after the IETF meets.

(I'll be available for anyone interested in helping with edits or
chatting about the draft from Monday evening through Tuesday.)

Thanks,

Joseph

---

From: jason bailey <[email-masked]>
Date: Tue, 26 Oct 2004 11:31:47 -0400

Would it be within scope to suggest that a future version of the SFTP
server provide a signed receipt of the file transfer(on request)?

This feature, would do much to alleviate a lot of our issues with secure
file transfers.

thanks

-Jason

---

From: Jon Bright <[email-masked]>
Date: Wed, 27 Oct 2004 08:16:02 +0200

Hi,

> This feature, would do much to alleviate a lot of our issues with secure

I'm not in a position to answer your question wrt. scope, but I'm 
curious as to what exactly the signature would signify?  That the server 
has definitely received the file at a given time?

-- 
Jon Bright
Silicon Circus Ltd.
http://www.siliconcircus.com

---

From: jason bailey <[email-masked]>
Date: Wed, 27 Oct 2004 08:09:40 -0400

Exactly. 

A receipt provides proof that a given file was delivered at a specific
time.

Receipts usually contains information like file name, a
fingerprint(hash), and a time stamp which is then signed by the servers
private key. It's part of a non-repudiation process.

To give an example. I deal with the transfer of data from my companies
production network to a variety of third party organizations. For the
majority of file transfers SFTP fits my needs. However there are
occasions where the data is sensitive enough, or we have some sort of
time based contractual obligation, where it is necessary for us to get a
receipt of transfer.

Right now my options are to go with some heavy weight B2B protocol which
requires a team of engineers and an architect to get set up properly. Or
implement our own solution, which inevitably results in placing some
form of server (of our devising) on another companies network. This
tends to make their network people very antsy.

If I could get a receipt capability into a standard like SFTP. It would
allow me promote SFTP over all the other options. It would make my
management happy, and would make my company dealings with other
companies a lot smoother.

Jason

---

From: "denis bider" <[email-masked]>
Date: Wed, 27 Oct 2004 15:54:03 +0200

Technically speaking, an extension to the SFTP protocol could be implemented
in which the server would testify (with a digital signature) that a file of
a certain name, attributes and contents existed on the server at a certain
time.

Whether or not this file was uploaded entirely by a certain user would be a
more complex challenge because SFTP has no such concept as "uploading" or
"downloading" a whole file. You have random access and you can pretty much
scratch anywhere you want in the remote filesystem.

An extension that would produce a certificate of a file's existence at a
certain time would be fairly straightforward. Provide an extension request
type for requesting the certificate, and define the contents of a receipt.

If providing a certificate of the file's existence on the server is
insufficient, and you must really provide a receipt which includes
information about the act of uploading, this could be done, too. For
example, a file for which you require an upload receipt must be opened with
a certain flag or set of flags which signal that you're going to do
receipted-uploading. When you open the file, you are allowed to append to
the file only (like uploading in TEXT mode). When you close it, the server
sends a RECEIPT message rather than STATUS. The format of the RECEIPT
message is what needs to be defined.

Whether or not this is something for a separate Internet-Draft (documenting
the SFTP extension) or something that can be added to SFTP itself as an
optional feature is, I guess, up for the workgroup or the SFTP draft editor
to decide. In my view, the first solution type (certificate of file's
existence) would be more apt for a separate draft because it is fairly
orthogonal to existing functionality. The second solution type (certificate
of the upload act with special flags for opening the file) might be better
documented in SFTP itself because of the flag's definition.

If the 2nd solution (upload receipt) is required, a question to answer is:
what if the user needs to upload an extremely large file and the connection
breaks? Should the "upload receipt" process allow for a file to be resumed?
If so, how do you know that the file being resumed is one that was actually
uploaded in a previous session by the same user, and that it wasn't lying
there on the server belonging to someone else?

denis

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of jason bailey
> Sent: Wednesday, October 27, 2004 14:10
> To: Jon Bright
> Cc: [email-masked]
> Subject: Re: future SFTP version question
> 
> 
> Exactly. 
> 
> A receipt provides proof that a given file was delivered at a 
> specific time.
> 
> fingerprint(hash), and a time stamp which is then signed by 
> the servers private key. It's part of a non-repudiation process.
> 
> To give an example. I deal with the transfer of data from my 
> companies production network to a variety of third party 
> organizations. For the majority of file transfers SFTP fits 
> my needs. However there are occasions where the data is 
> sensitive enough, or we have some sort of time based 
> contractual obligation, where it is necessary for us to get a 
> receipt of transfer.
> 
> Right now my options are to go with some heavy weight B2B 
> protocol which requires a team of engineers and an architect 
> to get set up properly. Or implement our own solution, which 
> inevitably results in placing some form of server (of our 
> devising) on another companies network. This tends to make 
> their network people very antsy.
> 
> If I could get a receipt capability into a standard like 
> SFTP. It would allow me promote SFTP over all the other 
> options. It would make my management happy, and would make my 
> company dealings with other companies a lot smoother.
> 
> Jason
> 
> 
> > > Would it be within scope to suggest that a future version of the 
> > > SFTP server provide a signed receipt of the file transfer(on 
> > > request)?
> > > 
> > > This feature, would do much to alleviate a lot of our issues with 
> > > secure file transfers.
> >

---

From: [email-masked] (Peter Gutmann)
Date: Thu, 28 Oct 2004 03:24:28 +1300

"denis bider" <[email-masked]> writes:

>to decide.

If you're going to produce signed receipts as proof-of-delivery, you're well
into S/MIME / PGP territory.  This seems to be going way beyond what SSH
should be doing (depending on how far you want to take this you'd need to
reinvent significant chunks of PGP or S/MIME), it'd be better to just define a
signed content type for one of those formats and use that.

Peter.

---

From: Jon Bright <[email-masked]>
Date: Wed, 27 Oct 2004 17:16:39 +0200

> signed content type for one of those formats and use that.

I agree that the spec for the signature itself belongs elsewhere - but I 
can see an argument for building the ability to ask the server to 
produce such a signature into SFTP.  The signature standard need only be 
incorporated by reference.

As I read it, Jason's only requirement is for the server to sign to say 
"file X (hash Y) was on this server at time T".  This could use 
signature schemes already in existence, providing that SFTP were able to 
ask the server to produce the signature and transfer the produced 
signature back to the client.

---

From: jason bailey <[email-masked]>
Date: Wed, 27 Oct 2004 12:51:07 -0400

You've made some very good points.

Initially I had considered a flag for the transfer which would result in
a returned RECEIPT rather then a STATUS.

However, given your observations,and reviewing what I am looking for I
see no problem with it being an extension to the protocol. 

I'm not certain that it is, as you put it, orthogonal to the current
functionality. There already appears to be an existing extension to
support a hashing of a file to verify the contents.

In my mind I see this as very similar. We are merely verifying the
contents and that it is on a particular server at a given time.

Jason

---

From: Joseph Galbraith <[email-masked]>
Date: Wed, 27 Oct 2004 10:54:10 -0600

This seems like a useful extension; I've got half
of a proposal worked up... I just haven't quite
managed to finish it yet.

Hopefully I can post something to be shot down
in a day or so.

- Joseph
(sftp draft editor)

jason bailey wrote:
>>Technically speaking, an extension to the SFTP protocol could be
> 
> implemented
> 
>>in which the server would testify (with a digital signature) that a
> 
> file of
> 
>>a certain name, attributes and contents existed on the server at a
> 
> certain
> 
>>time.
>>
>>Whether or not this file was uploaded entirely by a certain user would
> 
> be a
> 
>>more complex challenge because SFTP has no such concept as "uploading"
> 
> or
> 
>>"downloading" a whole file. You have random access and you can pretty
> 
> much
> 
>>An extension that would produce a certificate of a file's existence at
> 
> a
> 
>>certain time would be fairly straightforward. Provide an extension
> 
> request
> 
>>type for requesting the certificate, and define the contents of a
> 
> receipt.
> 
>>example, a file for which you require an upload receipt must be opened
> 
> with
> 
>>receipted-uploading. When you open the file, you are allowed to append
> 
> to
> 
>>the file only (like uploading in TEXT mode). When you close it, the
> 
> server
> 
>>Whether or not this is something for a separate Internet-Draft
> 
> (documenting
> 
>>the SFTP extension) or something that can be added to SFTP itself as
> 
> an
> 
>>optional feature is, I guess, up for the workgroup or the SFTP draft
> 
> editor
> 
>>orthogonal to existing functionality. The second solution type
> 
> (certificate
> 
>>of the upload act with special flags for opening the file) might be
> 
> better
>

---

From: [email-masked] (Niels Möller)
Date: 28 Oct 2004 09:22:26 +0200

[email-masked] (Peter Gutmann) writes:

> should be doing [...]

I agree this seems to be beyond what standard sftp is supposed to do.
And I also don't see why sftp extensions are crucial for supporting
the given use case. One could use plain sftp (or *any* file transfer
mechanism, for that matter) and the following convention:

  * Client uploads the file "foo" into a particular directory or using some
    particular naming scheme.

  * When upload is complete (sftp close), the server processes the
    file using the signature mechanisms of its choice (pgp, s/mime,
    whatever), and writes a receipt as a new file "foo.receipt".

  * The client downloads "foo.receipt". Everyone is happy.

Adding extensions to sftp to do this have the potential advantage of
letting us standardize it, but I seriously doubt it's worth the
effort; it seems too obscure and specialized.

Regards,
/Niels

---

From: [email-masked] (Niels Möller)
Date: 28 Oct 2004 09:41:56 +0200

Joseph Galbraith <[email-masked]> writes:

>    from client-to-server for server to pick one.

As I said before, I really don't think this is the right way to go.

I can accept a "fancy version negotiation"-extension only if it is
specified in such a way that implementations that don't care about
disjunct version intervals (e.g. the common case of an implementation
supporting only version x, or supporting only version x and version
x+1) don't need to implement or otherwise care about the extension.

I view the fancy-version-negotioation thing only as a temporary kludge
needed for the unfortunate version 3 -> version 6+ upgrade. It will be
totally unneeded bloat and uglyness once this is over and the protocol
specification stabilizes. (I don't see any value whatsoever in fancy
version negotiation in the longer run. Simply advertising the highest
supported version number provides for adequate backwards compatibility
for everybody else's protocols, and I think it really ought to work
also for sftp).

And I don't think it is appropriate at all to consider things like
round-trip optimization for a temoprary kludge. Just solve the problem
at hand in a simple a way as possible. Both the problem and the
solution will be forgotten a few years from now.

Regards,
/Niels

---

From: "denis bider" <[email-masked]>
Date: Thu, 28 Oct 2004 10:42:41 +0200

Niels,

if the version extension scheme we're talking about is what I proposed, it
does seem to solve the problem at hand in the simplest complete way.

If in the future the 3->6 problem indeed becomes forgotten and obsolete,
clients will simply send "23" as their version (or whatever the highest
version exists at that time) and will bomb out if the server responds with
anything lower than "17". Fine.

If a client is concerned about the 3-6 problem though and wants to avoid the
chance that the server might reply to 6 with 4 or 5, I believe my proposal
is simple and complete.

If you don't care to implement it, it won't burden your client
implementation at all - you can continue to send the highest version you
support and that will work just as fine.

On the server side, this method will burden you only if you wish to offer
your highest version to clients which support non-contiguous version ranges.
But do you not wish your server to be well interoperable with such clients?
Is it not legitimate for clients to have a non-contiguous version
requirement?

Note that the solution has almost 0 impact on sessions where the client is
not one that supports non-contiguous version ranges.

I don't know why you label it as "fancy", because fancy is not what it is.
It just solves the problem with the smallest possible negative impact. The
only alternative with even less burden is to ignore the problem exists in
the first place.

I, for one, see myself implementing a client with non-contiguous version
support. And I would appreciate if the new servers allowed my client to do
so.

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Thursday, October 28, 2004 09:42
> To: Joseph Galbraith
> Cc: denis bider; [email-masked]
> Subject: Re: SFTP v6?
> 
> 
> I can accept a "fancy version negotiation"-extension only if 
> it is specified in such a way that implementations that don't 
> care about disjunct version intervals (e.g. the common case 
> of an implementation supporting only version x, or supporting 
> only version x and version
> I view the fancy-version-negotioation thing only as a 
> temporary kludge needed for the unfortunate version 3 -> 
> version 6+ upgrade. It will be totally unneeded bloat and 
> uglyness once this is over and the protocol specification 
> stabilizes. (I don't see any value whatsoever in fancy 
> version negotiation in the longer run. Simply advertising the 
> highest supported version number provides for adequate 
> backwards compatibility for everybody else's protocols, and I 
> think it really ought to work also for sftp).
> 
> And I don't think it is appropriate at all to consider things 
> like round-trip optimization for a temoprary kludge. Just 
> solve the problem at hand in a simple a way as possible. Both 
> the problem and the solution will be forgotten a few years from now.
> 
> Regards,
> /Niels
>

---

From: [email-masked] (Niels Möller)
Date: 29 Oct 2004 11:23:19 +0200

> does seem to solve the problem at hand in the simplest complete way.

I think it is the need for completeness that I want to challenge. The
sftp version negotiation mechanism was decided a long time ago, and it
appears to be adequate for most other protocols. I believe it should
work well to live with that decision and its limitations, and *not*
try to retrofit a very different version negotiation scheme into the
protocol.

To make it a little more concrete, I'll make a counterproposal, and
explain the assumptions that are behind it, to see if the rest of you
will shoot it down. First, assumptions:

  * sftp version 3 is widely deployed now.

  * sftp version 6 will be (we hope) widely deployed soon.

  * current deployment of sftp version 4 and 5 is quite marginal. (If
    you want to correct me on this, please differentiate between
    server and client deployment, it makes a difference).

The simple version negotiation scheme is

  V_S = max { version numbers supported by the server }
  V_C = max { version numbers supported by the client }
  V_U = min (V_S, V_C)    # version to use

  If V_U is supported by both parties, negotiation is successful, and
  V_U is the version that is used on the wire. Otherwise, negotiation
  fails.

This scheme is used in lots of protocols, including ssh and sftp (at
least up to now).

If both server and client supports a continuous set of version
numbers, this negotiation scheme is perfect: if there is any version
that both parties supports, V_U = max { version numbers supported by
both parties }.

The limitation of this scheme, is that if client or server supports a
discontinuous set of version numbers, negotiation can fail even if
there is some version that both sides support.

Our problem case is: A supports versions { 3, 4 }, B supports { 3, 6
}. Then V_U = 4, which isn't supported by B, and hence negotiation
fails. Hence, A are B not interoperatable, while a different
negotiation scheme could come up with V_U = 3 and succeed.

My first comment is that this can happen only if the client or server
actually advertises 4 (or 5) as V_C/V_S. How much deployed code out
there actually do that?

My counterproposal is to keep the old version negotiation, and add the
following note on version negotiation:

  The sftp protocol uses the traditional version negotiation mechanism
  where each side advertise the highest version number it supports,
  and the smallest of these two numbers are selected for use.
  Communication is possible if this selected version is supported by
  both sides.

  This mechanism is widely used in Internet protocols, but it also has
  some limitations. For example, if one side supports versions 3 and
  4, and the other supports versions 3 and 6, the output of the
  version negotiation is that version 4 should be used. Communication
  fails, even though there is one protocol version, namely 3, that is
  supported by both sides. This is expected to be a rare problem in
  practice, since at any given time, deployment is usually dominated
  by at most two different versions.

  In order to make communication possible also in situations like the
  above example, where the version negotiation fails, implementations
  that support more than one version of the protocol SHOULD provide
  some mechanism to manually or automatically configure which of its
  supported versions it advertises. In the above example, if either
  side is configured to advertise version 3 as the highest supported
  version, i.e. to pretend that the support for the higher version
  protocol is non-existant, then the output from the version
  negotiation is version 3, which both sides support.

This proposal sacrifices trivial out-of-the-box interoperability
between implementations that support version 3 and 6, and
implementations supporting version 3 and 4 (interoperability requires
either manual configuration, or implementation kludges like sometimes
reconnecting and trying a different version). I'm willing to do that
sacrifice in order to keep the protocol nice and clean, and because
the 3,4-vs-3,6-interoperability problem is rare and temporary.

Am I alone in that?

> support and that will work just as fine.

Is that really correct? If I connect to a 3,6 server, that server will
advertise version 3 (it *has* to, if it wants to interoperate with old
3,4 clients, and that is the entire point of the extended version
negotion, right?). If I'm a version 6 only client, and I don't
implement the proposal, my interpretation of this will be that the
server is an old version 3 server, and that I can't communicate with
it, so I will just display an error message and disconnect.

> your highest version to clients which support non-contiguous version

As I understand it, a server or client that supports only version 6
still has to implement the proposal (i.e., advertise version 3, which
it does *not* support at all, in the initial version exchange, and
negotiate version 6 in the new second phase), or else it will not
interoperate at all with clients that support version 3 and 6. That's
why I find it so ugly.

> I don't know why you label it as "fancy", because fancy is not what

It's "fancy", i.e. different and more complicated, compared to the
good old way of doing version negotiation. I didn't mean any more than
that. Sorry for any confusion.

Best regards,
/Niels

---

From: "denis bider" <[email-masked]>
Date: Fri, 29 Oct 2004 13:28:37 +0200

Niels,

I will reply with the technical aspects first.

> > implementation at all - you can continue to send the 
> > highest version you support and that will work just as fine.
>
> Is that really correct? If I connect to a 3,6 server, that 
> server will advertise version 3 (it *has* to, if it wants to 
> interoperate with old 3,4 clients, and that is the entire 
> point of the extended version negotion, right?).

If your client sends version 6 and the server supports version 6 it can
respond with version 6.

If the client supports a contiguous range of versions, we can make it OK for
the client to simply advertise the highest version it supports. It would
work fine.

> As I understand it, a server or client that supports only 
> version 6 still has to implement the proposal (i.e., 
> advertise version 3, which it does *not* support at all, in 
> the initial version exchange, and negotiate version 6 in the 
> new second phase), or else it will not interoperate at all 
> with clients that support version 3 and 6. That's why I find 
> it so ugly.

This can be fixed. If the client supports a contiguous range of versions, it
should send the highest version it supports in INIT. Otherwise it should set
the highest of the first contiguous set of versions it supports, e.g. if it
supports 3,4,6, it should send 4.

The server should respond with min{advertisedClientVersion,
highestServerVersion} in VERSION. However if advertisedClientVersion is
lower than highestServerVersion, the server should include the
"higher-versions" extension (or whatever it's called) enumerating the higher
versions it supports. In this case the server MUST then accept the
subsequent "SELECTVERSION" packet from the client.

I haven't yet had the chance to read Joseph's wording in the SFTP 6 draft so
I don't know if his wording supports this. If it doesn't we can fix.

Now for the practical aspects of this.

> This is expected to be a rare problem in practice, since at any given
> time, deployment is usually dominated by at most two different versions.

I believe this to be too optimistic. A situation like 3-6 can reoccur with a
future set of versions. Could be several years from now if the need arises
for new features requiring an incremented version. It won't hurt to have
this interoperability support in place then. Or if the past is any
predictor, something like that could even happen again now. Suppose 6 is a
major thing, then a few people implement 7, and then 8 is the next major
thing.

Note that even in a few years' time we will still likely have SFTP versions
3, 4, 5 around.

Right now our SSH server still supports SFTP version 2 only. The features of
SFTP version 2 are all that's needed in a Windows server, so we didn't yet
upgrade. We will upgrade, but right now we have a fairly considerable base
of deployed installations featuring SFTP version 2. A good portion of our
users won't bother to upgrade to the next version and will continue to
advertise SFTP version 2.

That's for how long a deployed version can be expected to persist. I expect
SFTP version 3 will continue to be around for some time to come.

> supported versions it advertises.

Niels - are any of your clients and servers graphical? Textual configuration
is easy. But providing good and well working graphical configuration for
ANYTHING is much, much more difficult than extending the protocol for it to
work automatically. Adding a new option can even take more than a day some
times. It's an important consideration when implementing anything.

In my experience, if the protocol doesn't provide an easy solution to a
common problem, the result will be chaotic differences between
implementations and haphazard interoperability.

> This proposal sacrifices trivial out-of-the-box 
> interoperability between implementations that support version 
> 3 and 6, and implementations supporting version 3 and 4 
> (interoperability requires either manual configuration, or 
> implementation kludges like sometimes reconnecting and trying 
> a different version). I'm willing to do that sacrifice in 
> order to keep the protocol nice and clean, and because the 
> 3,4-vs-3,6-interoperability problem is rare and temporary.

I'm not willing to do that sacrifice because the burden of implementing this
non-contiguous version negotiation is in my view much smaller than
application-specific remedies that will need to be implemented otherwise,
and the user frustration and loss of interoperability that will result.

Besides - a client that supports a contiguous set of versions does not need
to be impacted in any way. Just the server needs to support this extension
to be interoperable at a higher version with clients implementing
non-contiguous versions.

> It's "fancy", i.e. different and more complicated, compared 
> to the good old way of doing version negotiation. I didn't 
> mean any more than that. Sorry for any confusion.

Look at it this way. If we both spent the time we are spending to discuss
this implementing this extension, we both would have had a server and client
implementation already. :-)

Best regards,

denis

---

From: jason bailey <[email-masked]>
Date: Fri, 29 Oct 2004 08:59:35 -0400

Allow me to clarify some of my terminology.

When I first made the suggestion, I had in mind a possible change to the
protocol where a receipt would be returned upon successful completion of
a file transfer. Upon further thought and input I recognized that an
extension would be more then suitable.To be accurate that is no longer a
receipt, it would be a manifest. Proof of existence of a specific file
on a particular server at a particular time, with no regard to how it
got there.

Whether it is a manifest file or a receipt, both provide similar
functionality, which is an aspect of non-repudiation.

The mechanism that you describe is technically correct. However it is a
simplification of the issue and misses some of of the inherent problems.
Yes, you can write a server (utilizing any protocol) that will accept a
connection , receive a file, manifest the file and then return the
manifest. In fact my current solution for this is done over https to a
customized tomcat server.

So if I have a solution in place why am I bringing this up?

My answer relates in part to the last observation

> but I seriously doubt it's worth the

Non-repudiation is a growing demand in the corporate world. It is
inextricably tied into secure file transfers. In a system which requires
non-repudiation, sftp fulfills all the necessary requirements ( key
handling, secure transmission, validation ) everything, except for the
digitally signed receipt/manifest. 

There are standards that implement the necessary non-repudiation. EbXML
and Rossetta are the two that come to my mind. However these are ugly,
bloated, protocols. Completely unsuitable to do something as simple and
as straight forward as moving a file from a to b.

The reason I am bringing this up is because from my point of view having
sftp adopt this as an extension would create a standard which would
result in quicker deployments, better integration with other companies,
happier customers, happier management, and costs savings that extend
into the 10's of millions of american dollars for my company alone.

I also see this as something that would be equally beneficial to the
third party companies I work with as it is to ours.

Is it obscure? Not from my perspective, nor the companies I work with.
Is it specialized? possibly.
Is it worth the effort. I really can't answer that from this groups
perspective.

I was surprised to realize that no one had brought this up before. I
don't have a problem if, at this time, the working group doesn't believe
that this is a suitable fit as an extension to the current protocol.
However I do believe it deserves more critical consideration.

-Jason

---

From: Joseph Galbraith <[email-masked]>
Date: Fri, 29 Oct 2004 14:06:10 -0600

>   V_U is the version that is used on the wire. Otherwise, negotiation

This is almost the way it works, but not quite.

V_C = max { version numbers supported by the client }
V_U = highest version supported by server <= V_C

(V_S doesn't actually exist on the wire.)

A minor difference... but...

[...]

> it, so I will just display an error message and disconnect.

No, this is not correct.

There is actually no problem with the existing version scheme for
_servers_ supporting discontigous protocol versions.  This is
because the _client_ MUST send it's version first.  And the
server doesn't actually send it's version, it sends the version
to use.

So, the server simply waits for the clients version, and picks
the highest protocol version that it supports that is less than
or equal to the clients version and sends that as V_U.

> interoperate at all with clients that support version 3 and 6. That's

See above for client supporting only version 6.

A client that wishes to support discontigous ranges
will behave as follows:

   1. It must send version 3 or higher during initial
      exchange, because if it doesn't the server can't
      send back an extension packet describing it's
      extended version options.

      This implies that in order to support server's
      that don't implement the new version scheme, which
      include v3,4, and 5 servers (of which I know of at
      least on of each), it MUST actually support the version
      3 protocol variant.  YUCK.

   2. It will look for the 'version' extension in the servers
      SSH_FXP_VERSION packet.  If the server doesn't send the
      packet, then the server doesn't support the protocol.

      This implies that a server that doesn't support the new
      protocol will interoperate with this client as if it only
      supported the clients lowest contiguous range of protocols.

   3. If the 'version' packet is available, and contains a version
      higher than that which is currently negotiated, the client
      sends a 'version-select' (I like that name much better than
      what I used in the draft.)

So, yes, if you implement a version 6 client, you'll need to be
able to support enough version 3 to get through the extension.
And I agree, that is gross.

So I'll make an alternate proposal:

SFTP (which stands for Secure File Transfer Protocol) is not
a file transfer protocol.  If it were a file transfer protocol,
it would have operations like get and put.  Instead it has
operations like open, read, write, and close.

Therefore, I suggest that we change the name of the subsystem
to 'file-share'

The 'file-share' protocol supports all versions of the 'sftp'
protocol, with the minor exception that version exchange
consists of:

Server and client both send (without waiting to receive):

FXP_INIT2
string version-list
<extension-data>

version-list is comma separated string of versions,
in order of preference.  For example.  "6,2,1"

The version to use is the first version on the clients
list that is also on the server list.

Now:

- People with existing implementations only need a little
   bit of parameterization based on subsystem name to use
   the same piece of code to support 'sftp' and 'file-share'.

   Unix out-of-process implementations, might, for example,
   simply hard-link the sftp server binary to sftpd and
   fileshared.  Or they might pass the subsystem name on
   the command line.

   In proc implementations have similar options available.

   In fact, implementors can (and SHOULD I think) make the
   switch to 'file-share' independent of implementing
   the latest protocol version.

- Older clients continue to work without change (they
   talk to the older 'sftp' subsystem.)

- Newer clients will have to attempt to start the
   'file-share' subsystem, and if that fails, start
   the 'sftp' subsystem.

   This could cost one additional round trip when talking
   to a downlevel server, but, when talking to newer
   servers, it actually saves half a round trip because
   the server no longer waits for the clients version
   before sending it's own, and in addition, appropriate
   client extensions can be batched up and sent during
   init (something that couldn't be done before because of
   compatibility problems.)  Which could save even more.

- The new version negotiation scheme is:

   a. familiar to all ssh implementors.
   b. flexible enough to handle any future needs I can think
      of.
   c. more graceful than the old, especially in the way it fails.
      In particular, the client can now tell the user what versions
      the server _does_ support (which is really useful to me
      as an implementor or sys admin try to help a befuddled
      user.)

I actually think the name change is a cleaner solution.
Niels does raise a good point about a version 6 only
server having to lie and support version 3 in order to
work with discontigous clients.  This does seem very
ugly to me.

However, I do think we need to do one or the other
of these proposals.  I don't think simple language
change is sufficient with implementation advice is
sufficient.

The problem is indeed, temporary, for some definition
of temporary.  But, I suspect we will be seeing version
3 in use for another 5 years, at least.  That definitely
puts it on my mid to long range radar.

My pessimistic evaluation is that it will be more like
5-7 years after we hit RFC status before we don't have
customers that want to interoperate with a version
client or server.

I also suspect if we give people with existing
implementations a way to get to the latest version with
less pain the problem will be more temporary (i.e., newer
versions will be out there sooner for people to start
upgrading to, and the sooner they start the more temporary
our problem.)

- Joseph

---

From: "denis bider" <[email-masked]>
Date: Fri, 29 Oct 2004 23:33:21 +0200

> Non-repudiation is a growing demand in the corporate world. 
> It is inextricably tied into secure file transfers. In a 
> system which requires non-repudiation, sftp fulfills all the 
> necessary requirements ( key handling, secure transmission, 
> validation ) everything, except for the digitally signed 
> receipt/manifest. 

I agree with Jason's assessment and disagree about this extension being "too
obscure and specialized". It is so simply because there is no simple way of
doing it. If we come up with an extension that provides a simple way of
doing it, I believe there will be plenty demand. I would implement it in our
server.

Someone needs to draft an extension though. That ought to be Jason because
he has background knowledge about the problem and can likely come up with
something that will be simple enough yet will work.

I propose this be done here because we've had at least 2 positive responses
(I don't recall if there were more) and just one negative. If this is an
optional extension, people who don't want to implement will not be impacted
by it, whether it's part of SFTP or a separate draft.

What does Joseph as the SFTP draft editor think about making this an
extension in the SFTP draft, one like MD5 hashes for instance? (which I
think were an excellent idea btw)

If this is too complex to be an extension in the same draft, I think Jason
can just go get xml2rfc or a similar utility and write up an Internet-Draft
for this - am I right? How would he submit it when he writes one?

denis

---

From: Joseph Galbraith <[email-masked]>
Date: Fri, 29 Oct 2004 16:05:56 -0600

denis bider wrote:
> I proposed in my previous email to the list the following simple change
> which solves this problem:
> 
> If the client supports a contiguous range of versions, it should send the
> highest version it supports in INIT. Otherwise it should send the highest of
> the first contiguous set of versions it supports, e.g. if it supports 3,4,6,
> it should send 4.
> 
> This means that if the client supports 4,6 it can send 4 and does not need
> to implement 3. No yuck here. :)

But, if the server only support 6, it must still respond
respond with 4, even though it does not support 4.

Then, if the client doesn't send a 'version-select' the
server must close the channel, because it can't speak
the protocol.

Rather than getting a nice error message about the server
not supporting the version, the user will likely see
strange behavior, because the version 4 client won't be
expecting a channel after version exchange is complete
to mean 'version unsupported.'  It thought the version
stuff was already sorted out.

Ugh.

On the other side of the coin, let's say we have a 4,6
client, which advertises 4 in it's INIT packet.  Now,
the server is a 3,6 server, so it responds 3.  The
client now needs to negotiate down to 3.

That's not so bad, I guess.  (I don't think I allowed
it in the original language, but that could be fixed :-)

The third problematic case is a a client that supports
1,2,6.  Such a client can not send 2, because 2 doesn't
support extensions.  So it must send 3.

Now, what if that client actually meets a version 3 server?
(A highly likely event.)  It is now stuck.  It can't jump
to six, nor can it fall back to 2 where it could actually
interoperate.  So a client really must support, at a minimum,
version 3.

Ugh.

> The SFTP module can be implemented separately from the SSH server, as we do.
> This proposal requires the SSH server to know about complexities of the SFTP
> protocol and to invoke the SFTP module differently depending on what verb
> was used to start it.

Not really; it just requires the SSH server to pass the name
of the subsystem being started to the subsystem.

This could be as simple as invoking the sftp-server module
by a different name, or having the config file setup in a
way such that the 'file-share' subsystem invokes
'sftp-server -file-share' and the 'sftp' subsystem invokes
'sftp-server' ... or it might be more complicated
in some implementations.

But it shouldn't require your SSH server to know anything about
the SFTP protocol... it just has to support a new subsystem, and
the subsystem has to know it's name (or you have to have separate
modules.)

In all honesty, I think that the new version exchange is how
it should have been done originally.  And I think that it solves
a number of outstanding problems in the protocol that we've
been reluctant to address because of backwards compatibility
issues, including:

   - the version problem we've been discussing
   - extensions from the client on startup (these have
     been very useful on the server side... originally they
     were there for the client to, but had to be removed for
     backwards compatibility.)
   - Better failure mode when there isn't a compatible version.
     (In fact the draft is currently silent on how to deal with
     this and there isn't a great way to do so with the current
     scheme.  With the new scheme, both sides know that the version
     exchange has failed, and which versions the other supports.)

     We've actually tried to make some changes for better failure
     mode before too.

> This also violates independence of SFTP from the SSH protocol. Each
> underlying protocol must now think up its own rules for starting old versus
> new versions of SFTP.

I'm not sure I follow this argument.

Since any hypothetical non-SSH implementation doesn't actually
have subsystems, it already had to think up its own rules for
starting the protocol.

I don't know of any actual non-SSH implementations.

Any new non-SSH implementation should just always use the
new version exchange, or in other words, the 'file-share'
version semantics.

It should also probably just use the latest protocol version
and not have any of this mess, since it is new and doesn't
have to worry about interoperating with old.

> I think that's much uglier.
> 
> Doesn't my proposal above solve all of the gripes about people having to
> advertise versions they don't actually support? It's a straightforward
> change and if that's the only real problem thus far identified we have a
> good solution I think (if Niels agrees).

Unfortunately, no, it really doesn't.  At least I don't see
how yet.

I was initially in favor of the version re-negotiate scheme
(it was basically the solution I had in the back of mind
if I ever got a chance to solve this problem.)

But as I've thought about the possible interoperability
interactions, I've become convinced that it would actually
be pretty fragile, and could lead to non-obvious failures
(like the one I described at the top.)

Thanks,

Joseph

---

From: Joseph Galbraith <[email-masked]>
Date: Fri, 29 Oct 2004 16:16:05 -0600

> for this - am I right? How would he submit it when he writes one?

Well, below is a _very_ rough version of an extension.  I'd be
more than happy to have Jason's expertise to help get this
fleshed out where it will work for him (and us.)

And since non-repudiation has recently whacked me over the head,
I'd agree that I suspect for most of us implementing sftp, it
will probably become non-obscure and not-so-specialized soon, if
it hasn't already.

(One alternate proposal I had was to put together a AS profile for
running of sftp, AS4... but I think that might be _really_
heavy weight, and what Jason seems to be saying is we already
have most of what we need anyway.)

- Joseph

---

9.2  File Signing

    This extension allows a client to request that the server provide a
    signature proving that it has recieve a correct copy of the data.

        byte   SSH_FXP_EXTENDED
        uint32 request-id
        string "sign-filename" / "sign-handle"
        string filename [UTF-8] / file-handle
        string pubkey-algorithm-list [from transport draft]

        byte   SSH_FXP_EXTENDED_REPLY
        uint32 request-id
    	string filename-used-in-signature [UTF8]
        int64  time-of-request
        attrib file-attrib
        string public-key [from transport draft]
        string signature  [from transport draft]

    filename
       Used if 'sign-filename' is specified; indicates the name of the
       file to use.  The hash will be of the file contents as it would
       appear on the wire if the file were opened with no special flags.

    file-handle
       Used if 'sign-handle' is specified; specifies a file handle to
       read the data from.  The handle MUST be a file handle, and
       ACE4_READ_DATA MUST have been included in the desired-access when

Galbraith                Expires April 29, 2005                [Page 46]

Internet-Draft         SSH File Transfer Protocol           October 2004

       the file was opened.

       If this file handle was opened in TEXT mode, the signature must be
       made of the data as it would be sent on the wire.

    pubkey-algorithm-list
       The algorithm list of publickeys the client is capable of
       supporting.  If the server cannot provide a signature using any of
       the  algorithm requested, it should respond with a STATUS
       response, with UNSUPPORTED error code.

       In the case of x509v3 (see below), no attempt is made to negotiate
       which algorithms are supported within the certificate chain.

    filename-used-in-signature
    time-of-request
    file-attrib
       The filename actually used, time-of-request, and file attribute
       structure which were used to make the signature.  The filename, in
       particular, may be difference from that specified if the server
       applied some sort of canonicalization, or may be empty if the
       request was a sign-handle and the server couldn't map from a
       handle to a filename.

    public-key
       The public-key the server used to sign the filename.  This is a
       data structure as specified in the secsh transport specification
       [2].

       This draft specifies an additional public format,
       [email-masked].  The key data in this case is the
       x509v3 certificate DER encoded.  The signature algorithm used for
       this key type is one of three types:

       ssh-dss
          Encoded as per the secsh transport specification [2].

       ssh-rsa
          Encoded as per the secsh transport specification [2].

       [email-masked]
          A DER encoded pkcs #7 signature with detached data, as
          specified by .

          The prefered signature algorithm is pkcs7.

Galbraith                Expires April 29, 2005                [Page 47]

Internet-Draft         SSH File Transfer Protocol           October 2004

    signature
       The signature over the following data, formatted as specified in
       the secsh transport specification [2] or as above.

            string filename [UTF-8]
            string username-of-signature-requestor [UTF-8]
            int64  time-of-request
            attrib file-attrib
            string file-data

       filename
          The filename of the file being signed.  If this is a
          'sign-filename' request, the filename MUST be included,
          although it MAY be canonicalized.  If this is a sign-handle,
          the filename SHOULD be included; however, if the server can not
          map backwards from a handle to a filename, the filename may be
          zero length.

       username-of-signature-requestor
          This is the username by which the client authenticated to the
          system.

       time-of-request
          This is the time that the signature request was made, format as
          described by file attributes. (Section 6.7)

       file-attrib
          File attribute structure describing the attributes of the file.
          The server SHOULD include any fields that it can, even if it
          would not normally retrieve the data.  In particular, the
          server SHOULD include owner, group, acl, permissions, and
          mime-type data if it can support these field.  The size and
          attrib-bits fields SHOULD also be included.

       file-data
          The file data as an SSH string.

---

From: "denis bider" <[email-masked]>
Date: Fri, 29 Oct 2004 23:15:25 +0200

denis

---

From: [email-masked] (Niels Möller)
Date: 30 Oct 2004 18:40:01 +0200

[...]

> digitally signed receipt/manifest. 

I'm afraid, I'm not really familiar with the non-repudiation area.

> However I do believe it deserves more critical consideration.

Then I think the first thing you have to do is to write up the
requirements. "Non-repudiation" is a very fuzzy concept to me, and
I'll have a hard time participating in discussion of details in a
non-repudiation mechanism.

I don't know if the others in the secsh wg are familiar with
non-repudiation, but perhaps you have to seek feedback elsewhere in
order to get the requirements right.

When the requirements are clear, we'll have a better foundation for
designing or evaluating extensions to sftp.

Regards,
/Niels

---

From: [email-masked] (Niels Möller)
Date: 30 Oct 2004 19:00:58 +0200

> I will reply with the technical aspects first.

Thanks.

> If your client sends version 6 and the server supports version 6 it can

I'm afraid that makes absolutely no sense to me. It is possible that I
have misunderstood the vesrion extension in some fundamental way, so I
should probably shut up until I've reread the proposal (which may take
a long time, since I'm pretty busy with work for the rest of this
year).

Anyway, I'll summarize my current understanding of the proposal.
Given:

  Server S, that supports version 3 and version 6 of the protocol.

  Client O (old), supporting version 3 and version 4.

  Client N (new), supporting only version 6.

S wants to interoperate with O. Using the traditional version
negotiation will fail (server advertises version 6, O advertises
version 4, which leads to version 4 being selected, which S doesn't
support). Hence S *must* implement the proposed extension. It always
advertises version 3, and intends to use the proposed extension to
negotiate version 6 in the second phase.

Now N connects to S. If N uses the traditional version exchange only,
communication will fail (N advertises 6, S advertises 3, which leads
to version 3 being selected, which N doesn't support. So if N wants to
interoperate with S, it too *must* implement the extension.

> upgrade.

When you upgrade, you will have a server that supports version 2 and
6, or version 2,3 and 6, right? I don't see why that should pose a
problem.

The problem is only implementations that *advertice version 4 and 5*.
Any body who knows about deployment of such versions, please speak up.
I'd like to see some evidence for the existence of the problem the
extension is trying to solve.

> times. It's an important consideration when implementing anything.

I don't usually to GUI programming. From my limited experience, if you
have some kind of options dialog or menu, adding one more checkbox to
that dialog is about the same amount of work as adding one more
command line option to a command line tool.

Regards,
/Niels

---

From: der Mouse <[email-masked]>
Date: Sat, 30 Oct 2004 17:22:13 -0400 (EDT)

>> Non-repudiation is a growing demand in the corporate world.  [...]

Non-repudiation is a way of arranging that it is impossible (well,
infeasible - about as impossible as anything gets in cryptography) to
later claim "no I didn't do that".  (As a non-crypto example, for
example, my signature on a document gives some non-repudiable assurance
I've read the document.)

It's not clear how it's being used here: is the client trying to
protect itself against a server saying "no I never received that file"
or is the server trying to protect itself against a client saying "no I
didn't send that file"?  (The mechanisms sketched appear to be designed
for the former, but that strikes me as an unlikely desire.)

When I first saw the mechanisms outlined, it appeared to me that they
were designed to give the client a certificate which could demonstrate
to a third party that the server did indeed receive the file; while
this does have non-repudiation uses, it seemed to me more designed to
allow the client to prove the fact of transmission to a third paty
without requiring the proof to involve contacting the receiving host or
its admins.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Damien Miller <[email-masked]>
Date: Sun, 31 Oct 2004 11:53:20 +1100

> This feature, would do much to alleviate a lot of our issues with secure

I don't think that this is within the scope of sftp at all. It would
more appropriately be done by a higher-level protocol, e.g. one that
uses sftp for the file transfer and something else (OpenPGP, S/MIME) for
the signed receipts.

-d

---

From: Joseph Galbraith <[email-masked]>
Date: Sun, 31 Oct 2004 10:02:50 -0700

> extension is trying to solve.

I have a version 4 deployment.  I'm pretty sure Richard Walen
(process software) has a version 4 deployment.

And Henrick Hellström says that he and others he knows of
have a v5 deployments.

So yes, the problem does exist in the wild and we aren't just
talking about the hypothetical problem.  There are known,
currently shipping implementations of every version of
sftp from 2-5.

I'm mildly surprised about the 2, by the
way. I would have thought that no one was currently
shipping version 2, though I suspected there was still a
number of v2 implementations in use.

That is one of the reasons I claim this problem isn't
quite as 'temporary' as you and I would wish.

Thanks,

- Joseph

---

From: [email-masked] (Peter Gutmann)
Date: Mon, 01 Nov 2004 16:39:27 +1300

[email-masked] (=?iso-8859-1?q?Niels_M=F6ller?=) writes:

>Then I think the first thing you have to do is to write up the requirements.
>"Non-repudiation" is a very fuzzy concept to me, and I'll have a hard time
>participating in discussion of details in a non-repudiation mechanism.

It's a fuzzy concept to everyone, so much so that after 20-odd years of trying
the X.509 guys gave up and renamed the nonRepudiation flag in certs to
something that actually had a meaning.  Calling it a "delivery receipt" would
be better, that's what S/MIME (which is the only major standard to
specifically address this) calls it.  In fact you could probably lift a lot of
the S/MIME stuff, since they've looked at it in some detail.

Peter.