Subject: Re: UTF8

From: [email-masked] (Markus Gyger)
Date: Fri, 18 Feb 2005 13:00:29 +0100 (CET)

Niels Moeller writes:
> Please leave the file system issues out of it for now. What's of
> primary importantance are the core drafts, and those deal with
> usernames and passwords in utf8 form, *not* file names.

Are there any directions on what encoding SSH_MSG_CHANNEL_REQUEST
with the "exec" request type should use for the command string
(for "subsystem" it is specified but there sems to be no info
for "exec" in connect-23)?

Markus

---

From: [email-masked] (Niels Möller)
Date: 18 Feb 2005 15:41:30 +0100

> for "exec" in connect-23)?

I've never thought about the encoding issues for this string before.
It's not specified. Note that also the channel data is often text, and
its encoding is also not specified at all by the ssh protocols. In
practice,

  * You have to guess what character system is used by the default
    session.

  * This will probably be a pain if local and remote systems use
    different character sets (say latin-1/utf8).

  * It's easier to let the client environment adapt to the server
    environment conventions than vice versa.

But for now, users have to ensure manually that they are using the
same character set locally and remotely. If you e.g. want to login to
a server that lives in utf8-land, and your local environment doesn't
use utf8, then you'd better use some local terminal emulator that
understands utf8.

Regards,
/Niels

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 18 Feb 2005 21:28:37 -0500

> use utf8, then you'd better use some local terminal emulator that

It's interesting that I was just thinking about this very problem earlier 
this week.  Ultimately, I came to the conclusion that it's probably best 
not to try to specify the character set of channel data, because 
operational experience shows that what we have now actually works fairly 
well, and trying to "fix" it may well create more problems than it would 
solve.

Fundamentally, I see ssh channels as doing two things:

(1) allowing the user to have a local terminal on a remote machine

(2) carrying non-textual protocols like sftp or X11 or arbitrary traffic
    via port-forwarding

I think it's pretty non-controversial that for (2), we don't want the ssh 
protocol modifying the data stream.

I'd like to suggest that case (1) is really the same thing.  In this case 
we're not carrying _text_; we're carrying communication between a terminal 
and programs using that terminal, which is actually a non-textual protocol 
that happens to carry a lot of text.

The ssh protocol already provides a means to inform the server software of 
the user's terminal type; servers (or the application software they run) 
will use this information to determine things like what are valid control 
sequences for the terminal, including how to determine what the available 
character sets are and how to switch between them.  It would be pretty sad 
if an application switched character sets and stopped working because the 
ssh client and/or server didn't notice the change.

Even worse, for a shell we actually can't tell the difference between (1) 
and (2).  For an interactive login (1) is probably the case, but scp runs a 
binary protocol over a shell connection, which is actually (2).

So if someone wants to define a set of extensions which define a UTF-8 
based network virtual terminal, I'd not object.  But changing the default 
behavior of the existing session channel is probably a bad idea.

-- Jeff

---

From: [email-masked] (Markus Gyger)
Date: Mon, 21 Feb 2005 13:35:21 +0100 (CET)

Niels Moeller writes:
> Markus Gyger writes:

In the long term, is the idea to use UTF-8 or to have it
configurable? Since currently only US-ASCII is used it's not
much of a problem (I don't know if there are e.g. servers that
use EBCDIC). But how about e.g. the names of environment
variables to be passed or the TERM names passed -- should
they use the same encoding as the channel data (once they
are outside of US-ASCII) or UTF-8 per default?

Markus

---

From: [email-masked] (Niels Möller)
Date: 21 Feb 2005 14:42:11 +0100

[email-masked] (Markus Gyger) writes:

> configurable?

It seems that the world is slowly moving towards utf-8, but that
development is not really in scope of the secsh wg to decide. We just
go with the flow.

When it has been necessary to deal with different character sets in
the ssh protocols, so far the answer has been to require utf-8 on the
wire.

> Since currently only US-ASCII is used it's not
> much of a problem

Huh? I'm using latin-1 over ssh sessions all the time. I'm sure others
are using utf-8 as well as a bunch of other encodings.

> But how about e.g. the names of environment

For names and values of environment variables, it's about as undefined
as the charset for channel data. For the TERM names, these are fairly
standardized, and ascii. We don't need to care until non-ascii
terminal names are added to termcap/terminfo, and I doubt that will
ever happen.

Using plain ascii for various identifiers which are intended to be
interpreted by machines rather than humans, seems to be a very deeply
rooted convention.

/Niels

---

From: Jeffrey Altman <[email-masked]>
Date: Mon, 21 Feb 2005 21:15:10 -0500

Let me remind the group that it is completely up to the terminal and the 
application to determine the character set and terminal type which are 
to be used across the data stream.  Both the terminal type and the 
character set may be negotiated as needed.  The SSH channel like the 
Telnet channel is a binary pipe.  It is not text data.

Jeffrey Altman