Subject: Re: IUTF8 pseudo-terminal mode

From: der Mouse <[email-masked]>
Date: Mon, 2 Jan 2006 01:00:06 -0500 (EST)

> Recent versions of the Linux kernel support an IUTF8 flag [...].

> I would like to allow this mode to be preserved by SSH, but there is
> no assignment for it at present.

> Could this line be added to the appropriate place in
> draft-ietf-secsh-connect and draft-ietf-secsh-assignednumbers to
> create this assignment?

It's probably too late to do this before they make it to RFC status.

> 42 seems like a reasonable place for it.

>           42    IUTF8       Assume input characters are UTF-8 encoded.

Actually, if you're going to be adding such things, how about adding
bits for ECHOPRT, ALTWERASE, NOKERNINFO, and character size (CS5..CS8)?
In my own implementation, I created a private channel request to carry
those bits....

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 2 Jan 2006 02:34:02 -0600

> In my own implementation, I created a private channel request to carry

Using an "env" channel request to set LANG/LC_CTYPE/LC_ALL to a UTF-8
locale should do, though if the pty requires special attention then the
server will have to know to interpret said environment variables
specially.

And, of course, this is definitely a POSIX-specific approach.

Nico
--

---

From: Jacob Nevins <[email-masked]>
Date: Mon, 2 Jan 2006 12:27:55 +0000

der Mouse writes:
> [Colin Watson:]

(You have CS7 and CS8 already, although you have to be sensible about
which combinations you enable.)

Well, the IETF seems to care more about UTF-8 than those other
things. :)

Given that the SSH protocol doesn't assign any semantics to these
terminal modes -- they are just tokens passed from one side to the other
-- it's a shame that they don't have extensibility and private-use
facilities, unlike most of the rest of the protocol.

If it's too late to get IUTF8 (etc) into the core drafts, and consensus
is that it's worth a new document, it's probably best to define a
channel extension like yours with textual names, private-use provisions,
etc, rather than just assign some new code points in the existing
mechanism (given how long it looks like it will take to get _any_
document through).

---

From: der Mouse <[email-masked]>
Date: Mon, 2 Jan 2006 12:13:32 -0500 (EST)

> which combinations you enable.)

Yes, it seems odd to have a CS7 boolean and a CS* boolean, rather than
a CS field which takes an argument like 7 or 8.

> Well, the IETF seems to care more about UTF-8 than those other

Given that there's PENDIN present, which has no business being present
at all (it's transient internal state which is exposed so that userland
can set it to provoke a reprint, not something that it makes any sense
to push across the wire at connection setup), I don't see why they
wouldn't include ECHOPRT, ALTWERASE, and NOKERNINFO.  CS5 and CS6 are a
little iffier....

There's also the question of what to do if the input and output
character sizes differ, since there's only one character-size field in
the protocol, not one in each direction.

> terminal modes -- they are just tokens passed from one side to the
> other -- it's a shame that they don't have extensibility and
> private-use facilities, unlike most of the rest of the protocol.

Sure they do.  Just not in the way you're thinking of.  You can always
do it the way I did - use a private CHANNEL_REQUEST to carry other
stuff.  Not interoperable between different independent versions, but
that's true of pretty much any private-use extensibility mechanism.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 2 Jan 2006 12:51:50 -0600

> that's true of pretty much any private-use extensibility mechanism.

Yes, but putting a pseudo-terminal into UTF-8 mode (including, perhaps,
specifying a normalization form) is something that strikes me as fairly
generic and which the IETF would care about.

---

From: Colin Watson <[email-masked]>
Date: Mon, 02 Jan 2006 22:04:26 +0000

In article <[email-masked]>,
>> It's probably too late to do this before they make it to RFC status.

(I thought this was probably the case, but it did no harm to ask.)

>server will have to know to interpret said environment variables

While setting LANG et al is indeed useful, it's not enough to make the
kernel's terminal driver do the right thing when erasing characters in
cooked mode. That's why the termios flag was invented.

It would probably indeed be possible to come up with some baroque shell
configuration that set IUTF8 if in a UTF-8 locale, but, since SSH
already supports passing pseudo-terminal modes, this seems like a more
correct approach in the long run ...

Cheers,

-- 
Colin Watson                                       [[email-masked]]

---

From: Bill Sommerfeld <[email-masked]>
Date: Mon, 02 Jan 2006 17:29:40 -0500

On Sat, 2005-12-31 at 11:05, Colin Watson wrote:
> Recent versions of the Linux kernel support an IUTF8 flag (see
> http://www.cl.cam.ac.uk/~mgk25/unicode.html#mod) which allows the
> character-erase function in cooked mode to handle UTF-8 characters
> correctly. I would like to allow this mode to be preserved by SSH, but
> there is no assignment for it at present.
> 
> draft-ietf-secsh-connect and draft-ietf-secsh-assignednumbers to create
> this assignment? 42 seems like a reasonable place for it.
> 

as others have suggested, it's already too late to modify -connect and
-assignednumbers.

So what needs to happen to get this standardized is for someone to write
an internet-draft documenting this extension and advance it as either a
working group item or as an individual submission.

> cooked mode. That's why the termios flag was invented.

I think Nicolas was implying that a request from the client to use a
UTF8 locale would cause the server to set up the pseudoterminal it
creates/allocates/... appropriately for the requested locale.  But
that's perhaps at odds with the existing strategy within the protocol of
making termios-level details visible on the wire.

						- Bill

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 2 Jan 2006 16:43:58 -0600

> making termios-level details visible on the wire.

I think I was a bit more explicit about this, but, yes, this would be a
hack, and very POSIX-specific (and I was explicit about that too).  But
such a hack would also probably make most users happy :^/ at the expense
of elegance and code complexity for SSHv2 servers running on POSIXy
platforms.

SSHv2's pty negotiation could certainly improve in this regard, I don't
deny it!

But I suspect saying "UTF-8" is not enough here.  What options are
there?  Should the pty driver reject non-UTF-8 sequences?  Should it
normalize?  Pass input through raw?  I suspect most users wouldn't want
much of a pty UTF-8 input mechanism as their clients, presumably, have a
decent UTF-8 input method already -- but then, maybe they don't.

I definitely think this should be a WG work item, if nothing else to
ensure it gets more eyeballs because I18N requires care.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 2 Jan 2006 17:30:34 -0600

On Mon, Jan 02, 2006 at 06:20:50PM -0500, Jeffrey Hutzelman wrote:
> The original request wasn't really about standardizing handling of UTF-8 in 
> SSH data streams.  That's really outside the scope of the protocol -- 
> unlike telnet, SSH doesn't provide a "virtual terminal"; it connects the 
> shell running on the server to the user's real terminal, and experience has 
> shown this is basically the right thing to do.

Well, yes, but I'm not entirely sure one wouldn't want to negotiate
additional behaviours -- I'm not saying one should as I've not really
thought this through.

> The request here was to enable SSH to pass a platform-specific TTY mode bit 
> which it doesn't currently handle.  The bit in question causes the tty 
> driver on Linux systems to behave in a particular way; specifically, it 
> tells the driver that the user is typing in UTF-8, and that when the user 
> types the ERASE character, the driver should remove a complete character 
> (possibly a multi-byte UTF-8 sequence) from the input buffer.

Got it.

> One could argue that an SSH server running on such a system should look at 
> the configured locale and configure the PTY appropriately, and that's 
> probably even a good idea.  However, a user using 'stty' to change terminal 
> modes at the remote end of an ssh connection has an expectation that the 
> change will propagate to the local terminal as much as possible, and the 
> point of defining a bit for IUTF8 is to help make that possible.

Did you switch local/remote here?

Nico
--

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 02 Jan 2006 18:55:58 -0500

I'm removing the Debian bug and original reporter from the CC list; I don't 
think they care about our protocol design discussion.

On Monday, January 02, 2006 05:30:34 PM -0600 Nicolas Williams 
<[email-masked]> wrote:

>> One could argue that an SSH server running on such a system should look
>> at  the configured locale and configure the PTY appropriately, and
>> that's  probably even a good idea.  However, a user using 'stty' to
>> change terminal  modes at the remote end of an ssh connection has an
>> expectation that the  change will propagate to the local terminal as
>> much as possible, and the  point of defining a bit for IUTF8 is to help
>> make that possible.
>

Nope.  If I type 'stty iutf8', stty does an ioctl on the pty slave on the 
remote machine (the ssh server).  My expectation is that sshd will notice 
the change, propagate it down the wire, and my ssh client will make the 
same ioctl on my terminal (assuming, of course, that both terminals support 
the bit).

-- Jeff

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 02 Jan 2006 18:20:50 -0500

-- Jeff

---

From: Joseph Galbraith <[email-masked]>
Date: Mon, 02 Jan 2006 17:38:43 -0700

Jeffrey Hutzelman wrote:
> I'm removing the Debian bug and original reporter from the CC list; I
> don't think they care about our protocol design discussion.
> 
> 
> Nope.  If I type 'stty iutf8', stty does an ioctl on the pty slave on
> the remote machine (the ssh server).  My expectation is that sshd will
> notice the change, propagate it down the wire, and my ssh client will
> make the same ioctl on my terminal (assuming, of course, that both
> terminals support the bit).

I don't think this currently happens with any of the bits; the
term bits (if I recall correctly) are sent only from client to
server when the pty is requested.  I don't believe there
is currently a way on the wire for either the or the server
to change their values after the pty is requested.

Thanks,

Joseph

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 3 Jan 2006 13:18:56 -0600

> to change their values after the pty is requested.

Correct.  The client has to put its terminal into raw, echoff mode in
order to let the pty on the server side handle stty settings on the
remote side correctly.