Subject: adding IUTF8 to encoded terminal modes in SSH Protocol Assigned Numbers

From: David Madore <[email-masked]>
Date: Fri, 29 Mar 2013 13:57:34 +0100

Hi,

I am writing to you since you are the authors of RFC 4250 "The Secure
Shell (SSH) Protocol Assigned Numbers", so I suppose you are a good
starting point towards seeking IETF consensus on adding a terminal
mode assigned in this protocol.  If not, could you please redirect me
to more appropriate recipients?

The problem is this: many Unix terminal drivers have historically been
broken on UTF-8 (or indeed, any multibyte) encoding because, for
example, the "backspace" key will remove only one byte from the buffer
whereas the terminal will remove one character.  To solve this, Linux
added an "iutf8" terminal input mode, which lets the kernel terminal
driver know that characters should be assumed to have their UTF-8
width (note that this concerns the terminal mode and is independent
from any locale setting).  Unfortunately, for this to have an effect
through SSH, it is necessary for the SSH client and server to
communicate on this, hence, to obtain an number assigned to do so.

(There are, of course, various approaches to handling UTF-8 terminals,
including full locale passing, but since the SSH terminal modes have
historically reflected the Unix terminal driver flags faithfully, and
since this approach does not preclude doing something better through
other mechanisms, it is not necessary to have an all-encompassing
solution to go forward.)

This is why I'd like to try to get things moving towards creating a
consensus around the assignment (ultimately to be added to section
4.5.2 of the RFC):

         42    IUTF8       Assume input characters are UTF-8 encoded.

matching the "iutf8" stty setting under Linux.

Now I realize, of course, that one doesn't just add a number to the
list.  But there seems to be a bootstrap problem: SSH implementors
will not implement the feature unless it is normalized (see <URL:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337041
 >), and the feature cannot be normalized until there is consensus on
its implementation.  How does one break this deadlock?

Is there some way to request at least a temporary, or provisional,
reservation, so that SSH implementations can be convinced to use this
number, which might be normalized later?

Best regards,

-- 
     David A. Madore
   ( http://www.madore.org/~david/ )

---

From: Mouse <[email-masked]>
Date: Fri, 29 Mar 2013 09:31:37 -0400 (EDT)

> But there seems to be a bootstrap problem: SSH implementors will not
> implement the feature unless it is normalized (see <URL:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337041 >),

Er, no.  One particular set of SSH implementors, perhaps.  moussh has
supported IUTF8 (as far as I can tell - I don't run anything that has
the bit, making it difficult for me to test) since shortly after I
first saw the issue mentioned.  (Supported it using the same extension
I use to support ECHOPRT and other bits not in the ssh RFCs, but
still.)

I don't know if any other implementations have.

> and the feature cannot be normalized until there is consensus on its
> implementation.  How does one break this deadlock?

Given what it is, in this case, I think the right way is to spec the
bit even in the absence of any ssh implementations actually using it.
(Not that I have any particular authority to do so.)  But, on the other
hand, if "tty drivers have this bit" were reason enough, they
wouldn't've left out the bits that prompted me to create the extension
I mentioned above.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: [email-masked] (Niels Möller)
Date: Fri, 29 Mar 2013 16:23:20 +0100

> matching the "iutf8" stty setting under Linux.

Great!

> its implementation.  How does one break this deadlock?

That's not required. Quoting earlier discussion:

: > So what needs to happen to get this standardized is for someone to write
: > an internet-draft documenting this extension and advance it as either a
: > working group item or as an individual submission.
: 
: That remains the case today.  The registration policy for that registry
: is "IETF Consensus", which means that getting a new number assigned
: requires publishing an RFC.  For this sort of thing, that could happen
: relatively quickly, once someone writes an internet-draft.
: 
: -- Jeff

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

---

From: David Madore <[email-masked]>
Date: Fri, 29 Mar 2013 18:32:08 +0100

> I use to support ECHOPRT and other bits not in the ssh RFCs, but

Please correct me if I'm wrong, but I seem to understand that moussh
supports the IUTF8 mode using a private channel approach, which
achieves the same purpose but in a different way from what I'd like to
see standardized.  I assume the reason you chose to use a private
channel is precisely that the value for the IUTF8 terminal mode is not
standardized: so this supports my statement "SSH implementors will not
implement the feature unless it is normalized" (although the word
"feature" was probably badly chosen: I meant something like "protocol
token").

Would you be willing to create - if only for testing purposes - a
version of moussh that implements the IUTF8 mode using protocol
encoded terminal mode 42?  I'm attaching a patch that I think will do
this, based on the source code I found in <URL:
http://ftp.rodents-montreal.org/mouse/local/src/moussh/moussh/
 >, which may or may not be the appropriate version to use.  It would
be great if we could check interoperability with the patched version
of OpenSSH I just posted on the Debian bug tracker (<URL:
ftp://quatramaran.ens.fr/pub/madore/misc/openssh-with-iutf8/
 >).

> (Not that I have any particular authority to do so.)

I'm willing to try this, but I somehow success that somebody is going
to say "you can't just appear out of nowhere and publish an Internet
draft that says 'now number 42 will mean this' to have it added to the
list".

>							But, on the other

I imagine nobody cared strongly enough about them.  But if the red
tape barriers do not turn out to be insurmountable and it is possible
to get IUTF8 added to the protocol assigned numbers, it might be an
occasion to add other missing ones as well.  Could you recall to me
what they are?

-- 
     David A. Madore
   ( http://www.madore.org/~david/ )
diff -puN moussh.orig/client.c moussh/client.c
--- moussh.orig/client.c	2012-12-05 02:32:52.000000000 +0100
+++ moussh/client.c	2013-03-29 18:16:52.000000000 +0100
@@ -3091,6 +3091,9 @@ static void request_pty(CSESSION *s)
 #ifdef IMAXBEL
     FOO(c_iflag,IMAXBEL);
 #endif
+#ifdef IUTF8
+    FOO(c_iflag,IUTF8);
+#endif
 #ifdef ISIG
     FOO(c_lflag,ISIG);
 #endif
diff -puN moussh.orig/msgs.h moussh/msgs.h
--- moussh.orig/msgs.h	2011-01-17 05:25:08.000000000 +0100
+++ moussh/msgs.h	2013-03-29 18:17:12.000000000 +0100
@@ -112,6 +112,7 @@
 #define TTY_OP_IXANY     39
 #define TTY_OP_IXOFF     40
 #define TTY_OP_IMAXBEL   41
+#define TTY_OP_IUTF8     42
 #define TTY_OP_ISIG      50
 #define TTY_OP_ICANON    51
 #define TTY_OP_XCASE     52
diff -puN moussh.orig/server.c moussh/server.c
--- moussh.orig/server.c	2012-12-05 02:32:59.000000000 +0100
+++ moussh/server.c	2013-03-29 18:17:52.000000000 +0100
@@ -352,6 +352,9 @@ static void set_tty_modes_std(PTY_REQ *r
 #ifdef IMAXBEL
        case TTY_OP_IMAXBEL: bit = IMAXBEL; } if (0) {
 #endif
+#ifdef IUTF8
+       case TTY_OP_IUTF8:   bit = IUTF8;   } if (0) {
+#endif
 #ifdef INLCR
        case TTY_OP_INLCR:   bit = INLCR;   } if (0) {
 #endif

---

From: Chris Lonvick <[email-masked]>
Date: Fri, 29 Mar 2013 11:47:33 -0700 (PDT)

Hi David,

On Fri, 29 Mar 2013, David Madore wrote:
<eliding some discussion>

> draft that says 'now number 42 will mean this' to have it added to the

That is the right way to do this.  I'll send you separately an xml2rfc 
template that you can fill in, with instructions on how to submit it to 
the ID repository.

Once you have published this as an Internet Draft, bring the discussion 
back to this group.  Update the draft if needed and I'll tell you what to 
do to get it published as an RFC.

Best regards,
Chris

---

From: Mouse <[email-masked]>
Date: Fri, 29 Mar 2013 21:52:27 -0400 (EDT)

> supports the IUTF8 mode using a private channel approach,

Yes.

> (although the word "feature" was probably badly chosen: I meant
> something like "protocol token").

Ah.  Then, yes.  The `feature' I as I understood it was something more
like "IUTF8 in the tty driver modes".

> encoded terminal mode 42?

Certainly...though, as I think I remarked, it is difficult for me to
test such a thing, since I don't run any OSes which have IUTF8.

> I'm attaching a patch that I think will do this, [...]

Offhand, your patch looks correct, though it's possible there's
something else which needs to be changed which doesn't come to mind
immediately - I'd grep for one of the other c_iflag bits to look for
any such possible places.  It's certainly very close.

> which may or may not be the appropriate version to use.

It's close enough for this.  The master copy of moussh is kept as a git
tree (clonable from git://git.rodents-montreal.org/moussh); in case you
care, I've just now updated the FTPable copy to match the current tip
of the master branch.  I'll create a branch IUTF8 and add IUTF8 as if
it were standard with value 42 there, to be merged (or, more likely,
cherry-picked) into master if-and-when appropriate.

>> bit even in the absence of any ssh implementations actually using
>> it.  (Not that I have any particular authority to do so.)
> draft that says 'now number 42 will mean this'

That much, I believe you can; I-Ds get published with some pretty wacky
content (draft-terrell-logic-analy-bin-ip-spec-ipv7-ipv8-08.txt comes
to mind; it's probably expired, but if there's no current version I can
send you a copy I saved).  The question is whether anyone will pay
attention to it.  In this case, I think people will...

> to have it added to the list".

Promotion from I-D to RFC, and implementations, are the interesting
parts here.  Once there's a spec, even as an I-D, I'd expect it to be
an easier sell to get ssh implementors to implement it, and, given a
few implementations, I'd expect (though I'm hardly an expert on this
point) it to be just a matter of bureaucratic hoop-jumping to turn it
into an RFC.  I'd certainly support it, and, while that's not usually
significant, in the case of ssh it might be.

>> But, on the other hand, if "tty drivers have this bit" were reason
>> enough, they wouldn't've left out the bits that prompted me to
>> create the extension I mentioned above.
> I imagine nobody cared strongly enough about them.

Apparently not.

> But if the red tape barriers do not turn out to be insurmountable and
> it is possible to get IUTF8 added to the protocol assigned numbers,
> it might be an occasion to add other missing ones as well.  Could you
> recall to me what they are?

ECHOPRT, ALTWRERASE, NOKERNINFO, and the CSIZE bits (CS5/CS6/CS7/CS8).
You can find the extension in question described in the moussh source;
search for MOUSETTY_OP_ and missing-pty-modes to find the relevant
pieces, including an English description in private-algs.txt.  I'm not
sure what the best way to handle the CSIZE bits within the standard
framework is; I don't think it's worth holding up the rest of them for
that.

I can also copy it here if anyone finds that easier, but I'm inclined
to doubt anyone cares who hasn't already grabbed the moussh source.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 30 Mar 2013 02:01:07 +0000

On 30 Mar 2013, at 01:52, Mouse <[email-masked]> wrote:

> just a matter of bureaucratic hoop-jumping to turn it

I can probably help there. If an I-D on this appears to have consensus I'd be happy to sponsor it to become an RFC. So far it looks good

S

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 29 Mar 2013 23:37:30 -0400

> draft that says 'now number 42 will mean this' to have it added to the

On the contrary, that's exactly what you should do.  What you should
generally _not_ do is appear out of nowhere and say "here's code that
uses this IANA-managed code point that hasn't been registered; please
distribute and run it", because that creates interoperability problems.
What happens when someone else comes along and uses that code to mean
something else?  Now you have two different programs floating around on
the Internet that claim to support the same protocol, but they don't
agree on what that protocol means.  It's not the same as making a change
that's only visible to programs that link against your library; this
change is visible to the entire Internet.

The answer is that we avoid that problem by having a central authority
(IANA) whose job is to maintain a record of which codes mean what, and
to avoid assigning or registering conflicting values.  In this case,
mostly because the number of available codes is small(*), the only way
to obtain a code is to publish an IETF consensus document(+) which
allocates the number and (presumably) describes its use.

That said, it's certainly not unreasonable to implement things that are
still under discussion to see whether they will work.  In fact, the SSH
community has traditionally been very proactive in this regard,
resulting in an unusually short delay from when a new feature is
standardized to when it becomes generally available.  The only part
that's a problem is letting such test code "escape" onto the open
internet, such that people think they're running an implementation of
SSH when really they're running an implementation of something slightly
different from SSH.

If you were to write an I-D, using the templates Chris said he'd
forward, you should expect no significant trouble getting it published.

-- Jeff

(*) In fact, my sense of the prevailing philosophy in the IETF at the
time SSHv2 was standardized was that, with certain notable exceptions,
such namespaces should have relatively high bars for registration, even
when space was not so scarce.  Things have changed somewhat since then,
but this is a scarce namespace and would likely still have a fairly
restrictive policy were the document published today.  Note that most of
SSHv2's namespaces are easily extensible by anyone with a registered
domain name, and that a number of others, such as message numbers, are
designed so that portions of the namespace can be reused across
different parts of the protocol.  We tried pretty hard to be liberal in
this regard, and it has paid off.

(+) essentially, an RFC which has gone through the correct series of
consensus calls and approvals on the way to publication

---

From: Mouse <[email-masked]>
Date: Sat, 30 Mar 2013 00:13:10 -0400 (EDT)

> Now you have two different programs floating around on the Internet
> that claim to support the same protocol, but they don't agree on what
> that protocol means.

We _already_ have that.  I've seen (relatively) numerous embedded
devices that claim to speak ssh, including using port 22, but actually
speak a closely related protocol in which the private-part@domain
extensibility mechanism does not work the way it does in ssh.  (I
haven't probed the envelope of the issue enough to know whether it is
completely busted or busted only partially; I just know that when I
don't turn that stuff off, they ungracefully close the connection on me
when talking with moussh.)

> The answer is that we avoid that problem by having a central
> authority (IANA) whose job is to maintain a record of which codes
> mean what, and to avoid assigning or registering conflicting values.

Of course, even this much is still only advisory in most respects.  The
IETF, IANA, and related bodies have no ability, either de jure or de
facto, to prevent you, me, or anyone else from running not-quite-ssh
software such as you'd now get by checking out moussh's IUTF8 branch
(which now includes IUTF8 with value 42, as discussed upthread).

...this is just as well, actually, because ssh as specified is
basically unimplementable on many - most? - Unix variants.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Sat, 30 Mar 2013 19:56:04 -0400

> when talking with moussh.)

<sigh>  What's supposed to happen here is that you match on the server
version, turn off the features that cause it to break, and then stop
buying from that vendor until they fix their stuff.  :-)

Unfortunately, for that to be effective, you usually have to be large
and/or numerous.  Either that, or you file a bug report, and they fix
it.  I've heard that sometimes that works, against all reasonable
expectations.  Sometimes.

> (which now includes IUTF8 with value 42, as discussed upthread).

True, and this is as it should be.  We do seem to have a lot of
influence over what actually gets deployed, so the system seems to
mostly work.  In the meantime, I like to think that being "opt-in" gives
us the flexibility to be fairly particular about things.  For that
matter, the same is true of the @domain stuff - since it's easy for
others to accept extensions, we don't have to standardize everything
anyone might want to do.

> basically unimplementable on many - most? - Unix variants.

Hm?  That seems fairly surprising, given that we had what claimed to be
working implementations before we ever finished the specs.  Perhaps
you'd care to elaborate (maybe in a new thread) ?

-- Jeff

---

From: Mouse <[email-masked]>
Date: Sat, 30 Mar 2013 21:04:36 -0400 (EDT)

>> I've seen (relatively) numerous embedded devices that claim to speak
>> ssh, including using port 22, but actually speak a closely related
>> protocol in which the private-part@domain extensibility mechanism
>> does not work the way it does in ssh.  [...]
> <sigh>  What's supposed to happen here is that you match on the
> server version, turn off the features that cause it to break, and
> then stop buying from that vendor until they fix their stuff.  :-)

Yeah.  And, if I had control over the relevant buying decisions, that
might actually happen.

> and/or numerous.

That too. :(  Name-and-shame occasionally works, too; if I could recall
which devices they were, I'd name them here.

>> The IETF, IANA, [etc] have no ability [] to prevent you, me, or
>> anyone else from running not-quite-ssh software such as [...]
> True, and this is as it should be.

Agreed.

>> [...] ssh as specified is basically unimplementable on many [...]
> Perhaps you'd care to elaborate (maybe in a new thread) ?

Absolutely.  I'll reply to just this part and change the Subject:.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Sat, 30 Mar 2013 21:57:30 -0400 (EDT)

> Hm?  That seems fairly surprising, given that we had what claimed to
> be working implementations before we ever finished the specs.

The basic problem is that various strings are specified as being UTF-8,
but things in question are things that the systems in question don't
store as character strings, but rather as octet strings.  This means
that either an ssh implementation has to have some configuration switch
telling it what the string encoding is in use or it will conform only
if the local admins stick to UTF-8 for those things.  (In some cases
it's conceptually possible to make the encoding setting user-specific,
but in other cases, such as usernames, it's not.)  There _is_ a third
option, sort of, in that the implementation can pretend that anything
that doesn't stick to the ASCII range - or, perhaps, which isn't a
valid UTF-8 string - doesn't exist, but that's really just a hardcoded
version of the "encoding in use is ASCII" (or "...is UTF-8")
configuration, combined with a particular error recovery technique upon
seeing octet sequences which are invalid for that encoding.

In moussh's case, I chose to treat those things as opaque octet
strings.  If the local system does not use UTF-8 for such things,
moussh will not conform to ths spec.  (This is outlined in moussh's
documentation, of course.)

Actually, it's not quite unimplementable.  The "reject anything
non-ASCII" technique sketched above would lead to conformant
implementation.  (One that could reasonably be seen has having a
quality-of-implementation defect, but it would conform.)

I don't know what implementations other than moussh do.  I just had a
look at the sshd_config manpage on one NetBSD machine, which gives me
some reason to think that OpenSSH (that being what that NetBSD version
provides) doesn't have an encoding setting.  This means either I missed
something (always a possibility), or it treats the relevant strings as
octet strings rather than character strings (and thus, like moussh,
does not conform unless the system uses UTF-8 for all such strings), or
it has some encoding assumption (ASCII? UTF-8? 8859-1?) effectively
hardwired into the code.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign