Subject: Re: UTF8

From: Sam Hartman <[email-masked]>
Date: Mon, 03 Jan 2005 20:39:00 -0500

>>>>> "der" == der Mouse <[email-masked]> writes:

    >>> All the same UTF-8 issues I've raised repeatedly,
    >> My requirement as an IESG member is that it be possible to have
    >> a properly internationalized ssh.  Among other things that
    >> means the characters in usernames and passwords need to belong
    >> to some character set.

    der> Why?  Perhaps this is the fundamental part I'm missing.  What
    der> does "properly internationalized" mean - or perhaps more
    der> precisely, what is there about being "properly
    der> internationalized" that demands that usernames, passwords,
    der> and filenames consist of character sequences rather than
    der> octet sequences?

I'd appreciate replies off-list as I believe we are outside of the
scope of this working group.

Humans use our software.  They typically enter characters using an
input method  for items such as usernames, passwords and
filenames.

One input method that is relatively common is a keyboard that maps
keycodes to characters.  This method tends to be mostly OK even if you
assume it produces octets not charactares.

Other input methods produce different octet sequences for semantically
similar content.  I gave an example of combining accents vs single
characters in the message introducing this thread.  So, the set of
octets produced depends not so much on what the user does, what keys
they press, or even what is displayed on their screen; it depends on
implementation details of the input method the user's OS happens to
use.  The user does not typically have enough control (and almost
certainly has insufficient knowledge) to end up with the other octet
sequences that are the same semantic content.

So, if we treat passwords as octet-strings then whether the user can
type their username or password will depend on what input method they
are using.  They may not even have control over this.

We, the IETF, e have decided for the most part that interoperability
requires that things work independent of what input method is used.

---

From: der Mouse <[email-masked]>
Date: Tue, 4 Jan 2005 00:23:55 -0500 (EST)

>> What does "properly internationalized" mean - or perhaps more
>> precisely, what is there about being "properly internationalized"
>> that demands that usernames, passwords, and filenames consist of
>> character sequences rather than octet sequences?

In general, perhaps, but insofar as it bears on implementing ssh, I am
inclined to disagree - which is why I'm replying on-list anyway.

> [...]

As I see it, this amounts to "the IETF position is that humans think of
these things as character strings, so we demand that they be handled as
character strings by the protocol".

What is the IETF position, then, on how someone such as me should
handle the situation I'm faced with: writing software specified from
this point of view (ssh, in my case) for systems on which these
entities are _not_ character strings (a fairly traditional Unix
variant, NetBSD in my case)?  I'm faced with an encoding-agnostic
filesystem interface and implementation, wherein filename components
are sequences of octets not including 0x00 and 0x2f, independent of any
characters; I'm faced with password hashing routines that work with
octet strings, not character strings; etc.

Are such systems beyond the pale for the IETF, and I can do anything I
want, with a suggestion that I try to stay within something like the
spirit of the spec?  Is it simply not possible to implement ssh (or
anything else specified with similar normalization rules) on such a
system within the spec without converting all the affected code
(filename, username, and password handling in ssh's case) to the
character-string paradigm?  Am I required to reject attempted non-ASCII
strings in these places for no reason other than an inability to know
what the user intended the character set - if any - to be?  (For that
matter, what grounds are there for assuming that octets in the ASCII
range are intended to correspond to ASCII characters, rather than, say,
KOI-7?)

Or what?

Given how common such systems are, it seems a bit odd that the IETF
would take a position so apparently incompatible with them.  As an
implementer I find the situation rather confusing; there's obviously
something I don't understand going on, and I'd like to know what the
IETF's idea of the right thing for me to do here is.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Jeffrey Altman <[email-masked]>
Date: Tue, 04 Jan 2005 02:41:26 -0500

> character strings by the protocol".

Absolutely not.  The IETF position is that if I am attempting to login
to machine H via SSH, I should be able to do so by knowing the necessary
bits: username, password, etc.

The requirement is that no matter what user interface I use to enter
these bits, I should be able to successfully authenticate.  Now if I 
happen to be in front of a keyboard based interface which is Unicode
aware and happens to generate "SMALL LETTER u WITH DIAERESIS" as two
code points represented as two 32-bit values or 8 octets instead of the 
non-Unicode aware system which uses a single code point represented as
a single byte, I have a problem.  I type exactly the same thing on
both keyboards and get extremely different octet strings.

Are you telling me that once I configure a login to work from one
particular platform and user interface configuration that I should
be locked into that choice exclusive of all the other system types
and user input methods which are available?

I would find it hard to believe that anyone could decide that this is
desireable.

> octet strings, not character strings; etc.

As an AFS developer I am very sympathetic to the situation. 
Unfortunately. there are no true raw octet strings.  Octet sequences are
created within a context and without knowing the context it is not
possible to properly manipulate the octets.  At the present time AFS
does not support a notion of storing character-set context information.
This causes severe problems for users who want to access the names
associated with directories and files from heterogeneous systems.
File names created from most Unix user interfaces in Western Europe
will produce strings using Latin-1 code points.  Those from Eastern 
Europe will use Latin-2.  Linux systems may store unnormalized UTF-8.
Windows systems will store one of the many IBM/MS DOS OEM code pages.
A name created on one system not only will be displayed to users of
another system something which is incorrect but the name may be 
something which is completely unparsable.

At the moment the only safe set of strings that can be used are those
restricted to US-ASCII.  This is because US-ASCII is the only common
set of values which will be properly interpretted without additional
context information which is not available.

In the the long run we are going to need to fix AFS to do one of two
things:

(1) store context information associating the character set used to
     create each name AND provide the means necessary for file servers
     to be able to translate names from one character set to all the
     other possible sets.

(2) provide support for a normalized character set which is inclusive
     of all characters which users may be able to enter.

Having worked on the character set translation capabilities of C-Kermit
I can tell you that storing context information and providing 
translation is lossy and imperfect.  UNICODE solves the problem in a
much nicer and heterogeneous manner.  It is by no means perfect but
biting the bullet and supporting it makes the end user experience oh
so much nicer.

In the coming year I will be adding UNICODE support to AFS.  I expect
that all file systems will have to provide support for it in the years
to come.   Operating systems which do not provide support for character
set processing will find a smaller and smaller percentage of users.

> range are intended to correspond to ASCII characters, rather than, say,

You have to draw the line somewhere if you are going to make progress
at improving cross platform user experience.  Systems without support
for character-set processing are useful only when all of the systems
they share information with are used in exactly the same context.
In a distributed heterogeneous environment such as the Internet,
this assumption cannot be made.

If a system wants to assume that all of its local input is in KOI-7
and an SSH implementation wants to be able to support that, then the
implementation must provide for character set translation from KOI-7
to UNICODE.  If you need such translation tables, they are available
from the UNICODE consortium and are implemented within a wide number
of open source packages.

> IETF's idea of the right thing for me to do here is.

You do what Kermit has done since 1981.  When moving information between
systems you convert from the local character set to a network neutral
form and then the receiver converts its local form.  Before the advent 
of UNICODE Kermit was forced rely on the user to choose an intermediary
character-set which would be inclusive of all characters used and be
understood by both systems.  When this was not possible, substitution
rules and best guesses forced the data stream to become lossy.

With the availability of UNICODE the available set of characters which
can be sent without loss has been greatly enhanced.  Normalization rules
are used to prevent multiple representations of a common input form
from preventing interoperability.  While this has a negative impact on
the ability to display strings to the end user after use; it enhances
the ability to provide for cross platform comparison and computation.

Jeffrey Altman

---

From: [email-masked] (Niels Möller)
Date: 04 Jan 2005 13:45:01 +0100

der Mouse <[email-masked]> writes:

> I'm faced with an encoding-agnostic
> characters;

Please leave the file system issues out of it for now. What's of
primary importantance are the core drafts, and those deal with
usernames and passwords in utf8 form, *not* file names. The issues for
filenames, e.g. in sftp, are slightly different, and not relevant to
the core drafts.

> I'm faced with password hashing routines that work with

> Am I required to reject attempted non-ASCII

I'm assuming you're talking about the server implementation now
(client side is comparatively trivial; convert input to utf8 based on
the current $LC_CTYPE). On the server side, problem is that at login
time, you don't know the user's $LC_CTYPE. My recommendation is as
follows:

1. Chose one default encoding (be that plain ascii, or latin1, or
   koi-7, or normalized utf-8, depending on your context and
   preference).

2. Provide an option for the sysadmin to say that on his or her
   particular system, some other character set is used for user names
   and passwords.

Then convert the usernames and passwords you get on the wire to the
selected encoding. That's almost solves the problem, and it's no big
deal.

Optionally, to support systems where different users use different
character sets for their usernames and/or passwords, use some per user
configuration or kludgery to figure out the user's character set.

I'll be happy to discuss these implementation issues (my
implementation doesn't get non-ascii quite right yet either), but we
should probably do that off-list.

> would take a position so apparently incompatible with them.

Do you have some numbers to back that up? I've seen quite some number
of unix systems, but as far as I can recall, I've *never* seen one
where usernames and passwords used non-ascii characters. (I *have*
seen plenty of non-ascii filenames, but as I said, that's a different
issue, and irrelevant to the core drafts). I live in latin1-land, not
asia, though.

Best regards,
/Niels

---

From: der Mouse <[email-masked]>
Date: Tue, 4 Jan 2005 11:14:07 -0500 (EST)

>> As I see it, this amounts to "the IETF position is that humans think
>> of these things as character strings, so we demand that they be
>> handled as character strings by the protocol".
> Absolutely not.  The IETF position is that if I am attempting to
> login to machine H via SSH, I should be able to do so by knowing the
> necessary bits: username, password, etc.

But which is (say) the username?  The character string g e-acute r a r
d, or the octet string 0x67 0xe9 0x72 0x61 0x72 0x64?  A human is more
likely to think of it as the former; the reality to the computer is
more likely to be the latter.  (At least assuming an encoding-agnostic
user database such is at issue here.)  So does "entering the username"
mean typing g e-acute r a r d (for any of the various ways of typing
those characters), or does it mean typing whatever is necessary to
generate 0x67 0xe9 0x72 0x61 0x72 0x64?  (Note that either or both may
be impossible to do under reasonably plausible circumstances.)

The stated IETF position on interoperability makes no sense unless it's
based on the former of those two positions, which is why I phrased my
gloss on it the way I did.

> particular platform and user interface configuration that I should be
> locked into that choice exclusive of all the other system types and
> user input methods which are available?

No; even if you go with the octet-string model, you are locked in only
to system types and input methods that permit you to generate that
octet string.

Very much the way, in fact, that the character-string model locks you
into the ability to generate the desired character string.

It's just a question of which lock you prefer to be in.

> (1) store context information associating the character set [...]
> (2) provide support for a normalized character set [...]

Only if AFS is (or becomes) philosophically committed to considering
file names to be character strings.  (While this may not be a wrong
choice, it is still a choice, and you seem to be arguing from a
position that is unaware of that.)

Character strings make a lot of sense from some points of view, yes -
and that's true not only of filenames but of other things, such as
usernames.  Character strings are a better match to the way most people
think of them, if nothing else.  But they bring a whole passel of
problems with them, some of which we're discussing here.

The biggest problem is perhaps the one that got me writing to the list
about this: a large body of existing code that takes the octet-string
point of view and what the best way is to impedance-match it to a spec
that takes the character-string point of view.

> at improving cross platform user experience.

I guess what I don't quite see is how rendering ssh unimplementable (or
implementable only crippledly, such as by restricting everything to
ASCII) on traditional Unix systems is going to improve anything.
Honestly, what I expect it to do is to create two imcompatible dialects
of ssh, one taking the character-string point of view and the other
taking the octet-string point of view, with humans rqeuired to deal
with the mismatch whenever they meet.  (There may be a third dialect
that imposes willy-nilly some guessed character set on the octet-string
environment....)

> Systems without support for character-set processing are useful only
> when all of the systems they share information with are used in
> exactly the same context.

I think that's too strong.  Rather, I would say, they allow mismatches
to show through in some form, usually in the form of text in one
character set being displayed in another and coming through as
nonsense.  This is not to say that they're _not_ useful in the face of
such things, just _less_ useful, or at least less transparently useful.

The corresponding upside, of course, is a simpler implementation and
more flexibility.

>> [...] I'd like to know what the IETF's idea of the right thing for
>> me to do here is.
> You do what Kermit has done since 1981.  When moving information
> between systems you convert from the local character set to a network
> neutral form

But this step cannot be done when I'm sending, because all I have is an
octet string.  I don't know what character set it's in; strictly
speaking, I don't even know whether it _is_ in a character set, though
for usernames and passwords it is extremely likely that it is, at least
in someone's mind (and for filenames it's reasonably likely).

> and then the receiver converts its local form.

And this is equally impossible, for similar reasons.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Tue, 4 Jan 2005 11:56:36 -0500 (EST)

> Please leave the file system issues out of it for now.

Okay.  The fundamental issues are the same anyway, as I see them.

>> Am I required to reject attempted non-ASCII strings in these places
>> for no reason other than an inability to know what the user intended
>> the character set - if any - to be?
> the current $LC_CTYPE).

Hm?  I have no LC_CTYPE in my environment and have got by fine without
it for years.  The only places I've been able to think of that anything
knows that I'm using 8859-1 (which is what I use by default) are:

- The font I use to display text;
- The Content-Type: header my MUA is configured to default to;
- $LESSCHARDEF (and that just knows which characters are printable,
  basically; it couldn't tell you 8859-1 vs 8859-2);
- The mapping table between what I might loosely call compose sequences
  and octet values my text editor uses;
- The display substitutes my text editor uses when displaying non-ASCII
  text on an ASCII-only display;
- The non-ASCII keycodes I've chosen to xmodmap onto real keys on my
  keyboard in my X startup scripts;
- My own mind.

I don't know what I should set LC_CTYPE to to indicate 8859-1.  I don't
even know where to look to find that out - I don't use non-ASCII for
much beyond inter-human communication (eg email).

> Do you have some numbers to back that up?  I've seen quite some
> number of unix systems, but as far as I can recall, I've *never* seen
> one where usernames and passwords used non-ascii characters.

Come to think of it, I've never seen one where they were actually used
(or where I knew they were used, at least - for example, I have known
very few passwords besides my own).  I don't know how many I've seen
where they would work if attempted.  I don't even recall hearing of
anyone attempting it, either, regardless of the outcome; I'll have to
ask the more internationalized of my friends.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Sam Hartman <[email-masked]>
Date: Tue, 04 Jan 2005 12:22:11 -0500

>>> precisely, what is there about being "properly
    >>> internationalized" that demands that usernames, passwords, and
    >>> filenames consist of character sequences rather than octet
    >>> sequences?
    >> I'd appreciate replies off-list as I believe we are outside of
    >> the scope of this working group.

    der> In general, perhaps, but insofar as it bears on implementing
    der> ssh, I am inclined to disagree - which is why I'm replying
    der> on-list anyway.

*sigh*  Teach me to try and explain something.

    der> What is the IETF position, then, on how someone such as me
    der> should handle the situation I'm faced with: writing software
    der> specified from this point of view (ssh, in my case) for
    der> systems on which these entities are _not_ character strings
    der> (a fairly traditional Unix variant, NetBSD in my case)?  I'm
    der> faced with an encoding-agnostic filesystem interface and
    der> implementation, wherein filename components are sequences of
    der> octets not including 0x00 and 0x2f, independent of any
    der> characters; I'm faced with password hashing routines that
    der> work with octet strings, not character strings; etc.

I think this is all mostly an open issue.  IN practice what unix
server implementers seem to do is to treat the username and password
as octet-strings.  Neils has proposed ways of doing significantly
better than that.

I do think there is significant room here for implementers to decide
what works best, write it up and publish it.  Based on timing it would
probably work better as an informational RFC than as input to the core
drafts.

--Sam

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 04 Jan 2005 10:30:19 -0700

> selected encoding. That's almost solves the problem, and it's no big

Precisely.  By doing this you increase interoperatability from
only those systems that use the same character sets (i.e., only
koi-7 system interoperate with each other) to interoperating in
with all clients, as long as the same character set is used
on the server for passwords.

If this is too restrictive (i.e., different users on the same
server use different character sets for their passwords), do

> configuration or kludgery to figure out the user's character set.

For example, a non-script dot-file in the users home directory
that you can read to get such useful information as $LC_TYPE,
preferred umask, etc.  (Things you'd really like to know, but
don't want to run a user script to find out.)

> issue, and irrelevant to the core drafts). I live in latin1-land, not

I will say that windows can and does use non-ascii usernames and
passwords, and it is not an uncommon operating system, though it
is not the most common of server platforms.

- Joseph

---

From: [email-masked] (Niels Möller)
Date: 04 Jan 2005 19:41:45 +0100

> knows that I'm using 8859-1 (which is what I use by default) are:

It's basically needed whenever information is transferred to and from
remote systems, using protocols that are charset aware. E.g. if you
use a text web-browser to access pages written in utf-8, your browser
will do a better job displaying the content if it knows if your terminal
is configured to use latin1, utf8 or koi8r.

I think the primary reason I started setting LC_CTYPE many years ago
was that GNU ls uses the standard C locale system and isprint() to
figure out which characters in filenames are printable, and which
should be displayed specially. Don't know how BSD ls behaves in this
respect.

  $ touch räksmörgås
  $ ls
  räksmörgås
  $ LC_CTYPE='' ls
  r?ksm?rg?s

>   basically; it couldn't tell you 8859-1 vs 8859-2);

These are application specific configurations. I think it's preferable
to use a single configuration mechanism, namely LC_CTYPE.

> I don't know what I should set LC_CTYPE to to indicate 8859-1.

If you're saying C locales are broken, I'll agree that they're broken
in several different ways. That's why I wrote "comparatively trivial",
not "trivial"...

IMO, it *ought* to work to simply set LC_CTYPE to "iso-8859-1" if
that's what you're using, and LC_PAPER to "a4" if you want that. But
the current state of affairs seems to be that the simple way doesn't
work, one has to figure out a corresponding geographic region. E.g. I
use LC_CTYPE=sv_SE, and that seems to work for me. If I ever wanted,
say, "b5" as the default paper size, I have no idea which region, if
any, that choice would correspond to.

Regards,
/Niels

---

From: Chris Lonvick <[email-masked]>
Date: Tue, 11 Jan 2005 11:28:03 -0800 (PST)

Hi,

We've received some coments recently.  I'll summarize them and note what I
intend to do with them in the core IDs.  I'd appreciate feedback.

1)  Double spaces after the period after an initial in a name.
Issue:  xml2rfc has started double spacing between the end of a sentence
and the start of the next.  I saw that problem with "etc." used at the end
of a sentence.
Proposed Resolution:  I'm not sure what I can do about this but I'll play
around and see.

2)  Use of commas in "..may, or may not be.."
Issue:  English grammar.
Proposed Resolution:  My Scribner Handbook of English (after cleaning off
all of the dust) says that it's OK to have "..may or may not.." without
commas.  I'll clean those up.

3)  Use of commas in "..display forwarding in SSH (or other, secure
protocols), combined with.."
Issue:  English grammar.
Proposed Resolution:  I'll make the change to "display forwarding in SSH
(or other secure protocols), combined with..".

4)  SSH_DISCONNECT_* descriptions
Issue:  (From the note by der Mouse)  In sections 4.2 and 4.2.2 of
[NUMBERS]-10, the table headings make it appear that the SSH_DISCONNECT_*
names *are* the "description" strings appearing in an SSH_MSG_DISCONNECT
packet, which I assume is false - if that were true, the "description"
would carry no information not already present in the "reason code".
Specifically, the (new) first two lines of text in 4.2.2 comes very close
to stating that the table shows the strings as they must appear in a
DISCONNECT packet.  The text around line 425 further implies that the
"description" strings are standardized, which I had thought was not the
case - and which makes no sense because they then carry no information and
might as well be skipped.
Proposed Resolution:  Can I get some feedback on this?  I was under the
assumption that the "descriptions" are the standardized text to send in
the "description" field.  I too felt that was a bit redundant but figured
that it was setting the precendence for the descriptions associated with
local namespaces.  What's the right thing to do here?

5)  Opcode Arguments in [NUMBERS]-10 and [CONNECT]-23
Issue:  TTY_OP_END is listed with the prefix of "TTY_OP_" but none of the
others include that prefix.
Proposed Resolution:  I need some feedback on this as well.  Do the opcode
arguments all get the prefix of "TTY_OP_"?

6)  PENDIN opcode
Issue:  Why does PENDIN have an assignment.  It is really a dynamic tty
driver internal state.  It may not be the sort of state bit that it makes
any sense at all to push across the network.
Proposed Resolution:  I need someone to verify that.

7)  SPKI
Issue:  (From note by Peter Gutmann) The incompletely-specified X.509
format has been removed but the totally unspecified SPKI format (RFC2693
specifies neither a cert nor signature format, that was in the
long-expired draft-ietf-spki-certstructure, so it's not possible to
provide either a cert or a "signature blob in format specific encoding")
and the incompletely-specified PGP format (RFC2440 specifies a large
number of options, signed attributes, packet types, etc etc for
signatures) are still present.  These should really be removed as well if
it's not possible for anyone to implement them.
Proposed Resolution:  I'll remove SPKI where appropriate unless anyone has
a better proposal.

8)  UTF8
Issue:  The Editor is not sure what to do with all of the emails flying
back and forth.
Proposed Resolution:  Unless we get some consensus on proposed text, I'm
not going to make any modifications to the IDs.

9)  KEX Message Numbers
Issue:  (From note by Ben Harris) Requests for assignments of new message
numbers in the range of 1 to 127 MUST be done through the STANDARDS ACTION
method as described in [RFC2434].
This seems slightly wrong to me, in that message numbers in the ranges
30-49 and 60-79 are effectively assigned by whoever owns the KEX or
USERAUTH method in use, not by IANA (thought of course some KEX and
USERAUTH names are assigned by IANA.
Proposed Resolution:  This sounds right to me.  Unless someone objects,
I'll integrate the wording proposed by Ben into [NUMBERS] and duplicate it
in [USERAUTH].

Thanks,
Chris

---

From: der Mouse <[email-masked]>
Date: Tue, 11 Jan 2005 15:51:50 -0500 (EST)

> We've received some coments recently.  I'll summarize them and note
> what I intend to do with them in the core IDs.  I'd appreciate
> feedback.

You asked for it, so I'll give it. :-)

> Issue:  xml2rfc has started double spacing between the end of a
> sentence and the start of the next.

There's nothing wrong with that, surely?  The space after the
punctuation ending a sentence normally *should* be wider than an
inter-word space, and in a monospaced font a double space is, I
thought, the canonical way to do it.  (The spacing ratio in
well-typeset text is reasonably close to 2:1....)

What the correct way to mark certain dot-space combinations as not
ending sentences is for xml2rfc I don't know.  But it does seem to me
that if xml2rfc is producing problems, maybe the right fix is to stop
using it and switch to something better behaved?  Is there any reason
XML (which I assume is involved from the xml2rfc name) needs to be
involved?

> 6)  PENDIN opcode
> Issue:  Why does PENDIN have an assignment.  It is really a dynamic
> tty driver internal state.  It may not be the sort of state bit that
> it makes any sense at all to push across the network.

Surely all the verification it takes is to grep for PENDIN in your
favourite open-source kernel's tty code and look at how it's used.

For the rest of the points listed, I'm happy with the resolution given
(well, except for the UTF-8 issue, and that I'm at least content with,
pending some kind of consensus on what the right thing for
encoding-agnostic systems is to do and what that means the core drafts
should say).

And, Chris...thank you for all your unstinting work taking care of the
drafts.  It's a mostly thankless job, one that is noticed only when
done badly, and I for one appreciate your dedication to it.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Martin Schröder <[email-masked]>
Date: Tue, 11 Jan 2005 23:16:17 +0100

> well-typeset text is reasonably close to 2:1....)

That depends on your local context (e.g. in the USA the
inter-sentence space is larger while in europe it's mostly the
same; in TeX it's \nonfrenchspacing vs. \frenchspacing).

Best regards
    Martin
-- 
                    http://www.tm.oneiros.de

---

From: der Mouse <[email-masked]>
Date: Tue, 11 Jan 2005 21:45:23 -0500 (EST)

>> The space after the punctuation ending a sentence normally *should*
>> be wider than an inter-word space, [...]
> inter-sentence space is larger while in europe it's mostly the same;
> in TeX it's \nonfrenchspacing vs. \frenchspacing).

I thought that was correlated more with language than with location; is
that a mistaken impression?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Jon Bright <[email-masked]>
Date: Wed, 12 Jan 2005 08:35:04 +0100

> that a mistaken impression?

I'm English, living in Germany.  British English does use longer 
inter-sentence spacing.  German doesn't.  I'm also of the opinion that 
this is language-specific, not locale-specific.

Regarding xml2rfc - I believe this is (nowadays) the standard tool for 
producing drafts.  Certainly, converting the core drafts to a different 
format at this stage would doubtless be significant effort.  I note from 
  http://xml.resource.org that they're now recommending that one submit 
both text and XML versions, so I think a different tool is probably 
excluded.

The two spaces issue seems to be controlled by the two_spaces procedure 
in xml2rfc.tcl.  They have a check there for abbreviations, which seems 
to use the regexps *[A-Z][A-Z] and *[A-Z][a-z][a-z] to detect an 
abbreviation.  It *seems* like this should pick up initials in names and 
not add a space.  But maybe my primitive understanding of TCL isn't 
sufficient.

I think one could probably disable the whole insert-two-spaces behaviour 
by changing the content of the two_spaces function to be just "return 
$glop" (I only think this, I haven't tested).  Not what one really 
wants, but maybe a quick and easy for the time being.  Contact with the 
xml2rfc people will probably produce a better answer, but may take longer..

-- 
Jon Bright
Silicon Circus Ltd.
http://www.siliconcircus.com

---

From: [email-masked] (Niels Möller)
Date: 12 Jan 2005 18:10:11 +0100

> > issue, and irrelevant to the core drafts). I live in latin1-land, not

The problem under discussion was systems, in particular unix servers,
where the particular character set used for non-ascii usernames and
passwords is tricky to determine, *and* non-ascii characters are
actually used.

My current belief is that such systems are very rare, so I'd like to
know if I'm wrong.

> is not the most common of server platforms.

I'm under the impression that those windows systems all use utf-8, not
different character sets on different systems or for different users.
Then, they don't matter, because they don't have a problem. Right?

Regards,
/Niels

---

From: [email-masked] (Niels Möller)
Date: 12 Jan 2005 19:30:31 +0100

> 4)  SSH_DISCONNECT_* descriptions

...

> local namespaces.  What's the right thing to do here?

The description is an arbitrary human readably strings describing the
reason for the disconnection in some more detail. The text in will
usually end up on the users terminal or in the servers log file
(perhaps only when running in verbose mode).

I think the text in draft-ietf-secsh-transport-22.txt is ok, it's just
the table header that must be changed, to something like

   Reason code   Symbolic name
   1             SSH_DISCONNECT_HOST_NOT_ALLOWED_TO_CONNECT

Or actually, the table is duplicated in NUMBERS, section 4.2.2, so
perhaps it could be deleted all together.

It would be desirable with a table describing under what circumstances
each value should be used, but perhaps we can live without that. Most,
but not all, of the symbolic names are self-explanatory.

> arguments all get the prefix of "TTY_OP_"?

It really doesn't matter. The wire protocol only needs the opcodes,
not the symbolic names. Implementations will use their own prefixes or
other namespace mangling when defining these constants, but that
doesn't matter for interoperability.

So I'd say no change is needed. It's fine to have the "meta opcode"
TTY_OP_END stick out in the list.

> Proposed Resolution:  I need someone to verify that.

It's not entirely clear to me in which way it is an "internal state
bit". It's documented as

  PENDIN    If PENDIN is set, any input that has not yet been read is
            reprinted when the next character arrives as input.

Is the difference that it is reset automatically by the tty driver
after the requested action is completed? Compare for example to

  NOFLSH    If NOFLSH is set, the normal flush of the input and output
            queues associated with the INTR, QUIT, and SUSP characters
            will not be done.

which is not an "internal state bit".

> Proposed Resolution:  I'll remove SPKI where appropriate unless anyone has

It's unfortunate that an RFC on the spki format doesn't exist
(the latest and outdated draft is at
http://theworld.com/~cme/spki.txt).

I agree we'll have to remove spki from the ssh spec.

> not going to make any modifications to the IDs.

The way I see it, the recent discussion has made clear that the
benefit of requiring usernames and passwords to be normalized when
sent as utf8 on the wire, is very questionable.

The current text (draft-ietf-secsh-userauth-25.txt) is ok with me, and
also pretty vague. It's seems somewhat immature for this wg to make
recommendations on particular normalization conventions, so perhaps
the sentence "SSH implementations that both store the passwords and
compare them SHOULD use [I-D.ietf-sasl-saslprep] for normalization."
should be deleted or changed to a "MAY".

But I think an even better solution is to delete the entire paragraph
(the only place where the userauth document speaks about
normalization). Instead, tweak the architecture document as follows:

Before:

   The client and server user names are inherently constrained by what
   the server is prepared to accept. They might, however, occasionally
   be displayed in logs, reports, etc. They MUST be encoded using ISO
   10646 UTF-8, but other encodings may be required in some cases. It
   is up to the server to decide how to map user names to accepted
   user names. Straight bit-wise binary comparison is RECOMMENDED.

The last sentence looks very wrong; it will surely break
interoperability between systems with different input methods (IIRC,
that sentence was added at the request of the IESG, but it's still
wrong). The "in some cases"-part also looks wrong to me, and the
"displayed in logs" doesn't seem very relevant; that's not the main
issue. Proposed new text:

   User names and passwords are inherently constrained by what the
   server is prepared to accept. They MUST be encoded using ISO 10646
   UTF-8. No normalization is required on the wire; it's the server's
   responsibility to apply any normalization steps needed before
   accessing the user and password database. For example, systems that
   use latin-1 for user names and passwords MUST make sure that the
   character set conversion treats different but equivalent
   representations of accented characters in the same way, and systems
   that use UTF-8 for user names and passwords should apply
   appropriate normalization, e.g. using [I-D. ietf-sasl-saslprep].
   Normalization on the receiver side ensures that when a user enters
   a username and password, the authentication process will work
   regardless of the operation system, client software and input
   method used on the sending side.

I intentionally wrote "should apply normalization" rather than "SHOULD
apply normalization" in the last-but-one sentence, because it's a
recommendation that must be applied consistently to all programs that
access the user database. It doesn't make sense as a hard requirement
on the ssh server implementation in isolation.

> I'll integrate the wording proposed by Ben into [NUMBERS] and duplicate it

That's a little cumbersome, but I think it's the right thing to do.

Regards,
/Niels

---

From: der Mouse <[email-masked]>
Date: Thu, 13 Jan 2005 00:21:48 -0500 (EST)

>             reprinted when the next character arrives as input.

Documented where?

I just had a look through the manpages I have at most ready hand
(NetBSD/sparc 1.4T) and the only documentation I find on it is in
termios(4) and reads

 PENDIN      /* XXX retype pending input (state) */

The string "PENDIN" occurs in only two of the drafts I have at hand,
assignednumbers-10 and connect-23.  Both document it the same way:

          62    PENDIN      Retype pending input.

> after the requested action is completed?

Not only that, it is *set and* reset by the tty driver when
appropriate, at least in the kernel code I looked at - it really is
exposing the internal state bit the tty driver uses, not just allowing
userland to request retyping.  Since it controls retyping of input
received but still buffered in the kernel, it makes no sense to push it
across the network, since there is no way to push that unread input
with it.

Indeed, I'm not sure why it is exposed to userland at all.  Providing a
way for userland to request retyping of pending input is not
unreasonable, but making it a state bit like this is a bizarre way of
doing that.  But that's not an issue for us.

> Compare for example to

> which is not an "internal state bit".

...and which is never changed except by userland (or by wholesale
resetting of tty state to whatever the kernel's idea of a clean tty
state is, upon something like revoke() or last close).

> sent as utf8 on the wire, is very questionable.

I see the value of specifying the charset on the wire at all as
questionable, since it demands that they be character strings; an
encoding-agnostic system has no way to convert its octet strings to
character strings.

As long as these things are specified as character strings on the wire,
I can't see anything sensible for an encoding-agnostic OS to do,
regardless of what encoding is chosen, regardless of normalization (if
applicable to the encoding), regardless of almost everything else in
this area.

> I intentionally wrote "should apply normalization" rather than
> "SHOULD apply normalization" in the last-but-one sentence, because
> it's a recommendation that must be applied consistently to all
> programs that access the user database.

*Exactly*.  It seems my choices as an implementer are to give up trying
to conform to the spec, or rework the entire OS and all third-party
software to use a character-string paradigm instead of the
encoding-agnostic octet-string paradigm it currently uses.

Is that really what we want?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Jeffrey Altman <[email-masked]>
Date: Thu, 13 Jan 2005 00:52:32 -0500

> Then, they don't matter, because they don't have a problem. Right?

Windows uses Unicode; a code page they call "ANSI" (who knows why it has
nothing at all to do with the organization) which is a variation on one
of the ISO Latin character sets; and a code page they call "OEM" which
comes from IBM's DOS.  Which character set is used depends on the API
which is being called.

Jeffrey Altman

---

From: [email-masked] (Niels Möller)
Date: 13 Jan 2005 12:19:18 +0100

> >             reprinted when the next character arrives as input.

Sorry, I should have given an reference. This description is copied from
http://www.mcsr.olemiss.edu/cgi-bin/man-cgi?termios. A search for
"PENDIN" and "termios" turns up the man pages for a few different
systems. The linux man page is even more cryptic,

       PENDIN all  characters  in  the  input queue are reprinted
              when the next character  is  read.   (bash  handles
              typeahead this way.)

[About utf8 passwords and usernames:]

> encoding-agnostic octet-string paradigm it currently uses.

I've outlined a simple way to solve the problem for the common cases.
I agree it's not 100% satisfactory, but from my experience, I think
you are exagerating the implementation trouble.

Regards,
/Niels

---

From: "denis bider" <[email-masked]>
Date: Thu, 13 Jan 2005 15:39:27 +0100

> I'm under the impression that those windows systems all use 
> utf-8, not different character sets on different systems or 
> for different users. Then, they don't matter, because they 
> don't have a problem. Right?

Windows is a dual-mode platform.

Windows NT/2000/XP/2003 natively use 16-bit Unicode, but the vast majority
of system APIs also come in so-called ANSI versions, which convert between
internal Unicode representation and, on the application side, whatever code
page is currently selected by the user or for the system. These code pages
are usually 8-bit but, as used in Asia in particular, they also include
multi-byte character sets (MBCS) with variable-length code sequences.

Windows 98/Me, a number of installations of which still persist, are
natively 8-bit and use internally whatever code page is configured for the
machine. Many system calls have a Unicode counterpart which translates the
strings to and from the internal 'ANSI' representation, but the windowing
subsystem (the graphical user interface) is totally ANSI and, on these
platforms, has no support for Unicode.

All versions of Windows from 98 and up provide a system character set
conversion API function which supports conversion to and from UTF-8. But the
internally used character set depends on the platform (16-bit Unicode for NT
platforms, or an 8-bit or variable-bit code page in Windows 9x/Me).

---

From: [email-masked] (Niels Möller)
Date: 13 Jan 2005 17:38:45 +0100

> Windows is a dual-mode platform.

But it still sounds like it doesn't have the problem der Mouse is
having with unix systems.

> of system APIs also come in so-called ANSI versions,

Then the ssh implementation (client or server) can stick to the
unicode api:s, and pass only unicode to the system (optionally with
normalization, if appropriate). 

> machine.

As long as this code page configuration is system wide, and available
to applications, there should also be no problem. The ssh
implementation only has to convert to and from utf8 (on the wire) and
the system charset, whichever that is.

Regards,
/Niels

---

From: Joseph Galbraith <[email-masked]>
Date: Thu, 13 Jan 2005 12:08:01 -0700

> Then, they don't matter, because they don't have a problem. Right?

True with the current drafts.  However, if we changed the drafts to
send passwords as 'octets' instead of 'utf-8', it would be a problem.

In fact, in such a scenario, I would either have to assume that
all clients are using 'utf-8' (BAD) or that all clients are using
the same thread locale as the windows operating system, for example,
for a japanese install of windows, SJIS.

In fact, geez, the problem seems fairly familiar!  It is the same
problem that 'octet' systems face now... except, instead trying to
guess what 'charset' the client is using, they are trying to guess
what 'charset' their password database is using.

So, we have two possible scenarios here:

a. We put utf-8 on the wire.  A 'octet' server must guess
    what charset the password database is using (or what charset
    the user was using last time they ran 'passwd')
b. We put 'octets' on the wire.  A server that stores username /
    passwords in a know charset must now guess what charset the
    client is using.

I suppose if we really want to solve this problem, we do the following,
we could introduce a new set of authentication messages where the
client sends BOTH the UTF-8 and the raw data entered by the user.
Then the server can pick which to use.  The client knows the users 
LC_TYPE, and can therefore safely translate to UTF-8.

I don't think we should do this, but if we want to do anything
in the space, I think this is what we have to do.

Thanks,

Joseph

---

From: Joseph Galbraith <[email-masked]>
Date: Thu, 13 Jan 2005 13:26:00 -0700

> applicable to the encoding), regardless of almost everything else in

I will argue strenuously against removing the UTF-8 for user names
and passwords.

There is a way for 'octet' systems to handle UTF-8.  It has
been discussed here previously.  I believe that in
my implementation for 'octet' systems, I can hit very very
close to 100% of all cases by allowing the user to specify
what charset his password is in in a user specific dot file
that I can read before authentication.  (And I believe I can
probably get 'good-enough' just by allowing the admin to
specify the charset of the passwd database.)

If we remove the requirement, I simply CAN NOT get 100%
in my windows servers.  My server would have to guess
the charset of the client.

This isn't actually something that an admin can reasonably
configure.  I could allow the each user to configure the
server for the charset of their client, but I can easily
imagine that lots of users use more than one client, and
some subset of those might use clients with different
charsets.  (Imagine a user working in a foreign office
that uses one charset when they connect from the office
machine, but a different charset when they connect from
their personal machine.)

So given that I think it is possible for an implementation
on an 'octet' machine to get 100% functionality using utf-8,
and impossible for a 'charset' machine to get 100% functionality
using octets, I think we ought to leave the draft alone.

If people find that completely unacceptable, I think we
have no choice but to introduce "publickey2", "password2",
etc., that require the client to send both the UTF-8
and the 'octet' form of the username and password.

Thanks,

Joseph

---

From: [email-masked] (Niels Möller)
Date: 14 Jan 2005 14:23:02 +0100

> LC_TYPE, and can therefore safely translate to UTF-8.

The proposal is mostly equivalent to having the client tell the server
what the client's local character set is, which might help servers
with no clue about the user's preferred character set. Such a message
from the client to the server would mean

  "Hey! I don't think you have any clue what character set is used for
   my account. But I know that I prefer koi8r, so please convert the
   data I provide into koi8r before you try to look me up in your user
   database."

Except that if we also send the octets, no actual conversion is needed
on the server side.

I think Joseph's idea is pretty good. I don't want to add it to the
core drafts, but interested parties could surely write up a separate
draft to experiment with.

It would also be possible to configure a client running on utf8 to
work with an "agnostic" server with koi8r (say) accounts, by having
the *client* convert the utf8 input into koi8r, and then putting it
into the "octets" field. This of course requires that the user is able
to explicitly configure the client to assume that the server is
actually using koi8r.

Regards,
/Niels

---

From: Sam Hartman <[email-masked]>
Date: Sat, 15 Jan 2005 17:02:45 -0500

>>>>> "Joseph" == Joseph Galbraith <[email-masked]> writes:

    >>> The way I see it, the recent discussion has made clear that
    >>> the benefit of requiring usernames and passwords to be
    >>> normalized when sent as utf8 on the wire, is very
    >>> questionable.
    >> questionable, since it demands that they be character strings;
    >> an encoding-agnostic system has no way to convert its octet
    >> strings to character strings.  As long as these things are
    >> specified as character strings on the wire, I can't see
    >> anything sensible for an encoding-agnostic OS to do, regardless
    >> of what encoding is chosen, regardless of normalization (if
    >> applicable to the encoding), regardless of almost everything
    >> else in this area.

    Joseph> I will argue strenuously against removing the UTF-8 for
    Joseph> user names and passwords.

I will block the drafts at the IESG if UTF8 is not sent.

I'll respond to the idea of new messages shortly.

--Sam

---

From: Sam Hartman <[email-masked]>
Date: Sat, 15 Jan 2005 17:16:11 -0500

>>>>> "Joseph" == Joseph Galbraith <[email-masked]> writes:

    Joseph> I suppose if we really want to solve this problem, we do
    Joseph> the following, we could introduce a new set of
    Joseph> authentication messages where the client sends BOTH the
    Joseph> UTF-8 and the raw data entered by the user.  Then the
    Joseph> server can pick which to use.  The client knows the users
    Joseph> LC_TYPE, and can therefore safely translate to UTF-8.

    Joseph> I don't think we should do this, but if we want to do
    Joseph> anything in the space, I think this is what we have to do.

Speaking as an individual, I would not support delaying the core
drafts for this change.

Speaking as an AD, I'm concerned about this proposal but perhaps not
sufficiently concerned to file a discuss.  We discussed a similar
proposal for Kerberos over in krb-wg and realized that we had
significant problems deciding what to do if the two strings were not
transforms of each other.  In some sense, the client is being given an
opportunity to try two usernames and passwords.

Are servers required to try both strings or just the one that is most
convenient to the server.  The obvious answer is that the server tries
the most convenient string.  But what happens when a client that
doesn't really know what its character set is talks to a unicode
server?  Again we find ourselves back in a situation where at least
one party needs to know what character set it speaks.

I'd like to draw the working group's attention to RFC 2277, the IETF's
character set policy.  Quoting section 3.1 of that document:

  3.1.  What charset to use

     All protocols MUST identify, for all character data, which charset
     is
        in use.

You would need to present a strong argument to the IESG why that
doesn't apply to you.

And as I said above, I'm concerned about the security implications of
sending two versions of the same string.

So if the working group decides to follow this approach, here is what
you must do.  First, you must describe RFC 2277 concerns and explain
why the charset policy does not apply or why you are violating a MUST
from that policy.  Second, you must explore the security implications
of your proposal and include them in the document you submit.  The
IESG will evaluate the proposal; I know my evaluation will focus on
these two issues.  It's reasonably lykely that the IESG will decide
not to publish such a proposal.

--Sam

---

From: der Mouse <[email-masked]>
Date: Sat, 15 Jan 2005 23:15:54 -0500 (EST)

>      All protocols MUST identify, for all character data, which
>      charset is in use.

> You would need to present a strong argument to the IESG why that

But it does apply-- vacuously.  Because the whole problem is what to do
when you *don't* have "character data", just an opaque octet string.
(It probably is character data at some conceptual level, in a human
mind somewhere, but even assuming that that's always so, that level is
totally inaccessible to software.)

Of course, this is my take on it.  Niels seems to think that a right
answer is to just impose a character set on the octet string by fiat,
and given how alone I am in holding up my end of this question, that's
probably about what the WG position is going to end up being.  (And in
that case there is no question of violating the snippet you describe.)

I find it somewhat odd that the IESG considers it so impossible for
these things to be octet strings rather than character strings that
they're willing to not only approve a protocol that makes no provision
for octet strings, but, you say, to actively block a protocol that goes
out of its way to make octet strings possible.  But apparently I'm
alone in wanting to be able to use protocol fields that are
fundamentally octet strings to represent octet strings rather than
character strings; I can only infer that nobody else cares about
traditional Unices any longer.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Chris Lonvick <[email-masked]>
Date: Sun, 16 Jan 2005 15:05:55 -0800 (PST)

Hi,

> perhaps it could be deleted all together.

Got it.  I'm assuming that this will then also apply to
SSH_EXTENDED_DATA_STDERR in [CONNECT] ("data" as the table header will
become "Symbolic name"), and the SSH_MSG_CHANNEL_OPEN_FAILURE table also
in [CONNECT].  Let me know if anyone disagrees with this.

> but not all, of the symbolic names are self-explanatory.

If we can't come to consensus on UTF8, I'll have plenty of time to work on
that.  :(

Thanks,
Chris

---

From: "Jeff P. Van Dyke" <[email-masked]>
Date: Sun, 16 Jan 2005 16:18:31 -0700

> If we can't come to consensus on UTF8, I'll have plenty of time to work on

I'd like to see us get to consensus on UTF8 soon...

For the purposes of this (or any IETF) working group, can someone remind
me how "consenus" is defined?

Jeff P. Van Dyke
[email-masked]

---

From: Sam Hartman <[email-masked]>
Date: Sun, 16 Jan 2005 18:53:02 -0500

>>>>> "Jeff" == Jeff P Van Dyke <[email-masked]> writes:

    Jeff> For the purposes of this (or any IETF) working group, can
    Jeff> someone remind me how "consenus" is defined?

The chair judges consensus.  If the chair claims there is a consensus
and someone disagrees they can talk to the chair about it or bring it
up on the list.  There is a formal procedure for dealing with appeals,
but we won't need that here;).

Bill can correct me if I'm wrong, but I don't think we are blocking on
the UTF 8 change.  It's already in the draft, no one has proposed
alternative text.

There's discussion for adding additional text (possibly in the core
drafts, possibly somewhere else) to handle the case where you don't
know what character set your encoding is.

To the extent that I started that discussion, my intent was to start a
non-blocking dsicussion.  If we come to a consensus on that thread in
time to make a change to the core drafts, then we do.  If not, that
discussion does not block the core drafts.

I think the issue we're blocking on is trademark and that requires no
action on this WG's part.

Technically speaking Russ is the only one holding a discuss on the ssh
drafts.

--Sam

---

From: Joseph Galbraith <[email-masked]>
Date: Mon, 17 Jan 2005 11:55:05 -0700

> Speaking as an individual, I would not support delaying the core

Definitely not.  To be honest, I don't really like the proposal
that much-- I was just tired of repeating the same debate over
and over again, so I figured we could move on to debating a
solution rather than the problem :-)

I also tend to be the type that figures if we've analyzed two
or three different solutions we are more likely to have found
the correct one than if we only considered one.

So, given the fact that I'm unwilling to hold up the core
drafts, think it terribly ugly that the entire auth draft
would have to be duplicated with multiple username and
password fields for each userauth-request message, there
are ambiguities and potential security issues in terms of
which username/password fields should be used by the server,
and reluctance from the AD, I'll withdraw my proposal.

For our unix implementations I believe that 'imposing' a
character set is a fine solution.

In fact, since all input systems, either local or remote,
connected to a 'octet' based system, MUST use the same
encoding system to provide passwords, I am quite confident
that allowing the administrator to specify a single charset
to translate usernames and passwords to will be sufficient
to produce interoperability superior to what would be
available on a locally attached terminal.

For example, if a user sits down at a X-Station and logs
on via XDM, the characters he types on the keyboard are
encoded in a certain way.  (This is irregardless of whether
the passwd database is storing octets or not.)

If the user then walks over to a VT320 terminal that is encoding
the keystrokes in a different way, he will not be able to log
in.  All the input devices capable of taking username/password
input MUST encode input characters identically.

So, from a certain viewpoint, in such an 'octet' system, the
passwd database is not really encoded in 'octets', it is simply
the charset used to encode input by the login devices in use.

I've felt all along that the current wording in the drafts
is fine, but now, over the course of this discussion and
and analysis of the problem, I've realized the limitations
of such an 'octet' based system, even to locally attached
terminals, so my brain now agrees with my gut.

I'm confident that a properly implemented SSH server that is
conformant to the drafts as currently written can produce
better interoperability than is available at the locally
attached terminal.

Equal interoperability could be obtained locally by having
the login, XDM, and passwd programs translate from the device's
input encoding to a common charset encoding... but that is a
discussion for a different mailing list :-)

So my vote is no change to the drafts.

Thanks,

Joseph

---

From: der Mouse <[email-masked]>
Date: Mon, 17 Jan 2005 14:04:56 -0500 (EST)

> In fact, since all input systems, either local or remote, connected
> to a 'octet' based system, MUST use the same encoding system to
> provide passwords,

I've seen others claim basically this, and it simply isn't true.

If my password is the octet string 0x20 0xe7 0x22, it doesn't matter
whether I type that 0xe7 in a way that the input method thinks is an
8859-1 c-cedilla or an 8859-7 eta or whatever; it still works.
(Assuming, of course, that the channel between the typing and the
checking is an octet-string channel.)

> For example, if a user sits down at a X-Station and logs on via XDM,
> the characters he types on the keyboard are encoded in a certain way.
> (This is irregardless of whether the passwd database is storing
> octets or not.)

> If the user then walks over to a VT320 terminal that is encoding the
> keystrokes in a different way, he will not be able to log in.

This is true only if the user not only insists on thinking of the
password as a character sequence rather than an octet sequence, but
also insists on typing it that way rather than adjusting to the
difference.  The login failure is really due to the mismatch between
the user's concept of the password as a character string and the
system's implementation of the password as an octet string.  (Such a
user would be better served by a character-string system, but that's
neither here nor there.)

> All the input devices capable of taking username/password input MUST
> encode input characters identically.

No, they simply must be capable of generating the necessary octets.
What characters the input method thinks they correspond to is
irrelevant.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: "denis bider" <[email-masked]>
Date: Tue, 18 Jan 2005 19:51:37 +0100

> > If the user then walks over to a VT320 terminal that is 
> > encoding the keystrokes in a different way, he will not
> > be able to log in.
> 
> This is true only if the user not only insists on thinking of 
> the password as a character sequence rather than an octet 
> sequence, but also insists on typing it that way rather than 
> adjusting to the difference.

Given that you present this as a special case ("not only ... but also ..."),
I wonder what kind of users your software is targeting.

I have yet to meet a user that wouldn't fall into this "special" category.

But then again, we write our software for people, not for cyborgs. :)

> The login failure is really due to the mismatch between the
> user's concept of the password as a character string and the
> system's implementation of the password as an octet string.

And when there is such a mismatch, who is right? The machine, or the user?

(My answer is, definitely, most always, the user.)

> (Such a user would be better served by a character-string system,
> but that's neither here nor there.)

And if I translate this into ordinary-speak ;) it says: virtually every user
would be better served by such a system.

It makes no sense to write our software as though it's aimed for cyborgs
when it's aimed for real people who mostly don't even know what the ASCII
code for the letter 'b' is (and probably don't even know where to look it
up, or what an ASCII code is in the first place); let alone being able to
manipulate input so as to successfully enter a password when the input
charset has been set incorrectly.

Maybe there are wiz kids out there who do this, but maybe they too would be
better off simply correcting the code page setting so that the password can
be entered normally.

The vast majority of people cannot enter binary data from a password prompt,
and most SSH clients will probably not even allow them to.

On the other hand, if you have programs connecting to other programs
unattended, you can use other modes of authentication better suited for that
purpose than arbitrarily-encoded binary passwords.

It seems to me too that interoperability and user expectations are better
served by treating usernames and passwords as character strings, not
arbitrary binary data.

Best regards,

denis

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Tue, 18 Jan 2005 19:40:14 -0500

On Tuesday, January 18, 2005 19:51:37 +0100 denis bider 
<[email-masked]> wrote:

> Given that you present this as a special case ("not only ... but also
> ..."), I wonder what kind of users your software is targeting.

Really, so do I.  Certainly I don't know anyone who thinks of their
username or password as being an octet string.  Nor do I know of anyone who 
would put up with this sort of behaviour from software, instead of just 
getting something that actually works for them.

I'm not the chair of this group, so I don't get to decide when we do or 
don't have consensus here.  However, it sure looks to me like we have 
"rough consensus" that these things are in fact character strings, not 
octet strings, and that the SSH protocol should behave accordingly.  Once 
such a consensus has been reached, it's generally considered bad form for 
the one dissenting voice to keep repeating the same argument over and over 
again, particularly with no new information.

Someone commented to me the other day that it seemed like almost no one in 
this group had actually _read_ IETF character set policy as defined in 
RFC2277.  I would suggest that anyone wishing to participate further in 
this discussion take a break right now to do so.  The entire document is 
only 9 pages long, and besides providing a variety of useful advice, it 
also represents the consensus of the IETF as to how protocols must behave 
with respect to character sets.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: der Mouse <[email-masked]>
Date: Tue, 18 Jan 2005 22:13:55 -0500 (EST)

> Certainly I don't know anyone who thinks of their username or
> password as being an octet string.

Neither do I, usually.  But I know that it _is_ an octet string to the
OS, fundamentally, and if I have trouble logging in, I take that into
account.

> However, it sure looks to me like we have "rough consensus" that
> these things are in fact character strings, not octet strings, and
> that the SSH protocol should behave accordingly.

"In fact"?  Humans think of them as character strings (mostly, at
least).  OSes, at least the ones in question, treat them as opaque
octet strings.  Which is the "fact"?  When it comes to actually
successfully logging in, I would say the octet string is the fact: it
doesn't matter whether the characters you type are the ones you think
of your username or password as containing, if the resulting octet
string doesn't match.

What I think we do have consensus on, even if only consensus by apathy,
is that this issue can be ignored; I appear to be the only person the
least bit concerned about it.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Derek Fawcus <[email-masked]>
Date: Wed, 19 Jan 2005 18:23:41 +0000

> least bit concerned about it.

I guess.  I happen to agree with your position to a large extent,  in
that I use unix systems which are using 8859-1 (from my POV).

But then since I happen to really only use the ASCII compatible part,
if it's an octet string or UTF-8 becomes irrelevant.  I don't have any
characters outside the ASCII range in usernames or passwords.

So it becomes academic.  The only other argument would probably be that
the original implementations (for unix I believe) would have been using
octet strings,  despite anything to the contrary in the spec.

Rather than have a new message defined now,  we could simply define a
new error code which mean 'need octet string input'.  Then on unix
machines,  if either the username or password contained octets with
values >= 128 this error could be returned.

Then in a new draft define new messages and a new error.  These being the
authentication forms with explicit interpretation as octet strings.
The error could be used to indicate 'need utf-8 string input' in case
some client way configured to start with the octet form.  An alternate
is simply a client -> server message meaning subsequent auth messages
are octet strings.  This would require client side config/choice to
send on a per server basis.

Mind,  the approach of having configuration (at the client) to say what
the server charset is,  seems simpler.  In which case the client would
convert to the proper octet string,  then just send the existing messages
but treating the fields as if they were defined as octets - i.e. possible
sequence which would be bad if interpreted as utf-8.

DF

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 19 Jan 2005 19:39:23 +0100

Derek Fawcus <[email-masked]> writes:

> But then since I happen to really only use the ASCII compatible
> part, if it's an octet string or UTF-8 becomes irrelevant. I don't
> have any characters outside the ASCII range in usernames or
> passwords.

Perhaps it would be useful to consider what problems you would have
interacting with EBCDIC systems, to imagine what the situation is for
Latin-1 or Unicode users.  ASCII isn't the only 7-bit encoding.

Regards,
Simon

---

From: Derek Fawcus <[email-masked]>
Date: Wed, 19 Jan 2005 21:45:04 +0000

> Latin-1 or Unicode users.  ASCII isn't the only 7-bit encoding.

That is part of the point.  My local machine is using UTF-8,  the Unix
machines I ssh into are in 8859-1,  and I do use a couple of 8 bit
characters.  However simply not in my username or password.  Hence the
local unicode characters (when UTF-8 encoded) are valid ASCII,  and
hence I can log in.  Once logged in,  I can use those 8859-1 characters
(i.e. The pounds sterling symbol).

You point wrt EBCDIC is valid,  but I don't now the appropriate answer.
Also,  do their authentication systems work with a charset,  or octets?

DF

---

From: Joseph Galbraith <[email-masked]>
Date: Wed, 19 Jan 2005 15:18:04 -0700

> Also,  do their authentication systems work with a charset,  or octets?

I don't know, but someone implementing a server on such a
system would!  And they could then convert the UTF-8
correctly to either EBCDIC if that is what is needed
(one presumes that if the authentication is not charset
aware the authentication devices attached to such a
system a probably producing EBCDIC), or to some other
charset if specified by the authentication system.

See that is the thing.  I don't know squat about EBCDIC,
and my OS doesn't know squat about EBCDIC and can't convert
to or from it.  So even if the server told me, please send
me EBCDIC, I just can't do it.

On the other hand, such a server implementation running
on an EBCDIC machine _does_ know about EBCDIC.  Hence,
the correct place for charset conversion is on the server.

Thanks,

Joseph

PS. If your 8859-1 systems had a comforming SSH implementation,
you would be able to use 8859-1 character in you password, which
would potentially double the security of your password.

A password guessor is probably going to stick to 7-bit characters;
he'll get most passwords there.  (And if you force him to include 8-bit 
characters, he has got a bigger space to cover-- not really double,
but significantly larger.)

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 19 Jan 2005 23:43:44 +0100

Derek Fawcus <[email-masked]> writes:

> hence I can log in.

That works as expected if both systems use ASCII.  I'd imagine that
your username in ASCII would be equally valid in EBCDIC, but would
mean something different.

The fact that UTF-8 and Latin-1 are identical for ASCII is a useful
property, but it may limit how you think of the problem.

> Also,  do their authentication systems work with a charset,  or octets?

If humans entered a name of the user using a keyboard at some point in
time, and the string was stored as octets, the string must have been
encoded using some charset.  You can't go from human symbols to octets
without using some kind of charset.

The only system I can recall that uses true opaque octets would be
some kind of PIN or one-time-pad system that only uses digits.

If the SECSH protocol uses ASCII today, the EBCDIC server would have
to convert the wire charset from ASCII to EBCDIC before it can compare
the string.

This is similar to the UTF-8 situation under discussion:

If the SECSH protocol uses UTF-8 tomorrow, the server will have to
convert the wire UTF-8 data to the local charset, before it can
compare it with the authentication systems.  Only the server
implementation can know the local charset.

As far as I understand, Unix /etc/passwd only support ASCII usernames,
so Unix SECSH implementations would naturally be limited to ASCII.

Other authentication databases may support Unicode (or Latin-1, or
EBCDIC...).

I'm not sure I see the problem.  Implementations that doesn't know
what charset their authentication database uses, will be limited to
ASCII, or whatever safe subset they can assume.  Implementations that
know what charset the authentication mechanism uses, can convert it as
appropriate.

Regards,
Simon

---

From: der Mouse <[email-masked]>
Date: Wed, 19 Jan 2005 19:11:57 -0500 (EST)

> As far as I understand, Unix /etc/passwd only support ASCII

I've seen this, or equivalent things, claimed before.  I finally tried
it, and it's not true, at least not for the Unix variant I have at
ready hand (five-year-old NetBSD).  I created a user whose name is 0xe5
0x67 0x65 (Latin-1 "åge" - I happen to know of someone named Åge).
vipw did not complain.  I set its password; passwd did not complain.
"su åge" worked.  "ssh localhost -l åge" worked, too, with an ssh
that's ssh 1.2.14 in all username-processing respects.

Sorry to go injecting fact into a good flamefst, but there you have it.
At least one octet-string system - with a legacy ssh - does in fact
support 8-bit usernames, at least minimally.  (I'd try an ssh2 test of
it, but the only ssh2 implementation I have on that system is my own,
which at the moment is encoding-agnostic even on the wire, so it would
prove nothing.)

> ASCII, or whatever safe subset they can assume.

Why?  Why should one octet-string system talking to another
octet-string system be unable to use non-ASCII octets?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 20 Jan 2005 10:55:58 +0100

> that's ssh 1.2.14 in all username-processing respects.

That's a useful data point, thanks.

> octet-string system be unable to use non-ASCII octets?

You shouldn't compare octet-strings with each other unless you know
which charset they were encoded in.

It doesn't follow that an ASCII string that is equal to an EBCDIC
string, octet-wise, mean the same thing.

In your example, an SECSH implementation would appear to need to know
what charset /etc/passwd is using.  In your example, it would be
ISO-8859-1.  This could be specified through a configuration file,
unless there is a system default.  Then, if SECSH used UTF-8, the
server would have to convert back and forth between UTF-8 and
ISO-8859-1 before comparing.

With this in place, your system could support Latin-1 usernames
reliably.

Try 'ssh -l åge remote-server' from a host using UTF-8 to realize why
the octet string view is not reliable.

Hope this helps,
Simon

---

From: der Mouse <[email-masked]>
Date: Thu, 20 Jan 2005 11:09:37 -0500 (EST)

> which charset they were encoded in.

(a) Take off your character-string blinders for a moment!  Your
statement is true only if both octet strings represent character
strings, and it's equality of the character strings that you really
care about.

(b) That aside, why shouldn't one octet-string system talking to
another octet-string system be unable to use non-ASCII octets?  If a
human tells them to do this, it's up to the human to ensure they use
sufficiently compatible character sets when character sets matter.  But
such cases are relatively common - if both systems are
encoding-agnostic, but the user has chosen to use the same character
set on both, for example.

> what charset /etc/passwd is using.

Perhaps it appears that way to you.  I see no particular need for it.
(Except for conformance to the fiat of UTF-8, which since it's the
point under discussion can be ignored for the moment.)

> In your example, it would be ISO-8859-1.

For that user, it is.  Another user of the same system may be of Greek
extraction and use 8859-7, with username, say, 0xec 0xe1 0xf1 0xea (mu
alpha rho kappa, which I can't insert directly because this message is
in 8859-1, and I'd rather not mess around with trying to create a
multipart/mixed message).  And what's wrong with that?

There appears to be an unstated assumption lurking beneath all the
character-string positions that users are incapable of making, or at
least unwilling to make, adjustments and allowances for certain
impedance mismatches - allowances such as Åge, when visiting Mark,
realizing that to log in he has to type epsilon g e as his username.

A lot of users doubtless are incompetent to do this.  More are
unwilling to.  But I do not see those as reason to prevent the others
from accepting the mismatch and doing useful things - especially since
I rather suspect that using an encoding-agnostic system like a
traditional Unix correlates positively and moderately strongly with
such ability and willingness.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Jon Bright <[email-masked]>
Date: Thu, 20 Jan 2005 17:48:57 +0100

> strings, and it's equality of the character strings that you really

...which they do, for pretty much every user ever.

No* user has ever said to themselves "Ah! The password field! I want to 
enter the octets 0xC4 0xD6 0xDC! Now, hm, I'm using <checks quickly>... 
ah, yes, I'm using ISO 8859-1.  So, to enter those octets... <looks up 
ISO 8859-1 table>... ah, I'll need to enter the characters Ä, Ö and Ü to 
do that.".

Seriously, nobody* has ever done this.  What users say to themselves is, 
"Ah! The password field! My password is Äpfel.  I'll type in Äpfel!". 
And they expect this to work.  And they expect it to work whether 
they're typing in their password on Windows, using Unicode, or some *nix 
using 8859-1, or whatever.  If they're using a system which only 
supports ASCII (or EBCDIC), they're not going to be able to enter Äpfel 
in the first place.

Users don't know, don't care, shouldn't have to care and probably never 
will care about what octets those characters are.  They care about words 
and the characters that make up those words.

* with the possible exception of yourself, since you're campaigning so 
vehemently for this.  If you do do this, then I'm sorry, but I'm pretty 
sure you're a lone voice, crying in the wilderness.  Presumably, crying 
in octets.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 20 Jan 2005 18:26:57 -0500

> point under discussion can be ignored for the moment.)

Actually, that point is _not_ up for discussion.  IETF policy, as described 
in RFC2277 which represents the consensus of the IETF, says exactly what we 
must do with character data.  I do not believe that revisiting that 
decision is in scope for us.

The current discussion centers around whether usernames and passwords as 
seen in ssh are character strings or not.

> multipart/mixed message).  And what's wrong with that?

Aha.  I do believe that you just admitted this is a character string, and 
you couldn't include it in your message because its character set did not 
match that of the rest of the message.  Note, incidentally, that you could 
have resolved this problem by using UTF-8 instead of ISO-8859-1 -- MIME 
does conform to IETF character set policy.

> realizing that to log in he has to type epsilon g e as his username.

Those assumptions are not at all unstated; I think we've made them quite 
clear.  The point of internationalization of text strings is that users 
should not even have to think about those kinds of "allowances", let alone 
actually make them.  Åge should be able to use Mark's keyboard to type Åge 
(hint - I'm having no trouble typing it on my US keyboard) and have it work.

But see, here is the real problem.  Suppose I have a nice, modern, 
Unicode-using system.  The one I'm sitting in front of at the moment uses 
UTF-8 for nearly everything.  Windows boxes also use Unicode.

If I have a compliant ssh client, and Åge's home machine has a compliant 
server, then he can type his username, which my client will encode on the 
wire as <0xe3><0xa5><0x67><0x65>.  His ssh server will then convert that to 
its character set, in which it is represented as <0xe5><0x67><0x65>, and 
he'll be able to log in.

On the other hand, if my ssh client and his insist on treating this 
character string as an "octet string" that does not require tagging or 
conversion, then Åge cannot log in at all, because on my UTF-8 system he 
_cannot_ produce the sequence <0xe5><0x67><0x65>, because that is not valid 
UTF-8.

Personally, I prefer the model which allows my Swedish friends to log in.

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 21 Jan 2005 00:41:45 +0100

> set on both, for example.

That paragraph allowed me to understand your position.  I think we
fundamentally disagree.  Here's my view:

The point of requiring that you (and everyone else) use UTF-8 in the
protocol is to make sure humans do not need to go through the hassles
of ensuring they use compatible charsets.  Few people understand these
matters to make an informed choice.  Further, this is an error prone
topic, so it is not reliable to rely on humans to perform the task.

I believe most users expect things to work.  In this case, things
can't work unless the entire world is using the same charset (not
going to happen), or the protocols use charset-labeled fields and
different systems convert as appropriate.  Or, as you would prefer,
users handle charset conversion by themselves.

I'm not sure you realize the importance of the problem -- non-ASCII
username and passwords is not a "nice thing".  It is not something
that you will rarely see in practice, so that educating those special
users about how to convert between charsets themselves is an option.
For the majority of potential users, it is critical that they can use
symbols they are familiar with.  Only a small part of the world uses
languages that are written in the ASCII alphabet.

You might want to read RFC 2277, it discuss these matters more
eloquently than I can hope to do.

Hope this helps,
Simon

---

From: Alan Barrett <[email-masked]>
Date: Fri, 21 Jan 2005 13:19:02 +0200

> multipart/mixed message).  And what's wrong with that?

Even if the underlying operating system thinks that the user name
is

	OCTET_SEQUENCE(0xec, 0xe1, 0xf1, 0xea),

the user probably thinks that the user name is

	CHARACTER_STRING(<greek small letter mu>, <greek small letter alpha>,
		<greek small letter rho>, <greek small letter kappa>).

and perhaps the sysadmin thinks that the user name is

	ENCODE_USING_CHARSET(iso-8859-7, CHARACTER_STRING(...)).

I see no reason why the ssh server implementation could not have a
set of rules or a database to tell it things like "The user whose
username is really OCTET_SEQUENCE(x) likes to think that his user name
is CHARACTER_STRING(y)", or "All users on this system like to think that
their usernames are character strings encoded in CHARSET(z)".  Then,
when CHARACTER_STRING(y) arrives in the relevant ssh protocol field, the
server could map it to the correct OCTET_SEQUENCE(x).  Users who really
do think of their user names as octet sequences can be handled by having
an rule to say "The user whose username is really OCTET_SEQUENCE(0x01,
0x02) will login via ssh using the name CHARACTER_STRING(<digit zero>,
<latin small letter x>, <digit zero>, <digit one>, <digit zero>, <digit
two>)".

So, the fact that the ssh protocol wants to use a character string does
not appear to prevent an implementation that wants to use an octet
sequence.

--apb (Alan Barrett)