Subject: Normalization of passwords in SASL and SSH

From: Sam Hartman <[email-masked]>
Date: Mon, 29 Nov 2004 11:13:15 -0500

Hi.  A discussion in the IETF 61 secsh meeting re-opened the issue of
how to handle password normalization for passwords received by the
server.  The ssh protocol had adopted a significantly different
solution to this problem than the sasl plain mechanism.  This concerns
me; I want to either solve the problem of password normalization in a
consistent manner or to understand why the ssh requirements are
different than the sasl requirements.  

As a result of that discussion I've been talking to several people
about the issue.  Now I think I understand things well enough to
propose a solution to both working groups.

First, I'd like to revisit the motivation behind normalization.  There
are a variety of semantically equivalent sequences of characters that
have multiple representations in Unicode.  The simplest example is
accented characters where one code point represents the accented
character and another series of code points represents the
semantically equivalent character plus a combining accent.  Other
examples exist.  Which sequence of code points gets used tends to
depend on the input method used to enter the characters.  That means
that it tends to depend on the OS, application and version of software
used.  So, telling users to type their password the same way is not a
sufficient answer to guarantee that you end up with the same sequence
of code points; the user may not have a choice of input methods.

The IDN working group developed an architecture called Stringprep
which allows protocol-specific profiles of rules for mapping some
semantically equivalent strings into the same sequences of code
points.  It is not perfect; as with any normalization discussion the
question of where to draw boundaries between equivalence classes is
difficult.  The SASL working group developed a profile of Stringprep
called Saslprep; this profile is targeted at simple usernames and
passwords.  Everyone I talked to over the past two weeks believes that
if you are going to normalize passwords for ssh or the SASL plain
mechanism, Saslprep is a reasonable profile to use.  Saslprep has the
advantage of already being in the rfc editor queue.

We'd like to tell people to normalize passwords all the time.  If they
actually did that then it would improve interoperability and the user
experience.  However most systems we use do not store plaintext
passwords.  Instead, they tend to store some function of the password
or to call out to something like PAM or RADIUS to do password
authentication.  Normalizing a password before you check it but not
when new passwords are stored may be very problematic.  

For example, consider an environment where the input method tends to
produce separate base character and accent code points. Without
normalization, when passwords containing accents are stored, any
computed hash will include both the base character and the accent.
However Saslprep will normalize such a string to use a single code
point when possible.  Thus in such an environment, normalizing only
when checking passwords would decrease interoperability because
accented characters would never work in passwords.

I'm proposing the following approach to address this situation.  For
both ssh and sasl plain, normalization should be done on the server if
at all.  Only the server knows what the password store technology in
use is.  We recommend normalizing when passwords can be normalized
both as they are set and as they are compared.  In environments where
passwords may not be normalized as they are set, implementations may
want to try comparing against both the normalized and supplied form of
the password.  Doing so may improve interoperability because it means
there is a password that if successfully set will work when used with
any input method.  However in many environments such as when a server
calls out to PAM or RADIUS, even the server cannot try multiple
passwords without incrementing failed password counters or having
other similar side effects.  Here are some examples of how I see these
recommendations working.

A sasl plain implementation that manages its own password database
would always normalize when setting a password and when checking
passwords.  

A Windows ssh service  that managed its own authentication database
and provided the only tool for setting passwords  would normalize both
for setting passwords and for comparing them.

An ssh implementation that used a unix password file would not
normalize passwords for comparison.  This is true even if the ssh
implementation sometimes was used to set passwords, because the passwd
command could be used independently of the ssh implementation.

An ssh implementation that called out to PAM or RADIUS would not
normalize passwords.  However some of the modules eventually doing the
password comparison might well do normalization if appropriate for the
password store in question.

Does this seem like a reasonable approach to people?  If so, we can
work on appropriate text.

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 29 Nov 2004 18:25:13 +0100

Sam Hartman <[email-masked]> writes:

> use is.

For what it's worth, I support this view.  I'd like to add one other
advantage, that I feel is a strong argument for this approach.

If client-side normalization is deployed based on SASLprep in SSH,
upgrading to a new revision of SASLprep will be more difficult.  This
is because the server may need to know which SASLprep revision is
used, to select the hashed password that was prepared using the proper
SASLprep version.  That could be solved by signaling the SASLprep
version in SSH, but I'd prefer to avoid that, if possible.

Further, if the revised SASLprep turn out to be backwards
incompatible, which doesn't seem unlikely (given the normalization
flaw in Unicode 3.2), the server would receive SASLprep(password).
Computing SASLprep2(SASLprep(x)) may not in general be equal to
SASLprep2(x).

Having normalization occur on the server also make it easier to switch
from SASLprep to something else in the future.  In my opinion, sending
SASLprep data over the wire should always be avoided, in favor of
sending Unicode code points.

I think we should get used to the idea of a SASLprep2, and design
protocols that use SASLprep with this in mind.

Thanks,
Simon

---

From: [email-masked] (Niels Möller)
Date: 29 Nov 2004 18:29:20 +0100

> command could be used independently of the ssh implementation.

I'd like to this description that I think the only sane way to support
passwords beyond ascii on unix systems is to use normalization (either
to utf8 + normalization, or to some system-wide fixed 8-bit character
set. The passwd program, and other programs handling passwords, must
be fixed to do the right thing.

> work on appropriate text.

If I understand your proposal, as it applies to ssh, you're suggesting
that we should

1. Strike the new text on normalization, in effect reverting to what
   was in older drafts (e.g. draft-ietf-secsh-userauth-18.txt says
   "Note that the password is encoded in ISO-10646 UTF-8. It is up to
   the server how it interprets the password and validates it against
   the password database.").

2. Add some new text saying that we recommend that systems supporting
   non-ascii passwords always normalize passwords and usernames
   whenever they are added to the database, or compared (with or
   without hashing) to existing entries in the database.

Am I missing something? 

To me, (2) seems slightly out of scope for the secsh wg, but I won't
object if such a recommendation is added, provided the text can be
kept short and to the point.

As for (1), I don't care very much either way. Normalizing identifiers
in wire formats is generally a good thing to do, but in this case, it
doesn't really solve the problems with non-normalized entries in
various legacy user databases. I'll also note that what you're
proposing now, appeared to be the wg consensus up to quite recently.

From the ssh point of view, I think it is very important to keep
changes minimal, so that we can get our severly delayed spec
published at last.

Regards,
/Niels

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 29 Nov 2004 21:00:14 +0100

>    without hashing) to existing entries in the database.

I believe the part about storing normalized strings in databases is a
poor recommendation.

If the password is stored, then I believe it would be better to store
the unprepared string than the prepared string.  Otherwise, there is
an upgrade problem when SASLprep2 is released.

Of course, if the database store a hashed password, then there is no
choice, and you have to prepare it before hashing.

I would recommend doing normalization when storing strings, to provide
early feedback for invalid strings, but store the unprepared string.
During validation, you can either prepare the string (using the
then-current preparation algorithm), or you could use a prepared copy
of the password stored in the database, for efficiency.  In the latter
case, the database should also remember which algorithm was used to
prepare the prepared string (SASLprep, SASLprep2, etc).

Thanks,
Simon

---

From: Sam Hartman <[email-masked]>
Date: Mon, 29 Nov 2004 15:39:09 -0500

>>>>> "Niels" == Niels Möller <[email-masked]> writes:

    Niels> If I understand your proposal, as it applies to ssh, you're
    Niels> suggesting that we should

    Niels> 1. Strike the new text on normalization, in effect
    Niels> reverting to what was in older drafts
    Niels> (e.g. draft-ietf-secsh-userauth-18.txt says "Note that the
    Niels> password is encoded in ISO-10646 UTF-8. It is up to the
    Niels> server how it interprets the password and validates it
    Niels> against the password database.").

    Niels> 2. Add some new text saying that we recommend that systems
    Niels> supporting non-ascii passwords always normalize passwords
    Niels> and usernames whenever they are added to the database, or
    Niels> compared (with or without hashing) to existing entries in
    Niels> the database.

And say that ssh implementations that both store the passwords and
compare them SHOULD use saslprep for normalization.

---

From: Sam Hartman <[email-masked]>
Date: Mon, 29 Nov 2004 16:15:26 -0500

>>>>> "Simon" == Simon Josefsson <[email-masked]> writes:

    Simon> [email-masked] (Niels Möller) writes:
    >> 2. Add some new text saying that we recommend that systems
    >> supporting non-ascii passwords always normalize passwords and
    >> usernames whenever they are added to the database, or compared
    >> (with or without hashing) to existing entries in the database.

    Simon> I believe the part about storing normalized strings in
    Simon> databases is a poor recommendation.

    Simon> If the password is stored, then I believe it would be
    Simon> better to store the unprepared string than the prepared
    Simon> string.  Otherwise, there is an upgrade problem when
    Simon> SASLprep2 is released.

Note that the sasl working group considered Simon's argument and ended
up disagreeing with him.

As  I recall, Simon  was concerned  about cases  where either  bugs in
Unicode (which do exist) or explicit changes to saslprep 2 designed to
be incompatible would create  a situation where saslprep2(foo) was not
the same as saslprep1(foo).  

The stringprep specification has a reasonable versioning model.  It is
generally true that if applying one version of a stringprep profile to
some string succeeds with unassigned code points prohibited, then
later versions of the same profile will yield the same result.

Simon's concern is that generally true is not the same as always true.

The SASL working group did not decide to adopt this interpretation.  I
don't think a consensus was declared on the reasoning.

---

From: Sam Hartman <[email-masked]>
Date: Mon, 29 Nov 2004 17:26:33 -0500

>>>>> "Bill" == Bill Sommerfeld <[email-masked]> writes:

    Bill> (WG chair hat off.  just my questions) Why should these
    Bill> proposed rules apply only to passwords and not also to login
    Bill> names?  It seems like the core justification for server side
    Bill> normalization -- that they're often stored in database
    Bill> maintained by a subsystem not under the control of the ssh
    Bill> server implementor -- also applies to usernames.

SASL believes they should apply to usernames as well.
Kerberos has adopted the same position.

    Bill> Is it ever the case that normalization functions would
    Bill> change the human-readable representation meaningfully?
    Bill> Examples?

I'd expect a normalization profile for passwords to remove direction
markers.  I'd expect it to map all forms of white space together.
You'd lose the difference between say a 1 em space and u+0x20, which
would be visible.  I'd say anything outside of these sorts of examples
would be a bad idea in a stringprep profile, especially for a security
application.

---

From: Bill Sommerfeld <[email-masked]>
Date: Mon, 29 Nov 2004 17:17:49 -0500

(WG chair hat off.  just my questions)

Why should these proposed rules apply only to passwords and not also to login names?  It seems like the core justification for server side normalization -- that they're often stored in database maintained by a subsystem not under the control of the ssh server implementor -- also applies to usernames.

Is it ever the case that normalization functions would change the human-readable representation meaningfully?  Examples?

If there are multiple available normalization functions it would seem that you might be able to either (a) try several, accept if one matches, and/or (b) "renormalize" already normalized strings using alternate functions, in which case erroneous client-side normalization could well be undone or worked around by the server.

								- Bill