Subject: How to treat utf8 text with overlong utf8 sequences?

From: [email-masked] (Niels Möller)
Date: 02 Oct 2004 15:18:19 +0200

What do you think about sending overlong / "non-minimum form" utf8
sequences in various utf8 strings in the protocol?

It matters the most for utf8 strings that are displayed to the user,
e.g. the prompt strings in SSH_MSG_USERAUTH_INFO_REQUEST, where the
specification recommends control character filtering.

I prefer doing the control filtering before converting the data to the
local character set, because it's pretty well defined which
ucs4/unicode values are control characters (namely u0000-u001f,
u007f-u009f).

If we allow overlong control character sequences, then e .g. ESC can
be represented in utf8 as

  0x1b, (0xc0 0x9b), (0xe0 0x80 0x9b) ... or (0xfc 0x80 0x80 0x80 0x80 0x9b)

Filtering gets easier if I can first check if the utf8 string contains
overlong sequences at an early stage, and treat that as a protocol
error.

About the same question applies for the utf8 encoding of ud800-udfff
(surrogates) and the non-characters ufffe and uffff, which are also not
supposed to ever occur in valid utf8 text.

RFC 2279 does not address these questions, as far as I can see.

I'm tempted to treat any use of overlong or otherwise invalid utf8
strings that I receive from the remote end as a protocol error.

* Do you think that is a reasonable thing to do?

* Does it violate the ssh specification?

* Will it cause any interoperability problems in practice?

Regards,
/Niels

---

From: Simon Tatham <[email-masked]>
Date: Sat, 02 Oct 2004 15:33:13 +0100

> sequences in various utf8 strings in the protocol?

Should be illegal, definitely.

> RFC 2279 does not address these questions, as far as I can see.

I think it does, if only obliquely. Section 2 says:

|   1) Determine the number of octets required from the character value
|      and the first column of the table above.  It is important to note
|      that the rows of the table are mutually exclusive, i.e. there is
|      only one valid way to encode a given UCS-4 character.

I think that adequately justifies considering any overlong sequence
to be completely invalid.

> * Do you think that is a reasonable thing to do?

Definitely.

> * Does it violate the ssh specification?

I don't think so.

> * Will it cause any interoperability problems in practice?

It _shouldn't_; but failure to disallow overlong sequences could
cause security problems, therefore it's reasonable to consider any
implementation currently generating them to require fixing.

All IMO, of course.
-- 
Simon Tatham         "Happiness is having a large, warm, loving,
<[email-masked]>    caring, close-knit family in another city."

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 02 Oct 2004 18:55:46 +0200

> sequences in various utf8 strings in the protocol?
...
> RFC 2279 does not address these questions, as far as I can see.

For what it's worth, RFC 2279 is obsoleted by RFC 3629, which include:

   Implementations of the decoding algorithm above MUST protect against
   decoding invalid sequences.  For instance, a naive implementation may
   decode the overlong UTF-8 sequence C0 80 into the character U+0000,
   or the surrogate pair ED A1 8C ED BE B4 into U+233B4.  Decoding
   invalid sequences may have security consequences or cause other
   problems.  See Security Considerations (Section 10) below.

I don't think it is evident that "MUST protect against" imply
"reject", though.

Thanks,
Simon

---

From: Derek Fawcus <[email-masked]>
Date: Mon, 4 Oct 2004 17:24:12 +0100

> sequences in various utf8 strings in the protocol?

Don't do it.

> strings that I receive from the remote end as a protocol error.

Agreed.

> * Do you think that is a reasonable thing to do?

Yes.

> * Does it violate the ssh specification?

No.

> * Will it cause any interoperability problems in practice?

No.  If so,  if the other end.  I happe to agree with the
comment that it could lead to security problems;  if not
immediately,  then eventually,  simply by the fact that
the 'unexpected' form may violate some assumptions.

DF

---

From: [email-masked] (Niels Möller)
Date: 05 Oct 2004 14:42:07 +0200

> sequences in various utf8 strings in the protocol?

Thanks to all those who answered (Simon, Simon and Derek).

Unicode-3.0 (the version I have on paper) is a little vague, decoders
are appearantly expected to decode overlong utf8 sequences, although
implementations are not allowed to generate them. Unicode-3.2 is more
explicit, see http://www.unicode.org/reports/tr28/,
in particular "Table 3.1B. Legal UTF-8 Byte Sequences".

This table explicitly excludes overlong utf8 sequences, and
utf8-encodings of unicode/utf16 surrogate characters. So it seems
fairly safe to treat use of such utf8 sequences as protocol errors in
ssh. Also RFC 3629 spells this out quite clearly.

Neither the table in tr28 nor RFC 3629 explicitly forbids the unicode
"non-characters" ufffe and uffff, and the corresponding utf8 sequences
(0xef 0xbf 0xbe) and (0xef 0xbf 0xbf), but in the ssh context, I think
it makes sense to treat them too as protocol errors.

I hope we don't need to clarify this explicitly in the ssh
specification, as its a general utf8 issue. However, I think we should
update our utf-8 references to refer to RFC 3629.

Regards,
/Niels

---

From: Jon Bright <[email-masked]>
Date: Tue, 05 Oct 2004 15:13:08 +0200

> update our utf-8 references to refer to RFC 3629.

Mentioning this issue in the Security Considerations section would seem 
like a good idea, though I agree that nothing further is necessary.

-- 
Jon Bright
Silicon Circus Ltd.
http://www.siliconcircus.com