Subject: [apps-discuss] FWD: RFC 2152 - UTF-7 clarification

From: John C Klensin <[email-masked]>
Date: Wed, 07 Oct 2015 07:00:34 -0400

Hi.

As many others presumably noticed, the message below recently
appeared on the IETF list.  It prompts me to wonder whether,
given increasing use of UTF-8, the "EAI" mail extensions for
non-ASCII addresses and headers (which require UTF-8), and the
observation that UTF-7 has never been a recommended form, it may
be time to formally deprecate the thing.  That could be done
either by moving it to historic or by writing a super-short
Applicability Statement that would reclassify it to "Not
Recommended".  Especially if Amichai is correct and there is an
ambiguity in the spec, but even in the interest of reducing the
number of forms that appear to be recommended, it seems to me
that the right recommendation for email today is:

(1) Use UTF-8
(2) If the transport environment is not suitable for 8-bit
characters, use Base64

I think experience suggests that Quoted-Printable should
probably not be recommended further unless the character
repertoire is entirely Latin Script and has a very high ratio of
Basic Latin (i.e., ASCII) characters to decorated/ectended ones.

Anyone interesting in working on this to the extent needed to
push something through?

    best,
      john

---------- Forwarded Message ----------
Date: Wednesday, October 07, 2015 11:21 +0300
From: "A. Rothman" <[email-masked]>
To: [email-masked]
Subject: RFC 2152 - UTF-7 clarification

Hi,

I hope this is the right place to discuss RFC 2152 - I couldn't
find a conclusive answer as to where and how one should comment
on a specific Request For Comments (which is a bit unsettling
:-) )

I'd like to raise an issue with the UTF-7 decoding process as
described in the RFC with respect to trailing padding bits:

"Next, the octet stream is encoded by applying the Base64 content
transfer encoding algorithm as defined in RFC 2045, modified to
omit the "=" pad character. Instead, when encoding, zero bits are
added to pad to a Base64 character boundary. When decoding, any
bits at the end of the Modified Base64 sequence that do not
constitute a complete 16-bit Unicode character are discarded. If
such discarded bits are non-zero the sequence is ill-formed."

The way I understand this is that after decoding the
modified-base64 data and grouping the resulting octets into
16-bit Unicode characters, any remaining zero bits at the end
(up to 15 bits, theoretically) should simply be ignored. I'm not
sure why an encoder would want to add extra zero bits at the end
beyond the minimum necessary, but it is arguably allowed to pad
'to *a* Base64 character boundary', not specifically *the next*
boundary. Perhaps an encoder would use some version of a standard
Base64 routine and then replace the padding '=' characters with
'A' characters (which are then decoded to all zero bits). Such
encoding would obviously be less space-efficient since it adds
unnecessary octets to the encoding - but it seems like there are
valid reasons to do so.

The issue is with the decoding though, and the reason it came up
is that I've checked various existing UTF-7 decoder
implementations and resources (e.g. iconv, uconv, icu, jutf7,
jcharset, Wikipedia, etc.) and they seem to disagree about this
issue. Some generate an error after a maximum number of trailing
zero bits (the maximum changes between implementations), and
some agree with my interpretation above where any leftover
partial group of zero bits is discarded and it's always valid.

So, since there is such discrepancy in practice in how this is
being interpreted, I submit that the description is not clear
enough to make this unambiguous. Could someone please clarify
what is officially valid/invalid according to the RFC regarding
trailing zero bits? Can we add errata that clarifies it in
either case?

Finally, if it helps, here are some concrete test cases to
consider:

+A-
+AA-
+AAA-
+AAAA-
+AAAAA-
+AAAAAA-
+AAAAAAA-

Which of these are valid inputs? Which are invalid? How many
0x0000 16-bit characters should each one be decoded into?

Thanks!

Amichai

---------- End Forwarded Message ----------

---

From: Tony Finch <[email-masked]>
Date: Wed, 7 Oct 2015 12:08:02 +0100

John C Klensin <[email-masked]> wrote:
>
> As many others presumably noticed, the message below recently appeared
> on the IETF list.  It prompts me to wonder whether, given increasing use
> of UTF-8, the "EAI" mail extensions for non-ASCII addresses and headers
> (which require UTF-8), and the observation that UTF-7 has never been a
> recommended form, it may be time to formally deprecate the thing.  That
> could be done either by moving it to historic or by writing a
> super-short Applicability Statement that would reclassify it to "Not
> Recommended".

UTF-7 is used in modified form for IMAP mailbox names, and RFC 2152 is a
normative reference from RFC 3501. So while it probably makes sense to
deprecate it for message bodies, it needs to remain as part of IMAP.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire: Easterly 4 or 5, increasing 6 at times. Slight or
moderate, but rough in southwest Viking. Showers later. Good, occasionally
poor later.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 07 Oct 2015 12:10:47 +0100

> deprecate it for message bodies, it needs to remain as part of IMAP.
Yes, it needs to remain as part of IMAP in short term, but I can take 
care of this, as I am revising RFC 3501.

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 07 Oct 2015 14:07:48 +0200

On 7 Oct 2015, at 13:00, John C Klensin wrote:

> it seems to me

Yes

> Basic Latin (i.e., ASCII) characters to decorated/ectended ones.

No reason to really use it then either.

Anyone know of software that can do Quoted-Printable and not Base64?

Anyone know of people reading (easily) UTF8 encoded with Quoted-Printable?

> Anyone interesting in working on this to the extent needed to

Interested, yes. Time, no.

But happy to help pushing.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 Oct 2015 09:07:06 -0400

--On Wednesday, October 07, 2015 12:10 +0100 Alexey Melnikov
<[email-masked]> wrote:

>...
>> UTF-7 is used in modified form for IMAP mailbox names, and
>> RFC 2152 is a normative reference from RFC 3501. So while it
>> probably makes sense to deprecate it for message bodies, it
>> needs to remain as part of IMAP.

> Yes, it needs to remain as part of IMAP in short term, but I
> can take care of this, as I am revising RFC 3501.

I had forgotten the IMAP case but, given the mess created by any
approach to message header downgrading that the EAI WG was able
to come up with (RFCs 6857 and 6858 are "least bad" approaches),
it seems to me that we should be pushing IMAP (and POP3) very
strongly in the direction of 6855 and 6856 respectively.  We
should be doing that independent of the implications to UTF-7,
but, if it eliminates the need for UTF-7, so much the better.
Following the precedent set in 5321 and 5322 and their
predecessors, would it make sense to incorporate the core
elements of 6855 into 3501bis and then describe the UTF-7
mailbox form as "SHOULD ACCEPT" for servers and "SHOULD NOT
SEND" for clients?

    john

---

From: "John Levine" <[email-masked]>
Date: 7 Oct 2015 18:55:26 -0000

>Anyone know of people reading (easily) UTF8 encoded with Quoted-Printable?

When the UTF-8 is mostly ASCII with the occasional accented latin
character or soft space, yes.

When the UTF-8 is Chinese, no.

R's,
John