Subject: [EAI] Downgrading an EAI

From: Freek Dijkstra <[email-masked]>
Date: Tue, 22 Apr 2014 22:14:07 +0200

Hi all,

First of all, let me state that I'm a novice when it comes to the SMTP
protcol. With that, and risking a heated discussion, let me ask the
question:

What's the best way for a library to support EAI for the recipient address?

Mostly for fun, I give students a little challenge at the end of a
yearly colloquium (on an unrelated topic) I give at a local university.

This year, the challenge was to drop me an email at √[email-masked],
hopefully challenging them to think about web standards (in this case
IMA, IDNA and IPv6 -- the later because this particular domain's
resolvers no longer support the deprecated IPv4 protocol ;) ).

Of course, my challenge was to make it work. To test, I wrote a simple
Python script [1], and also created a bug report to support EAI for the
Python smtplib library [2].

Of course, the question came up how to handle MTAs that do not support
the SMTPUTF8 extension. According to RFC 6530 sect 8, the proper way is
to "return the message to the originating user so the message may be
manually downgraded to the traditional form" (e.g. try again with an
ASCII address).

We interpreted that we should modify the smtplib.sendmail() method to
both take a UTF-8 encoded mail address as well as a related ASCII mail
address for the to_address. This seemed clumsy to us: when a user has to
specify a ASCII variant in the first place, what would be the value of
adding another UTF-8 variant?

So I asked how a MTA like Postfix handled the situation [3] and got a
response that it will not be supported unless SMTPUTF8 provides a path
for gradual migration, and without an option for a SMPTUTF8-capable MTA
to forward email with an SMPTUTF8 sender address to a non-SMPTUTF8 MTA,
that SMTPUTF8 will fail.

(Some participants in the discussion even stated that for this reason,
thay hope that SMTPUTF8 will fail, but Wietse pointed out that he still
hopes that some clever person will come up with a tweak to the proposed
protocols that avoids the need to bounce email with a non-ASCII sender,
or non-ASCII whatever).

Rather than admitting defeat and declaring SMTFUTF8 a failure without a
way to downgrade a non-ASCCI mail address in transit, I had a look at
what current MUAs do right now.

Here is what I've seen so far:

Apple Mail: Convert to quoted printable format.
Evolution: Convert to quoted printable format.
Thunderbird: supports IDNA, but not non-ASCII characters
    in the local part of the recipient address.
Mutt: Send as-is in UTF-8 format.
MailMate: Send as-is in UTF-8 format.
GMail: supports neither IDNA nor EAI.  (This is the webinterface,
    GMail SMTP servers forward it as-is though)
Squirrelmail: supports neither IDNA nor EAI.
Alpine: supports neither IDNA nor EAI.
Roundcube: supports neither IDNA nor EAI.

On the server side I've only checked Postfix. Interestingly enough, if a
MUA sends a mail with UTF-8 encodeded recipient, or quotes-printable
encoded recipient, the message arrives just fine, and I can even
retrieve it with the dovecot IMAP server and Thunderbird IMAP client. So
after announcing the √[email-masked] email address, I could see a
significant amount of email with users running Apple Mail.

For good measure, I also tried to list all possible formats that arrived
at my MTA, listed by popularity:

9× <=?utf-8?Q?=E2=88=9A9?=@3.xn--14-nbc.eu> (local part quoted
    printable, IDNA for domain)
2× <√[email-masked]> (UTF-8 encoded as-is)
0× <utf-8;\[email-masked]\x{3C0}.eu>  utf-8-addr-xtext with
    Embedded Unicode Chars [RFC 6533]
0× <=?utf-8?Q?=E2=88=9A9=403=2E14=CF=80=2Eeu?=>  (fully quoted
    printable)  [RFC 6857, as suggested for IMAP/POP]
0× <=?utf-8?b?4oiaOQ==?=@3.xn--14-nbc.eu>  (local part BASE64
    encoded, IDNA for domain)

Interestingly, the behaviour of Apple Mail and Evolution seems most
popular, and that is also what is kind of suggested in RFC 6530 sect 8:

> [...] message-originating systems SHOULD be prepared to [either ... or]
> return the message to the originating user so the message may be
> manually downgraded to the traditional form, possibly using encoded
> words [RFC2047] in the message headers.

Where RFC 2047 defines this quoted printed format.

Now I'm a bit confused if such a downgrade is a good or bad thing. From
the users perspective it is clearly good.

The odd thing seems to be the fact that RFC 6530 sect 8 and RFC 6409
sect 6.5 suggest that downgrading a mail format is good, using out of
band information. But RFC 6530, sect 9 clearly states that this should
not happen automatically.

Now I'm a bit confused as to what a library such as Python's smtplib
should do:

* allow the user to specify the downgraded recipient address in
conjunction with the non-ASCII address, and fail when no such
alternative is provided (and basically accepting that no-one will use
STMPUTF8, because users won't bother to create a clumsy non-ASCII
address in the first place)
* use the printed-quotable translation for MTA that do not support
SMTPUTF8, so that it arrives in practice.
* use the printed-quotable translation for MTA that do not support
SMTPUTF8, but provide a way for the user to specify an alternative
downgraded recipient address.

Similarly, for a MTA 'in transit' such as Postfix: what should be the
behaviour here?

I'm happy to explain the provided RFCs to the students who took the
challenge to drop me an email at the above address, but I was not able
to find background information why the WG did choose not to make the
system backward compatible, giving rise to the opposition I've come
across while implementing the proposed standard.

Regards,
Freek

[1] https://github.com/macfreek/homescripts/blob/master/mail/smtptest.py
[2] http://bugs.python.org/issue20084
[3] http://thread.gmane.org/gmane.mail.postfix.user/241125

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 23 Apr 2014 12:38:10 +0200

> What's the best way for a library to support EAI for the recipient 
address?

IMO, to support sending mail to both EAI addresses and to ASCII addresses.

That's two disjoint sets of addresses. There could be a [email-masked] 
to match [email-masked], but the Günthers of this world don't seem 
eager to do that. The people who really want EAI are the ones who don't 
have a natural and easy ASCII transcription.

So, EAI doesn't say that an ASCII address has to accompany each non-ASCII 
address. If you create and use उदाहरण@उदाहरण.in, you're not going to be 
able to send mail to [email-masked], and he may or may not be able to 
send to you. But on the other hand, you don't have to decide whether दा is 
transcribed with one a or two.

Arnt

---

From: James Cloos <[email-masked]>
Date: Wed, 23 Apr 2014 14:29:25 -0400

>>>>> "FD" == Freek Dijkstra <[email-masked]> writes:

FD> Interestingly, the behaviour of Apple Mail and Evolution seems most
FD> popular, and that is also what is kind of suggested in RFC 6530 sect 8:

I'd go that route (use utf8+qp) if I were writing the application.

I'd support utf8+b64 on the receiving end, but prefer qp when sending.

-JimC
--
James Cloos <[email-masked]>         OpenPGP: 0x997A9F17ED7DAEA6

---

From: [email-masked]
Date: Thu, 24 Apr 2014 11:52:34 +0900 (JST)

> From: Freek Dijkstra <[email-masked]>

If you are planning to write E-mail client (Mail user agent),
please read draft-ietf-eai-email-clients-01 and comment it.

  http://tools.ietf.org/html/draft-ietf-eai-email-clients-01

If it is useful for implementors, we will restart it as individual submission.

> adding another UTF-8 variant?

In my opinion, the library should take one email address (UTF-8 or ASCII only).
If the library detects errors, it returns errors.
E-mail clients need to detect errors and inform the errors to users.
Then, users need to change configurations. (UTF-8 address to ASCII only)

> to forward email with an SMPTUTF8 sender address to a non-SMPTUTF8 MTA,

See section 3.2 of RFC 6531.

      it SHOULD reject the message.  As usual, this can be done either
      by generating an appropriate reply during the SMTP transaction or
      by accepting the message and then generating and transmitting a
      non-delivery notification.

> what current MUAs do right now.

'downgrade' was rejected.

--
Kazunori Fujiwara, JPRS <[email-masked]>

---

From: John C Klensin <[email-masked]>
Date: Thu, 24 Apr 2014 08:11:00 -0400

--On Wednesday, April 23, 2014 14:29 -0400 James Cloos
<[email-masked]> wrote:

> FD> Interestingly, the behaviour of Apple Mail and Evolution
> seems most FD> popular, and that is also what is kind of
> suggested in RFC 6530 sect 8:
> 

Fascinating.

First, I think it would be reasonable to summarize 20+ years of
experience with Quoted-Printable as "works ok for strings and
blocks of text that are predominately ASCII or at least
[extended] Latin script; not much better than Base64 or Hex
encoding for anything else (and sometimes worse)".  Second,
remember that, unlike IDNA A-labels, where we had lots of reason
to believe that it was reasonable to assume that anything
starting in "xn--" could safely be interpreted as such a label
(or invalid), random strings of characters from the Base64
character set have been valid as local parts, with no decoding
implications, since about the time intersystem email appeared on
ARPANET hosts.

For email local parts, this is all about aliasing.  If you want
to create a local alias that connects a non-ASCII string with
what you think of as its Base64 or Q-P mapping, or a traditional
hex mapping (with or without "%d" or "0x"), or what you think of
as a translation into some other language or script, go for it.
That has always been possible and orderly and the EAI work just
expands the possible options a bit.  

The problems arise (and always have) if the sending side --or,
worse, an intermediary system-- tries to guess at what aliases
are present.  Such guessing is likely to yield negative results
(and hence bounces or message rejection).  Algorithmic guesses
are less likely, but still plausible, to yield false positives
and deliver to the wrong address.  Again, nothing new here.  If
some system looks at "James Cloos <[email-masked]>" and
guesses that mail could be sent to [email-masked],
[email-masked], or [email-masked]
because it knows something about the sometime relationship
between "James" and "Jimmy", "Джеймс", or something about
Base64 , there should be no expectation of accurate delivery
even thought the operator of the delivery server for
"jhcloos.com" might choose to make one or more of those work.

What concerns me about "just map to Q-P" or "just map to Base64"
is that, while a user getting some form of a "UTF-8 in addresses
or headers not supported here" message is not desirable, it is
at least clear and accurate.  If the sending MUA or intermediary
looks at a non-ASCII UTF-8 string and automatically converts it
to some ASCII-compatible encoding and the message is then
rejected as undeliverable, the sender has just been convinced
that email with non-ASCII addresses is unpredictable -- an
especially bad message if the original, non-ASCII address really
could have been delivered-- and that sets all of us back.

    Just my opinion, of course,
     john

---

From: James Cloos <[email-masked]>
Date: Thu, 24 Apr 2014 12:38:43 -0400

>>>>> "JK" == John C Klensin <[email-masked]> writes:

JK> Quoted-Printable as "works ok for strings and blocks of text that
JK> are predominately ASCII or at least [extended] Latin script;

You're right.  I was thinking of euro names when I wrote that.

In retrospect, I'd send qp where that makes sense else b64.

JK> For email local parts, this is all about aliasing.

I tried a test before sending, but further testing today shows that I
misread the results.  Dumb.

It delivered due to a local catchall rule (avoiding that domain's normal
processing) and the qp was converted *after* the mta, rather than *by*
as I had presumed.

So we need to patch MTAs to decode qp and b64 when comparing addresses
against permission and routing databases.  That probably is needed even
without utf8, for security and robustness.  Which may be as hard as
getting SMTPUTF8 support. ☹

(Obviously those conversions must be done in a manner which avoids any
new attacks, such as SQL-injection or the like.)

-JimC

---

From: John C Klensin <[email-masked]>
Date: Sun, 27 Apr 2014 10:54:32 -0400

--On Thursday, April 24, 2014 12:38 -0400 James Cloos
<[email-masked]> wrote:

> JK> Quoted-Printable as "works ok for strings and blocks of
> text that JK> are predominately ASCII or at least [extended]
> Latin script;
> 

"Makes sense" turns out to be a little subjective.  If all
characters are in (estended) Latin script and most words or
names (including those in strings turned into local parts by
catenation or abbreviation) contain only ASCII characters with
those that contain non-ASCII ones typically containing only a
single one, then Q-P is pretty reasonable, in part because one
can usually guess when was intended even without decoding.
Those conditions are common for Western European languages and
names.  I would suggest that thinking mostly about them was what
brought us Q-P in the first place.  By contrast, if one has a
local part in even such a closely-related script as Greek or
Cyrillic, Q-P makes quite a mess... and is typically less
compact and no easier to decode than Base64 or even HTML-like
%-encoding.

> I tried a test before sending, but further testing today shows
> that I misread the results.  Dumb.
> 
> It delivered due to a local catchall rule (avoiding that
> domain's normal processing) and the qp was converted *after*
> the mta, rather than *by* as I had presumed.
> 
> So we need to patch MTAs to decode qp and b64 when comparing
> addresses against permission and routing databases.  That
> probably is needed even without utf8, for security and
> robustness.  Which may be as hard as getting SMTPUTF8 support.
> ☹
> 
> (Obviously those conversions must be done in a manner which
> avoids any new attacks, such as SQL-injection or the like.)

It is a matter of taste, but I suggest that patching MTAs, even
final delivery MTAs, is a bad idea.  If you want Q-P or B64
encodings to "work" in local-parts, do it in aliases.  Remember
that, if non-ASCII characters appear in a backward-pointing
field in UTF-8 and you convert them to some other encoding form,
there is no guarantee at all that the differently-encoded form
will be recognized by the originator if it shows up in a reply.
Your outgoing submission server/ MTA could try to undo the
conversion but then you face a tricky heuristic problem.  The
same sorts of "tricky string" issues that make for good phishing
attacks, including mixed script and oddly-appearing strings, are
good protections (even if somewhat security-by-obscurity)
against dictionary or algorithm based spamming of addresses.
For example making the pre-subaddress local part of your address
"сlоos"  (first and third characters are U+0441 and U+043E
respectively), you would probably see less random spam than you
do with "cloos".   Similar comments would apply, probably even
more strongly given dictionaries, if you used "jаmеs" (U+0430
and U+0435 in the second and fourth positions) instead of
"james" in a local-part.

Similar comments apply to Q-P look-alikes.  Sticking "=20" or,
for that matter, "=7F" or even "=FF"  in the middle of a string
would be pretty good protection against dictionary attacks.  For
the people who believe that "foobar at example.com" is effective
at preventing address harvesting relative to
"[email-masked]", then "foo=[email-masked]" or
""foo=NNbar at example.com" should be lots more effective, with
"NN" being two random hex digits and a filter in the delivery
MTA that simply removes all three-character sequences starting
in "=".   Those approaches are as reasonable and legitimate as
treating such strings as Q-P and encoding or decoding them to
UTF-8 (or some other charset) -- the problem is that a receiving
system just can't tell.

So, again, if you want Q-P or B64 aliases for your incoming
non-ASCII local part, go for it.  But, at least IMO, you are far
better off treating them as local aliases than trying to do the
conversions algorithmically.

    john

---

From: James Cloos <[email-masked]>
Date: Sun, 27 Apr 2014 11:12:36 -0400

>> In retrospect, I'd send qp where that makes sense else b64.

JK> "Makes sense" turns out to be a little subjective.

Indeed.  I suspect less than four or five non-ascii characters.
Or, more likely, less than ten or so percent of the string.

I won't know for sure what I'd choose without testing; visual
appearance would be the most important issue.

But it is a moot point unless we could get MTAs to decode them
when routing mail.  And since the working group chose not to
support qp or b64 that may not be possible.

I wouldn't have replied at all had my test not been compromized.
(Cf my earlier retraction.)

-JimC

---

From: James Cloos <[email-masked]>
Date: Sun, 27 Apr 2014 11:20:56 -0400

-JimC

---

From: John C Klensin <[email-masked]>
Date: Sun, 27 Apr 2014 14:09:19 -0400

--On Sunday, April 27, 2014 11:12 -0400 James Cloos
<[email-masked]> wrote:

>...

My explanation and examples aren't intended to represent or
analyze WG consensus, but should give you some idea of at least
one of the reasons why the WG decided to not do that.  Another,
as you and others pointed out earlier, is that simply supporting
UTF-8 addresses and headers may be no more difficult (and
possibly even less so) than fussing with problem-prone
conversion approaches.

    john