Subject: Re: [EAI] RFC5336bis and trace fields

From: John C Klensin <[email-masked]>
Date: Tue, 28 Jun 2011 09:37:24 -0400

--On Saturday, June 25, 2011 16:10 -0700 Dave CROCKER
<[email-masked]> wrote:

> On 6/25/2011 4:01 PM, Jiankang Yao wrote:
>> It seems that you favor the Unicode-pure for the trace field;
> 
> I believe that that is consistent with the current goals of
> the working group, yes.

>...
> However as I noted earlier today, support for hybrid
> environments was removed as a goal for the working group and
> it has greatly simplified all other portions of these two
> documents.

Dave,

This is personal opinion... any declaration of consensus, etc.,
will need to come from Joseph.

I think we have a small disconnect about definitions here, one
that is making a conclusive discussion harder.

>From one perspective, the WG decision was simply to remove the
idea of doing downgrading on the fly.  All of the other
concessions to, and design decisions for, operating in an
environment in which legacy systems (those that did not support
the EAI extensions) would send mail to and receive mail from
EAI-conforming systems would remain or at least be part of
separate discussions.  Such legacy systems obviously could not
send or receive messages that required the capabilities that the
EAI extension enabled, but that is the nature of SMTP extensions.

The other perspective --the one that I think you have
consistently assumed-- is that the in-transit downgrading was
the essential feature of a "hybrid" environment and that
elimination of in-transit downgrading as a goal was a WG
decision to drop all support for such "hybrid" environments.  (I
use quotes because the term does not appear in any EAI
WG-produced spec nor, as far as I know, has it ever been clearly
defined.)

There are a number of situations in which the difference between
those two definitions (or, if one prefers, models) is important.
As a trivial example, suppose we have SMTP client that has no
EAI (or IDNA) support and that is trying to send a message to
[email-masked] (with "someuser" entirely in ASCII and
"some.domain" containing characters that correspond strictly to
the preferred hostname syntax of RFC 1034 (no IDNA involved).
Suppose that SMTP client does the required lookup on
"some.domain" and gets:
    some.domain. IN MX 10 smtp.some.domain.
                    MX 20 smtp.other.domain.
but that "other.domain" contains A-labels and that the host
identified by that FQDN is EAI-capable (and advertises that in
its EHLO response).

We've advised that EAI-capable target sites not list sites as
lower-preference MXs unless they are also EAI capable.  We've
not given advice that EAI-unaware sites avoid having
lower-preference MXs that are EAI-capable (partially because
such advice would be ignored and would almost certainly retard
EAI deployment), so the above case is plausible.

Now, if smtp.some.domain accepts the connection, we have a pure
SMTP environment in which EAI is not involved in any way.
However, if it cannot be reached, the connection would go to
"smtp.other.domain".   It is EAI-capable, so, by some ways of
reading the recent threads, it should insert trace subfields
(particularly "Received: ... from ...") that include its own
domain name in U-label (UTF-8) form.  A message containing UTF-8
data in its trace field(s) (or any other header field) cannot be
passed on to any host that does not support EAI, nor can it be
bounced to such a host, so having smtp.other.domain make that
decision would immediately drop the message into a black hole as
neither forwardable to smtp.some.domain nor bouncable to the
original client without violating the spec.  To me, that is an
obvious silly state, independent of what we have to say about
smtp.other.domain and hybrid models.   On the other hand, if it
inserts its name in A-label form, things are just fine even
though the host that sent to it and the host to which it intends
to relay are both EAI-ignorant.

So it seems to me that, to retain the integrity of the legacy
email system but permit EAI-capable hosts to be listed in
lower-preference MX records for non-EAI-capable targets, the
host at smtp.other.domain must either always use A-labels in any
IDNs it needs to place in "Received...from" subfields or must
use them when relying to a server that does not support EAI (but
may use U-labels when relaying to an EAI-capable server).  

I see things as more complicated (and ironically as making less
difference for the reasons Ned, John Levine, and others give) if
the message that the intermediate host receives requires EAI
features (e.g., non-ASCII address information) and the
submission client asked for those features with a MAIL
parameter.   For those cases, we've already told people that
having EAI-incapable hosts in the MX chain is a bad idea, the
originating host is presumably EAI-capable, and a little UTF-8
leakage is unlikely to significantly compound our problems.  My
instinct is that we would be better off with A-labels in any
IDNs that appear, mostly because it would lower the odds of
systems getting the decisions wrong, but I recognize the
arguments of Shawn and others that U-labels may be better.  And,
again, I don't think it makes a lot of difference -- people will
do what they think is easiest and that may depend on the
internal logic of MTAs.  

But for what I think you are describing as a hybrid case, at
least the one illustrated above, I think the intermediate/relay
host has better behave as if it were not EAI-capable.  Indeed, I
think that we added that MAIL parameter (partially at your
insistence, if I recall) precisely to let the relay host make
the decision about whether EAI capabilities are needed (of
course, if an SMTP client sends non-ASCII UTF-8 address or
header material without sending that parameter, it is on its
own, but that is not the case I'm concerned about).

best,
   john

---

From: "John Levine" <[email-masked]>
Date: 28 Jun 2011 16:14:26 -0000

>Now, if smtp.some.domain accepts the connection, we have a pure SMTP
>environment in which EAI is not involved in any way.  However, if it
>cannot be reached, the connection would go to "smtp.other.domain".
>It is EAI-capable, so, by some ways of reading the recent threads, it
>should insert trace subfields (particularly "Received: ... from ...")
>that include its own domain name in U-label (UTF-8) form. ...

I don't think so.

We've defined two parallel mail streams, legacy and EAI.  Each SMTP
transaction is one or the other. Mail can move freely from legacy to
EAI, but only sometimes and with difficulty from EAI back to legacy.
This is so for any EAI MTA, no secondary MX required.

Since it will be a long time until the world is all EAI, as a quality
of implementation issue, MTAs should leave mail in the legacy stream
whenever possible to maximize the chance that it can be delivered to
a legacy-only destination. So if I were writing an MTA, I would put
A-labels in the Received: headers in mail in the legacy stream, and
UTF-8 in the Received: headers in the EAI stream.

We had a discussion a few months back essentially about how an MTA
would handle the two streams. One possiblity is to keep an EAI flag
with each message based on which stream it arrived from, and changed
only when forwarded from the legacy stream to an EAI address.  Another
is to do what I called Deep Message Inspection (DMI), look through the
envelope and contents of a message at the time it's relayed to see if
it needs EAI. As far as we decided anything, we decided that those
were both reasonable approaches, with the former being a lot faster
and the latter possibly rescuing mail that had arrived via the EAI
stream but turned out not to need EAI features.

You could argue that in a DMI environment, you might have just added a
UTF-8 header to an otherwise legacy message that arrived via the EAI
stream, which you now can't deliver.  Even if you believe that's a
likely problem (I don't), again, it's a quality of implementation
issue to downgrade a Received: header that it just added if that's all
it needs to do.

So anyway, I don't see a problem. EAI mail is EAI mail, legacy mail
is legacy mail.

R's,
John