Subject: Re: [EAI] RFC5336bis and trace fields

From: Shawn Steele <[email-masked]>
Date: Sun, 26 Jun 2011 05:38:48 +0000

>> the current text in rfc5336bis is "   Except in the 'For' clause and
>> 'Reverse-path' clause where
>>   internationalized domain name with the U-label form MAY be used,
>>   internationalized domain names in Received fields MUST be transmitted
>>   in the form of A-labels. "
>
> then a rule requiring use of ASCII-only encoding does NOT seem to be
> appropriate.
>
> That sort of rule is appropriate to a hybrid environment, not a
> Unicode-pure environment.
>

I disagree with John's take for the same reasons :)  "tracing" is complicated if there is artificial encoding that may or may not be accurately applied, eg: conversion to a-labels.  I'd prefer that stuff be left in unaltered UTF-8.  I realize that this has some risk of loss, but it modern systems it's trivial to forward stuff as attachments or whatever for debugging.  I'd prefer to err on the side of having new systems be more tracable than a chance of having obsolete systems more traceable at the expense of new systems.

Note that, if this goes to a downlevel legacy SMTP thing, then these RFCs don't apply.  Presuming that the system's smart enough to know about EAI and could follow any rule like the one above would also presume that the system can pass back the UTF-8.

In my opinion, I find logs and tracing type files much easier to understand when it's plaintext.  If it says xn--gobbledegook, then I have no clue if it's a Chinese domain, a Russian domain or whatever, I'd have to find a tool to see if its the domain expected.  (Not to mention which, leaking punycode everywhere is Evil, at least in the long term).  If it's in Unicode, I can at least see that the scripts similar to whatever's expected, and can even think "yup, it's got a character with a house or 3 squiggles or a backwards R" or whatever, and see if it's the expected string or not.  I can still use a tool if really needed to get to punycode.  (My experience is that I end up seeking Unicode code points though).

Anyway, I missed this before, I really don't want to require (or even encourage) use of A-Labels.  A-Labels belong to the legacy SMTP behavior, which these specs are not.

-Shawn

 
http://blogs.msdn.com/shawnste

---

From: SM <[email-masked]>
Date: Sun, 26 Jun 2011 02:40:08 -0700

Hi Shawn,
At 22:38 25-06-2011, Shawn Steele wrote:
>labels.  I'd prefer that stuff be left in unaltered UTF-8.  I 
>realize that this has some risk of loss, but it modern systems it's 
>trivial to forward stuff as attachments or whatever for 
>debugging.  I'd prefer to err on the side of having new systems be 
>more tracable than a chance of having obsolete systems more 
>traceable at the expense of new systems.

As an anecdote, my experience is different.  The level of brokenness 
of modern systems is understated.  Adherence to standards is 
flaky.  It can be quite an effort to get mail headers for 
debugging.  Any loss of relevant information increases that effort.

Regards,
-sm

---

From: "John Levine" <[email-masked]>
Date: 26 Jun 2011 14:46:40 -0000

>of modern systems is understated.

Indeed, but this discussion is basically an attempt to guess in what
way broken mail systems will behave once they have ignored our "don't
do that" and relayed an EAI message to a non-EAI MTA.  I can imagine a
wide variety of things that might happen to a message: it might pass
through unscathed because the MTA happens to be 8-bit clean (what my
MTA does).  Or it might helpfully encode stuff as base64. Or it might
chop stuff down to 7 bits, Or it might guess that the 8-bit stuff is
ISO-8859-x or Windows-1252 and downcode it to nonsense.  Or it might
do something else.

I don't see any particular reason to expect any of these behaviors to
dominate the broken MTA market, so I'm reluctant to make any changes
intended to work around what we anticipate broken MTAs might do.  No
matter what we do, including doing nothing, it shouldn't be hard to
tell where the broken relay is.

R's,
John

---

From: Shawn Steele <[email-masked]>
Date: Sun, 26 Jun 2011 19:12:01 +0000

And I think munging to a different form causes that kind of loss... Because it's not predictable :-D

- shawn ()

Sent from my Windows Phone 7
________________________________
From: SM
Sent: 6/26/2011 2:40 AM
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] RFC5336bis and trace fields

Hi Shawn,

Regards,
-sm

---

From: [email-masked]
Date: Sun, 26 Jun 2011 09:33:26 -0700 (PDT)

> ISO-8859-x or Windows-1252 and downcode it to nonsense.  Or it might

One of the more likely outcomes is that all uses of 8bit gets turned into
encoded-words with some guess being made as to the charset. The reason it's
common is it actually works pretty well as long as the 8bit wasn't used in
addresses, which prior to EAI you wouldn't expect it to be.

But with 8bit appearing in addresses ... a lot of the agents that turn stuff
into encoded words don't exactly have the world's best parsers, and will
probably end up converting EAI addresses into encoded-words.

> intended to work around what we anticipate broken MTAs might do.

We need to be very clear here: The "brokenness" occurred when some EAI system
improperly forwarded an EAI message to a non-EAI-capable MTA or other
component. Once that happens all bets are off - with only a few exceptions, our
standards generally don't specify what various components are supposed to do
when the receive what is from their perspective egregiously incompliant
material. So you can and should expect behaviors ranging from outright message
rejection (likely result with our MTA, for example) to various attempts to fix
the message, to the message silently disappearing. (This last is often the end
result of an initial bounce - the bounce itself bounces and that's that.)

> No

Yes, in most cases tracking down the system that sent an EAI message to a
non-EAI system is fairly simple. Getting such things fixed is another matter
entirely, but unless someone has some truly novel approach to suggest I'm
not seeing anything we can do about it in our specifications.

				Ned

---

From: Joseph Yee <[email-masked]>
Date: Mon, 27 Jun 2011 15:32:41 -0400

Speaking as individual (and listening from everyone)...

One of my concern, besides relay, are programs that analyze trace information.  Some analyze on IP, some on domains, some on both and beyond.  And with current ASCII-only standard, these programs who concern on domain infos treat domains as is (A-label only).  If we allowed non-ASCII Unicode in trace fields, what is the impact?  My personal estimate is that we will break many of these programs at MUA and mail-store level initially, and it will hurt adoption and deployment.  And I would love to be convinced that the impact is minimal.

I agreed with Shawn that it is painful to human eye (especially end users) to see A-label in trace info, but I am curious how much end users care about the trace info (that is if they read from raw message source a lot), I assume it matters more to mail administrators and programs.

And I would like to ask everyone (still as individual, but start getting blurry):

Who cares of trace info? (human? at what role? program? at what level of mail transaction?)

It's not consensus question, but to keep discussing how the WG perceive trace info in different perspectives.

Best
Joseph

On 2011-06-26, at 3:12 PM, Shawn Steele wrote:

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: [email-masked]
Date: Mon, 27 Jun 2011 14:24:33 -0700 (PDT)

> One of my concern, besides relay, are programs that analyze trace
> information.  Some analyze on IP, some on domains, some on both and beyond. 
> And with current ASCII-only standard, these programs who concern on domain
> infos treat domains as is (A-label only).  If we allowed non-ASCII Unicode in
> trace fields, what is the impact?

IMO the mistake here is in thinking we have any real control over this
happening. For better or worse, large numbers of email applications create
identifiers and fill in these sorts of fields with values derived from various
configuration settings that also control what ends up in various addressing
fields. It's simply not realistic to believe that all of the code out there
that does this will be changed to deal with those settings moving to utf-8 as
part of EAI.

So there are going to be leaks. Probably lots of them. Which means even if we
prohibit utf-8 in trace fields, it's going to end up in there anyway. In fact
there's nothing stopping 8bit material from ending up in trace fields already
(yes it's a standards violation but there's never any shortage of violators out
there), and since trace field analysis tends to be rather critical when it is
done, I rather suspect the tools in this space have at least learned to
tolerate 8bit rather than fall over.

The alternative, which I strongly prefer, is to say, "EAI trace can contain
utf-8 information and tools need to be updated accordingly. Deal with it."

Given that EAI causes utf-8 to appear in MAIL FROM and RCPT TO, this means that
there's already no choice but to upgrade the large number of tools that process
transaction logs to deal with it, I'm just not seeing this as being all that
difficult. In fact the work involved seems completely trivial compared to the
effort needed to add EAI support to, say, a message store.

> My personal estimate is that we will break many of these programs at MUA and
> mail-store level initially, and it will hurt adoption and deployment.  And I
> would love to be convinced that the impact is minimal.

I don't think the impact is minimal, but again, the mistake is in thinking that
a prohibition will fix the problem. And again, I don't think this is anywhere
close to being the biggest impediment to EAI deployment. In fact given what
store EAI support is starting to look like I seriously doubt it's of any real
significance at all.

> I agreed with Shawn that it is painful to human eye (especially end users) to
> see A-label in trace info, but I am curious how much end users care about the
> trace info (that is if they read from raw message source a lot), I assume it
> matters more to mail administrators and programs.

End users don't know and don't care.

> Who cares of trace info? (human? at what role? program? at what level of mail

It's overwhelming consumed by people in my experience, but there are any
number of automatic processors.

But if you want to look at something specific, consider Sieve. Sieves that
piddle around in Received: fields are pretty common (in fact an entire Sieve
extension, index, primary purpose is looking at Received: fields). But not only
is Sieve natively utf-8, it automatically decodes encoded-words in Received:
fields to utf-8. So Sieves already have to deal with utf-8 in Received: fields.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 27 Jun 2011 23:07:41 -0000

>tolerate 8bit rather than fall over.

Having written a certain amount of it, I'd agree.  A little UTF-8 in
the headers is no more broken than a lot of stuff we have to deal with
already.

>utf-8 information and tools need to be updated accordingly. Deal with it."

Right.

R's,
John