Subject: [EAI] RFC5336bis and trace fields

From: John C Klensin <[email-masked]>
Date: Sat, 25 Jun 2011 15:54:38 -0400

Hi.
Personal opinion only...

My experience in trying to track missing or broken email
messages across multiple environments leads me to believe that
information in trace fields should be confined to the more
universal forms possible and that they should not rely on any
sort of extended, advanced, or enhanced features if that can
possibly be avoided.  

I recognize, and agree with, the view that, if we were designing
a new mail system today, we would want to accept characters
drawn from a large subset of Unicode anywhere that a "character"
could occur.  But I also recognize that, while some communities
will probably adopt EAI very quickly, we are likely facing a
long transition before the whole Internet catches up.  

I have also experienced messages being forwarded, extracts
getting made and copied into others, and so on, often in ways
that have little or nothing to so with any standards we might
produce.  As just one example, the guidance from many mail about
the first step to be taken when a message appears to be
unreadable or damaged is to expose all of the headers of that
message and then copy and paste them into a new message to be
sent off for diagnosis.  Unfortunately, most of us who have
worked in internationalization know that copy and paste
operations of non-ASCII text are not particularly reliable,
especially when the text originated outside the system on which
the operation is being performed. 

There is no question that prohibiting non-ASCII text it trace
fields will lose information.  It is likely that forcing domain
names in such fields into A-label form will be inconvenient in
many cases.  But it is almost equally certain that being
permissive about use of non-ASCII text will prevent debugging in
some situations or at least make it vastly more difficult.

The WG made a different decision in 5336 than is now proposed
for 5336bis.  Circumstances were different.  Even if they were
not, nothing says that the WG was right then and is therefore
wrong now.  But the opposite doesn't apply either: it is not
automatically true that the change is an improvement.

The tradeoff is complicated.  If the WG makes an informed
decision that the current text in 5336bis is what is wanted and
strikes the best balance between maximum information and minimum
risk of ending up with unusable data at a critical time, I have
no problem going along with that decision.  But I would
encourage everyone to think carefully about this an reach a
conclusion about the tradeoff you want, not to assume that
others have magically gotten it right or that a slogan about,
e.g., "UTF-8 everywhere" is an adequate way to make the decision.

       john

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 25 Jun 2011 14:43:56 -0700

> long transition before the whole Internet catches up.

You appear to be continuing with the working group's old model, rather than 
applying the new model.  The old model worried about hybrid environments, where 
some places support Unicode and some do not.

To repeat what has now been repeated many times:

      The new model that was adopted months ago is of a Unicode-pure environment.

      There will, indeed, need to be gateways between the ASCII-only legacy 
world and the new, Unicode-pure world, but the work to deal with that has been 
explicitly moved outside of the current two documents.

So while your estimate of the future is no doubt correct, it is not relevant to 
the current two documents.  The current two documents start from the assertion 
that Unicode is supported.

> produce.

This is, of course, true.  But it is true for many scenarios, not just for 
scenarios involving the trace fields.  Hence this is a more general issue and 
does not warrant specialized handling for trace fields.

> fields will lose information.

Correct.  And that's why it is a mistake to impose restrictions, for an 
environment that has already been declared to be Unicode-pure.

> The tradeoff is complicated.

Indeed it is... for the hybrid model.  The working group tried to deal with that 
complexity.  The decision some months ago was, instead, to make things simple. 
The only challenge now is to make sure that things are /kept/ simple.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "Jiankang Yao" <[email-masked]>
Date: Sun, 26 Jun 2011 06:27:06 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, June 26, 2011 5:43 AM
Subject: Re: [EAI] RFC5336bis and trace fields

...
>
> Indeed it is... for the hybrid model.  The working group tried to deal
> with that complexity.  The decision some months ago was, instead, to make
> things simple. The only challenge now is to make sure that things are
> /kept/ simple.
>

Does it mean that the current wording in RFC5336bis about this issue is ok
for you?

Jiankang Yao

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 25 Jun 2011 15:45:41 -0700

If the current wording is the text that you copied to the list, a few minutes ago:

> the current text in rfc5336bis is "   Except in the 'For' clause and 'Reverse-path' clause where
>   internationalized domain name with the U-label form MAY be used,
>   internationalized domain names in Received fields MUST be transmitted
>   in the form of A-labels. "

then a rule requiring use of ASCII-only encoding does NOT seem to be appropriate.

That sort of rule is appropriate to a hybrid environment, not a Unicode-pure 
environment.

   d/

-- 

   Dave Crocker

---

From: "Jiankang Yao" <[email-masked]>
Date: Sun, 26 Jun 2011 07:01:56 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Jiankang Yao" <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, June 26, 2011 6:45 AM
Subject: Re: [EAI] RFC5336bis and trace fields

> That sort of rule is appropriate to a hybrid environment, not a 
> Unicode-pure environment.
>

It seems that you favor the Unicode-pure for the trace field;

 John prefers ASCII-pure for it.

 the current wording is a tradeoff between easy debugging
and clear UTF-8 environment.

I prefer the current wording since it was discussed and decided in the WG 
many months ago.

If we need to update the current wording, it is better that one of the chair 
can call the rough consensus for this issue.

Jiankang Yao

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 25 Jun 2011 16:10:26 -0700

> It seems that you favor the Unicode-pure for the trace field;

I believe that that is consistent with the current goals of the working group, yes.

> and clear UTF-8 environment.

The only references to "debugging" that I recall concerned operation in a hybrid 
environment, with some places supporting Unicode and some places supporting 
ASCII-only.

However as I noted earlier today, support for hybrid environments was removed as 
a goal for the working group and it has greatly simplified all other portions of 
these two documents.

I do not understand the reason for saying that the more complicated hybrid model 
is needed here but not elsewhere.

d/

-- 

   Dave Crocker

---

From: "Jiankang Yao" <[email-masked]>
Date: Tue, 28 Jun 2011 10:10:05 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Jiankang Yao" <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, June 26, 2011 6:45 AM
Subject: Re: [EAI] RFC5336bis and trace fields

> That sort of rule is appropriate to a hybrid environment, not a Unicode-pure 

I dig the WG's decision in IETF77, which said

"   question
                  - "Does the WG support moving forward on the standards track
                  with the core documents (4952bis, 5335bis, 5336bis, 5337bis,
                  5721bis and the in- progress drafts) while dealing with necessary
                  end-point 7-bit header interoperability, but omiting support
                  for in-transit downgrading or alt-addresses?"
          
                 Result
                  - most say yes, no visible objections in the room
"

I think that this current wording about trace field in RFC5336bis may be a kind of "dealing with necessary end-point 7-bit header interoperability".

Jiankang Yao

---

From: Joseph Yee <[email-masked]>
Date: Tue, 28 Jun 2011 09:38:46 -0400

On 2011-06-25, at 5:43 PM, Dave CROCKER wrote:

> You appear to be continuing with the working group's old model, rather than applying the new model.  The old model worried about hybrid environments, where some places support Unicode and some do not.
> 

Speaking as co-chair:

Up to now the term "Unicode-pure" was used as the new informal working model suggestion, but it was never adopted by the WG formally, and the suggestion of the "Unicode-pure" itself was not clearly defined.  If it's truly Unicode-pure, field names can be Unicode, day & month in "Dates:" can be Unicode.  There are details not discussed + addressed.

Reviewing past WG discussions most agreed (although no formal declaration has been made) on the new ABNF approach where messages are treated as Unicode-pure. The revised ABNF allows comments, atoms, quoted-strings, unstructured text to contain Unicode.  Certainly the WG has not discussed limiting certain fields values to ASCII-only in depth, much less to the point of a conclusion being reached.  This current thread has some discussions going on, and so far no issues have been repeated after WG conclusions were reached.

So a consensus call will be made very soon on trace fields, and the WG may assess all other fields in similar situation, and determine (another consensus call) whether to adopt Unicode-pure model or not.

Also, the terms "old model" and "hybrid environment" are not adopted by WG.  What's dropped from "old model" is in transit downgrade mechanism defined in RFC5504.  EAI is still an extension of SMTP RFC5321, still supports ASCII-only mail exchanges defined by RFC5321 and RFC5322. 

> 
>     There will, indeed, need to be gateways between the ASCII-only legacy world and the new, Unicode-pure world, but the work to deal with that has been explicitly moved outside of the current two documents.

One of the formal consensus result (http://www.ietf.org/mail-archive/web/ima/current/msg04031.html), makes it clear that gateway is not mandatory if ASCII-only legacy and EAI-capable MTA exchanges ASCII-only messages.

This consensus still stands regardless whether EAI is Unicode-pure or not.  The statement above is not correct.

Best Regards,
Joseph Yee, co-chair of EAI

> 
> So while your estimate of the future is no doubt correct, it is not relevant to the current two documents.  The current two documents start from the assertion that Unicode is supported.
> 
> 
> This is, of course, true.  But it is true for many scenarios, not just for scenarios involving the trace fields.  Hence this is a more general issue and does not warrant specialized handling for trace fields.
> 
> 
> 
> Correct.  And that's why it is a mistake to impose restrictions, for an environment that has already been declared to be Unicode-pure.
> 
> 
> 
> Indeed it is... for the hybrid model.  The working group tried to deal with that complexity.  The decision some months ago was, instead, to make things simple. The only challenge now is to make sure that things are /kept/ simple.
> 
> d/
> -- 
> 
>  Dave Crocker

---

From: "Jiankang Yao" <[email-masked]>
Date: Tue, 28 Jun 2011 23:09:14 +0800

----- Original Message ----- 
From: "Joseph Yee" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, June 28, 2011 9:38 PM
Subject: Re: [EAI] RFC5336bis and trace fields

>
>
> So a consensus call will be made very soon on trace fields, and the WG may
> assess all other fields in similar situation, and determine (another
> consensus call) whether to adopt Unicode-pure model or not.
>

I hope that the a consensus call  can be sent ASAP and the result can be
sent to the list a few days before the draft deadline (11 July).
RFC5336bis is waiting for the consensus result.

thanks a lot.

Jiankang Yao

---

From: Joseph Yee <[email-masked]>
Date: Wed, 29 Jun 2011 15:05:55 -0400

All,

I will announce the consensus result at July 3 (00:01am US Eastern) regarding trace fields.  Please continue all discussions regarding trace fields with the current conversation thread.  I am monitoring all time.

If anyone's not satisfy of the July 3 result, please send to WG mailing list with your reasons.  During this phase I'm looking for issues not addressed by WG previously.  If necessary, a revised final consensus will announce at July 5, 23:59 US Eastern.

I had already captured views and opinions from several participants already.  If you have not express your view yet, please do so as soon as possible and before the July 3 deadline.  If you have any concerns or questions, please raise them here too, constant frequent discussions help.

Again, the focus is whether trace field values support Unicode beyond ASCII, or ASCII-only.  All other fields will be discussed in separate thread.

Best Regards,

On 2011-06-28, at 11:09 AM, Jiankang Yao wrote:

> 
> ----- Original Message ----- From: "Joseph Yee" <[email-masked]>
> To: <[email-masked]>
> Cc: <[email-masked]>

---

From: John C Klensin <[email-masked]>
Date: Wed, 29 Jun 2011 16:06:22 -0400

--On Wednesday, June 29, 2011 15:05 -0400 Joseph Yee
<[email-masked]> wrote:

>...
> I had already captured views and opinions from several
> participants already.  If you have not express your view yet,
> please do so as soon as possible and before the July 3
> deadline.  If you have any concerns or questions, please raise
> them here too, constant frequent discussions help.
> 
> Again, the focus is whether trace field values support Unicode
> beyond ASCII, or ASCII-only.  All other fields will be
> discussed in separate thread.

Joseph,

I think there are actually two separate questions:

	(1) Domain names in the "from" clause of "Received:".
	
	(2) Everything else.

I won't reprise my earlier note but, for the first question,
there are several cases in which an EAI-aware system may be
called upon to send, forward, or return a message that has no
[other] internal requirements for EAI capabilities to an
EAI-unaware system.  For that situation, we could specify "ASCII
always" (i.e., use A-labels if the domain is an IDN regardless
of the EAI status of the message or destination), "ASCII if
needed" (use A-labels iff the destination if not EAI-capable),
or "U-labels if IDNs are involved (which, given that scenario,
involves either sending a non-conforming message or trashing or
droping it).

The situation for all other trace field cases is different:
there will not be a requirement for representation of characters
outside the ASCII repertoire unless the incoming message
requires EAI capabilities.

      john

---

From: Joseph Yee <[email-masked]>
Date: Thu, 30 Jun 2011 03:14:32 -0400

> 	(1) Domain names in the "from" clause of "Received:".

Thanks, and yes, these 2 are different.  I didn't make the question specific enough in first mail.

Joseph

---

From: "John Levine" <[email-masked]>
Date: 30 Jun 2011 13:47:23 -0000

>> For that situation, we could specify "ASCII always" (i.e., use
>> A-labels if the domain is an IDN regardless of the EAI status of
>> the message or destination), "ASCII if needed" (use A-labels iff
>> the destination if not EAI-capable), or "U-labels if IDNs are
>> involved (which, given that scenario, involves either sending a
>> non-conforming message or trashing or droping it).

That still strikes me as a quality of implementation issue.  For best
backward compatibility, you should keep messages in the legacy stream
whenever you can, but using A-labels in the Received: headers is just
part of that.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Jun 2011 12:33:02 -0400

--On Thursday, June 30, 2011 13:47 +0000 John Levine
<[email-masked]> wrote:

>>> For that situation, we could specify "ASCII always" (i.e.,
>>> use A-labels if the domain is an IDN regardless of the EAI
>>> status of the message or destination), "ASCII if needed"
>>> (use A-labels iff the destination if not EAI-capable), or
>>> "U-labels if IDNs are involved (which, given that scenario,
>>> involves either sending a non-conforming message or trashing
>>> or droping it).
> 
> That still strikes me as a quality of implementation issue.
> For best backward compatibility, you should keep messages in
> the legacy stream whenever you can, but using A-labels in the
> Received: headers is just part of that.

If we say "put something useful into the Received... from"
subfield, that might make it a quality of implementation issue
(Jiankang probably needs significant advice on how to write that
text).  If we say "you SHOULD use UTF-8 forms, rather than any
other Unicode encoding form, if possible", then we take on some
obligation to at least explain this issue and when it is not
possible or we turn using the A-labels into non-conformant
behavior, not a quality of implementation issue.  

There is also an issue of conformance to 5321.  We may decide we
don't care --although that would be a somewhat odd attitude to
take in a standards-track document, IMO-- but having a client
send non-ASCII header material to a server that does not
advertise EAI is pretty clearly a violation of 5321, not just a
quality of implementation issue.

IMO, this is really pretty easy if we accept the fact that an
EAI MTA must be able to simulate a MTA that is strictly
conforming to 5321 (I believe that is a general requirement of
the SMTP Extension mechanism; YMMD).   Then:

	An EAI-capable client MUST either be able to send a
	5321-conformant message to a non-EAI-capable server or
	MUST reject or return the message.  Only the second is
	plausible in the absence of downgrading facilities if
	the message requires EAI capabilities.
	
	An EAI-capable client SHOULD send messages what require
	EAI functionality in UTF-8 form to servers that do
	support EAI functionality.

Now, if we can agree on those two principles, I can see text
that permits either A-labels or U-labels in "Received...from"
subfields and the choice being a quality of implementation issue
as long as the principles aren't violated and we don't back
ourselves into a corner in which it is necessary to bounce
messages that started out 5321-compliant.

   john

---

From: "John R Levine" <[email-masked]>
Date: 30 Jun 2011 12:53:39 -0400

> the SMTP Extension mechanism; YMMD).

Right, unless it's in a closed all-EAI network or something else exotic. 
If a message comes using the normal SMTP commands, no EAI parameter on the 
MAIL FROM, it's a legacy message and it would be nice if the MTA kept it 
as a legacy message if it relays it to a legacy address.

That's why I keep talking about the legacy stream and the EAI stream. 
For mail in the legacy stream, the MTA has to be a legacy MTA and put 
legacy 5321 headers on the mail.  The quality of implementation issue (for 
me at least) is how hard the MTA works to keep messages in the legacy 
stream, e.g., if a message arrives in the EAI stream, but its contents and 
envelope happen not to need any EAI features, it would be nice if the MTA 
treated it as a legacy message.  But I wouldn't want to mandate that, 
since that requires deep message inspection which is more than some MTAs 
may want to do.

I think we agree what we want here, although perhaps we differ about how 
best to describe it.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Jun 2011 13:10:55 -0400

--On Thursday, June 30, 2011 12:53 -0400 John R Levine
<[email-masked]> wrote:

> Right, unless it's in a closed all-EAI network or something
> else exotic. If a message comes using the normal SMTP
> commands, no EAI parameter on the MAIL FROM, it's a legacy
> message and it would be nice if the MTA kept it as a legacy
> message if it relays it to a legacy address.
> 
> That's why I keep talking about the legacy stream and the EAI
> stream. For mail in the legacy stream, the MTA has to be a
> legacy MTA and put legacy 5321 headers on the mail.  The
> quality of implementation issue (for me at least) is how hard
> the MTA works to keep messages in the legacy stream, e.g., if
> a message arrives in the EAI stream, but its contents and
> envelope happen not to need any EAI features, it would be nice
> if the MTA treated it as a legacy message.  But I wouldn't
> want to mandate that, since that requires deep message
> inspection which is more than some MTAs may want to do.
> 
> I think we agree what we want here, although perhaps we differ
> about how best to describe it.

I have thought so, but wasn't sure until now.  

And, fwiw, building that description around the notion that EAI
MTAs need to handle incoming messages that don't request EAI
features as legacy (5321-conforming) messages works for me...
I'm fairly agnostic about whether than is described as "legacy
stream and EAI stream" or not.   I get nervous when people say
things that sound like "EAI-capable MTAs always need to behave
as if UTF-8-everything is the norm, independent of what they
receive".  I think that, in your terms, the latter is "one
stream only for EAI-conformant systems" and leads to completely
unnecessary message rejections.

    john

---

From: [email-masked]
Date: Thu, 30 Jun 2011 10:04:21 -0700 (PDT)

> as a legacy message if it relays it to a legacy address.

Agreed. As long as the addresses in the envelope remain in the legacy space
there should be no reason to promote the message to EAI in an MTA. I don't
think an outright ban is a good idea though - there are a lot of wildly varying
use-cases out there.

> That's why I keep talking about the legacy stream and the EAI stream.

It's a useful way to think about it.

> stream, e.g.,

Exactly.

> if a message arrives in the EAI stream, but its contents and

This is both a design and a configuration issue. We have customers running MTAs
with CPU to burn who have no problem performing deep inspection of every
message and even insist on it. But there are also sites processing huge amounts
of mail (thousands of messages a second) where the overhead of deep inspection
is flatly unacceptable. So we're going to have to have some knobs to turn for
this.

				Ned