Subject: [EAI] UTF-8 in Message-IDs

From: Frank Ellermann <[email-masked]>
Date: Sat, 13 Aug 2011 22:23:22 +0200

Hi,

looking for something else (= state of the art wrt EAI
and SPF) yesterday I stumbled over the "81 IETF EAI
meeting minutes" in the mailing list archive, with a
note about "UTF-8 in Message-IDs".

That was certainly (bad) news from my POV, and whining
about it off list I got the tip that waiting for an
IETF Last Call is a bad plan when this could be also
addressed before WG Last Call.

>From my POV the "globally unique forever Message-ID"
in mail and news is in essence an early form of what
is now a GUID/UUID in other protocols.  Maybe a bit
more so, in theory there could be UUID collisions,
and that's not possible for Message-IDs (in theory).

This theory is relevant for NetNews, gateways from or
to NetNews, mail and news threading, APOP, and likely
other use cases I'm not aware of.

Nobody would say replace hex. by UTF-8 in GUID/UUIDs,
therefore I fail to see the point of trying something
in this direction in Message-IDs.  Gateway operators
stating that UTF-8 in Message-IDs would be no problem
in practise, or that this would be only a minor point
when they adopt EAI, might convince me that I'm only
paranoid, but I doubt it.

And the "EAI experiment" phase did not test this plan,
there's no evidence I'm aware of that UTF-8 in Message-
IDs is harmless.  Did you really check all RFCs using
or mentioning Message-IDs for potential side-effects
of introducing UTF-8 in the "local", "unique", "left"
etc. part of a Message-ID?  E.g., does it fly with the
Message-ID concept in APOP and CRAM-MD5, or do you
intend something in the direction of "updates 2195"
to make it work?

-Frank

---

From: [email-masked]
Date: Sat, 13 Aug 2011 15:08:48 -0700 (PDT)

> IDs is harmless.

I'm sure it is not, just as utf-8 in addresses is far from harmless and is
going to require all sorts of infrastructure changes.

But I fail to see how this is in any way relevant. We're defining a new message
format here that *cannot* be downgraded to the old format and retain all
semantics. This is true irrespective of how message-ids are handled. As such,
utf-8 in message-ids is a small additional cost.

It's also a cost that I doubt can be avoided no matter what the specification
says. Once utf-8 is in domains in messages it's going to leak into message-ids
no matter what the specifications say.

> Did you really check all RFCs using
> etc. part of a Message-ID?

Nope. Given the approach the WG is taking to this problem, such an analysis
would be a pointless waste of time.

Maybe, if we were taking the MIME approach, such a check would make sense.
But we're not doing that.

> E.g., does it fly with the

Why would that be coupled to this new message format?

				Ned

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Aug 2011 10:17:45 +0100

> utf-8 in message-ids is a small additional cost.

Yes, but you have to consider all the other protocols using mail-like  
formats and their use of the Message-ID. For exmaple, if EAI were to be  
carried over into Netnews (quite a likely development) it would NOT be  
regarded as a "new message format" since the transport paths for Netnews  
are already 8-bit clean and it would simply be necessary for those who  
wish to take advantage of the new facilities to ensure that their user  
agents were suitably upgraded. There will never be a need for  
"downgrading" except at gateways back into the email system.

But within Netnews the Message-ID plays a crucial role, so it is  
reasonable to ask whether UTF-8 in it would cause problems. According to  
the current standards it would not be allowed, of course, but in practice  
the transport paths might or might not barf. So the question has to be  
asked (and I do not know the answer off the top of my head).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: [email-masked]
Date: Mon, 15 Aug 2011 08:19:44 -0700 (PDT)

> formats and their use of the Message-ID.

These protocols don't make use of email addresses? 

> For exmaple, if EAI were to be

And all netnews applications properly handle the construction and display of
utf-8 addresses in message headers? Just as one example, netnews already
suports the proper form of Unicode normalization on input for this. Right?

An 8-bit clean transport is nowhere near sufficient to accomodate the other
changes to the format we're making here. Message ids are actually the least of
it since they are machine generated.

> asked (and I do not know the answer off the top of my head).

I'm sorry, but the entire approach of this WG is to define a format that is NOT
downgradeable to previous formats without substantial information loss. Given
that's the context here, how can the question of "how well does this downgrade
in this particular case" possibly be relevant?

Look, I'll be the first to state that the approach being taken here is risky.
In fact I would never have chosen this approach myself. But that doesn't
change the fact that there's a consensus to proceed in this fashion.

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 15 Aug 2011 08:46:42 -0700

> in this particular case" possibly be relevant?

+1

> change the fact that there's a consensus to proceed in this fashion.

+1

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Mon, 15 Aug 2011 12:47:01 -0400

Ned, Charles, and others,

(strictly personal opinion)

Let me give a different perspective on this, in the hope of
drawing us together on the conclusions if not on the reasoning.

I'm not (personally) wildly comfortable with unrestricted
Message-IDs.  I don't find arguments based on "we chose this
general model and therefore Message-IDs can't/shouldn't be
restricted" persuasive, precisely because we've said "protocol
identifier" before and, IMO, our ability to say that is critical
to getting any i18n design process right.  In particular, I note
that we have not permitted non-ASCII characters in new/optional
header field names by modifying RFC5322's <optional-field> to
permit <ftext> to be a much less restricted range of characters.
I think that it would be a mistake to make that extension
(indeed, I personally think that we'd be much better off if
<ftext> were restricted to ASCII letters, digits, and hyphen
rather than excluding only non-printable characters and colon).
But, if one argues that Message-ID has to be permitted to be
non-ASCII because we've adopted a "UTF-8 nearly everywhere"
model, the case for excluding field names has not been made (or
even really discussed)... and a field name of, e.g.,
   דרעקק: ...
has a certain appeal (as well as some nightmare aspects).

However, other arguments are much more persuasive to me, with
the two critical ones being:

	(1) An MUA, or MTA pretending to be an MUA for message
	analysis purposes, that can handle non-ASCII UTF-8 in
	addresses or other header fields will need to go to very
	small marginal extra effort to handle a Message-ID with
	non-ASCII content.  Indeed, most sensible
	implementations would see a restriction that Message-IDs
	be ASCII as a requirement for added code to support an
	unnecessary requirement.    We have not had good luck
	with restrictions like that.
	
	(2) Given what we know about how Message-IDs are
	constructed in practice, we can expect non-ASCII forms
	to appear no matter what the spec says.  There is no
	point in writing a spec that we know will be ignored.
	For that reason, I think those arguing to restrict
	Message-IDs to ASCII need to come up with a clear,
	concise, and powerful explanation of why non-ASCII
	Message-IDs would be harmful -- an explanation that can
	be included in our documentation to persuade
	implementers that the requirement is really worth
	enforcing.  

To me, the "unnecessary requirement" and "will be ignored in
practice" positions strongly suggest that the burden of
demonstrating likely harm -- and producing that clear, concise,
and powerful explanation-- falls on those who would like to keep
Message-IDs ASCII-only.   If one believes in the "we adopted a
different model" story, it just strengthens that position.

In all of these discussions, I haven't seen that explanation
emerge.

I can think of one set of scenarios that might provide a
foundation for it, even though I'm not convinced that they are
strong enough to restrict Message-IDs.  The scenario goes
somewhat like this (and parallels some of the discussion about
message changes in Submission Servers).  

	(1) Suppose I'm a user who receives a message.  
	
	(2) Suppose I have an EAI-capable delivery MTA, message
	store, MUA, and whatever connects the MUA to the message
	store (i.e., none of the edge cases that are a headache
	for POP and IMAP apply).  Suppose the message I receive
	contains a non-ASCII Message-ID and very little other
	non-ASCII header field content.   As one example among
	many, suppose that only the contents of the Subject
	header field are non-ASCII (in addition to the
	Message-ID).  

	(3) For simplification, suppose that my outgoing mail
	server's primary domain name does not contain any IDN
	components and Message-IDs I generate are all-ASCII.
	
	(4) Now I decide to reply to the originator (note that
	neither my address nor the address that appears in the
	"From:" header are non-ASCII).  I also add a recipient
	to the reply whose system is not EAI-capable (I know
	that from out of band information -- how is not
	important).   So, in addition to adding the additional
	address, I edit the Subject line so it is ASCII-only.
	(Remember this is a reply and I'm the user, so both of
	those changes are perfectly legitimate according to
	everything we've specified).

Now there is a problem.  An automatically-generated In-reply-to
header field, generated as recommended in 5322 (unchanged in
5335bis) is going to contain that non-ASCII Message-ID.  That
requires EAI handling (i.e., the UTF8SMTPbis extension) for the
outgoing message even though the In-reply-to field is the only
header field containing non-ASCII characters.  Given the
scenario above, that makes the message non-deliverable to my
intended new recipient.

That is bad news... and bad news that could have been completely
prevented by requiring that Message-IDs be restricted to ASCII.

The question, IMO, is whether cases of that type are likely to
be prevalent enough to justify a restriction.  I contend that
they are more than sufficient to justify some explanatory text
and a recommendation that there are cases in which having a
sending system be conservative enough to generate only all-ASCII
Message-IDs will provide some marginal robustness against
delivery rejections.  But I don't think it is nearly persuasive
enough to justify an outright ban on non-ASCII Message-IDs.

YMMD.

     john

---

From: "John Levine" <[email-masked]>
Date: 15 Aug 2011 17:52:14 -0000

>intended new recipient. ...

If I may try to peek inside the can while trying to keep the worms
from escaping, I think many of us have an informal mental model of a
poorly specified downgrade process at submission time, since that's
the point at which it's more likely that software will have access to
address books with alternate addresses and so forth.  With that in
mind, it would be nice to have an ASCII message-id to make the poorly
specified process easier.  But I don't think we should go there,
except perhaps in an entirely separate document that makes it as clear
as possible that it's not part of the normal EAI transmission process.

I find the most compelling argument the one that there'll be UTF-8 in
message-ID's no matter what we say, so let's not specify something
that we expect implementors to get wrong.

R's,
John

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Aug 2011 21:13:38 +0200

Sorry, apparently I forgot the CC: to the list, resent:

On 15 August 2011 17:19,  Ned Freed wrote:

> I'll be the first to state that the approach being taken here is risky.

Maybe my original idea to start the whining in an IETF Last Call was not
completely wrong, because I certainly missed whatever resulted in this
consensus, and just *assumed* that EAI just moves on to "standards track",
because the Internet at large obviously survived the EAI "experiment".

In another mail you wrote about checking all Message-IDs in other RFCs:

| Maybe, if we were taking the MIME approach, such a check would make
| sense. But we're not doing that.

That is rather disturbing for me, "not taking the MIME approach" is not
something I'd consider.  Apart from breaking the concept of Message-IDs
without any immediately visible reasons after more than three decades,
what else does it encompass?  Whatever it might be, shouldn't this be
stated in 4952bis, preferably in upper case?

>From my POV the A in IDNA means Applications, and the first I in IRI is
not an U, no matter what XML or HTML5 say or wish.  E.g., if email users
try to use a mailto:acsii@example?in-reply-to=mid link in a mailing list
archive this is supposed to result in a message/rfc822 with an ordinary
In-Reply-To header field, unless this user intentionally uses some EAI
features.

Message-IDs don't need to be "pretty" or "human readable" or something,
their main point is to work as designed under almost all circumstances.

-Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Aug 2011 21:29:36 +0200

> that we expect implementors to get wrong.

Why do you all say that implementors will get this simple point wrong,
while you expect them to get far more convoluted syntax details right?

I don't see anything "compelling" in that line of reasoning, otherwise
we could also say that using 8bit in DNS queries works, and therefore
it's okay to use UTF-8 in host names (example).  Bugs in new apps can
be fixed, but limitations in old apps will never go away (until nobody
uses such old apps anymore).

-Frank

---

From: [email-masked]
Date: Mon, 15 Aug 2011 10:32:30 -0700 (PDT)

> restricted" persuasive,

That's good, because AFAIK nobody is making that argument. The argument we're
making is threefold:

(1) Because of structural issues in the RFC 5322 ABNF, it's much easier
    to make some changes to low-level rules than to try and add utf-8 at
    a higher leve. But one consequence of the low-level approach is that
    message-ids are also affected. This is fixable with a handful of
    additional rules that are still much cleaner and simpler than the
    higher level approach, but it's still something of a wart.

(2) An overwhemlming amount of deployed software generates message-ids by
    blindly appending @domain to some random string. A smaller but significant
    number also place some portion of the local part of the user's address
    on the left hand side. When domains and local parts start having
    utf-8 in them, the chances of all these code paths noticing and doing the
    right thing (which I suppose would be to encode it, but the encoding
    needs to be consistent or collisions are possible) are very, very low.
    So the issue of utf-8 in message-ids has to be faced and dealt with
    irrespective of what the standards say. 

(3) Most of the software that will be affected by utf-8 in message-ids is also
    going to be affected by utf-8 in addresses. (Actually, in most cases
    addresses present a much more difficult problem.) As long as you're prying
    the lid off, might as well deal with the issue a little more generally.

Given all this, the argument is we might as well bite the bullet and allow
it. Frankly, given how boundary markers are generated by some clients, I
think we're going to have enough fun dealing with that leakage case.

> precisely because we've said "protocol

That's because none of the arguments regarding message-ids apply to <ftext>.
<ftext> isn't affected by the ABNF changes, there's essentially no leakage from
domains or addresses into field names, and common code paths between field name
processing and address or message-id processing would seem to be ... unlikely.

> has a certain appeal (as well as some nightmare aspects).

See above.

> 	with restrictions like that.

Another good point.

> 	implementers that the requirement is really worth

This is the second point above, more or less.

> Now there is a problem.

No, not really. Again, this work is premised on semantic-preserving downgrades
not being necessary. You are considering an odd corner case where were it not
for the utf-8 in the message-id, such a downgrade would be possible.

So what? If you really care about being able to send *any* subset of EAI
messages to non-EAI recipients without semantic loss, then this entire effort
is misdesigned: We should be using some 7-bit encoding scheme and not utf-8.

> An automatically-generated In-reply-to

> prevented by requiring that Message-IDs be restricted to ASCII.

Again, so what? We could send an even larger portion of those EAI messages to
the non-EAI recipient if we were to impose similar restrictions on addresses,
which we could easily do by using some other encoding of Unicode. And so on.
This logic puts you on a slippery slope that unavoidably leads to most of the
design decisions this group has made being wrong.

> enough to justify an outright ban on non-ASCII Message-IDs.

Well, while I don't find your example here persuasive, I also don't have a
problem with saying utf-8 SHOULD NOT be used in message-ids. My guess is this
will have no more effect on utf-8 showing up there than an outright ban will,
but it's defensible recommendation.

				Ned

---

From: [email-masked]
Date: Mon, 15 Aug 2011 13:37:42 -0700 (PDT)

> because the Internet at large obviously survived the EAI "experiment".

First of all, I don't think the EAI experiment showed us diddley-squat about
how the Internet "survived" EAI - it was much too small for that.

What the EAI experiment did show was that the original EAI approach, which
provided for message downgrade by including the information needed for
downgrade in each message, was unworkable. It's unworkable because people are
not going to spend the time to provide that information. And without that
information useful downgrading of message cannot occur.

In hindsight, this should have been obvious. Providing alternate addresses is
work that the originator needs to do but the direct benefits of that work only
accrue to the non-EAI recipient. As such, even assuming there weren't any UI
issues (and I'll bet there were), it should not be surprising that people were
unwilling/unable to do this.

Given that result, the options for moving forward boil down to either:

(1) Stop worrying about semantic-preserving downgrades in order to preserve
    the no-new-encodings goal the group has had from the start.

(2) Abandon the no-new-encodings goal. (In other words, the approach MIME
    took.)

I would have been a lot more comfortable with (2) personally, but the WG
consensus went the other way.

> something I'd consider.

Then your objection is to the WG's entire approach, and should have been
registered months or years ago when the group was rechartered and this 
direction was chosen.

And while I'm sympathetic to your position, I'm a lot more sympathetic to
not endless revisiting decisions we've already made.

In other words: You're much too late, and this is out of order.

> Apart from breaking the concept of Message-IDs

4952bis should already state that downgrades to non-EAI formats are not a
consideration, period. I don't mean to be overly dismissive, but against that
backdrop message-ids seem like extremely small beer.

> their main point is to work as designed under almost all circumstances.

And once again I come back to: What about addresses? There's no downgrade
path for them either and they have a lot more immediate importance.

				Ned

---

From: [email-masked]
Date: Mon, 15 Aug 2011 14:03:49 -0700 (PDT)

> while you expect them to get far more convoluted syntax details right?

Because getting those convoluted syntax details right and getting the handling
of message-id wrong are how email code works today. It's all sunk costs, in
other words.

The addition of utf-8 to various syntactic elements isn't convoluted. In fact,
having looked at the impact on our code base, it's going to be pretty easy to
add.

The additional negotiation and failure handling code are more difficult, but it
all follows well established design patterns. When you get right down to it the
impact of EAI on MTAs isn't going to be that large. (MUAs are another story,
but because of addresses and other elements, not message-ids.)

> it's okay to use UTF-8 in host names (example).

Except that it doesn't - the DNS protocols may support 8-bit but there are
specific restrictions on host names that some deployed software honors. And if
this wasn't the case you'd probably hear exactly this argument being made.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 15 Aug 2011 17:22:37 -0400

Ned,

Let me try again with a note short enough that my conclusion and
intentions are not obscured (I think that happened the last
time; my apologies).

I think several of us have reasoned to the conclusion that, on
balance, Message-IDs should not be restricted to ASCII (in the
formal syntax or more generally).  Some of us find some of those
arguments more persuasive than others; others of us would choose
a different mix, but the conclusion is the same.

Given the multiple reasons for that conclusion; the apparent
consensus about it in email discussions before, during, and
after IETF 80; and the fairly general impression that an ASCII
restriction would be generally ignored because of the way
Message-IDs are often contructed, I believe that anyone who
continues to believe that non-ASCII Message-IDs should be
prohibited needs to persuasively demonstrate to the WG that they
would cause significant harm.

That demonstration has not appeared.  We can create edge cases
that show that messages with Message-IDs with non-ASCII content
are slightly less robust that Message-IDs that are ASCII-only,
but far more likely cases can be shown to demonstrate that
messages with only ASCII addresses are more robust than messages
that contain non-ASCII text in addresses, etc.  To go down that
path is to argue that any message with UTF-8 strings in _any_
header field is less robust than a corresponding message with
only ASCII in those fields.  While that is undoubtedly true, the
WG (and the IETF by issuing the WG a charter) have decided that
the advantages of having internationalized addressing and header
fields far exceed the disadvantages of that drop in robustness.
Moreover, the marginal drop due to non-ASCII Message-IDs alone
(once the risks of any non-ASCII material are accepted) appears
to be close to trivial... making a persuasive demonstration of
harm even less likely.

I urge Joseph to review the history of this discussion and then
close it out.

      john

p.s. As far as "SHOULD keep Message-IDs in ASCII" is concerned,
I could live with it but would actually oppose it.  The reasons
are implicit in the above, in your recent notes, and in other
recent discussions: (i) restrictions that we unlikely to be
obeyed are just bad for standards and (ii) to whatever extent
Message-IDs (including values in In-Reply-To and other fields)
are ever examined by humans, forcing id-right to use A-labels
for the most obvious and common cases is just inconsistent with
multiple design goals.  So I would prefer a bit of
implementation advice that points the issue out, not a
conformance statement.  YMMD but, if you agree even slightly,
let me try to draft a paragraph that we can then figure out
where to put.

--On Monday, August 15, 2011 10:32 -0700 Ned Freed
<[email-masked]> wrote:

>...
> That's good, because AFAIK nobody is making that argument. The
> argument we're making is threefold:
> 
> (1) Because of structural issues in the RFC 5322 ABNF, it's
> much easier     to make some changes to low-level rules than
> to try and add utf-8 at     a higher leve. But one consequence
>...

---

From: [email-masked]
Date: Mon, 15 Aug 2011 19:26:42 -0700 (PDT)

> intentions are not obscured (I think that happened the last

No problem!

> a different mix, but the conclusion is the same.

I'm actually more concerned about this dragging out than which way it gets
decided.

> would cause significant harm.

My comment here is that harm has to be associated with operation inside the EAI
"zone". Downgrading issues don't count.

> to be close to trivial... making a persuasive demonstration of

OK, now I see what you were getting at. Sorry for not seeing it the first
time.

+1

> let me try to draft a paragraph that we can then figure out

I certainly can do that.

				Ned

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 17 Aug 2011 11:32:42 +0100

> These protocols don't make use of email addresses?

Some do some don't. In Netnews they are not used except to reply to the  
author or an article by email.

> suports the proper form of Unicode normalization on input for this.  

No. If you want to participate in newsgroups where strange characters are  
likely to be used, then you need to ensure your MUA has the necessary  
capability. That's your problem.
> it since they are machine generated.

It is sufficient for the existing Netnews transport to deliver articles  
with utf-8 headers to participants; its their problem to get them  
displayed. The only two headers that are necessary in the transport layer  
are the Newsgroups header and the Message-ID. It has already been  
demonstrated that a Newsgroups header with utf-8 will be delivered  
correctly, but a standard would be needed to ensure proper normalization.  
The Message-ID is crucial to the correct operation of the Netnews  
transport, and the effect of using utf-8 in it has not been investigated  
(but should be). It is expected that some ancient transport agents might  
not like it but even that might not be a show stopper, since Netnews is  
very good at navigating around such obstacles.

> downgradeable to previous formats without substantial information loss.

Utter rubbish!

We are not designing a protocol that goes out of its way to make  
downgrading impossible. We are designing a protocol in which downgrading  
is not REQUIRED to be possible, but we should still avoid putting  
unnecessary obstacles in its way.

And, as I have said, in Netnews downgrading is not an issue (except at  
gateways out of Netnews).

Returning now to email, the Message-ID is only important when used in the  
References and In-Reply-To headers, where is is generally used by MUAs for  
threading purposes. So the worst that can happen if Message-IDs get  
garbled in transport is that threading fails to work.

My suggestion is that we retain a requirement that Message-ID remain in  
pure ASCII but add a remark that this position is likely to be reviewed in  
a future version of the standard.

If then, as is likely, they start to appear, then we will see how they  
cope and decide how best to regulate them in a future standard. In the  
meantime, they will do little harm. But when that future standard comes,  
it have to pay attention to exactly what sorts of things can appear in  
them (as indeed 5322 places quite a lot of restrictions) and, in addition,  
it will have to pay careful attention to normalization if threading it to  
continue to work.

And I don't think we want to go into that sort of detail at this present  
stage of our work.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 17 Aug 2011 15:59:47 +0200

On 17 August 2011 12:32, Charles Lindsey wrote:

> threading purposes. So the worst that can happen if Message-IDs get garbled
> in transport is that threading fails to work.

That includes the cute mailto:list@example?In-Rreply-To=unique@example use
to get correct threading in replies to messages in list archives.  Covered
in RFC 6068 section 6.1 and AFAICT invented by "pipermail" many years ago.

Likewise RFC 5538 section 6 used a crystal ball to claim that whatever EAI
will do, it won't touch Message-IDs for the spectacle of a news URI turned
into a news IRI without anybody knowing what happened, and what consequences
it might have on NNTP servers.  RFC 3977 section 10.3 "outstanding issues"
discusses newsgroup names in its i18n considerations, and I think EAI uses
some ABNF from this RFC.  Its Message-ID concept could live without an "@",
but otherwise <A-NOTGT> is "ASCII excluding '>'".  RFC 5536 section 4 does
not mention that "there be dragons" wrt "prettier" Message-IDs, because it
is easier to change the syntax at a low level in EAI and see what happens.

-Frank

---

From: [email-masked]
Date: Wed, 17 Aug 2011 06:54:56 -0700 (PDT)

> unnecessary obstacles in its way.

Funny, I don't see anything about that in the charter or any of the
current architecture descriptions.

> gateways out of Netnews).

Nor is support of netnews mentioned anywhere.

> a future version of the standard.

It is completely unrealistic to believe such a requirement will be honored,
for reasons already given.

> And I don't think we want to go into that sort of detail at this present

Well, your position is noted.

				Ned

---

From: Chris Newman <[email-masked]>
Date: Wed, 17 Aug 2011 15:36:04 -0700

Thank you for raising this use case.

I believe a downgraded EAI message constitutes a "subsequent revision" as 
described in RFC 5322 section 3.6.4. Specifically, it is likely that 
attempts to reply to the author, sender and recipients of a downgraded EAI 
message will fail so it is a fundamentally different message. And thus it 
is important that it gets a new message id.

So perhaps we should change the advice in RFC5335bis to say:

  The Message-ID SHOULD include at least one UTF-8 character.

By including a UTF-8 character, any gateway to a non-EAI system will have 
to replace the Message-ID with a new one, thus correctly indicating that 
the resulting downgraded message is a subsequent revision. Systems wishing 
to identify the original EAI message id for a damaged downgraded message 
can look at the Downgraded-Message-ID header.

		- Chris

--On August 13, 2011 22:23:22 +0200 Frank Ellermann 
<[email-masked]> wrote:

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 17 Aug 2011 19:16:55 -0400

--On Wednesday, August 17, 2011 15:36 -0700 Chris Newman
<[email-masked]> wrote:

> By including a UTF-8 character, any gateway to a non-EAI
> system will have to replace the Message-ID with a new one,
> thus correctly indicating that the resulting downgraded
> message is a subsequent revision. Systems wishing to identify
> the original EAI message id for a damaged downgraded message

Chris, while I agree with your analysis of "subsequent
revision", I don't get from their to your conclusion for two
reasons:

(1) Other than in the POP/IMAP downgrade path, there is no such
thing as a Downgraded-Message-ID header.   One can imagine a
number of scenarios that would involve a need to send a message
back out in ASCII-only form that do not involve that header or
any downgrading we are going to specify.  Note especially that
the user generating a reply may have sufficient out of band
information available to supply deliverable addresses for some
or all of the author, sender and recipients of the message as
appropriate.

(2) Per one of my examples of some days ago, if we allow UTF-8
in Message-IDs at all, it is quite possible to have a message
that requires UTF8SMTPbis extension handling but that contains
no non-ASCII data in either header fields or addresses other
than that UTF-8 Message-ID.  Indeed, one interpretation of the
statement you suggest above virtually guarantees a lot of
messages of that type (probably it can be rewritten to reduce
the number, but that doesn't change things much).  Now we have
an interesting problem: the message is reply-able (and, more
important, forward-able without creating a message/global body
part) and new recipients with all-ASCII addresses can be added.
Certainly in the forwarding case, there is no "subsequent
revision".   It can be handled strictly as a legacy message
except for that pesky Message-ID.  And, again, except for
replacing or otherwise dealing with the Message-ID, there is no
need to downgrade anything.

As I've said before, I don't think those scenarios (including
Frank's) are adequate to make a case for a protocol restriction
to ASCII.  I think they are sufficient to justify a
recommendation that systems sending mail into environments they
consider fragile should confine themselves to ASCII message
ID's.  Requiring non-ASCII characters in Message-IDs would not
only defeat that recommendation, it would run counter to the
core of the "if we tell them ASCII-only, they will ignore us"
observation that Ned, I, and others have been making.  

To review that observation (at least my version), MUAs have ways
to construct Message-IDs, usually by cramming some local
information and a domain name together.  For all sorts of
reasons, they are likely to continue doing whatever they are
doing, using a domain name with U-labels in it if they consider
such a domain name primary.  If we ask them to do anything else,
it is going to create a requirement for extra code and extra
work -- whether that be converting the domain name to use
U-labels or to artificially add a non-ASCII character because
some other header field contains at least one non-ASCII
character.  We can provide them very little motivation for doing
that other than the WG's exploration of transition cases and
edge cases.  Experience indicates that extra work, especially
extra work that creates additional code paths, with no even
slightly compelling motivation, translates into an unimplemented
requirement.

(personal opinion only)
    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 18 Aug 2011 16:34:12 +0100

On Wed, 17 Aug 2011 23:36:04 +0100, Chris Newman <[email-masked]>  
wrote:

> I believe a downgraded EAI message constitutes a "subsequent revision"  
> as described in RFC 5322 section 3.6.4. Specifically, it is likely that  
> attempts to reply to the author, sender and recipients of a downgraded  
> EAI message will fail so it is a fundamentally different message. And  
> thus it is important that it gets a new message id.
>

Absolutely NO-WAY!

If a message is evidently the SAME message, even though different  
recipients may encounter it in slightly different forms (possibly  
dowgreaded, possibly garbled, but still with the majority of it readable)  
then it MUST have the same Message-ID.

So if, by some complex rerouting, encapsulating, forwarding, someone  
manages to acquire copies of it in both forms, it will be clear it is the  
same (e.g. it will not need to be replied to twice, and in any threaded  
list both will appear together with the same parents and same children).

An essential property of <msg-id>s, which we managed to achieve during the  
discussions that lead to RFC 5322, is that two <msg-id>s can be compared  
for equality by a simnple byte-for-byte comparison. Threading algorithms  
need to rely on this, and within Netnews the same property is essential  
for the transport mechanism to sork (even if the transport proceeds by  
gatewaying into email and then nack into Netnews).

Therefore, IF we are going to permit utf-8 in <msg-id>s, then it is  
essential to include sufficient mandatory requirements on Normalization,  
and to scrutinize the set of allowed characters for troublesome cases. If  
this WG is prepared to put in the work to do then, then well and good, and  
I will support it.

But otherwise, please count me as OPPOSED.

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 16 Aug 2011 07:47:50 -0700

> having looked at the impact on our code base, it's going to be pretty easy to

That mirrors the effect I saw when making generic changes to the ABNF, rather 
than by treating UTF-8 as an extended set of special cases around the syntax.

> this wasn't the case you'd probably hear exactly this argument being made.

In practical terms (de facto group consensus), the recent version of the EAI 
effort has chosen to design for a hypothetically-pure 8-bit environment. 
Dealing with the much messier hybrid world is deferred, here.

The Unicode effort for the DNS chose to work within the hybrid world and, 
therefore, had to make fundamentally different design choices.

Design DNS support for a truly pure UTF-8 environment, end-to-end, and it well 
might be reasonable to 'just do 8 bits'.  But they didn't have that luxury.

d/

-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Tue, 23 Aug 2011 10:42:40 -0400

--On Tuesday, August 16, 2011 07:47 -0700 Dave CROCKER
<[email-masked]> wrote:

>...
> In practical terms (de facto group consensus), the recent
> version of the EAI effort has chosen to design for a
> hypothetically-pure 8-bit environment. Dealing with the much
> messier hybrid world is deferred, here.

As you know, opinions differ about exactly what that design
decision was, how far it extends, and what implications can be
drawn from it.  Nonetheless...
 
> The Unicode effort for the DNS chose to work within the hybrid
> world and, therefore, had to make fundamentally different
> design choices.
> 
> Design DNS support for a truly pure UTF-8 environment,
> end-to-end, and it well might be reasonable to 'just do 8
> bits'.  But they didn't have that luxury.

To strengthen the distinction, one would need actually need a
lot more than "truly pure UTF-8 environment".  Identifier
comparison in a non-ASCII environment is actually quite
difficult in the general case, requiring a lot of additional
comparison rules and/or restrictions (see RFC 6055 for a partial
discussion).  While we take it for granted these days, that is
even true in the ASCII environment because we have to specify
choices between case-dependent and case-independent comparison.
But, in the DNS case, even if we substituted "pure UTF-8" for
the IETF-invented Punycode encoding, we would still need fairly
extensive rules about permitted characters, normalization, etc.,
to make things work -- and even more extensive rules if any sort
of mapping (or non-exact comparison) was permitted.

See current discussions in PRECIS, the IAB work on identifiers,
the Unicode work on identifiers, and the periodic debates in
DNSEXT and ICANN about extended aliases, matching, variant
names, and so on for examples of how complicated those issues
get.

We are spared most of this problem with email addresses because,
for other reasons, we adopted a very strong "no one gets to
interpret an address other than the delivery server" rule.
Preserving that rule into the EAI work permits us to push "pure
UTF-8" much further than was the case with IDNs and IDNA, which
had no such advantages.  We are probably spared them for other
header fields as well, although, for example, I've seen problems
with comparison and threading of subject lines and searching on
display names that use encoded words.  Those problems will
almost certainly carry forward and become more extensive as we
encourage arbitrary UTF-8 in those fields.  From my point of
view, EAI doesn't change those problems because one can have all
of them when the data are expressed in encoded-word form.  But
we should be careful about assuming that a "pure UTF-8"
environment will be as problem free (or problem-minimal) as
ASCII letters and digits.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Aug 2011 14:26:16 +0100

> of mapping (or non-exact comparison) was permitted.

All of which is exactly why I oppose allowing UTF-8 in Message-IDs (which  
get treated like identifiers in many situations). Fine if the job is done  
properly by specifying the normalizations etc, but not otherwise.