Subject: Re: [EAI] Re: Discussion of draft-ietf-eai-downgrade-03.txt

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 02 Apr 2007 12:58:52 +0100

On Sat, 31 Mar 2007 12:34:04 +0100, Frank Ellermann  
<[email-masked]> wrote:

> Charles Lindsey wrote:
>
>>>> Content-Disposition: attachment; filename="mañana"

> Are you talking about what I see, a Latin-1 &ntilde; ?
> Or was it meant to be an UTF-8 &ntilde; ?

Since my MUA is configured to use IS) 8859-1, what you saw was a Latin-1  
&ntilde;, but what the line

>>>> Content-Disposition: attachment; filename="mañana"

was intended to illustrate was a header containing a UTF-8 &ntilde;, since  
that is what the Utf8headers document proposes to allow; hence its RFC  
2231 encoding would be

>>      Content-Disposition: inline; filename*=utf-8''ma%C3%B1ana

> I tried to 2231 encode the undeclared F1 octet.  Your example
> didn't state that it's a new MIME version allowing raw UTF-8
> in Content-* header fields.

It is what the current Utf8headers draft proposes to allow, without any  
change in the MIME version.

>>> Doesn't it ?  My 2231-attempt was intended for 7BIT, why
>>> does 8BITMIME care about 8bit crap in MIME part headers ?
>
>> Because there is no way to downgrade them to 7bit for a further MTA
>> which does not advertise 8BITMIME. So you need to downgrade it to
>> filename*=utf-8''ma%C3%B1ana before it leaves the UTF8SMTP universe.
>
> Maybe start with a new example where it's clear what you're talking
> about.  MIME 1.0 with an unknown %xF1 garbage is very different
> from "MIME 2.0" with UTF-8 %xC3.B1.  In both cases I don't see
> why 8BITMIME should care about 8bit stuff in the body, it has a
> parameter BODY=8BITMIME.  The definition in RFC 1652 says:

And indeed, RFC 1652 requires it to be transported correctly to other  
systems supporting 8BITMIME. But RFC 1652 also requires it to be  
downgraded before it is passed to a Non-8BITMIME MTA, and there is no way  
to do that downgrade within a MIME part header, because such headers do  
not come withn the scope of any Content-Transfer-Encoding header.
>
> | The value associated with the BODY parameter indicates whether the
> | content body which will be passed using the DATA command consists of
> | a MIME message containing some arbitrary octet-aligned material
> | ("8BITMIME") or is encoded entirely in accordance with [1] ("7BIT").
>
> IMO "some arbitrary octet-aligned material" does _not_ imply that
> servers are supposed to check the MIME syntax, let alone version.

They have no choice but to check the MIME syntax, and in fact that is  
precisely what they do. We have already discused this in connection with  
the source code of Sendmail, which indeed does a full MIME recursive  
descent in order to find bodies of MIME subparts and contained  
message/rfc822 objects where is can then change the C-T-E. But if it  
encounters a MIME part header, or a header in a message/rfc822, that  
contains 8bit stuff, then it is stuck (because such headers are not  
allowed to exist in the present universe).

>> How about
>>    subject: =?utf-8*de?Qtest_?= ?utf-8?Q?%C3%BC?= =?utf-8*en?Q?_this?=
>
> I think that's perfect.  You could also add the spaces to the second
> word, leaving the adjacent words alone -- for cases where you have no
> clue what charset / language / encoding it might be - but there can't
> be any encoding different from "Q" or "B", or can it ?.

RFC 2047 includes provision for new encodings to be invented in the  
future, hence my suggestion for an "I" encoding:

>>     Subject: =?utf-8*es?I?mañana=?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 02 Apr 2007 23:51:40 +0200

> It is what the current Utf8headers draft proposes to allow,
> without any change in the MIME version.

It has to be fixed, creating its own MIME version explicitly, or
in some obscure way, where an implicitly identified message/utf-8
implicitly also redefines MIME.  One "implicit" too many for my
tastes, I stick to version 1.0 as specified:

| The "multipart" boundary delimiters and header fields are always
| represented as 7bit US-ASCII in any case (though the header
| fields may encode non-US-ASCII header as per RFC 2047)
 [2046 5.1]

| However, in no event are headers (either message headers or body
| part headers) allowed to contain anything other than US-ASCII
| characters.
 [2046 5.1.1]

| The message header fields are always US-ASCII in any case, and
| data within the body can still be encoded, in which case the
| Content-Transfer-Encoding header field in the encapsulated
| message will reflect this.  Non-US-ASCII text in the headers of
| an encapsulated message can be specified using the mechanisms
| described in RFC 2047.
 [2046 5.2.1]

|     MIME-Version: 1.0
|
| The presence of this header field is an assertion that the message
| has been composed in compliance with this document.
 [2045 4]

| Using the MIME-Version, Content-Type, and Content-Transfer-Encoding
| header fields, it is possible to include, in a standardized way,
| arbitrary types of data with RFC 822 conformant mail messages.  No
| restrictions imposed by either RFC 821 or RFC 822 are violated, and
| care has been taken to avoid problems caused by additional
| restrictions imposed by the characteristics of some Internet mail
| transport mechanisms (see RFC 2049).
 [2045 10]

And then there was this series of four MUST in 2047 which should have
told us that any attempt to 2047-encode header field bodies already
containing encoded words to get a "downgrading" effect is moot... :-(

Tough, back to the drawing board.  
  
 [8BITMIME]
> RFC 1652 also requires it to be downgraded before it is passed to a
> Non-8BITMIME MTA, and there is no way to do that downgrade within a
> MIME part header, because such headers do not come withn the scope
> of any Content-Transfer-Encoding header.

8BITMIME to 7BIT is the job of those gatewys, not of UTF8SMTP.  If they
can't do this they're forced to reject the mail.  Of course I fail to 
see any problems with 8bit in MIME part headers because I don't intend
to allow this in the first place (for MIME 1.0 as quoted above).

> They have no choice but to check the MIME syntax, and in fact that
> is precisely what they do.

If any decent 8BITMIME server receiving BODY=8BITMIME from a client,
and relaying it (as is) to another 8BITMIME mailer, ever looks for
8bit characters in MIME part headers in the body, I'd really wish to
know *why*.  What's it supposed to do with it, reject the mail as 
unsuited for 7bit conversions in a pure 8BITMIME environment ?

> We have already discused this in connection with the source code
> of Sendmail, which indeed does a full MIME recursive descent

That was an 8BITMIME-to-7BIT gateway, not an 8BITMIME-to-8BITMIME
relay.  I assume that UTF8SMTP-to-7BIT is functionally equivalent
to UTF8SMT-to-8BITMIME followed by 8BITMIME-to-7BIT.  If that's the
case EAI only needs to solve UTF8SMTP-to-8BITMIME.  If the 2nd step
(to 7BIT) doesn't work for any reasons, then doing it in one instead
of two steps also won't work.  

And of course I'm talking about "MIME version 1.0" as quoted above.

> if it encounters a MIME part header, or a header in a message/rfc822,
> that contains 8bit stuff, then it is stuck (because such headers are
> not allowed to exist in the present universe).

It's not stuck, it rejects the garbage mail, it's invalid.  Or as the
similar EAI problem description puts it:

| Reject the message or, if necessary, return a non-delivery
| notification message, so that the sender can make another plan.

If in doubt reject, most likely it was anyway spam.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 03 Apr 2007 10:21:48 +0100

> know *why*.

Nobody has suggested that would be necessary. It is only required when  
downgrading to Non-8BITMIME.

> case EAI only needs to solve UTF8SMTP-to-8BITMIME.

Exactly. Which means that if MIME body part headers contain UTF-8 (whether  
as an extension to MIME 1.0, or as some new-angled MIME 2.0 makes no  
difference), then they have to be downgraded (in a 2231 manner) as soon as  
they leave the UTF8SMTP unoiverse, even if they are still remaining in the  
8BITMIME universe, because the 8BITMIME universe does not othersise know  
how to downgrade them to Non-8BITMIME.

> If the 2nd step

Exactly, which is why the problem has to be fixed in the first step.

---

From: [email-masked]
Date: Tue, 10 Apr 2007 19:18:03 +0900 (JST)

Thank you very much for your many comments.

I'm updating downgrade document.

> From: "Charles Lindsey" <[email-masked]>
> As compared to the Utf8headers and Smtpext drafts, this one still seems a  
> long way from completion.
> 
> 1.  Introduction
> 
>     Downgrading in UTF8SMTP consists from following two parts:
>                                      ^^^^
>                                      of the

fixed

>     This document assumes that MIME headers are not extended to use UTF-8
>     characters because it is not clearly defined in
>     [I-D.ietf-eai-utf8headers].
> 
> On the contrary, [I-D.ietf-eai-utf8headers] now makes it very clear that  
> those MIME headers are to be extended, and the downgrading of those  
> headers now needs to be addressed.
> 
> 3.1.  Timing and conditions of downgrading
> 
>     Conditions:  SMTP client detects that non-ASCII characters are
>        included in the SMTP envelope or mail header fields in the SMTP
>        DATA.
> 
> That needs to include mail header fields at any level within the MINE  
> structure.

changed to:
	or mail header fields and MIME body part headers in the SMTP DATA.

> 3.2.  Requirements
> 
>     4.  Downgrading and upgrading should be easy and lightweight as it is
>         possible to do with MTA like 8BITMIME encapsulation.
> 
> But even downgrading 8BITMIME is not "easy and lightweight" if down  
> properly (e.g. as sendmail does it). Granted that the machinery for doing  
> 8BITMIME downgrading can easily be adapted to do header downgrading.

removed this requirement.

> 4.  Downgraded header
> 
>     downgraded  = "Downgraded:" [CFWS] field-name ":" dvalue CRLF
>     field-name = 1*ftext
>     ftext      = %d33-57 / %d59-126 ; printable ASCII except ":"
>     dvalue     = 1*any-ascii
>     any-ascii  = %d32-126
> 
> No, the proper syntax there is
> 
>     downgraded  = "Downgraded:" [CFWS] field-name ":" unstructured CRLF
> 
> Since <unstructured> is essentially what RFC 2047 is defined to work upon.  
> And I am not sure whether you want that "[CFWS]" there. Shouldn't it be  
> "[FWS]", because this header is always automatically generated, so there  
> will never be any comments in it? Or do you want downgrading  
> implementations to be able to add comments in there to cover unanticipated  
> situations (just like Received headers have acquired comments in such  
> ways).

"[CFWS]" is changed to "[FWS]".

> 3. You change the syntax to
> 
>     downgraded  = "Downgraded:" [CFWS] field-name ":" encoded CRLF
>     encoded     = *([FWS] (utext / encoded-word) [FWS]
> 
> where <utext> is the strict RFC 2822 version, and <encoded-word> comes  
>  from RFC 2047.

Rewrited ABNF as 3.

>     To preserve SMTP envelope downgraded information, another new header
>     is required, and it is specified as "Envelope-Downgraded".  Details
>     are described in Section 5.  The header field syntax is specified as
>     follows:
> 
>     fields =/ edowngraded
>     edowngraded = "Envelope-Downgraded:" [CFWS] field-name ":" value CRLF
>     field-name = "From" / "To"
>     value = CFWS "<" uPath ">" CFWS "<" Mailbox ">" CFWS
> 
> Several problems here:
> 
> 1. Same question about whether CFWS rather than FWS is appropriate in a  
> machine-generated header.

changed to FWS.

> 2. Please don't use <value> as an ABNF name. Too much risk of confusion  
> with <value> as used in MIME parameters

changed to "edowngraded-value"

> 3. Would it be better to have a notation (e.g. using some 'arrow' like  
> "->") which immediately suggested which of the two addresses was the  
> original and which was the downgraded?

I think two angle-addrs are enough.

> 4. Since this header will itself inevitably have to be downgraded,  
> wouldn't it be simpler to do it all in one step by providing for the  
> <uPath> to be encoded (e.g. by RFC 2047). All you would need would be a  
> syntax like
> 
>     value = CFWS "<" encoded-word ">" CFWS "<" Mailbox ">" CFWS

changed to this line and add some comments.

> 5.  SMTP Downgrading
> 
>     One SMTP session may contain multiple recipients.  Downgrading SHOULD
>     be performed for each SMTP recipient address individually.  First,
>     split a multiple recipients session to each sessions.  If the
>     recipient address is downgradable, the SMTP session to the recipient
>     is downgradable.
> 
> It is not clear what you mean by the term "session" here. Is it defined  
> anywhere?

Removed lines which contains "session".

>     Also note that by appending "Envelope-Downgraded:" headers, MUA/MTA
>     MUST perform Email header Downgrading described in Section 6.
> 
> And that is the unnecessary step I want to avoid by defining the  
> Envelope-Downgraded header to be already pure ASCII.

I agreed and replaced.

>     ESMTP ORCPT parameter is used for Delivery Status Notifications
>     (DSNs) [RFC3461].  ORCPT parameters contain mail addresses.  After
>     extending ORCPT parameter to support Non-ASCII mail addresses, ORCPT
>     parameter downgrading should be defined here.
> 
> Then please do it, or agree with Chris that it is to be defined in the DSN  
> document, and cross-reference it from here (but I think it would be better  
> to do it here, and I think we are agreed that it has to be done by %hex  
> encoding of the UTF-8).
> 
> 6.  Email header Downgrading
> 
>     This section defines conversion method to US-ASCII for each header
>     which may contain Non-ASCII characters.  If the whole mail header
>     does not contain Non-ASCII characters, email header downgrading is
>     not required.
> 
> You need to add that headers in MIME body parts and in contained  
> message/rfc822 objects also need to be examined.
> 
>    Otherwise, email header downgrading checks each header
>     whether it contains UTF-8 characters or not.
>                         ^^^^^^^^^^^^^^^^
>                         <UTF8-xtra-char>s
>                  {or whatever they are eventually called}

Changed to 'non-ASCII' which is defined in framework document.

>     .......... If the header contains
>     UTF-8 characters, convert the header in a suitable method for of each
>     header.  Each header's downgrading method is described below:
> 
>     This section defines conversion method to US-ASCII for each header
>     which may contain Non-ASCII characters.  If the whole mail header
>     does not contain Non-ASCII characters, email header downgrading is
>     not required.  Otherwise, email header downgrading checks each header
>     whether it contains UTF-8 characters or not.  If the header contains
>     UTF-8 characters, convert the header in a suitable method for of each
>     header.  Each header's downgrading method is described below:
> 
>     From, To, CC, Reply-To, Return-Path:
> 
> You need to add the Sender header, and also say that it applies to any  
> other header that might be defined to use a <mailbox>.

Added Sender and Resent-{From,To,CC}, removed Return-Path.

> And please s/header/header field/ in all places where it is not already  
> done.
> 
>        The header field body is composed of single or multiple angle-
>        addr/addr-spec fields defined in [I-D.ietf-eai-utf8headers].
>             ^^^^^^^^^
>           utf8-addr-spec
>        If the header has no 'angle-addr' or 'utf8-addr-spec' which
>        contains UTF-8 characters, only "display-name" part contains UTF-8
>                 ^^^^^^^^^^^^^^^^                                    ^^^^^
>                 <UTF8-xtra-char>s                              
> <UTF8-xtra-char>s
>        characters, encode the header by [RFC2047] with UTF-8 tag and
>        ^^^^^^^^^^         ^^^^^^^^^^
>                        that <display-name>
>        replace it.

fixed.

> Generally speaking, when refering to objects defined by ABNF, it is better  
> to enclose them within <.....> as recommended in RFC 4234, than to enclose  
> them within '.........'.

Ok, I will change.

>        ... When
>        this address header downgrading fails, this downgrading fails and
>        the mail MUST be bounced.
> 
> For some reason, we decided to use "rejected", rather than "bounced".

fixed.

>     Subject:
>        Encode the header by [RFC2047] with UTF-8 tag and replace it.
> 
> There are other headers where this would be the proper treatment. E.g.  
> Keywords, and header defined as <unstructured> by other standards (e.g.  
> the Summary header in Netnews), and all X-headers.

added.

> You also need to mention Envelope-Downgraded in this list, unless you have  
> adopted my suggestion to make it already all-ASCII when it is generated.
> 
>     Downgraded headers should be inserted or replaced at the same
>     position of the original header.
> 
> Yes, I think that is correct, although it might be argued that they are  
> "Trace" headers.
> 
> 6.1.  Downgrading address headers
> 
>     This procedure targets "From:", "To:", "CC:", "Reply-To:", "Return-
>     Path:" headers which contains Originator/Destination address(es).
> 
> You need to add Sender: to that list. Even Bcc:, though it should not  
> really be there. But it would also apply to any other header that might be  
> defined in future with a <mailbox> in it (it should be permitted for  
> agents that recognixe it, though we cannot REQUIRE it for headers newly  
> invented after our document is published).

agreed and added Bcc and Sender.

> And somewhere around here (section 6.2 maybe) you have to describe the  
> downgrading of MIME <value>s, which are allowed to contain UTF-8 according  
> to Utf8headers. The downgrade should, of course, use RFC 2231, and a  
> Downgraded header should be provided as well.

I will write this.

> You also need at least a placeholder for downgrading List-* headers,  
> although discussion of these in ongoing in a separate thread. And maybe a  
> placeholder for any downgrading that may sries from the DSN draft.

I will add List-* headers.

> 7.  Upgrading downgraded header

Upgrading is moved to appendix and I will rewrite.

> 8.  Implementation consideration

>     MUA MAY encode UTF-8 in Subject header with the same encoding of body
>     part while downgrading.
> 
> That also applies to other unstructured headers, and to display-names,  
> etc. It is essentially the point I made earlier about upgrading possibly  
> undoing more than the earlier downgrading did.
> 
>     UTF8SMTP compliant MUA MUST upgrade downgraded mail and MUST show
>     Non-ASCII mail addresses on display.
> 
> s/MUST/SHOULD/ (twice). There is no interoperability - just confusion :-( .
> 

This part is removed.

> 10.  IANA Considerations
> 
>     IANA is requested to add the "Envelope-Downgraded:", and
>     "Downgraded:" new headers to the registry with the entries pointing
>     to this specification for its definition.
> 
> You need to provede proper templates for these, as required by RFC 3864.

OK, added.

> Appendix A.  Examples
> 
> A.1.  Downgrading example 1
> 
>     In this example, there are two sessions, one is To:, the other is
>     CC:.  Both sessions are downgradable.  Figure 4 shows To: session
>     downgrading.
> 
> 
> I don't think "sessions" is the right word here. There is only one SMTP  
> "session", which lasts from the EHLO until the connection is broken. What  
> you are talking about here is just a part of the processing of one  
> message, namely that which applies to a single RCPT. Perhaps someone else  
> can suggest a better word here, but "session" is definitely wrong.

changed "session" to "recipient".

>     Result of the header downgrading.
> 
> 
>     Downgraded: Envelope-Downgraded: From:
>                 MIME(<NON-ASCII-FROM>) <ASCII-FROM>
>     Downgraded: Envelope-Downgraded: To:
>                 MIME(<NON-ASCII-TO>) <ASCII-TO>
> 
> s/MIME/RFC2047/ (or some other word, e.g. "DECODE")

changed to RFC2047

> And at the end of this downgraded example, please show how a Return-Path  
> (derived from the MAIL FROM) would look. And how would you show an  
> <alt-address> in it? Is it accompanied somehow by a Downgraded:  
> Return-Path: ... . I don't think we ever discussed how that would work.

added.

Return-Path: will be downgraded <alt-address>.
Decoding Envelope-Downgraded header may restore it as non-ASCII address, 
but it is not target of this document.

> A.2.  Upgrading example
> 
> And somehow you have to show how a Return-Path got upgraded (if at all)  
> here.
> 
>                 Figure 10: Downgraded header upgraded message
> 
>     As a result, in this simple example, all headers are preserved.
>                                             ^
>                                          the original

fixed.

--
Kazunori Fujiwara, JPRS

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Apr 2007 07:57:34 -0400

--On Tuesday, 10 April, 2007 19:18 +0900 [email-masked]
wrote:

>...
>>        ... When
>>        this address header downgrading fails, this
>>        downgrading fails and the mail MUST be bounced.
>> 

Please be careful about this so as to not introduce new
problems.  "rejected" is intended to imply "detected during the
receiving function of the SMTP system and handled by sending a
reply code back to the would-be sender".   Unless the downgrade
machinery is invoked before the connection from the sender is
closed, any notification to the sender that downgrading is not
possible will have to be sent as a DSN, not a provided in a
"reject" context.  While it would be possible to do so in at
least some cases, in general trying to keep that incoming
connection open until it is determined whether successful
downgrading can be accomplished would not be a good idea.

>> There are other headers where this would be the proper
>> treatment. E.g.   Keywords, and header defined as
>> <unstructured> by other standards (e.g.   the Summary header
>> in Netnews), and all X-headers.
> 
> added.

Certainly not for "all X-headers" since one has no idea what is
in them.  This strikes me as dangerous and I would prefer that
we be very conservative about it.

>...

regards,
   john

---

From: [email-masked]
Date: Wed, 11 Apr 2007 11:22:10 +0900 (JST)

> From: John C Klensin <[email-masked]>

I understand.

I deleted "rejected" and rewrited as "Downgrading fails".
After failed downgrading, the message will be bounced but it is
written in smtpext document.

--
Kazunori Fujiwara, JPRS

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Apr 2007 01:07:28 -0400

--On Wednesday, 11 April, 2007 11:22 +0900 [email-masked]
wrote:

>> problems.  "rejected" is intended to imply "detected during
>> the receiving function of the SMTP system and handled by
>> sending a reply code back to the would-be sender".   Unless
>> the downgrade machinery is invoked before the connection from
>> the sender is closed, any notification to the sender that
>> downgrading is not possible will have to be sent as a DSN,
>> not a provided in a "reject" context.  While it would be
>> possible to do so in at least some cases, in general trying
>> to keep that incoming connection open until it is determined
>> whether successful downgrading can be accomplished would not
>> be a good idea.
> 
> I understand.
> 

That is, I think, an excellent solution.
 thanks,
    john

---

From: [email-masked]
Date: Wed, 11 Apr 2007 15:36:32 +0900 (JST)

> we be very conservative about it.

Encapsulating undefined headers (including all X-headers) is reasonable?

I listed headers from RFC 4021. (There may be some omission.)
Then, I categorized 7 rewriting header methods.

Please check the list of downgrading target headers below.

And then, I have a question about Content-* header downgrading.
Content-* header appears in RFC 2045 section 5.

Which downgrading method is suitable for Unknown Content-* header,
      (7)Encapsulating into Downgraded or (5)RFC 2231 ?

- List of downgrading target headers

  1. Originator and destination addresses headers
     ->  preserve original header into Downgraded: header
         and rewrite utf8-addr-spec as alt-addr
	 (comment encoded by 2047)

        From:
        Sender:
        Reply-To:
        To:
        Cc:
        Bcc:
        Resent-From:
        Resent-Sender:
        Resent-To:
        Resent-Cc:
        Resent-Reply-To:
	Disposition-Notification-To:

  2. ASCII only value with non-ascii comment
     -> encode by RFC 2047

        Date:
        Message-ID:
        In-Reply-To:
        References:
        Resent-Date:
        Resent-Message-ID:
	MIME-Version:
	Content-ID:

  3. text header
     -> encode by RFC 2047

        Subject:
        Comments:
        Keywords:

  4. trace header
     -> Remove uFor clause

        Received:

  5. MIME headers (with UTF-8 string)
     -> encode by RFC 2231

	Content-Type:
	Content-Transfer-Encoding:
	Content-Description:

  6. URI headers
     -> encode by RFC 3987

	List-Archive:
	List-Help:
	List-Owner:
	List-Post:
	List-Subscribe:
	List-Unsubscribe:

  7. Unknown/Undefined headers
     -> encapsulate into Downgraded header

	X-headers:
        other unstructured headers:
        other structured headers:

---
Regards,

--
Kazunori Fujiwara, JPRS

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Apr 2007 02:39:21 -0400

Yuri,

This is a personal perspective.  I hope that the authors of the
downgrade draft, and others, will respond to you also.

I think you have captured the key issues here.   I believe that
there are three, philosophically different, ways to look at the
problem for which the downgrade draft is a proposed solution.
Each one has implications for, e.g.., likely rate of deployment
and quality of interworking during the transition period.  There
is not  complete agreement about what those implications might
be, i.e., different people have different theories about which
approach is most likely to lead to rapid deployment of the EAI
features.

These are listed in order of increasing complexity, not
preference.

(1) No downgrading at all.   An address is an address.
Addresses being unreachable are old news and, while I might want
to tell a correspondent to use a hotmail address if my normal
one doesn't seem to be working, I do that in the body of the
message and allow the sending user to deal with it, rather than
requiring additional syntax to express the alternatives.  In
this model, if a UTF-8 address cannot be reached, the message
just gets rejected or returned.   Problems arise when the UTF-8
address is backward-pointing (so there is no place for an
ASCII-only system to send a reply or delivery status
notification) or if the envelope is all-ASCII but the message
headers contain UTF-8 material.  For the second of these cases,
the worst-case problem is only constructing a DSN, which is
inherently less general than the problem of constructing a
machine-intelligible downgraded message.

(2) Downgrading only.   Here, the downgrading operation is
treated as a one-way gateway transformation, with the target
being to create as near an approximation to the original message
as possible in all-ASCII form.  Other than trace information
that would indicate that the transformation occurred, there
would be no serious attempt to preserve pre-downgrade
information that couldn't be cleanly expressed in encoded words,
no attempt to preserve digital signatures, etc.  Probably there
would be no new headers, eliminating the problems you identify.
However, an internationalized client sitting behind an
ASCII-only system or relay would see only the ASCII addresses
and have no mechanism to generate a reply to the UTF-8 address
(unless there were comments in the body text), even if it has a
UTF8SMTP-capable path in the reverse direction.

(3) Downgrading with upgrade capability at the endpoints.   If
one thinks it is important to permit a UTF8SMTP-capable mail
reader or MUA at the receiving end to see an approximation of
the original message, headers, and addresses, despite
downgrading on the path to it, then one needs to, somehow,
preserve most of the original information -- much more
information than is required by (2).  How far one wants to go,
and how far one can go, is so far still an unresolved question,
with trying to preserve or restore the integrity of envelope
information and of digital signatures over header fields as the
most stringent possible targets.

Clearly, the downgrade document reflects the third view.  If one
were to adopt one of the other views instead, the specifications
would be quite different.

regards,
    john

(2) 

--On Tuesday, 10 April, 2007 22:56 -0700 Yuri Inglikov
<[email-masked]> wrote:

> I am trying to catch up on the progress of this WG and
> wondering if the following was ever debated or not. The
> downgrading with Downgraded: and Envelope-Downgraded header
> fields apparently has the following issues:
> 
> 1. Address header fields (such as To, Cc,...) going out of
> sync with corresponding Downgraded: header fields. There are
> applications which modify address header fields, such as
> address rewriting agents or workflow applications. This means
> that upgrade of address header field may not always be
> possible. I think that the method which is based on
> introducing unstructured redundant information like this is
> flawed. My preference, if we need to keep a path to upgrade
> header fields, would be to do this on a per-recipient basis
> and just drop the Downgraded: information if corresponding
> recipient not found in any address header. Also we should
> introduce as little redundancy as possible, e.g. there is no
> need to duplicate display names in Downgraded: header as
> display name can be safely 2047-encodded without data loss.
> 
> 2. Some MUA potentially can preserve the Downgraded: header
> fields in replies and especially in forwarded messages. While
> this may not be a problem for RFC2822 or MIME header fields
> (it is not worse than original header field being preserved),
> the net result is that such Downgraded: headers can appear in
> a wrong message, i.e. not the one which was originally
> downgraded. This, on the first glance, can be a real problem
> for Envelope-Downgraded headers, e.g. MAIL FROM address, is
> it? E.g. could it cause P1 sender of a message being replaced
> with original sender while upgrading the envelope information?
> Is there such thing as envelope information upgrade? If not,
> then why bother preserving such information (other than as
> another trace field)?
> 
> I personally think that it can be acceptable to define the
> downgrade process as not reversible, e.g. that downgrade can
> just drop the UTF8 address and use alt-addresses instead.
> There should be no such thing as upgrade, the downgraded
> message should be handled as normal pre-EAI message with
> appropriate information encoded in a backwards-compatible
> fashion. What are disadvantages of this approach? I assume we
> want the downgrade mechanism to be less and less necessary
> over time when everybody upgrades to support EAI.
> 
> My another feeling is that unknown header fields downgrade can
> just use RFC2047 encoding by default. The reasoning is that no
> header fields defined before EAI are supposed to contain UTF-8
> and no downgrade should be necessary for such header fields,
> except those specifically supported by EAI (such as Subject,
> address header fields). Of course new header fields can be
> defined after EAI is introduced which may want to take
> advantage of UTF8. In such case they either should be defined
> as unstructured headers for which default downgrade mechanism
> (RFC2047) is sufficient, or those are new structured headers.
> For new structured headers EAI could define a way to identify
> such header for EAI downgrade purposes and a common open
> format (like the one defined for a Content-Disposition header
> with parameters) with provisions how to identify parts which
> potentially can contain UTF8 and how to downgrade them
> (RFC2231?).
> 
> I.e. then downgrade process could be defined as dropping
> redundant information (UTF8 addresses) and orderly encoding of
> remaining UTF8 information. No upgrade process should be
> defined.
> 
> Sorry if I am bringing back something which was already
> debated and rejected, I still have a lot to catch up.
> 
> Thanks,
> Yuri

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 11 Apr 2007 19:00:19 +0200

> Encapsulating undefined headers (including all X-headers) is

As long as you don't use 2047 / 2231 on "words", because you
don't know what a "word" is without knowing the structure, as
in the ...

xxx: (=?ascii*fr?Q?no?= encoded =?ascii*en?Q?words?=)

...example, if xxx header field bodies are unstructured (or
allow no comments).

> Which downgrading method is suitable for Unknown Content-*
> header, (7) Encapsulating into Downgraded or (5) RFC 2231 ?

None, Content-* MUST be ASCII for MIME Version 1.0, and the
old pre-MIME Content-Type was also ASCII.

> From: Sender: Reply-To: To: Cc: Bcc:

Bcc should be empty...

> Resent-From: Resent-Sender: Resent-To: Resent-Cc:

...or add Resent-Bcc: (what crap, sigh).

> Resent-Reply-To:

Strike that, it doesn't exist.

> Disposition-Notification-To:

Can somebody _please_ start the next "decruft experiment" ?

>         Keywords:

Keywords isn't unstructured, <obs-phrase> can have a comment,
see above.

>         Received:

Received-SPF, DKIM-Signature.

>   5. MIME headers (with UTF-8 string)

Reject as bogus, or encode as unstructured garbage (like subject).

>         List-Unsubscribe:

Archived-At (will be an RFC before Downgrade, I hope).  I think
there can be comments in List-*.  Add "List-Id:" to your second
section.

>         other structured headers:

IMO that kills it, structured and unstructured MUST NOT be in the
same group.  For X-anything you can say "who cares, let's assume
it's unstructured like a subject", but unknown + structured is a
showstopper for any "2047-encode 'words'" strategy, because you
can't tell what a 'word' is.

Frank

---

From: Yuri Inglikov <[email-masked]>
Date: Wed, 11 Apr 2007 11:20:27 -0700

Thank you.

I think that even if WG decides to standardize the most complex solution #3, which seems to be the direction, then #2 and #1 could still be left legal and the choice left to implementers.

BTW apparently #1 is very close (in sense of the effect to end user) to a "failure to downgrade", i.e. bouncing the message when there is no alt-address to downgrade to - some addresses are just effectively unreachable. Logical extension to that is a "failure to downgrade" when MTA decides to not implement downgrade.

As for #2, somebody can probably decide that the name "gateway" versus "MTA" is not a big deal and implement this solution anyway?

Yuri

-----Original Message-----
From: John C Klensin [mailto:[email-masked]]
Sent: Tuesday, April 10, 2007 11:39 PM
To: Yuri Inglikov
Cc: [email-masked]
Subject: RE: [EAI] Discussion of draft-ietf-eai-downgrade-03.txt

Yuri,

regards,
    john

(2)

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 11 Apr 2007 20:18:45 +0100

> Encapsulating undefined headers (including all X-headers) is reasonable?

The general opinion is that X-headers are to be considered 'unstructured'.  
Certainly, RFC 2047 gives Carte Blanche to encode them, so we might as  
well follow that lead.

Of course, unrecognised headers are quite a different thing (and RFC 2047  
signally fails to deal with them). If you could find the RFC that defines  
them, you might well find that they had a structure, and that structure  
might even have places where UTF-8 might be sensible. But all that  
software that does not recognize them can do is to use a Downgraded  
header, and hope that whoever eventually upgrades them can do something  
useful with them.
> Then, I categorized 7 rewriting header methods.

Yes, I think your list is more or less correct.

> 	 (comment encoded by 2047)

And you could add to your list that it is acceptable to use the same  
technique on any header that might be defined in the future and that  
involves <mailbox>es, even though some software might only be able to use  
the default Downgraded header on them.

>      -> encode by RFC 2047

Again, add a general permission for any header that is defined as  
"unstructured" (the Netnews Summary and Organization headers, for example).

> 	Content-Transfer-Encoding:

AFAIK, all the currently defined Content-* headers are safe under this  
(i.e. they either have <parameter>s, or UTF-8 is inappropriate within them  
(apart from any <comment>s). Furtunately, many existing MUAs which have  
already implemented 2231 will upgrade them as appropriate.

But note also that many newer RFCs are begining to use <parameter>s in  
other situations (e.g. the new Injection-Info in Netnews). But AFAICS it  
is always safe to use 2231 on them.

---

From: John C Klensin <[email-masked]>
Date: Thu, 12 Apr 2007 00:10:18 -0400

Some small possible disagreements...

> ...example, if xxx header field bodies are unstructured (or

Right.  The good news is that we've made other requirements that
anything that would need to be downgraded must be Unicode
encoded in UTF-8.   So, if it is absolutely clear in the text of
the I-D how a "Downgraded" header is structured, you could use
one of the techniques discussed in draft-klensin-unicode-escapes
without much (or any) risk.

> old pre-MIME Content-Type was also ASCII.

Almost true.  The only MIME headers that can have non-ASCII
strings in them are those that can contain non-ASCII text if our
"utf8-headers" document says so.  For example, consider local
system pathnames on a Unicode-based file system, specified in a
"name=" parameter, which might well be in UTF-8.

My personal ruthless instinct is that it is better to drop these
things than to encode them and risk fouling them up, but there
is as good a case for a Downgraded header for them as for just
about anything else.

> Can somebody _please_ start the next "decruft experiment" ?

Please raise this in the context of RFC2822bis (assuming that
draft ever gets posted).

> Received-SPF, DKIM-Signature.

Apologies for being rigid about this, but I'm unaware of any
published RFCs that specify non-ASCII header material for either
of these.  If the header lines contain only ASCII text, they
don't need to be downgraded.

> Reject as bogus, or encode as unstructured garbage (like

See discussion of Content-* above.

> there can be comments in List-*.  Add "List-Id:" to your second

This identifies a problem with "downgrade" that we at least need
to identify.   New headers are added and standardized from time
to time.  We are going to need a general theory about how to
deal with them.  That theory might be expressed as a requirement
on standardization of new headers, or their registrations, that
they specify how they should be downgraded.   I don't think  it
is necessary to work this out in a comprehensive way for this
round of experimental RFCs, but there should at least be a
placeholder noting the issue.

> IMO that kills it, structured and unstructured MUST NOT be in
> the same group.  For X-anything you can say "who cares, let's
> assume it's unstructured like a subject", but unknown +
> structured is a showstopper for any "2047-encode 'words'"
> strategy, because you can't tell what a 'word' is.

Yes, I think that is correct.  See comments above about the
possibility of simply using escaped Unicode.

      john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Apr 2007 15:09:33 +0200

John C Klensin wrote:

> The good news is that we've made other requirements that
> encoded in UTF-8.   So, if it is absolutely clear in the
> text of the I-D how a "Downgraded" header is structured,
> you could use one of the techniques discussed in
> draft-klensin-unicode-escapes without much (or any) risk.

I'm not sure, it could run into similar problems like 2047-
encoding something that already is (in parts) 2047-encoded.

One of the techniques discussed in your I-D is to use NCRs.
I know that you don't like them, but they're handy for a
quick example (using Latin-1 instead of UTF-8 here):

| Subject: &#160; ist für blöde Brwser besser als &#xA0;

Downgrading the UTF-8 version of this subject with NCRs
would result in an ASCII string like this:

| &#160; ist f&#xFC;r bl&#xF6;de Browser besser als &#xA0;

When that's "upgraded" later the literal &#xA0; would be
replaced by UTF-8 for u+00A0, while the &#160; survives
as is, the "upgraded" subject would be displayed as:

| Subject: &#160; ist für blöde Browser besser als  

IMO we need a new technique that can completely ignore the
structure or not of the header field, and the existence of
anything within the header field using a similar technique.

Or we say (for the same hex. NCR example) that any original
"&" has to be encoded as &#x26;   This would result in:

| &#x26;#160; ist f&#xFC;r bl&#xF6;de Browser besser als &#x26;#xA0;

As you said we'd be in a hopeless situation if there are
any non-ASCII octets not belonging to a valid UTF-8.  For
that case we'd need a technique encoding "raw" octets, or
in other words B64-encode the complete header field body,
no matter if it's proper UTF-8 or not, structured or not,
or folded or not.  We'd have to introduce our own folding
of course, 1000 B64-characters per folded line or similar.

> strings in them are those that can contain non-ASCII text
> if our "utf8-headers" document says so.

IMO it can't.  What about a multipart Content-Type with a
weird UTF-8 comment, would that result in the following raw
example ?

-- good boundary
Downgraded: Content-Type: Base-64(UTF-8 header field body)

-- lost boundary
...
-- lost boundary --
-- good boundary --

The "lost boundary" would be hidden within whatever the
Downgrade mechanism did.  The former multipart would be
a single part without MIME header fields.  Maybe a smart
Downgrade mechanism could fix that somehow, preserving a
copy of the old Content-Type without its UTF-8 comment.

> For example, consider local system pathnames on a
> Unicode-based file system, specified in a

Yes, I know why Charles wants it.  IMO adding "MIME 2.0"
(explicitly or implicitly) to the EAI experiment is too
much, email addresses (as the name EAI says) are already
complex enough for the moment.

If some folks really "must" have UTF-8 in filenames and
comments of Content-* they can manage this with RFC 2231,
at least until we know how the EAI core works without
this additional feature.

This MIME 2.0 idea scares me.  It's about as attractive
as integrating 4408 into 2821bis, or Bruce's great plan
to integrate MIME into 2822bis.

> My personal ruthless instinct is that it is better to
> drop these things than to encode them and risk fouling
> them up, but there is as good a case for a Downgraded
> header for them as for just about anything else.

My instinct is to leave MIME 1.0 Content-* alone.  All
these "look into the body and parse its MIME structure"
concepts make me nervous.  I'd prefer to ignore all 7BIT
routes as irrelevant for the purposes of EAI.  We should
just assume that 8BITMIME is the norm, and where reality
disagrees it's an 8BITMIME problem, not a job for EAI.

> Please raise this in the context of RFC2822bis

Diposition-Notification-To would be 3798bis, also used in
4356.  The new Jabber-ID header field is also interesting,
it's in essence an IRI in URI format, like Archived-At.
Jabber-ID is also no RFC yet, but it might be ready soon.

> Apologies for being rigid about this, but I'm unaware of
> any published RFCs that specify non-ASCII header material
> for either of these.

Charles wants UTF-8 comments everywhere.  For Received-SPF
there will be an update to allow a new UTF8SMTP MAIL FROM
address, and in theory also a message/utf-8 PRA if somebody
cares to update 4405 and 4406.

> If the header lines contain only ASCII text, they don't
> need to be downgraded.

These trace header fields are added in transit, of course
only ASCII on 8BITMIME routes.  But for UTF8SMTP routes
they will have to support UTF8SMTP addresses, and a mailer
trying to downgrade them has then to know how.  One of the
reasons why I prefer a downgrade mechanism that can ignore
the syntax of obscure header fields.

> This identifies a problem with "downgrade" that we at least
> need to identify.   New headers are added and standardized
> from time to time.  We are going to need a general theory
> about how to deal with them.

Yes, my theory is a "syntax agnostic downgrade mechanism".

> That theory might be expressed as a requirement on 
> standardization of new headers, or their registrations,
> that they specify how they should be downgraded.

Possible, but too late for already existing header fields,
or for the two "ready for IRI" candidates mentioned above.

Maybe we could mix it:  For some well-known header fields
offer a "smart" downgrade mechanism, like 3987-IRI-to-URI
in the case of Archived-At (only an example, of course
it's not important enough to get a non-default handling).

And for most header fields a "B64-everything" mechanism.

> there should at least be a placeholder noting the issue.

Yes.  We could start with "B64-everything" and then add
the most important exceptions when they are identified.

Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 12 Apr 2007 10:00:30 -0400

> anything within the header field using a similar technique.

I agree.  Unfortunately, we have so many different ways of
encoding things floating around that there is a great deal of
reluctance (at least by me) for creating a new one.  Worse, we
have to optimize for two different types of MUA endpoints.  One
is a legacy MUA that has no knowledge at all of this stuff.  It
is going to have to display the escaped form and hope that users
can make sense of it.  It was concern for those sorts of MUAs
that brought us Quoted-Printable which many of us consider to
have been largely a failure.   The other is an MUA that knows
all about the internationalized forms but that is on the far
side of something that had to downgrade.  It probably will want
to decode (or "upgrade") later.

My comment about knowing things are in UTF-8 was addressed
largely toward the latter, but it suggests that probably a
downgrade procedure that encounters a coded-word that was coded
in terms of ISO 8859-1 should convert it to a coded-word (or
some other encoding) based on Unicode.  But, in the general
case, that is impossible because it means that the downgrading
MTA has to have conversion tables from Just About Everything to
Unicode. 

I think that means that the downgrading procedure needs a way to
clearly distinguish between things that were already encoded but
that it does not know how to encode/recode and things that it
encodes.  There are lots of ways to do that, but they tend to
require some knowledge of what is/was going on and to be ugly
enough, or clearly encodings enough, to be very hard on those
legacy systems and their users.  For example, if our main
interest was in upgradability and we didn't care about much of
anything else, the value of a Downgraded header field would be a
Base64 string that contains, in encoded form, _exactly_ what
appeared in the original header.  That would permit unambiguous
restoration, but would turn the original header information to
trash for anyone who didn't have an upgraded receiving MUA.
 
> | &#x26;#160; ist f&#xFC;r bl&#xF6;de Browser besser als

You are moving closer to the above, i.e., making the string
completely unintelligible in the interests of accurate coding
and decoding. 

> of course, 1000 B64-characters per folded line or similar.

Right.
 
> copy of the old Content-Type without its UTF-8 comment.

Right. Very good example.

> complex enough for the moment.

I agree, fwiw.   But that leaves me without knowing quite what
to do with those file names other than drop them.  I would
happily do that since I've believe the parameter was a bad idea
from the beginning, but I know I'm in a tiny minority about
that.

> at least until we know how the EAI core works without

Makes sense to me, but see above.

> to integrate MIME into 2822bis.

When I suggested it, I intended it as a discussion-stopper.
Most of the people who understand its implications got that, in
one form or another.   For those who are surprised by my saying
that (a list that almost certainly does not include you, Frank),
the problem is that the "MIME-version" header has become noise
because we had no theory about how versioning mechanism would
work going forward.  So we have systems that don't check for its
presence at all, systems that check for its presence and then
ignore it, and systems that might actually notice if the version
number had changed.  Of the latter, some would assume that 2.0
was at least upward compatible with 1.0 and others would assume
that it meant the message was completely incompatible, i.e.,
that they couldn't even interpret Content-type headers unless
they had 2.0 support.

FWIW, that experience, and a few similar ones, are the reason I
get so resistant when people propose parameters to support
future extensions that they don't understand and can't predict
and specify.

> disagrees it's an 8BITMIME problem, not a job for EAI.

And that takes you down the slippery slope toward "if the
address is not deliverable, or relay-able, exactly as written,
the message should be rejected" with no more "downgrade" or
"alternate address" plan than we have for non-deliverable mail
with all-ASCII address and headers.   While I will continue to
try to figure out how to make downgrading work, you might find
me waiting for you at the bottom of that slope.

> Jabber-ID is also no RFC yet, but it might be ready soon.

Ack.
 
> cares to update 4405 and 4406.

Well, first, I don't see how we can write specifications against
an update that hasn't been written yet (or at least hasn't been
reviewed by the wider community yet).   Let me address the rest
of this in a separate note.

    john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 12 Apr 2007 22:57:02 +0200

> to decode (or "upgrade") later.

The work in "scenarios" has singularly failed to establish convincing 
more-than-transient scenarios where this functionality is needed. So (my 
personal technical opinion) we have very strong reasons to optimize for the 
first case.

That means that the *result* of the transformation needs to either be 
ignorable or decodable by MUAs that don't support UTF8SMTP.

In practice, I think that leaves us with "remove or encoded-word".

                         Harald

---

From: John C Klensin <[email-masked]>
Date: Fri, 13 Apr 2007 07:36:23 -0400

--On Thursday, 12 April, 2007 22:57 +0200 Harald Tveit
Alvestrand <[email-masked]> wrote:

>> have to optimize for two different types of MUA endpoints.
>> One is a legacy MUA that has no knowledge at all of this
>> stuff.  It is going to have to display the escaped form and
>> hope that users can make sense of it.  It was concern for
>> those sorts of MUAs that brought us Quoted-Printable which
>> many of us consider to have been largely a failure.   The
>> other is an MUA that knows all about the internationalized
>> forms but that is on the far side of something that had to
>> downgrade.  It probably will want to decode (or "upgrade")
>> later.
> 
> The work in "scenarios" has singularly failed to establish
> convincing more-than-transient scenarios where this
> functionality is needed. So (my personal technical opinion) we
> have very strong reasons to optimize for the first case.
> 
> That means that the *result* of the transformation needs to
> either be ignorable or decodable by MUAs that don't support
> UTF8SMTP.
> 

Sadly, I agree.   "Sadly", because I think it really implies
that, if a downgrading system doesn't understand --explicitly
recognize-- a header that contains non-ASCII information-- it
probably needs to drop it.   Anything else gets us involved with
guessing what the information there is actually about and about
what the receiving MUA may or may not know. 

A different way to look at this is that, while losing
information is unattractive, it is better than risking changing
material in a way that appears to retain information but
actually alters it.  If we try to preserve every possible bit in
the expectation of later upgrading, we are likely to increase
the number of situations in which messages are rejected or
returned rather than downgraded.

   john

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Apr 2007 12:49:53 +0100

> encoding something that already is (in parts) 2047-encoded.

The proper thing to do in that case is for the downgrader to leave the  
already-encoded bits alone and to encode any UTF-8 bits in between them.  
You have to be careful with SPs where they occur, and we recemntly  
discussed some examples and it seems they could be managed by inserting  
"_"s at suitable places within the new encoded-words.

But the downgrade document should probably include warnings about this. If  
you read RFC 2047 carefully, you should always be able to discover at  
least one valid way of doing it.

> quick example (using Latin-1 instead of UTF-8 here):

What is an NCR? If they are anything like the examples you give, then we  
are better off without them :-( .

> anything within the header field using a similar technique.

That is what the Downgraded header is supposed to do.

> any non-ASCII octets not belonging to a valid UTF-8.

Not supposed to happen, so Garbage-in Garbage-out. Actually, most systems  
will probably not notice them, and will simply encode the octets as if  
they were UTF-8, leaving the man at the other end to sort out the mess (as  
he would have had to do anyway even if it had not been downgraded).

> Downgraded: Content-Type: Base-64(UTF-8 header field body)

If you want to give examples, then please give examples that are valid  
MIME, otherwise it is impossible to see what you are getting at. There is  
no such Content-Type as "Base-64".

The only things that need to be considered with a Content-Type that occurs  
in front of some part of a Multipart are the following:

    -- good boundary
    Content-Type: text/plain; name="mañana" (mañana is Spanish)

{Note, I am writing in Latin-1 here, but assume that UTF-8 was intended}

That will downgrade to

    Content-Type: text/plain; name*=utf-8''ma%C3%B1ana
        (=?utf-8?Q?ma=C3=B1ana?= is Spanish)

The <parameter> was encoded according to RFC 2231, and the <comment> was  
encoded accoerding to RFC 2047. There was NO need to a Downgraded header.  
There is no possibility of any  "--lost boundary". All existing agents  
should handle that header correctly, and existing MUAs should be able to  
display it in upgraded form if they implement 2231 and 2047 correctly and  
have the capability to display utf-8 (not that they are usually called  
upon to display body mime part headers).

So it you think there is some other problem with Content-* headers, then  
please produce a valid example.

>> that they specify how they should be downgraded.

If anybody standardizes a new header, they are welcome to define a special  
downgrade mechanism for it (which may or may not include creating a  
Downgraded header as well). However, already written UTF8SMTP-capable  
agents will not instantly be able to use that downgrade mechanism, and  
will just replace it by a Downgraded header, so people had better not  
invent new headers where such existing agents would cause problems.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 14:12:22 +0200

> In practice, I think that leaves us with "remove or encoded-word".

The point of my articles was that "2047 encoded word" is impossible.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Apr 2007 14:10:33 +0100

> MTA has to have conversion tables from Just About Everything to

If an UTF8SMTP-comploiant MUA generates some headers with RFC 2047
encoding in is0-8859-* and some headers in raw UTF-8 (or even both in the
same header), then that is rather stupid even if it is legitimate.
However, I think it safest for a subsequent downgrading agent to leave it
that way and just encode the raw utf-8 bits (see my reply to Frank). Not
that re-encoding everything in UTF-8 would not necessarily be wrong, but
it would probably lead to even more complications.
> trash for anyone who didn't have an upgraded receiving MUA.

I think we have to accept that it will not be possible for an upgrader to
distinguish between stuff that was encoded by the originating MUA and
stuff that was encoded by an intermediate downgrader. I am not sure that
this matters, except possibly when some DKIM-like signatures were involved
(and even those are probably solvable by some special canonicalization
mechanism, but not until this WG has completed its primary tasks).

And making the Downgraded header encode _exactly_ what was in the original
header will lead to double encodings, which can be hideously difficult to
disentangle, and so are better avoided.

> Right. Very good example.

Actually no. A very bad example. See my reply to Frank.
> from the beginning, but I know I'm in a tiny minority about

You are seeing problems where none exist. RFC 2231 (assuming it is
implemented at the receving end - and if it isn't it will have problems
even without UTF8SMTP) is a complete solution, and is still valid within
MIME 1.0.

> that they couldn't even interpret Content-type headers unless

Yes, that is why I don't want to invent MIME 2.0, and just to allow
extensions to MIME 1.0 that are only allowed to be seen within the
UTF8SMTP universe, and can be downgraded to things that are then
strict-current-MIME-1.0.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 15:08:29 +0200

Charles Lindsey wrote:

> If you read RFC 2047 carefully, you should always be
> able to discover at least one valid way of doing it.

NAK, only if you know the structure.  Modifying one of
my torture examples (read raw Latin-1 as raw UTF-8):

| xxx: (=?utf-8*nds?q?ä?= ö =?utf-8*de?q?=FC?=)

In an unstructured field that's no comment, and there
are no encoded words.  In a structured field it might
be a comment (not necessarily, it could be a part of a
quoted string, or a field not allowing comments, etc.)

Let's assume that it's a structured comment, then it
contains one encoded word =?utf-8*de?q?=FC?=

A 2047-downgrade mechanism would have to pick either
=?utf-8?q?=F6_?= or =?utf-8?q?=F6?= to preserve the
space to its right.  (Or equivalent 'B' encodings)

It must not start with _=F6, as that would insert a
space to its left (in addition to the one SP that's
already there).

Last but not least I don't trust that any downgrade
mechanism based on 2047 gets the non-encoded "word"
(=?utf-8*nds?q?ä?= right.

It's just too tricky, even if we'd decree "treat all
UTF-8 header fields as unstructured for the purpose
of 2047-downgrading" it's still too tricky.

And hopeless for raw non-UTF-8 octets.  Downgrade
needs its own mechanism, hex. NCRs after protecting
all input "&" is an idea.  Or the perl mechanism
mentioned in John's "escape" I-D, but in that case
I don't know the proper way to protect input using
this style.

> What is an NCR?

A numerical character refeence like &#xA0; for u+00A0.

NCRs are quite popular because they're supported in XML
(and FWIW in HTML) and almost all browsers.  The last
browser insisting on decimal instead of hexadecimal
NCRs is under my firm control.

NCRs (or the perl-style) won't help us with non-UTF-8
octets, but we could use a mixed downgrade strategy:

If possible (valid UTF-8) use hex. NCRs, otherwise
use "B64 everything".

 [non-UTF-8]
> simply encode the octets as if they were UTF-8

You're kidding.  You'd have to use UNKNOWN-8BIT.

> please produce a valid example.

It was an example showing why a "B64 everything"
strategy won't fly for MIME part headers (Content-*).

Anything Downgraded: Content-[...] is impossible in
a MIME part header.  The real error is the attempt
to touch any bit in the body.  8-to-7 bits magic is
no job for EAI.  Good 8-to-7 bits magic should be on
standards track, or deserves its own experiment.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Apr 2007 14:23:04 +0100

> ignorable or decodable by MUAs that don't support UTF8SMTP.

However, using downgrades via 2231 or 2047 should be acceptqable to  
existing MUAs. Moreover, most existing MUAs, when faced with a header they  
do not recognize but observing apparent <encoded-word>s within them will  
try to decode them before display, so hopefully even a Downgraded header  
will be displayed in a readable form.
> In practice, I think that leaves us with "remove or encoded-word".

I disagree. Encoded-words in places where they are expected will surely  
work. In places where they might not be expected (as in the Downgraded  
header) they will probably work. Stuff encoded via 2231 should be handled  
correctly (if the MUA has bothered to implement it). So I see no situation  
where "remove" would be needed.

In the worst case, the reader will see the encoded-form and will have to  
decide whether the header in question is important enough for him to  
bother trying to disentangle it (to which the usual answer will be "no, it  
isn't").

---

From: John C Klensin <[email-masked]>
Date: Fri, 13 Apr 2007 09:42:34 -0400

> UTF8SMTP universe, and can be downgraded to things that are

But, if they are used and appear --even in encoded form-- on the
other side of a downgrade process, they are no longer "in the
UTF8SMTP universe" but in a universe in which they may not have
a clear and unambiguous interpretation.  Please see Harald's
recent note.

And, FWIW, I don't share your optimism about putting things that
look like encoded words into fields that the receiving system
may not understand and assuming that the right things will
happen.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Apr 2007 21:35:29 +0100

On Fri, 13 Apr 2007 14:08:29 +0100, Frank Ellermann  
<[email-masked]> wrote:

> are no encoded words.

Yes, for two reasons:
   1. The '(' and ')' should have been inside the (alleged) encoded-words;
   2. One of the (alleged) encoded-words contains a non-ASCII character.

So a really Bloody-Minded downgrader, when it saw that in a Subject, would  
say to itself "there are no encoded-words there, so obviously the sender  
wanted his Subject to contain all those "=?" and "?=" and "utf-8"s to be  
rendered just as you see them above, so it would downgrade them to

{Note that, although I am nominally downgrading from utf-8, I am actually  
showing hex for the Latin-1 characters, just as you did, because it is too  
much hassle to look up the utf-8 encodings.}

    Subject: =?utf-8?q?(=3D=3Futf-8*nds=3Fq=3F=e4=3F=3D_=F6_?=
             =?utf-8?q?=3D=3Futf-8*de=3Fq=3F=3DFC=3F=3D)?=

However, it is more than likely that the sender (or rather his broken  
software) really did intend those to be encoded words, and most MUAs take  
a more liberal view and attempt to decode anything that looks remotely  
like an encoded-word. For sure, they ignore reason #1, and quite likely  
reason #2 (certainly, upon offering your original example, and many  
variations on it, in a Subject to my own MUA caused it to display "(ä ö  
ü)". So I would not care overmuch if a downgrader tried to be as liberal  
as typical MUAs are and produced something like

    Subject: =?utf-8*nds?q?(=e4?= =?utf-8?q?_=f6_?= =?utf-8*de?q?=FC)?=

> space to its right.  (Or equivalent 'B' encodings)

No it wouldn't, because it would try to get both the ä and the ö within  
the same encoded-word.

So a Bloody-Minded downgrader would produce

    (?utf-8?q?=3D=3Futf-8*nds=3Fq=3F=e4=3F=3D_=F6_?= =?utf-8*de?q?=FC?=)

And a more liberal one would do much the same as my liberal Subject  
version. As I said before, I would not be particularly upset it  
downgraders did pedantically incorrect things which most typical MUAs  
regularly do anyway, hence making it more likely that any final and  
liberal upgrader would show what liberal MUAs already regularly show.

So, essentially, I still claim that a Bloody-Minded downgrader can still  
always find a pedantically correct way to encode any utf-8 stuff whilst  
leaving genuine pre-existing encoded-words untouched, and a liberal  
upgrader (much easier to write) can still get things at least as correct  
as the average MUA gets them.

> (=?utf-8*nds?q?ä?= right.

That may well be the case but, as I have said, they will get it as right  
as typical MUAs are likely to get it.
> of 2047-downgrading" it's still too tricky.

Well we aren't proposing that. If some downgrader genuinely knows which  
headers are unstructured and which not, then good luck to it. But if it  
doesn't, then we should also realise that most MUAs don't know that  
either, and so not to worry if it takes similar shortcute as typical MUAs  
already do.
> all input "&" is an idea...
>
>> What is an NCR?
>
> (and FWIW in HTML) and almost all browsers.

But not by MUAs I hope. But even if some MUAs do implement them, UTF8SMTP  
should take no special action on them. If some MUAs now happen to do the  
"right thing" with some UTF-8 encoded that way, then good luck to them,  
but let us not get involved.

> You're kidding.  You'd have to use UNKNOWN-8BIT.

Pedantically, they should do as you say, but in the Real World that is  
unlikely to happen (since we can have no control over what is done with  
inadmissable input).

> strategy won't fly for MIME part headers (Content-*).

It was not a valid example that could actually happen. Therefor it proved  
nothing.
> a MIME part header.

But there is never any necessity, with any proposal currently before us,  
for a Downgraded header to appear in a MIME part header. All such headers  
are supposed to start with Content-*, and all known Content-* headers can  
be downgraded withour creating a Downgraded header. So no problem.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Apr 2007 22:43:42 +0100

> a clear and unambiguous interpretation.  Please see Harald's

Not so. The only downgrades that can arise on Content-* headers are if the  
header contains a comment (they never do, of course), or if they contain a  
<parameter> of the form filename="mañana". There is never any need to  
generate a Downgraded header as well.

In the one case we then get an RFC2047-encoded <comment>, and in the other  
case an RFC 2231-encoded <value>, and existing software is supposed to be  
able to handle both of those situations without problem.

The only other situation where we would allow things not envisaged by  
current MIME 1.0 is if we permit UTF8SMTP messages to appear with a  
message/rfc822 (but only within the UTF8SMTP universe).

In that case, the downgrade process needs to descend through the complete  
MIME structure (which it will have to do in any case if we do not have  
that HRD=UTF8 parameter in order to ensure that no MIME body part headers  
have any UTF8 in them). Whenever it encounters a message/rfc822 it will  
then downgrade it, and so on recursively.

So, in the non-UTF8SMTP universe you will then see message/rfc822 objects  
that are in fact downgraded from UTF8SMTP. But then you already expect to  
see messages of that sort in the non-UTF8SMTP universe anyway, so what's  
new?
> may not understand and assuming that the right things will

It is no worse than what happens in the present ASCII universe when  
encoded-words turn up in header fields that we invented after the piece of  
software was written (nearly all current software is liberal enough to do  
the "right thing" in those cases).

---

From: [email-masked]
Date: Tue, 17 Apr 2007 21:28:41 +0900 (JST)

> From: Frank Ellermann <[email-masked]>

I changed the Downgraded header definition in my local downgrade-04 text.

downgraded =  "Downgraded:" [FWS] field-name ":" unstructured CRLF
And added as 'this header will be encoded by RFC 2047.'

> old pre-MIME Content-Type was also ASCII.

Adding UTF-8 capable MIME header list to utf8header document is one idea.

> > From: Sender: Reply-To: To: Cc: Bcc:

agree.

> Strike that, it doesn't exist.

Ok, I checked RFC 2822 Appendix B 26.

> Keywords isn't unstructured, <obs-phrase> can have a comment,

Encapsulating Keywords in Downgraded header is reasonable?

--
Kazunori Fujiwara, JPRS

---

From: Bill McQuillan <[email-masked]>
Date: Tue, 17 Apr 2007 09:20:15 -0700

> agree.

Disagree!

RFC 2822 - 3.6.3 states that the "BCC:" field may be used in several ways,
including sending each blind recipient a copy of the message with only
his/her address in the "BCC:" field or possibly sending only to the blind
recipients a copy of the message with a "BCC:" field containing all of the
blind recipients' addresses.

-- 
Bill McQuillan <[email-masked]>

---

From: [email-masked]
Date: Wed, 18 Apr 2007 02:27:36 +0900 (JST)

> From: Bill McQuillan <[email-masked]>
> 

My 'agree.' is not good...
In many cases, Bcc: header is removed by MUA.

> blind recipients' addresses.

and more, RFC 2822 ABNF says:
bcc             =       "Bcc:" (address-list / [CFWS]) CRLF

empty Bcc: value is allowed...

I considered Bcc is not exist or have some value.

Thank you.

--
Kazunori Fujiwara, JPRS

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 17 Apr 2007 22:37:16 +0200

> Disagree!

When downgrading gateways see it it should be empty, RFC 2821 says:

| Each recipient address from a TO, CC, or BCC header field SHOULD
| be copied to a RCPT command (generating multiple message copies if
| that is required for queuing or delivery).  This includes any
| addresses listed in a RFC 822 "group".  Any BCC fields SHOULD then
| be removed from the headers.  Once this process is completed, the
| remaining headers SHOULD be checked to verify that at least one
| To:, Cc:, or Bcc: header remains.  If none do, then a bcc: header
| with no additional information SHOULD be inserted as specified in
[RFC 2822].

| Addresses that do not appear in the message headers may appear in the
| RCPT commands to an SMTP server for a number of reasons.  The two
| most common involve the use of a mailing address as a "list exploder"
| (a single address that resolves into multiple addresses) and the
| appearance of "blind copies".

Admittedly RFC 822 and 2822 mention a problematic case, where the Bcc
is either kept as is (only in the blind carbon copies), or trimmed to
one corresponding address per blind carbon copy.  The RFC 2821 recipe
appears to be more robust.

Frank (waiting how long it takes the "headers" police to find 2821bis)

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 18 Apr 2007 11:20:03 +0100

> When downgrading gateways see it it should be empty, RFC 2821 says:

I think the safest thing to say is the a Bcc is downgradable, just to  
cover the occasional circumstance where it turns up, but point out that  
there will rarely be anything there to downgrade.
> | that is required for queuing or delivery).

But it may be the case, for example, that a MUA that finds itself talking  
to a non-UTF8SMTP MTA will do the downgrade operation _before_ it sets  
about generating RCPT commands for the MTA.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 18 Apr 2007 11:25:49 +0100

> Encapsulating Keywords in Downgraded header is reasonable?

keywords        =       "Keywords:" phrase *("," phrase) CRLF

It should be allowed to downgrade a <phrase> anywhere that it occurs using  
RFC 2047, and without any need for an additional Downgraded header.

We already do that for <display-name>s, which are the only other place  
where <phrase>s get used according to RFC 2822.