Subject: [EAI] ISSUE: Trivial downgrading

From: John C Klensin <[email-masked]>
Date: Tue, 05 Jun 2007 15:16:57 -0400

Hi.

Kari and I have been having an offlist discussion that has 
inspired a thought.

Suppose one has a message that has UTF-8 headers, but there are 
no non-ASCII addresses in the headers (either forward or 
backward-pointing).  This may actually turn out to be a common 
case for people who stick with ASCII-addresses (either for 
interoperability or because, e.g., their names don't happen to 
require non-ASCII characters even if their languages are written 
using extended "Latin" scripts and do).   But, if they are using 
UTF8SMTP-aware MUAs and Submission servers (MSAs), they are 
likely to be producing non-ASCII subject lines (as is the case 
today), but sending them as UTF-8 rather than as encoded words. 
There are some other cases, but this is the important one.

At present, our rules appear to sent such messages through the 
entire downgrade process, generating Downgrade headers, etc.

I wonder if, to make that class of cases efficient and maximize 
its utility, we ought to have a different downgrade path that 
simply:

    * Converts the known header fields containing UTF-8 data
    into the same fields with encoded words.

    * Documents the conversion only in trace information (i.e.,
    no "Downgraded:" headers).

    * If any body parts or information require downgrading,
    treat them strictly according to the 8BITMIME rules, which
    would obviously apply.

This would, of course, apply only if no non-ASCII addresses were 
present and all of the header fields that contained UTF-8 data 
were known (see below).  The present downgrade procedure, and 
the conditions associated with it, would continue to apply to 
all other cases.

The reason for a "known header" restriction is fear of 
information loss if we start tampering with headers that are not 
recognized, i.e., those for which explicit rules about encoding 
of non-ASCII information does not exist already.  But the net 
effect of this approach would be to downgrade by recreating 
_exactly_ the types of messages that can be sent today (without 
the EAI extension), with no new headers or other material, from 
messages that are just the UTF-8 rendering of their encoded 
words ... almost exactly symmetric with the intent of the 
8BITMIME transformations.

Having two downgrade styles does add complexity, but because 
this one is simple enough and has no expectation about anything 
UTF8-aware on the far side of the downgrade (even aware enough 
to pass Downgraded header fields), maybe it would be more widely 
implemented and perceived as worth the trouble.

Thoughts?   I think that, if there is interest, Kari and I could 
write it up fairly quickly (for the record, he doesn't know that 
I'm volunteering him).

      john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Wed, 06 Jun 2007 11:51:46 +0900

I haven't thought through every single detail of this,
but I very much agree that it should be looked into further.

Regards,    Martin.s

At 04:16 07/06/06, John C Klensin wrote:
>Hi.
>
>Kari and I have been having an offlist discussion that has inspired a thought.
>
>Suppose one has a message that has UTF-8 headers, but there are no non-ASCII addresses in the headers (either forward or backward-pointing).  This may actually turn out to be a common case for people who stick with ASCII-addresses (either for interoperability or because, e.g., their names don't happen to require non-ASCII characters even if their languages are written using extended "Latin" scripts and do).   But, if they are using UTF8SMTP-aware MUAs and Submission servers (MSAs), they are likely to be producing non-ASCII subject lines (as is the case today), but sending them as UTF-8 rather than as encoded words. There are some other cases, but this is the important one.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Kari Hurtta <[email-masked]>
Date: 06 Jun 2007 07:35:44 +0300

John C Klensin <[email-masked]> writes in gmane.ietf.ima:

> Suppose one has a message that has UTF-8 headers, but there are no
> non-ASCII addresses in the headers (either forward or
> backward-pointing).  This may actually turn out to be a common case
> for people who stick with ASCII-addresses (either for interoperability
> or because, e.g., their names don't happen to require non-ASCII
> characters even if their languages are written using extended "Latin"
> scripts and do).   But, if they are using UTF8SMTP-aware MUAs and
> Submission servers (MSAs), they are likely to be producing non-ASCII
> subject lines (as is the case today), but sending them as UTF-8 rather
> than as encoded words. There are some other cases, but this is the
> important one.

Yes. 

Some background (some of these I have mentioned ealier, but not all). 
I have noted:

    I think that case, where message is turned originally (when sent)
    to UTF8SMTP only because subject was UTF-8, should be transparent.

    This means that, when message is later forwarded as mime attachment
    (with type message/rfc822 or message/utf8smtp), this mime attachment
    need to be downgraded to regular RFC2822 message/rfc822 on border
    of UTF8SMTP universe.

That non-ASCII Subject often occur on Finnish mail (grep from one
folder -- most of mails (but not all) are Finnish:
hurtta$ formail -c < List/grp | fgrep Subject:  | fgrep '=?' | wc -l
   1308
hurtta$ formail -c < List/grp | fgrep Subject:  | wc -l
   3815
hurtta$ 
)

Non-ASCII letters, which occurs on Finnish, are ä and ö (mostly ä).

These (ä and ö) are are not very often used on names. It is is hard to 
estimate. It is more common on surname.

Most common first names (given names) on Finland:
            https://192.49.222.187/Nimipalvelu/nimipalvelu_etunimitop.asp?L=3
on 10 most common there is no ä or ö

Most common surnames (family names) on Finland:
                https://192.49.222.187/Nimipalvelu/nimipalvelu_sukunimitop.asp?L=3
on 10 most there names there is 4 with ä

So it is likely that UTF8SMTP mail addresses are never very common.

This combination makes likely that great part of UTF8SMTP messages because
of UTF-8 Subject and not UTF8SMTP address.

( And it is likely that mail server is changed according on what
  network you are connected. So MSA can suddenly come to UTF8SMTP aware :-)
  Port 25 is closed by order of Finnish Communications Regulatory Authority 
  (FICORA) on consumer subscription line.  (That order of course do not
  prevent sending mail via home server with port 587 or 465 (non standard)). )

Using of UTF8SMTP on MUA may be configuration switch or automatic.
I noted earlier on one message:
     MUA which is used to send message is UTF8SMTP capable.
     Because UTF8SMTP is negotiated proprerty, there is no 
     "use UTF8SMTP" switch on MUA. That just confuses users.

This perhaps gives some perpective, how I look situation :-)

> Thoughts?   I think that, if there is interest, Kari and I could write
> it up fairly quickly (for the record, he doesn't know that I'm
> volunteering him).
> 
>       john

Hmm. Now I know...   That "Trivial Downgrading" did not occured on discussion :-)

/ Kari Hurtta

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 07 Jun 2007 07:27:17 +0100

John C Klensin wrote:

> At present, our rules appear to sent such messages through the entire 
> downgrade process, generating Downgrade headers, etc.
>
> I wonder if, to make that class of cases efficient and maximize its 
> utility, we ought to have a different downgrade path that simply:
>
> present and all of the header fields that contained UTF-8 data were 
> known (see below).  The present downgrade procedure, and the 
> conditions associated with it, would continue to apply to all other 
> cases.

This seems sensible. The only tricky bit is properly describing the two 
types of messages.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 07 Jun 2007 14:06:28 +0100

On Tue, 05 Jun 2007 20:16:57 +0100, John C Klensin <[email-masked]> wrote:

> backward-pointing).  This may actually turn out to be a common case for  
> people who stick with ASCII-addresses (either for interoperability or  
> because, e.g., their names don't happen to require non-ASCII characters  
> even if their languages are written using extended "Latin" scripts and  
> do).   But, if they are using UTF8SMTP-aware MUAs and Submission servers  
> (MSAs), they are likely to be producing non-ASCII subject lines (as is  
> the case today), but sending them as UTF-8 rather than as encoded words.  

No they don't.
>     no "Downgraded:" headers).

And that is EXACTLY what the current downgrading draft says, so what are
you proposing that is new?

And it is also what happens with <comment>s in ANY header (assuming there
is no other UTF-8 in that header).
>     treat them strictly according to the 8BITMIME rules, which

But that bit is not so clear to me. Suppose some body part has a
Content-Description header, or a Content-Disposition with a UTF-8
filename. The 8BITMIME rules cannot change that, but our downgrade
document covers that case (you use RFC 2047 or RFC 2231, with NO
Downgraded header). Granted that the code you will find in well-written
MTAs that does 8BITMIME downgrading can easily be adapted to do this extra
job.

Again, no evidence is left in the final message to indicate that
downgrading has taken place, but who cares?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: [email-masked]
Date: Tue, 19 Jun 2007 14:38:35 +0900 (JST)

Is partial downgrading a solution for the trivial downgrading?

No downgrading is always allowed.

>     into the same fields with encoded words.

      * Don't touch trace headers. If trace headers contain
	non-ASCII characters, the downgrading fails.

      * Don't touch unknown headers. If unknown headers contain
	non-ASCII characters, the downgrading fails.

      * Don't touch address headers except in comments.
        If any address header contains non-ASCII addresses,
	the downgrading fails.

      * Don't touch the MIME body part headers.
        If the MIME body part headers contain non-ASCII characters,
	the downgrading fails.

This is one of trivial downgrading, I think.

Is it true?

--
Kazunori Fujiwara, JPRS

> From: John C Klensin <[email-masked]>
> Hi.
>

---

From: Martin Duerst <[email-masked]>
Date: Tue, 19 Jun 2007 14:52:54 +0900

I think the difference between John's proposal and this one
is that the way I understood it, John is proposing that there
are two ways of downgrading, and that in many cases, mail
servers would implement both. The list below suggests that
a server in general would choose just one.

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Tue, 19 Jun 2007 15:30:44 -0400

> a server in general would choose just one.

Martin,

I think Fujiwara-san is correct and that "trivial" is just one
special case.  I apologize for any additional confusion I may
have contributed.

There are still questions here that I think the WG needs to
consider.  For example:

(1) Is the explanation in the downgrading document clear enough
as to what is permitted, what is permitted to go in (e.g., that
downgrading is not possible if any of the header addresses are
non-ASCII and do no have ASCII forms), and what comes out under
various circumstances.  My personal belief is that the answer is
"no" with the current posted version.  The editors (design) team
is working on that issue, but I hope that, once the next version
is posted, other participants in the WG will make suggestions
about how clarity can be further improved where that is
appropriate.

(2) Are there sufficient options in, and paths through, the
downgrading process, depending either on the input or on server
choices, to cause an interoperability or "user surprise" problem
and, if so, is there a way to reduce them?  I think the answer
is "no", but that is just my opinion of the moment.  It is clear
to me that making downgrading work cleanly in the maximum number
of cases, without either causing long-term harm in a world after
most mail systems have been upgraded or delaying upgrading, is
the largest challenge facing this WG; I'm not sure we are in the
right place on that path yet.

best,
   john

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 20 Jun 2007 10:51:40 +0100

> a server in general would choose just one.

But since, in fact, what John is proposing seems to me to be identical to
what is already in the downgrade drafts, I am at a loss to understand what
difference will actually exist. Unless I have missed some feature of
John's idea.

---

From: Randall Gellens <[email-masked]>
Date: Mon, 25 Jun 2007 17:13:45 -0700

>    prevent sending mail via home server with port 587 or 465 (non 

This is a bit off-topic, but why would the MSA change depending on 
the connection network?  Generally, the idea of submission as a 
separate service is that one always submits using one's home MSA, 
regardless of how one is connected to the Internet.

So, if my home MSA is the one run by my usual ISP, but I happen to be 
travelling and using an 802.11 connection in a coffee shop, I still 
submit through my usual ISP's MSA.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Wealth is not only money.
Wealth can be happiness in your sweetheart or your honey.
  --(Fortune cookie)