Subject: Re: [EAI] Status of utf8header draft

From: Randall Gellens <[email-masked]>
Date: Wed, 17 Jan 2007 18:14:20 -0800

At 1:02 PM +0800 12/21/06, Jeff Yeh wrote:

>  Hi all:
>
>  While I'm trying to update the header draft, there comes some 
> issues (solved/un-solved). Listed below:
>
>  * Removing the "Header-Type" header (was UTF8SMTP)
>  The basic idea to have a "Header-Type" new header is for 
> identifying a email is EAI or non-EAI. And the new header could 
> also carry some useful information like downgrade 
> information/language tag. However, this kind of self-labeling 
> mechanism was found useless as the history of "MIME-Version" 
> header. For the robustness of email software, programmers usually 
> ignore the "MIME-Version" header, directly check the presents of 
> MIME headers and the MIME encoded stuff. This is also happened in 
> TWNIC's implementation on EAI testbed, "Header-Type" header is 
> ignored, we check the presents of UTF8 instead. Thus, the demand of 
> the "Header-Type" is being raise up again.
>  Although we already have consensus about this point on San Diego 
> meeting, it's still necessary to bring on the mailing list for 
> everyone's opinion.
>  <some quote from jabber scribe>
>  --harald offers choices:
>       a) leave UTF8SMTP header field in:one vote
>       b) take it out: 10 votes
>       c) mime-version 2.0: none
>       d) can't decide: 5 votes
>  --harald: will take proposal to list to take it out
>  </some quote from jabber scribe>

Defining the header introduces a silly state where the header doesn't 
match reality.

(1) If the header says "ASCII" or isn't present, the message could 
still contain UTF8.

(2) If the header says "UTF8" the message might be all ASCII.

The effect of (1) would potentially be bad.  The effect of (2) is 
either harmless, or in the worst case, results in a message being 
bounced that could have been delivered.

Because of (1), receiving systems MUST NOT trust a a label of 
"Header-Type: ASCII".  If a message is received that claims to be 
ASCII, or isn't labelled, the receiver still needs to examine the 
message for non-ASCII characters, as well as other conditions that 
make the message invalid.

So, I'd be OK with leaving the header, provided we add text saying 
that a value of "ASCII" mustn't be trusted (and even with a value of 
"UTF8" the message should still be checked to be sure it is otherwise 
valid).

>  * The alt-addr syntax
>  There is a roughly consensus on <UTF8 <ASCII>>, but still some 
> concern on parser updating effort. By TWNIC's EAI implementation 
> experience, <UTF8 [ASCII]> working just fine with a little bit 
> change on parser. Though there is a possible conflict to 
> domain-literal, but the use of domain-literal is quite few, I think.

I'd prefer the embedded angle-brackets: "<utf8 <ascii>>".

>
>  * Trace Field
>  Solved, will update in the upcoming version.
>  "Received" must put in ASCII, "for" could ignore (optional). 
> "Return-Path" syntax need to update to allow UTF8 characters.

It would be a shame to throw away information containing in "for" 
(even though it is generally only used when a message has a single 
recipient, this is a very common case).  Did we discuss something 
like "for8" that would contain an encoded version of the address? 
(Using one of the array of encodings, such as base64 or 2047.)

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
One should always be a little improbable.          --Oscar Wilde

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Jeff Yeh <[email-masked]>
Date: Thu, 18 Jan 2007 11:37:06 +0800

> "UTF8" the message should still be checked to be sure it is otherwise 
I'm kind of agree to Randall. You have to scan over the headers for UTF8 
anyway, no matter the "Header-Type: UTF8" exists or not.
> recipient, this is a very common case).  Did we discuss something like 
> "for8" that would contain an encoded version of the address? (Using 
> one of the array of encodings, such as base64 or 2047.)
This idea is first introduced, I like it, it makes the whole solution 
more complete. However, if this is adopted, there will be another 
encoding (even it use 2047). I'm wandering this will make the solution 
more complicate.  And also, lots of mail server implementation does not 
insert "for" on trace fields, since it is optional in 2822 definition.

Jeff Yeh
-TWNIC

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 18 Jan 2007 08:40:22 +0100

Jeff Yeh wrote:
> I'm kind of agree to Randall. You have to scan over the headers for 
> UTF8 anyway, no matter the "Header-Type: UTF8" exists or not.
I'm more happy with dropping it. Adding more header cruft that we know 
to be useless is a Bad Thing.
> more complicate.  And also, lots of mail server implementation does 
> not insert "for" on trace fields, since it is optional in 2822 
> definition.
The RFC 2822 Received: format isn't extensible. (for 2047-encoding) is a 
perfectly valid comment, but that's the only trouble-free way of 
preserving the information I see - we shouldn't (I think) try to mandate 
the content of comments, but might suggest it.

                        Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 18 Jan 2007 15:27:18 -0000

>>> (1) If the header says "ASCII" or isn't present, the message could  

But whoever put the header there would be lying and deserves what he gets.

One of the reasons for having this header is to relieve agents from the  
burden of having to check the whole document for 8bit stuff that, if that  
header says "ASCII", is 99.99% unlikely to be there. Note that you have to  
check the body as well, if it is a MIME message, because there might be  
internal headers that really ought to be checked as well.

Much better to let "ASCII" be treated as ASCII, and if that causes the  
message to be munged/lost, then what's new? Garbage in - Garbage out.

>>> (2) If the header says "UTF8" the message might be all ASCII.

A waste of resources, and hence to be discouraged, but no harm arises.

>>> "UTF8" the message should still be checked to be sure it is otherwise  

-1

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Jan 2007 10:49:24 -0500

--On Thursday, 18 January, 2007 15:27 +0000 Charles Lindsey
<[email-masked]> wrote:

>>>> Defining the header introduces a silly state where the
>>>> header doesn't   match reality.
>>>> 
>>>> (1) If the header says "ASCII" or isn't present, the
>>>> message could   still contain UTF8.
> 

If "he" would "get" the consequences of his lies, that would be
fine.  Unfortunately, he lies and the receiving user pays the
penalty.

> One of the reasons for having this header is to relieve agents
> from the burden of having to check the whole document for 8bit
> stuff that, if that header says "ASCII", is 99.99% unlikely to
> be there. Note that you have to check the body as well, if it
> is a MIME message, because there might be internal headers
> that really ought to be checked as well.

Yep.   Because the robustness principle doesn't permit the
receiver to assume that the sender followed the rules if doing
so causes the receiver to exhibit severely bad behavior.  Again,
this isn't "sender pays the penalty for bad behavior" but "bad
behavior by the sender inflicts pain on the recipient".  The
recipient system needs to check --and check whatever needs to be
checked-- at least to the extent necessary to avoid consequent
bad behavior.

> Much better to let "ASCII" be treated as ASCII, and if that
> causes the message to be munged/lost, then what's new? Garbage
> in - Garbage out.

Munged doesn't bother me very much, although message corruption
without notice to the user is very bad news.  Lost does.  And
having the receiving MUA get into unrecoverable trouble as the
result of a badly-formatted incoming message is a terrible
situation.   Receiving MUAs have to protect themselves against
attacks.  If someone says "ASCII", the necessary protection will
almost always require scanning the entire message (perhaps
incrementally).   In that context, the only thing that having
the header permits will be the generation of a message that said
"sender lied, message was supposed to be all-ASCII and wasn't".
But common practice says that message won't be generated:
8bit-capable receiving systems will just make the corrections
and go on because, if the corrections can be made, the user
won't care what came in.

> A waste of resources, and hence to be discouraged, but no harm

But it is further evidence that the header is nearly useless.

>>>> So, I'd be OK with leaving the header, provided we add text
>>>> saying   that a value of "ASCII" mustn't be trusted (and
>>>> even with a value of   "UTF8" the message should still be
>>>> checked to be sure it is otherwise   valid).

Adding headers has a cost, sometimes a significant one.  If we
care about those headers getting through filters,
institutional-boundary firewalls, the IMAP (and lemonade?)
header abstraction, and so on, we have to persuade all sorts of
folks to revise their implementations, data structures, and
abstractions.  If we don't care enough (and maybe even if we
do), they every receiving implementation has to be prepared to
deal with the no-header case because the header may have been
trashed, making the value of the header even more marginal.  

I don't see that a persuasive case has been made that the header
has enough value to justify that cost.

     john

---

From: Randall Gellens <[email-masked]>
Date: Thu, 18 Jan 2007 12:02:30 -0800

At 8:40 AM +0100 1/18/07, Harald Alvestrand wrote:

>  Jeff Yeh wrote:
>>
>>  Randall Gellens wrote:
>>>  Defining the header introduces a silly state where the header 
>>> doesn't match reality.
>>>
>>>  (1) If the header says "ASCII" or isn't present, the message 
>>> could still contain UTF8.
>>>
>>> message for non-ASCII characters, as well as other conditions 
>>> that make the message invalid.
>>>
>>> saying that a value of "ASCII" mustn't be trusted (and even with 
>>> a value of "UTF8" the message should still be checked to be sure 
>>> it is otherwise valid).
>>  I'm kind of agree to Randall. You have to scan over the headers 
>> for UTF8 anyway, no matter the "Header-Type: UTF8" exists or not.
>  I'm more happy with dropping it. Adding more header cruft that we 
> know to be useless is a Bad Thing.

I'm fine with dropping it, but there was an objection posted to the 
list stating that without this header, downgrade information is lost. 
If we want to keep it, we need to be clear what it means.

>>> (even though it is generally only used when a message has a 
>>> single recipient, this is a very common case).  Did we discuss 
>>> something like "for8" that would contain an encoded version of 
>>> the address? (Using one of the array of encodings, such as base64 
>>> or 2047.)
>>  This idea is first introduced, I like it, it makes the whole 
>> solution more complete. However, if this is adopted, there will be 
>> another encoding (even it use 2047). I'm wandering this will make 
>> the solution more complicate.  And also, lots of mail server 
>> implementation does not insert "for" on trace fields, since it is 
>> optional in 2822 definition.
>  The RFC 2822 Received: format isn't extensible. (for 2047-encoding) 
> is a perfectly valid comment, but that's the only trouble-free way 
> of preserving the information I see - we shouldn't (I think) try to 
> mandate the content of comments, but might suggest it.

RFC 2822 defers to 2821 for the format.  2821 doesn't include an 
extension mechanism for new arbitrary fields, but presumably one 
could be added by a document which extended 2821.  I'm not sure that 
would be justified in this case, though.  The "protocol" field is 
extensible per 2821, but we already have a problem with an explosion 
of new protocol atoms defined for combinations of SMTP extensions. 
It would be good to define a structured "protocol" value that could 
indicate which relevant SMTP extensions were negotiated.

-------------- Randomly-selected tag: ---------------
We like because, we love although.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 19 Jan 2007 02:53:09 +0100

> for UTF8 anyway, no matter the "Header-Type: UTF8" exists or not.

As long as the deduced result is NOT message/rfc822 for raw UTF-8
header fields I also feel safe without a Header-Type: header field.

But we still need an explicit "downgraded" indicator, or don't we ?

>> Did we discuss something like "for8" that would contain an
>> encoded version of the address? (Using one of the array of
>> encodings, such as base64 or 2047.)

> This idea is first introduced, I like it

+1

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 19 Jan 2007 03:27:34 +0100

Randall Gellens wrote:

> 2821 doesn't include an extension mechanism for new arbitrary 
> fields, but presumably one could be added by a document which
> extended 2821.

There's an IANA registry for VIA and WITH types, but nothing
about new <Opt-info> clauses, tough.  If UTF8SMTP updates the
821-syntax for <mailbox> the <for>-clause would inherit this
updated syntax.

And smtpext-02 in fact updates <mailbox> (or rather its 2821
<Mailbox> counterpart), what am I missing here ?  

Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Jan 2007 22:12:05 -0500

--On Friday, 19 January, 2007 03:27 +0100 Frank Ellermann
<[email-masked]> wrote:

> about new <Opt-info> clauses, tough.

That was, if I recall, discussed and decided against during the
DRUMS process.   Part of the problem (but only part) is that the
four Opt-info components use keywords that are really
delimiters.  Had it been defined, long ago, so that 
   Opt-info  = ["," Via] ["," With] ["," ID] ["," For]
but it wasn't, and adding extra fields would risk rather
seriously fouling up existing conforming implementations.

> If UTF8SMTP updates the

As long as, and only as long as, the message was being past to
something that had accepted the use of the extended syntax via
the SMTPEXT option.

> <Mailbox> counterpart), what am I missing here ?  

Legacy implementations of 2821.

     john

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 19 Jan 2007 08:51:53 +0100

> <Mailbox> counterpart), what am I missing here ? 
The fact that downgraded UTF8SMTP messages will expect to be RFC2822 (no 
extensions) compatible.
And the fact that altering trace headers is considered a Very Bad Thing.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 19 Jan 2007 14:28:33 -0000

> so causes the receiver to exhibit severely bad behavior. ...

Well robustness is a fine thing, but if it comes at a price you have to  
decide whether the benefit is worth the price.

Let us look at a comparable example - 8BITMIME. Suppose some message  
announces Content-Transfer-Encoding: 7bit, and then proceeds to include  
8bit stuff further down. Or suppose it omits the 8BITMIME parameter in the  
MAIL FROM command and nevertheless sends 8bit stuff. Is current practice  
for agents that support 8BITMIME to check through the whole body looking  
for 8bit stuff before passing the message on to an agent which does not  
support 8BITMIME? That would be robust, but I doubt current agents go to  
that trouble.

But if 8BITMIME agents DO go to that trouble, then I accept your point.  
But if not, then how do you justify such a check in this case?

> without notice to the user is very bad news.  Lost does.

Actually, I think complete loss is less likely than munging, since an MTA  
is likely to reject if it is not given a RCPT TO that looks like something  
it knows where to send to. Unless it is storage agents (IMAP/POP3) that  
canot tell where to put it just by looking at the To header.

> But it is further evidence that the header is nearly useless.

No worse that writing Content-Transfer-Encoding: 8bit when 7bit would have  
done.

But this header was never intended to guide MTAs in how to process  
messages, since our smtpext has got that well covered. I wrote on 21st  
December giving 7 reasons why this header was needed, and they were all  
related to situations where the transprt mechanism was not smtp, or where  
the message was being processed after it has left the smtp enviroment (and  
hence the envelope information was no longer available). If you believe  
this head is useless, then please reply, point by point, to that message.
> abstractions.

If the message is beiong processed by UTF8SMTP-upgraded agents, then we  
can assume that they will be aware of this header, and will either make  
use of it, or at least ensure it is passed on to other agents that process  
the message. It is only when the message passes into a non-UTF8SMTP  
environment that this header might get lost, but at that point the message  
is supposed to be ASCII-safe anyway and that header will certainly not say  
"UTF8", so the opnly real loss is that some "downgraded" may get lost, and  
that may be a nuisamce but is not a show stopper.

>  If we don't care enough (and maybe even if we

But if it has been trashed, then the message has already been downgraded,  
so it does not matter. The use of the header is in environments which DO  
understand it, and have to decide whether further processing of it (e.g.  
forwarding it to some alias) has to be done in a UTF8-safe manner.  
Remember that even UTF8SMTP-capable agents will still have a lot of pure  
ASCII messages to handle as well.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 19 Jan 2007 21:06:58 +0100

John C Klensin wrote:

> the four Opt-info components use keywords that are really
> delimiters

Yes, we better don't touch this <opt-info> set.

>> what am I missing here ?

> Legacy implementations of 2821.

That could be reduced to a relatively simple MUST in the
downgrading instructions.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 19 Jan 2007 21:00:37 +0100

Harald Alvestrand wrote:

  [<for> inherits <mailbox>]

> The fact that downgraded UTF8SMTP messages will expect to be
> RFC2822 (no extensions) compatible.

That limits the "for8" problem to removing the optional <for>
clause in the downgrade procedure.

> And the fact that altering trace headers is considered a

Forbidding <for> everywhere in UTF8SMTP is also very bad.  One
of the UTF8SMTP axioms is that downgrading will rarely happen.

I've my doubts about this axiom.  But if we use it for some
decisions we can as well use it consistently whenever it helps:
"remove the optional <for> if necessary" is no rocket science.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 25 Jan 2007 08:46:54 +0200

Randall Gellens <[email-masked]> writes in gmane.ietf.ima:

> > identifying a email is EAI or non-EAI. And the new header could also
> > carry some useful information like downgrade information/language
> > tag. However, this kind of self-labeling mechanism was found useless
> > as the history of "MIME-Version" header. For the robustness of email
> > software, programmers usually ignore the "MIME-Version" header,
> > directly check the presents of MIME headers and the MIME encoded
> > stuff. This is also happened in TWNIC's implementation on EAI
> > testbed, "Header-Type" header is ignored, we check the presents of
> > UTF8 instead. Thus, the demand of the "Header-Type" is being raise
> > up again.

IMHO

        Header-Type: UTF8

may be replaced with ESMTP option

        HDR=UTF8

But 
        Header-Type: Downgraded

may be still needed.

Yes,

        Header-Type: ASCII

is not needed.  There may be
        HDR=ASCII
for UTF8SMTP capable MTAs. I do not
think that silly state is more problem
than on case
        BODY=7BIT

When only

        Header-Type: Downgraded

is needed, perhaps it need to be renamed.

Various parameters from Header-Type header field
can be put to own header fields.

Only question what class
        
        Header-Type: UTF8

may need to answer, that is headers of
mime subparts UTF8. And 

        Header-Type: UTF8

do not answer that. In here case c

        Mime-Version: 2.0

may make some sense :-)

I do not think that MTAs scan whole
mail and parse MIME structure on general.

Only exception is that if BODY=8BITMIME
processiing tirggers it.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Jan 2007 11:43:06 -0000

> may be replaced with ESMTP option

Except when the message is being sent by non-SMTP mechanisms.
> is not needed.

It is only there for the sake of tidiness, and would not usually appear.  
But omitting Header-Type entirely means the same thing, and carries the  
same risks, if any.

>  There may be

Exactly. That would only ever be used for tidiness, and has no semantic  
effect.
> is needed, perhaps it need to be renamed.

But no, you still need the UTF8 form for non-SMTP, and for messages that  
have left the SMTP environment (for reading, storage, gatewaying to other  
media, whatever else).
> may need to answer, that is headers of

Yes. An encapsulated message/rfc822 might need to be downgraded, and the  
presence of that header would indicate that (or, rather, its absence would  
indicate a malformed message if there was 8bit stuff in it, and "garbage  
in - garbage out" would be justified).
> do not answer that. In here case c
[Smiley noted]
> mail and parse MIME structure on general.

Exactly. If they want to be excessively liberal, they can do it, but maybe  
the price of liberality is too much wastge of machine cycles.
> Only exception is that if BODY=8BITMIME

And nobody has yet answered my question as to whether current 8BITMIME  
downgraders bother to scan the whole message if that BODY=8BITMIME is  
absent.

---

From: Kari Hurtta <[email-masked]>
Date: 25 Jan 2007 16:57:30 +0200

> downgraders bother to scan the whole message if that BODY=8BITMIME is

I can only say about sendmail.  Sendmail detects if on mail there
is some bytes with 8-bit set.  That is done on receiving of mail.
That does not mean that mime structure is parsed.

But if there is 8-bit bytes and Mime-Version header field, then
following triggers (headers.c)

        /* check to see if this is a MIME message */
        if ((e->e_bodytype != NULL &&
             sm_strcasecmp(e->e_bodytype, "8BITMIME") == 0) ||
            hvalue("MIME-Version", e->e_header) != NULL)
        {
                e->e_flags |= EF_IS_MIME;
                if (HasEightBits)
                        e->e_bodytype = "8BITMIME";
        }

This does not depend is mail receibed via SMTP or is sendmail invoked
directly and mail is received from standard input of sendmail.

If next hop does not support 8BITMIME, that means normal 8BITMIME
processing, so on that situation mime structure is parsed (because
of downgrading). [1] 

/ Kari Hurtta

[1]  Actually this is not completely true. If MaxMimeHeaderLength
     or MaxMimeFieldLength is set, MMME structure is always parsed.
     These options are set by default starting from sendmail 8.11.7 
     Quote from RELEASE_NOTES

8.11.7/8.11.7   2003/03/29
<...>
        To provide partial protection for internal, unpatched MTAs that may be
                performing 7->8 or 8->7 bit MIME conversions, the default
                for MaxMimeHeaderLength has been changed to 2048/1024.
                Note: this does have a performance impact, and it only
                protects against frontal attacks from the outside.
                To disable the checks and return to pre-8.11.7 defaults,
                set MaxMimeHeaderLength to 0/0.

---

From: Randall Gellens <[email-masked]>
Date: Thu, 25 Jan 2007 19:27:43 -0800

>  may be replaced with ESMTP option

It seems to me that a HDR=UTF8 SMTP extension could only be used in 
the presence of UTF8SMTP, and isn't needed in that case, so I think I 
must be missing the intent of such an extension.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The attention span of a computer is only as long as its electrical cord.

---

From: Kari Hurtta <[email-masked]>
Date: 26 Jan 2007 06:38:39 +0200

> must be missing the intent of such an extension.

You missed earlier discussions.  Server announces UTF8SMTP.
SMTP client can send plain ASCII message still althoug server
announces UTF8SMTP.

1)  Client tells with HDR=UTF8 that it will use UTF8SMTP
   for mail headers.

2) UTF8SMTP extension is needed when MAIL FROM or RCPT TO is utf-8, but 
   headers are ascii. 
   
Therefore you can be

MAIL FROM:<[email-masked]> ALT-ADDRESS=[email-masked]
RCPT TO:<[email-masked]>
DATA
Received: by mta.example.com id k8DJRPxc015650; Wed, 13 Sep 2006 22:27:25 +0300
Date: Wed, 13 Sep 2006 22:27:25 +0300
From: [email-masked]
To: [email-masked]
Subject: Esimerkki posti

jotain.
.

Precense of ALT-ADDRESS does not indicate that headers are UTF-8.

So I disagreed with "isn't needed in that case" part with you.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 26 Jan 2007 07:09:27 +0200

> Except when the message is being sent by non-SMTP mechanisms.

That does not prevent these non-SMTP mechanisms to have some own
indication. Of course thay need some.

But there is also need that UTF8SMTP messages are not given
to non-UTF8SMTP aware agents.   Header-Type: UTF8   does
not imply that kind negation, so something other is
needed anyway on non-SMTP mechanisms.

Then there is problem that if UTF8SMTP messages are stored
on /var/mail/{user} -- they are not supposed to be seen on
non-UTF8SMTP aware mail user agents. That storage to
/var/mail/{user} or equivalent is one of these non-SMTP 
mechanisms.

Then there is POP and IMAP mechanism. Yes, they need also some
indication, but they also need some negation that UTF8SMTP messages
are not seen by non-UTF8SMTP aware mail user agents.

If header field Header-Type: UTF8 is dropped, then
draft-ietf-eai-imap-utf8 and draft-ietf-eai-pop need
invent some other mechanism :-)

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 11:46:45 -0500

>...
> Let us look at a comparable example - 8BITMIME. Suppose some
> message announces Content-Transfer-Encoding: 7bit, and then
> proceeds to include 8bit stuff further down. Or suppose it
> omits the 8BITMIME parameter in the MAIL FROM command and
> nevertheless sends 8bit stuff. Is current practice for agents
> that support 8BITMIME to check through the whole body looking
> for 8bit stuff before passing the message on to an agent which
> does not support 8BITMIME? That would be robust, but I doubt
> current agents go to that trouble.

Actually several do make that check, but not necessarily where
you think they do.  Others implement 8BITMIME as a variation on
"just send 8" and basically ignore the parameter, treating all
incoming messages as if they were 8bit and forwarding them to
the next-hop relay regardless of whether it advertises the
capability  or not.  Of course, they are in violation of the
spec.

When the check is made, it is at least as often made by scanning
the input data stream as it arrives, looking for characters with
the high-order bit set, as it is made as part of an analysis of
the body of a received message.
 
>...

>>> causes the message to be munged/lost, then what's new?
>>> Garbage in - Garbage out.
>> 
>> Munged doesn't bother me very much, although message
>> corruption without notice to the user is very bad news.  Lost
>> does.
> 
> Actually, I think complete loss is less likely than munging,
> since an MTA is likely to reject if it is not given a RCPT TO
> that looks like something it knows where to send to. Unless it
> is storage agents (IMAP/POP3) that canot tell where to put it
> just by looking at the To header.

Any "storage agent" that figures out where to put mail by
looking at the To header is seriously and fatally broken.
Storage locations and other delivery actions have to be
determined by envelope processing and nothing else.

> No worse that writing Content-Transfer-Encoding: 8bit when
> 7bit would have done.

I disagree, and this is, IMO, fairly fundamental.
Content-Transfer-Encoding must exist as a header.  The reasons
for that are well established -- MIME doesn't work without it
and systems that process MIME must be prepared for it and to
process it.  What is being proposed here is to _add_ a new
header.  Creating a new header, standardizing it, getting it
implemented and deployed -- all have costs, and those costs are
significant.

> But this header was never intended to guide MTAs in how to
> process messages, since our smtpext has got that well covered.
> I wrote on 21st December giving 7 reasons why this header was
> needed, and they were all related to situations where the
> transprt mechanism was not smtp, or where the message was
> being processed after it has left the smtp enviroment (and
> hence the envelope information was no longer available). If
> you believe this head is useless, then please reply, point by
> point, to that message.

In the hope that we can move beyond this, let me try to do that.
I believe that all, or almost all, of this has been said before.
But, maybe by drawing things together into one place, we can
make progress or at least avoid revisiting the topics about
which Harald has just issued a consensus call.

--On Thursday, 21 December, 2006 17:41 +0000 Charles Lindsey
<[email-masked]> wrote:

> AFAIAC, the Header-Type header field is VITAL, for the
> following reasons:
> 
> 1. It is the only way to tell that a message is UTF8SMTP in
> the case where the transport mechanism is _not_ SMTP (a
> situation which the draft now claims to allow).

Allowing a non-SMTP transport is different from an assertion
that all of the possible real or imaginary needs of non-SMTP
transports are satisfied.   We have a long history of saying
that, if we are carrying information in the SMTP envelope that
some other transport environment needs to have in headers (or
elsewhere), perhaps because it doesn't have an envelope, then it
is the responsibility of the protocol specification for the
other transport system to specify where to put the information
and how to get it there.  The fact that some other mail system
might require this header, or a header specifying the location
of the nearest Chinese restaurant, does not make it a
requirement for this specification or document.

> 2. Even when the transport mechanism _is_ SMTP, you cannot
> tell in the case where all the addresses are in pure ASCII;
> there could still be Subject and other headers with UTF8 in
> them, and there could be embedded MIME objects with UTF8 in
> their headers (e.g. message/rfc822 or filenames in
> Content-Disposition). It would need an SMTP parameter such as
> BODY-UTF8SMTP to indicate that.

It is fundamental to SMTP that accepting a message involves
accepting responsibility for that message on behalf of all
downstream systems.  The strongest manifestation of that
principle is the rule that message-acceptance must be taken
seriously with messages thereafter being either delivered or
rejected in some way, not accidentally or deliberately lost.  In
essence, the SMTP extension model simply extends the principle
to say that, if one accepts, or agrees to accept, a message with
a certain set of capabilities, one is taking the same
responsibilities as one had under 821 (and earlier) for
accepting the message itself.

The EAI model essentially provides that, if one offers the
UTF8SMTP option, one is willing to accept, and take
responsibility for, _any_ UTF-8 header information permitted by
the spec.  So one doesn't need additional headers to specify
that some other header might contain UTF-8 information in an
SMTP environment, even if they would "work".   For a non-SMTP
environment, see (1), above.

Also please remember that 2822 and its predecessors have a
strong bias against needing headers to be in a specific order.
If header reordering causes the "header-type" to show up after
the fields that it is supposed to protect one from, then one
needs to be able to parse, and possibly process, those fields
without whatever information "header-type" would provide.  One
could avoid that problem by scanning all headers for
"header-type" before doing anything else, but that does
considerable violence to both 2822 and to the way many
contemporary MUA and delivery MTA implementations are written.

> 3. Once a message has passed "final delivery" (whatever that
> is), the SMTP information is no longer available, but you may
> still need to know whether it is a UTF8SMTP message. Even if
> IMAP and POP3 contrive to keep this information somehow, they
> are not the only storage mechanisms in use; for example, good
> ol' mbox format is still around, and has no other way to store
> that information.

There is a long history of special headers, flags, and similar
information being kept with the stored form of a message.  We
have never tried to standardize that information or how it is
stored, much less required that a placeholder header be passed
around the network to accommodate it on storage.  Note that this
information includes things as simple (and common) as a "seen"
flag.  If we were to decide to standardize a way to handle that
information, or the deliveryMTA-> messageStore interface more
generally, it wouldn't be a task for this WG.

> 4. You NEED that information before you can gateway the
> message to some other medium (gatewaying is always dificult in
> the general case, and the other medium may or may not require
> some downgrade, and even if it doesn't it may need to know so
> that it in turn can gateway it on to something else).

You have not demonstrated that the information in the envelope
and the associated constraints are insufficient.  Nothing in the
work of this WG prevents you from creating a header (or
equivalent) in the gateway to the other environment. 

This is not a separate issue; it is just a different way to
restart (1).  So are some of the others, but they are better
disguised.

> 5. You NEED that information before you can Resend the message
> (or maybe even forward it, or encapsulate it as a
> message/rfc822 within something else).

MUA problem, outside the scope of these specs or this WG.   If
the MUA doesn't know what it is handling before trying to do
some of those things, it is in bad trouble.   Encapsulation is a
tricky case, as as been discussed many times before, but you
have offered no evidence that having a Header-type, especially
an unreliable one, floating around helps with that problem.   If
the MUA needs a reliable Header-type, then it needs to parse the
message to be sure it isn't being lied to.

> 6. Your MUA may NEED that information before it can display it
> (e.g. does it need to upgrade it first?), or else you NEED it
> in order to determine whether it requires to be downgraded
> before submission to the MUA, or even whether it needs to be
> upgraded first.

This appears to me to be just a variation on (3).  I'm not going
to repeat myself.

> 7. You NEED that information if the message has been
> DKIM-Signed, so you can either upgrade it before checking the
> signature, or else report "Sorry! This signature cannot be
> verified because of downgrading".

Anything that changes the format or content of the headers of a
message is going to pose a challenge for DKIM and anything else
that depends on integrity validation of headers.  If I were
inclined to depend on DKIM to get a message some kind of
preferential treatment, I'd probably be very sure that I didn't
supply an ALT-Address and would be checking the various pieces
of text to make the conditions under which a UTF8SMTP system
MUST NOT mess with headers in any way, i.e., that, if it hit a
barrier, it needed to reject the message rather than getting
creative.

Then I would go to the DKIM WG and say "look, there are going to
be messages with UTF-8-containing envelopes and headers floating
around the Internet, and some of it may involve downgrading, do
we need to adjust the DKIM machinery to deal better with that
situation?".  And, yes, there are reason why the work should be
done there, rather than here.  At least two of them are: (i)
They know the constraints and exceptions they need to have to
make a DKIM signature work, and are capable of changing them if
needed.  We don't and can't.   (ii) EAI is an effort to expand
and enhance the basic Internet email fabric to accommodate
address i18n and a number of things that seem to go with it.
DKIM is an overlay, additional, service to carry information to
help establish prioritization or preferences for email messages
but to carry the needed information in-band.   That makes it
_their_ problem to deal with our extensions... or to prove that
we have somehow made their problem impossible and then force an
IETF consensus decision about priorities.

> Note that there is a feature in DKIM whereby you can say, in
> the signature, "here is how such-and-such a header was at the
> point where it was signed". You might use that feature to
> record the original From, etc, but more importantly if you
> used it on the Header-Type (which itself MUST NOT be included
> in the signed hash, because it can change en route) then you
> could detect that the Header-Type was "UTF8" when it was
> signed but it now shows as "downgraded", and so you know
> exactly where you stand.

I think this proves the point I am trying to make above.
Specifying the interactions between DKIM and downgraded i18n
messages is a DKIM problem.

      john

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 26 Jan 2007 20:50:14 -0000

> implemented and deployed -- all have costs, and those costs are

You miss my point. If absent, the C-T-E defaults to 7bit, so it is not  
always required. However, for systems that routinely generate 8bit stuff,  
there is always a temptation to put in C-T-E: 8bit every time, without  
checking whether the particular message actually requires it. Likewise,  
there will be a temptation to write Header-Type: UTF8 even if the message  
happens to be pure ASCII. In both cases it is bad practice, because it may  
cause intermediate agents to do sxtra unnecessary work; nevertheless, it  
will cause no visible external harm.

As for creating new headers, we are already proposing to create two other  
new headers, and of those "Downgraded:" is going to cause problems if  
ASCII-only MTAs fail to transmit it. The loss of a Header-Type: in an  
ASCII-only MTA is a nuisance, but not a show stopper (because it can only  
say "downgraded" or "ASCII" in that context).

> requirement for this specification or document.

It has always been the case that the format of Internet Mail Messages has  
been standardized separately from the transport mechanism(s), so that the  
various agents involved don't have to worry about different formats in  
different contexts. What we have done, primarily, is to invent a new  
message format (a superset of the old one), by writing an extension to RFC  
2822. It is necessary to be able to tell whether a given message is an  
extended message or not, because numerous other agents will need to know,  
and it is ridiculous to suggest that every mechanism that handles mail  
should invent its own out-of-band machinery to solve (in a different way)  
a problem that is going to affect most of them. And it is not sufficient  
to say that they can always do an exhautive scan through the whole message  
in order to find out, because that is not a simple task (Hint: it does not  
suffice merely to check for the absence of a bit8 anywhere, it involves a  
recursive descent through the MIME structure to do it properly).

Common problems are most easily dealt with by common solutions.

Moreover, you are making a basic assumption that every transport mechanism  
will need to be redefined so as to handle UTF8SMTP messages, and that  
therefore it can be provided with the necessary mechanisms to transport  
this essential extra bit of information at the same time. That is simply  
not true. There are existing transport mechanisms (UUCP and NNTP for a  
start) that are quite capable of handling extended UFT8SMTP messages as  
they stand. It would be ridiculous to require them to be upgraded just  
because we had invented a new format that was not self-identifying.

> The EAI model essentially provides that,

No it doesn't, because our smtpext document provides no way to indicate,  
in the envelope, that a given message is an extended UTF8SMTP one.

> contemporary MUA and delivery MTA implementations are written.

There was a time when a single pass through the headers was all that could  
be tolerated. With modern computers, with vastly greater amounts of  
storage available, it is no problem at all to read in the full set of  
headers into a suitable internal structure before starting to process the  
message, and AIUI that is quite commonly done.

> information being kept with the stored form of a message.

Resulting in repeated reinvention of the same wheels. Since this  
particular piece of information needs to accompany the message throughout  
the whole of its travels, it is much easier to incorporate it within the  
format of the message. It is clearly a protocol requirement that it be  
present at all stages somehow. Once it is there, then if it saves the  
invention of further special headers and flags that is an added bonus.  
Storage and other agents don't _have_ to use it, but if it is there  
anyway, they probably will.

> equivalent) in the gateway to the other environment.

Creating the information within the gateway is not the problem. Ensuring  
that the gateway has access to the information in the first place is.
> restart (1).

Indeed, (1) is the fundamental issue. If it is once solved there, then all  
the rest fall into place.

> MUA problem, outside the scope of these specs or this WG.

Just as with gateways, MUAs NEED that information, and ensuring that it is  
available to them is most certainly within the scope of this WG. What the  
MUA does with it then is, of course, its own business.
>   If
> some of those things, it is in bad trouble.

Precisely. So how is it supposed to know?

> This appears to me to be just a variation on (3).

True.

> supply an ALT-Address

Unfortunately, you don't get to choose that. DKIM signing is likely to be  
done by the MSA, with or without your knowledge or approval (yes, DKIM is  
a can of worms and there seems to be no coherent strategy how to use it  
:-( ).

I think as it is currently specified, it will work OK on a message that is  
UTF8SMTP and remains in a UTF8SMTP environment (even if alt-addresses  
happen to be in some headers). But with downgraded messages it gets, shall  
we say, "interesting".

> situation?".

Too late. The DKIM-base is in the very last stages of being published as a  
standards-track RFC. They take the view that they comply with all the  
current email standards, and that if there is a bunch of people inventing  
some extension of those standards, then it is up to that bunch of people  
to fix the problem.

>  And, yes, there are reason why the work should be
> needed.  We don't and can't.

Sadly, they don't and can't either.

>> point where it was signed".

And using that feature, I can just about see a method whereby downgraded  
messages MIGHT be handled through DKIM. But even if that turned out to be  
infeasible, it would at least enable the situation to be recognized when  
it occurred.

The least we can do at the moment is to leave that possibility open.

> messages is a DKIM problem.

Then you go and try to convince them of that.

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 17:52:26 -0500

--On Friday, 26 January, 2007 20:50 +0000 Charles Lindsey
<[email-masked]> wrote:

>> implemented and deployed -- all have costs, and those costs
>> are significant.
> 
> You miss my point. If absent, the C-T-E defaults to 7bit, so
> it is not always required. However, for systems that routinely
> generate 8bit stuff, there is always a temptation to put in
> C-T-E: 8bit every time, without checking whether the
> particular message actually requires it. Likewise, there will
> be a temptation to write Header-Type: UTF8 even if the message
> happens to be pure ASCII. In both cases it is bad practice,
> because it may cause intermediate agents to do sxtra
> unnecessary work; nevertheless, it will cause no visible
> external harm.

I understood that point perfectly well.  But, unless someone can
depend on "Header-type: ASCII", the argument above demonstrates
that the header is useless.   That is not true of C-T-E, not
because of "7bit", but because of "Base64" and "Q-P", both of
which are, after all, 7bit and hence not definitively detectable
by a simple, non-heuristic scan. 

> As for creating new headers, we are already proposing to
> create two other new headers, and of those "Downgraded:" is
> going to cause problems if ASCII-only MTAs fail to transmit
> it. The loss of a Header-Type: in an ASCII-only MTA is a
> nuisance, but not a show stopper (because it can only say
> "downgraded" or "ASCII" in that context).

Any new header needs, IMO, strong justification.  That
justification hasn't been shown for Header-type and some of the
arguments for it, including the above, look like arguments
against it to me.  The questions, IMO, are "do we need this",
"does it provide information that we can't get elsewhere", and
"can it be trusted and, if not, can the problem be detected if
it makes any difference".  If the other two header proposals
don't yield satisfactory answers to those questions, you will
sooner or later see someone going after them too... I assure you.

>>> AFAIAC, the Header-Type header field is VITAL, for the

Ok.  I've read through your comments on my comments.  It is
clear that our perspective on what has been done in the past
(and maybe even the facts) and why, and how high-performance
mail servers are designed and operated, are different.  We also
have a difference of opinion about whether one should try to
standardize someone just because one might be able to write a
standard, or whether there are some things the IETF should not
touch absent much more compelling demand and, apparently,
different history as to what is necessary to design into a
gateway to maintain robust and conforming environments on both
sides.   That list could probably go on.  

Finally, the relevant ADs are quite aware that there are loose
ends wrt DKIM and i18n email.  Either they will get sorted out
at the right times, or the presence of DKIM signatures will be
taken in practice to encourage message rejection rather than
downgrading, or users (and MSAs) will end up with a choice
between internationalized email and DKIM.  Too bad, but pulling
one or two of the inhabitants out of a can of worms (your
description) in order to admire them and feed them growth
hormones does not seem to me to be an appropriate obstacle to
set in the path of this WG, nor am I convinced that reforming
one worm with an extra header will cause all of the other worms
to start crawling in the same direction.

So, I don't think I am going to convince you, and I don't think
you are going to convince me.  Our respective positions and
perspectives have now been enumerated and explained (and you
have responses to your seven or eight assertions, whether you
agree with them or not).  I don't think continuing this
discussion further is useful unless some other group of people
in the WG tell us they want it.

     regards,
      john

---

From: Chris Newman <[email-masked]>
Date: Fri, 26 Jan 2007 22:19:22 -0800

> by a simple, non-heuristic scan.

At the time MIME was designed, computer systems were much more limited and the 
cost of scanning the message to determine its type was more expensive than the 
cost of the silly states.  Today it is common practice for every MTA to scan 
every message (sometimes multiple times).  Whether it's for spam, virus, DKIM 
signatures, a Sieve body or loop action, IMAP mailstore insertion, mailstore 
search indexing or various other reasons, we routinely scan the entire message. 
Given that, the cost of the extra code paths to deal with the header parsing 
and silly states is more expensive than the cost of scanning the message 
(because scanning the message is a sunk cost).

If MIME were designed today, I would replace "7bit", "8bit" and "binary" with a 
single "identity" label and I'd certainly ignore the distinction between those 
three labels in any robust MIME processing agent I was writing today.

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: 27 Jan 2007 10:03:00 +0200

Chris Newman <[email-masked]> writes in gmane.ietf.ima:

<...>
> At the time MIME was designed, computer systems were much more limited
> and the cost of scanning the message to determine its type was more
> expensive than the cost of the silly states.  Today it is common
> practice for every MTA to scan every message (sometimes multiple
> times).  Whether it's for spam, virus, DKIM signatures, a Sieve body
> or loop action, IMAP mailstore insertion, mailstore search indexing or
> various other reasons, we routinely scan the entire message. Given
> that, the cost of the extra code paths to deal with the header parsing
> and silly states is more expensive than the cost of scanning the
> message (because scanning the message is a sunk cost).
> 
> If MIME were designed today, I would replace "7bit", "8bit" and
> "binary" with a single "identity" label and I'd certainly ignore the
> distinction between those three labels in any robust MIME processing
> agent I was writing today.
> 
>                 - Chris

Someone asked is MTAs scanning messages. I looked sendmail.

When sendmail does 8BITMIME downgrading, I was quessing that
it selects, which subparts to downgrade, by looking content-transfer-encoding.
That was not true. Instead it scans subparts twice (mime.c):

                /* remember where we were */
                offset = sm_io_tell(e->e_dfp, SM_TIME_DEFAULT);
                if (offset == -1)
                        syserr("mime8to7: cannot sm_io_tell on %cf%s",
                               DATAFL_LETTER, e->e_id);

                /* do a scan of this body type to count character types */
                while (sm_io_fgets(e->e_dfp, SM_TIME_DEFAULT, buf, sizeof buf)
                        != NULL)
                {
                        if (mimeboundary(buf, boundaries) != MBT_NOTSEP)
                                break;
                        for (p = buf; *p != '\0'; p++)
                        {
                                /* count bytes with the high bit set */
                                sectionsize++;
                                if (bitset(0200, *p))
                                        sectionhighbits++;
                        }

                        /*
                        **  Heuristic: if 1/4 of the first 4K bytes are 8-bit,
                        **  assume base64.  This heuristic avoids double-reading
                        **  large graphics or video files.
                        */

                        if (sectionsize >= 4096 &&
                            sectionhighbits > sectionsize / 4)
                                break;
                }

That scanning is done so that sendmail is can do smart selection between
quoted-printable and base64.

So it may be that Content-Transfer-Encoding with values 7bit, 8bit, binary
is just used as identity. Otherwise values are ignored (and scanning is done
instead.)

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Sat, 27 Jan 2007 11:38:46 -0500

--On Friday, 26 January, 2007 22:19 -0800 Chris Newman
<[email-masked]> wrote:

>> I understood that point perfectly well.  But, unless someone
>> can depend on "Header-type: ASCII", the argument above
>> demonstrates that the header is useless.   That is not true
>> of C-T-E, not because of "7bit", but because of "Base64" and
>> "Q-P", both of which are, after all, 7bit and hence not
>> definitively detectable by a simple, non-heuristic scan.
> 
> At the time MIME was designed, computer systems were much more
> limited and the cost of scanning the message to determine its
> type was more expensive than the cost of the silly states.
> Today it is common practice for every MTA to scan every
> message (sometimes multiple times).  Whether it's for spam,
> virus, DKIM signatures, a Sieve body or loop action, IMAP
> mailstore insertion, mailstore search indexing or various
> other reasons, we routinely scan the entire message. Given
> that, the cost of the extra code paths to deal with the header
> parsing and silly states is more expensive than the cost of
> scanning the message (because scanning the message is a sunk
> cost).
> 
> If MIME were designed today, I would replace "7bit", "8bit"
> and "binary" with a single "identity" label and I'd certainly
> ignore the distinction between those three labels in any
> robust MIME processing agent I was writing today.

Chris, I can certainly understand your removing the distinction
between "7bit" and "8bit", and maybe between "8bit" and
"binary", but it is not clear to my how you would definitively
determine the difference between "7bit" and the two downgrade
encodings ("Base64" and "Quoted-printable").   If what you
decided by heuristic scan was different from what the label
indicated, you'd be faced with a choice that was discussed
during the MIME work -- which to believe and the possibility
that the sender intended to transmit the apparently-encoded form
as a message, intending that it not be decoded.

Or am I missing something?

     john

---

From: Chris Newman <[email-masked]>
Date: Mon, 29 Jan 2007 11:56:24 -0800

> Or am I missing something?

I was proposing that MIME designed today would have three CTE labels: 
"identity", "quoted-printable" and "base64" (actually, I'd get rid of 
quoted-printable, but that's besides the point).  As you observe, 
distinguishing between "identity" and "base64" is important.  It's critical to 
distinguish between content and content-transfer-encoding.  An example where 
this was botched is the MacOS X Mail client which assumes any attachment with a 
zip magic number used zip as a content-transfer-encoding.  This breaks badly on 
the standard Open Document Formats which are zipped XML files where the zip is 
canonical to the format, and not used as a content-transfer-encoding.

                - Chris

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jan 2007 19:31:51 -0500

--On Monday, 29 January, 2007 11:56 -0800 Chris Newman
<[email-masked]> wrote:

>> Chris, I can certainly understand your removing the
>> distinction between "7bit" and "8bit", and maybe between
>> "8bit" and "binary", but it is not clear to my how you would
>> definitively determine the difference between "7bit" and the
>> two downgrade encodings ("Base64" and "Quoted-printable").
>> If what you decided by heuristic scan was different from what
>> the label indicated, you'd be faced with a choice that was
>> discussed during the MIME work -- which to believe and the
>> possibility that the sender intended to transmit the
>> apparently-encoded form as a message, intending that it not
>> be decoded.
>> 
> I was proposing that MIME designed today would have three CTE
> labels: "identity", "quoted-printable" and "base64" (actually,
> I'd get rid of quoted-printable, but that's besides the
> point). 

Aha.  I gave the wrong interpretation to "identity" and it was
downhill from there.  We are in agreement -- including about Q-P.

> As you observe, distinguishing between "identity" and
> "base64" is important.  It's critical to distinguish between
> content and content-transfer-encoding.  An example where this
> was botched is the MacOS X Mail client which assumes any
> attachment with a zip magic number used zip as a
> content-transfer-encoding.  This breaks badly on the standard
> Open Document Formats which are zipped XML files where the zip
> is canonical to the format, and not used as a
> content-transfer-encoding.

Of course, related issues with general-purpose zip files are
why, after some discussion, we rejected a proposal or three for
CTE: zip and some equivalents.

Incidentally, Charles, I don't know if Chris would agree, but,
if I were designing a mail store, I'd either make sure that
everything was stored in native form (Chris's "identity") or I
would record (somewhere) whatever the scanning process figured
out about what was present because I might want to have that
information handy to decide what sorts of mail-reader inquiries
to accept, etc.  But that, again, is not a case for an
on-the-wire message header.

      john

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 30 Jan 2007 17:47:51 -0000

> on-the-wire message header.

Absolutely. In an end-to-end world, that "identity" is the only reality.  
The only reason one might keep the encoded form in the mail store is for  
diagnostic/forensic purposes.

That was, incidentally, one of the problems I had with DKIM. They sign the  
body of a message in its on-the-wire form, and so subsequent changes of  
encoding (e.g. for downgrading in 8BITMIME) will break it. They get around  
that by RECOMMENDING that all signed messages should be sent in 7bit form,  
which is the last thing we would want to be doing with our UTF8SMTP  
messages.

Anyway, I am invited to prepare a draft for a canonicalization which  
always signs the "identity", and I may well do just that when Round Tuits  
become available.