Subject: [EAI] CONSENSUS CALL: Removal of Header-Format: marker

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 26 Jan 2007 11:13:49 +0100

The discussion of whether or not we need a header marker for marking 
UTF8SMTP messages that actually use UTF-8 characters has reached a point 
where it seems that no new technical information is being provided.

At this point, the chairs wish to see if the group is near a possible 
consensus on the issue.

PROPOSED RESOLUTOIN:

There is no significant operational or implementation benefit in having a 
marker in the headers of UTF8SMTP messages, and significant additional 
complexity. Therefore, the proposed "Header-Type" section, 
draft-ietf-eai-utf8headers-02 section 5, will be removed.

Note that this is not a call on the proposal to have a similar marker in 
the SMTP protocol; that's a separate issue.

POSSIBLE RESPONSES:

1. I agree with this resolution
2. I disagree with this resolution, for the following technical reasons:
3. I have another opinion, which is:

RESPOND:

- To the list, if you want to present an argument or just want to make your 
position visible at once
- To the chairs, if you don't want to respond on-list.

The deadline for responses is THURSDAY, FEBRUARY 1, 23:59 GMT
The chairs will prepare a summary, with names listed for each position, 
hopefully on Friday, Feb 2.

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Tony Finch <[email-masked]>
Date: Fri, 26 Jan 2007 10:54:55 +0000

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: NORTHWESTERLY 5 TO 7. ROUGH OCCASIONALLY VERY ROUGH. ISOLATED SHOWERS.
GOOD.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 13:45:11 +0100

> 3. I have another opinion, which is:

I can't judge it at the moment.  If the next draft tries to get away
without it, okay, but maybe we'll find later that an explicit header
field helps in some cases.

I'd feel far more confident that the WG is on the right track if we
nail the message/rfc822 vs. message/utf8smtp (or similar) question.

A redefinition of message/rfc822 to "who knows, you're free to scan
the header for non-ASCII octets and check if that happens to be UTF-8"
IMO can't work.

Frank

---

From: Bill McQuillan <[email-masked]>
Date: Fri, 26 Jan 2007 09:31:01 -0800

> draft-ietf-eai-utf8headers-02 section 5, will be removed.

I agree with this resolution

-- 
Bill McQuillan <[email-masked]>

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 26 Jan 2007 17:44:27 -0000

On Fri, 26 Jan 2007 10:13:49 -0000, Harald Tveit Alvestrand  
<[email-masked]> wrote:

> PROPOSED RESOLUTOIN:
>
> There is no significant operational or implementation benefit in having  
> a marker in the headers of UTF8SMTP messages, and significant additional  

I disagree with this resolution, for the following technical reasons:

I gave these reasons on December 21st, and nobody has commented on my  
objections. I repeat below what I wrote then:

AFAIAC, the Header-Type header field is VITAL, for the following reasons:

1. It is the only way to tell that a message is UTF8SMTP in the case where  
the
transport mechanism is _not_ SMTP (a situation which the draft now claims  
to
allow).

2. Even when the transport mechanism _is_ SMTP, you cannot tell in the case
where all the addresses are in pure ASCII; there could still be Subject and
other headers with UTF8 in them, and there could be embedded MIME objects  
with
UTF8 in their headers (e.g. message/rfc822 or filenames in Content-
Disposition). It would need an SMTP parameter such as BODY-UTF8SMTP to
indicate that.

3. Once a message has passed "final delivery" (whatever that is), the SMTP
information is no longer available, but you may still need to know whether  
it
is a UTF8SMTP message. Even if IMAP and POP3 contrive to keep this  
information
somehow, they are not the only storage mechanisms in use; for example,  
good ol'
mbox format is still around, and has no other way to store that  
information.

4. You NEED that information before you can gateway the message to some  
other
medium (gatewaying is always dificult in the general case, and the other
medium may or may not require some downgrade, and even if it doesn't it may
need to know so that it in turn can gateway it on to something else).

5. You NEED that information before you can Resend the message (or maybe  
even
forward it, or encapsulate it as a message/rfc822 within something else).

6. Your MUA may NEED that information before it can display it (e.g. does  
it
need to upgrade it first?), or else you NEED it in order to determine  
whether
it requires to be downgraded before submission to the MUA, or even whether  
it
needs to be upgraded first.

7. You NEED that information if the message has been DKIM-Signed, so you  
can
either upgrade it before checking the signature, or else report "Sorry!  
This
signature cannot be verified because of downgrading".

Note that there is a feature in DKIM whereby you can say, in the signature,
"here is how such-and-such a header was at the point where it was signed".  
You
might use that feature to record the original From, etc, but more  
importantly
if you used it on the Header-Type (which itself MUST NOT be included in the
signed hash, because it can change en route) then you could detect that the
Header-Type was "UTF8" when it was signed but it now shows as "downgraded",
and so you know exactly where you stand.

OK, there may still be lots of problems with DKIM-Signatures on UTF8SMTP
messages, and we will need to look at them later on; but in the meantime  
let
us not do anything that would make those propblems worse.

And, on top of all those reasons, I would add:

8. It will not be possible to extend this scheme to Netnews (Usenet)  
without this header, because there will be no way to know whether special  
attention is needed when gatewaying an article into email (a special case  
of point #4. Indeed, you would have the same problem it you tried to send  
email via NNTP (which has been suggested, though I do not see how it would  
work myself).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 26 Jan 2007 20:06:32 +0200

Harald Tveit Alvestrand <[email-masked]> writes in gmane.ietf.ima:

> UTF8SMTP messages that actually use UTF-8 characters has reached a
> point where it seems that no new technical information is being
> provided.
> 
> There is no significant operational or implementation benefit in
> having a marker in the headers of UTF8SMTP messages, and significant
> additional complexity. Therefore, the proposed "Header-Type" section,
> Note that this is not a call on the proposal to have a similar marker
> in the SMTP protocol; that's a separate issue.

I can not decide yet.   Specially removal of

   Header-Format: Downgraded

marker.

For 
   Header-Format: UTF8

I prefer marker on outside of headers.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 13:59:59 -0500

--On Friday, 26 January, 2007 17:44 +0000 Charles Lindsey
<[email-masked]> wrote:

>...
> 8. It will not be possible to extend this scheme to Netnews
> (Usenet) without this header, because there will be no way to
> know whether special attention is needed when gatewaying an
> article into email (a special case of point #4. Indeed, you
> would have the same problem it you tried to send email via
> NNTP (which has been suggested, though I do not see how it
> would work myself).

See comments in the note I sent a few hours ago (with Subject
line "Headers that define other headers and non-SMTP
environments (was: Re: [EAI] The Header-Type header (was: Re:
Status of utf8header draft))" and, in particular, the comments
about gateways and other systems.

Two comments on this new point in particular:

(1) Nothing would prevent:

(i) Defining a Header-format (or other) header for use in the
Netnews environment.

(ii) Requiring that a Mail-to-News gateway map the envelope
information into that header, or that it scan the message
headers and body and generate the header.

(iii) Requiring that a News-to-Mail gateway map the header into
appropriate envelope information or scan the message (including
headers) to determine whether the envelope extension information
needed to be present (or required).

Scenarios that have both the envelope information and a
redundant header field containing essentially the same
information for the same message at the same time risk all sorts
of "what happens if they are inconsistent and for what reasons"
cases.   It is an elementary design principle that such
situations are best avoided.  In the email case, that principle
is backed up by a huge amount of nasty experience that shows us
that, even if clear precedence is defined between the two
information locations, implementations(deliberately or
accidentally) get it wrong.   The "Errors-to" header, which is
needed for some systems that do not use SMTP envelopes or the
equivalent, used to be a major example of this problem.

(2) If I correctly understand the moves to make (or declare)
Netnews to be 8-bit-clean, the scan permitted or required by
(1)(iii) above would be required in any event, because messages
might originate in the Netnews environment that contained 8bit
information but didn't come from mail and hence did not have the
header.  Conversely, one could require that Header-type _always_
appear in Netnews files that contain 8bit information that might
migrate into the headers.

The thing that all of those cases and scenarios have in common
is that they point to this being a Netnews and/or Netnews <->
SMTP/Internet gateway problem, not an EAI problem.

     john

---

From: Ted Hardie <[email-masked]>
Date: Fri, 26 Jan 2007 11:30:12 -0800

>cases.  

While I'm generally sympathetic to the idea that you shouldn't
set up protocols that are liable to silly states, I do have some
questions here.  The first is how many different gateway types
we're actually likely to need here.  With a single type
(mail-to-news), it's pretty obvious that you can put the logic
on either side of the gateway provided you have any marker
at all.  But if there are lots of different types of gateways that
will need this, then a standard (or at least "usual") transport-independent
marker might be useful.  That makes it a bit easier for those
writing gateways.  There is also an issue of what happens when
the message passes through multiple gateways (imagine
some mix of mail, news, and mms a la 4356); knowing that
you'll see the same marker no matter what set of gateways a message
took to reach  from originator to receiver does seem useful.

>It is an elementary design principle that such

It's hard to argue with that experience, but it does seem like
a precedence *could* be established.  If you have access to the
SMTP transport this header is ignored; if you don't the header is
honored.

Note:  no hats on; just trying to understand the design space
and constraints.
				Ted

---

From: Kari Hurtta <[email-masked]>
Date: 26 Jan 2007 21:42:02 +0200

> Note:  no hats on; just trying to understand the design space

And actually there is header field which correspond to "Errors-to".

Return-Path -header field is standard. Trick is that it is valid
only when mail is not on SMTP environment.

Is similar structure needed on here?

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 22:17:31 +0100

> SMTP/Internet gateway problem, not an EAI problem.

The USEFOR draft states that netnews is a kind of subset of the
Internet Message format as defined in RFC 2822, and IIRC one of
the MIME RFCs claims that netnews is a subset of message/rfc822.

The USEFOR WG apparently intends to get rid of message/news at
some point in time - Harald decided that the second WGLC wasn't
an ideal time to finish it off.  If USEFOR is published as RFC
anybody could try to deprecate (OBSOLETE) message/news as a
clerical task.

It could depend on the "change controller", but IIRC Henry also
wanted to get rid of it.  <rant> The IANA registry is a PITA, I
don't find the original registration templates for various MIME
types. </rant>

Allowing message/utf8smtp in netnews is a job for a USEFOR-bis.

How gateways "downgrade" a message/utf8smtp to a message/rfc822
is our job (i.e. defining how to get this right).  If it happens
to be at a mail2news gateway there won't be many special netnews
considerations, after all netnews is already "8BITMIME"-ready.

Please correct me if I'm wrong, but IMO mail2news is no obstacle
for UTF8SMTP, but message/rfc822 is an issue.  Some weeks ago
you wrote that a MIME expert you've asked said "ugh", and IIRC
you said you'd write more about "MIME considerations" later (?)

Frank

---

From: Chris Newman <[email-masked]>
Date: Fri, 26 Jan 2007 21:57:29 -0800

I agree with this resolution.

                - Chris

---

From: Yangwoo Ko <[email-masked]>
Date: Sat, 27 Jan 2007 15:33:18 +0900

> 1. I agree with this resolution

+1

This header may provide some optimization if all required conditions 
(e.g. no liars and so on) are met. But, expected improvement seems 
marginal while (also quite naturally) expected problems are a lot more 
overwhelming.

---

From: Chris Newman <[email-masked]>
Date: Fri, 26 Jan 2007 23:06:41 -0800

Charles Lindsey wrote on 1/26/07 17:44 +0000:
> transport mechanism is _not_ SMTP (a situation which the draft now claims  to
> allow).

The _only_ way to tell if a message is a UTF8SMTP message is to scan the 
message and MIME headers to see if they conform to UTF8SMTP syntax.  A 
Header-Type field does not provide that information, it merely provides a 
non-authoritative hint about the message type which any robust processing agent 
needs to ignore.

> Disposition). It would need an SMTP parameter such as BODY-UTF8SMTP to

Same answer as statement 1.

> information is no longer available, but you may still need to know whether  it
> somehow, they are not the only storage mechanisms in use; for example,  good
> ol'
> mbox format is still around, and has no other way to store that  information.

A mbox-style file which contains UTF8SMTP messages would be a different file 
format from today's mbox format.  There are multiple ways to distinguish the 
two file formats, including by file extension, by out-of-band typing (e.g. 
MacOS creator code), by file location, or by putting a magic number at the 
beginning of the file (a UTF-8 BOM followed by "From " would be a good magic 
number for a UTF8SMTP mbox file).

A final delivery agent which put a UTF8SMTP message in a vanilla mbox file 
would have a software bug that needs to be corrected.

> 4. You NEED that information before you can gateway the message to some  other

I do not understand why a gateway would require a header containing a 
non-authoritative hint about the message type.   A gateway can scan the headers 
for 8-bit content and when it sees it process that content accordingly.  A 
robust gateway should already do this today.

> 5. You NEED that information before you can Resend the message (or maybe  even

This information can be gained by scanning the message and MIME headers so it 
is not needed.

> 6. Your MUA may NEED that information before it can display it (e.g. does  it
> need to upgrade it first?), or else you NEED it in order to determine  whether
> it requires to be downgraded before submission to the MUA, or even whether  it

An MUA can be designed so it doesn't need that information.  It can also be 
designed such that the mailstore caches that information in an authoritative 
fashion, which would actually be useful, unlike a header which contained an 
advisory hint about the message type.

> 7. You NEED that information if the message has been DKIM-Signed, so you  can
> either upgrade it before checking the signature, or else report "Sorry!  This

If upgrading is desirable, there will be some sort of Downgraded-Info trace 
field that can be used to detect that a downgrade occurred.  Of course a DKIM 
signature either verifies or it doesn't.  If an agent looks at something else 
to determine whether or not a message should be verified I can assure you 
spammers will quickly adopt that something else and make any sort of "can't be 
verified because of blah" message meaningless.  So the scenario you're 
describing is itself a design error.

> 8. It will not be possible to extend this scheme to Netnews (Usenet)  without
> this header, because there will be no way to know whether special  attention
> is needed when gatewaying an article into email (a special case  of point #4.
> Indeed, you would have the same problem it you tried to send  email via NNTP
> (which has been suggested, though I do not see how it would  work myself).

A robust news->email gateway should already scan the message for standards 
conformance and downgrade any 8-bit which appears in headers.  I do not see why 
our specifications should be concerned with non-robust software.

I have yet to see a single solid technical reason why a Header-Type field is 
useful.

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: 27 Jan 2007 11:24:06 +0200

> I prefer marker on outside of headers.

Seems that parsing of MIME structure by every MTA on path
is quite common.  So when MIME structure is parsed anyway,
that UTF8 indicator is not needed.

So question is, do we want upgrading of downgraded
header fields. If that is wanted,

and 

      Header-Format: Original

can be used. 

Without 

heade ffiled upgrading must not be done.  I think
automatic 'upgrading' always does too much damage.

For connecting isolated UTF8SMTP islands I have
suggested encapsulation  -- drawback is that this
requires manual configuration for tunnel -- it
is not automatic.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 27 Jan 2007 11:37:19 +0200

This means also that upgrading of UTF8SMTP downgraded messages
   is NOT done.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 27 Jan 2007 11:48:50 +0200

>    This means also that upgrading of UTF8SMTP downgraded messages

  In past I was suggested required-fields paramater Header-Type
  header fields. That was added to draft-ietf-eai-utf8headers-02.
  When Header-Type is removed, I will suggest "UTF8-required-fields"
  header field to draft-ietf-eai-utf8headers instead.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 27 Jan 2007 17:02:32 +0100

Chris Newman wrote:

> A mbox-style file which contains UTF8SMTP messages would be a different
> file format from today's mbox format.  There are multiple ways to
> distinguish the two file formats, including by file extension,
[...]
> A final delivery agent which put a UTF8SMTP message in a vanilla mbox
> file would have a software bug that needs to be corrected.

Yes, and that's an important statement, we've to make sure that the
affected drafts (e.g., maybe the mailinglist I-D wrt "digest mode")
clearly say so.

Another example is a Web mail interface I've used, it allowed to
download selected messages, and used a simple variant of the mbox-
format, not MIME multipart/digest.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Sun, 28 Jan 2007 17:13:41 -0000

> 8. ...

And here is another addition:

9. If you remove the Header-Type header, as currently defined, then you  
will need to invent a Header-Language header, since AIUI it is an IESG  
requirement that wherever you specify the use of a non-ASCII charset you  
have to have also a means to specify the language which is using that  
charset. The existing Content-Languags header is a model of what would be  
required.

---

From: "Charles Lindsey" <[email-masked]>
Date: Sun, 28 Jan 2007 17:48:03 -0000

On Fri, 26 Jan 2007 19:30:12 -0000, Ted Hardie <[email-masked]> wrote:

> at all.....

The basic RFC 822 format has now been used by Netnews, HTTP and Sip (and  
maybe others). It will likely be used by further protocols as time goes  
on. If you have N protocols, then you have in principle (N-1)**2/2  
gateways, not that they all necessarily make sense). Then you have mailing  
list expanders, which are a little like gateways too. In addition, you  
have gateways to/from protocols that do not use the RFC 822 format (e.g.  
X.400 mail and Fidonet).

>  There is also an issue of what happens when

But the real problem is that email (and all the others) does not always  
get processed by well-written software running well-defined protocols.  
People write all sorts of ad-hoc scripts to process mail after it has been  
received, or to generate it by automatic means. There will copy it from  
the clipboard into a file, losing all its context, and subsequently expect  
to be able to send to any email addresses within it, or even to forward  
the whole message. They will generate messages with text editors, and then  
telnet to some port 25 and paste the message in after the DATA command.  
Such people might just about be persuaded to take note of a Header-Type  
header if is present (or to generate it where needed). But there is no way  
they are going to go to the trouble of writing complete scans through the  
whole message into their scripts, as has been suggested.
>> equivalent, used to be a major example of this problem.

The intention of this header is that it should describe (definitively -  
since it is a MUST generate) the status of the message in which it is  
contained, travelling with it wherever it goes, since it is primarily of  
interest to the end-points (MTAs will normally pass it on without looking,  
though if they care to be liberal and handle cases where it has not been  
correctly provided, then good luck to them). The only case where this  
header is of interest bwtween the end-points is where downgrading is  
called for. Our documents will describe very clearly the proper way to  
downgrade, and if it is not implemented in strict conformance with those  
documents, then all sorts of things are going to go wrong.
> SMTP transport this header is ignored; if you don't the header is

The current proposals for the SMTP transport do not provide any mechanism  
to carry this information in the envelope. IMO, that is as it should be.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 28 Jan 2007 20:45:53 +0100

Charles Lindsey wrote:

> AIUI it is an IESG requirement that wherever you specify the use of a
> non-ASCII charset you have to have also a means to specify the language
> which is using that charset.

The chapters 3 and 4 in RFC 2277 are completely independent.  Of course
language tagging / identification / negotiation isn't bound to non-ASCII.

An e-mail address typically has no "language", and for anything else it
is possible to use RFC 2231 chapter 5 =?charset*langtag?x?y?= words, or
the weird RFC 2231 constructs for MIME parameters.

Language tags for the complete header are a dubious idea, I can imagine
scenarios where it depends on the header field or on single 2047-words.

Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 29 Jan 2007 00:34:52 +0100

Speaking as the author of RFC 2277:

Frank is right.

The need to identify language is independent of the need to allow for 
multiple charsets (or at least UTF-8).

There are many languages written in ASCII.

Section 3 of 2277 refers to "character data". Section 4 refers to "text". 
E-mail addresses may be many things, including being placed in text, but 
they are not text.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 29 Jan 2007 00:40:02 +0100

> automatic 'upgrading' always does too much damage.

As technical contributor, I have protested long and hard against the term 
"upgrade" as anything but something that can optionally be applied by the 
end-recipient after final delivery.

> requires manual configuration for tunnel -- it

TCP is an excellent tunneling protocol.

I believe the set of systems that *can* send SMTP mail to each other, but 
*cannot* establish TCP connections to each other, or to a relay friendly to 
both parties, is a fairly small set. Perhaps so small that we can ignore it.

(the canonical exception is, of course, the port-25-relaying, 
SMTP-interpreting firewall that still doesn't want to hear about EHLO......)

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Jan 2007 10:59:33 -0000

On Sun, 28 Jan 2007 23:40:02 -0000, Harald Tveit Alvestrand  
<[email-masked]> wrote:

> As technical contributor, I have protested long and hard against the  
> term "upgrade" as anything but something that can optionally be applied  
> by the end-recipient after final delivery.

Mostly agreed. The exception is a MDA which redirects a message based on  
something in its /etc/aliases file, which may need to do a limited upgrade  
in order to detect cases which need to be redirected.

Also, our IMAP document speaks of upgrading but that happens, of course,  
after final delivery.

---

From: Kari Hurtta <[email-masked]>
Date: Mon, 29 Jan 2007 13:01:55 +0200 (EET)

>>> Harald Tveit Alvestrand
<...>
> SMTP-interpreting firewall that still doesn't want to hear about EHLO......)

Even if it is heard about EHLO.

It is suggested, that firewall filter these EHLO keywords,
which it does not know.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Jan 2007 11:13:38 -0000

> There are many languages written in ASCII.

But I suspect those languages mostly do not need the language information  
to assist in the display process, whereas I would imagine that it is  
useful for a user agent to know whether it is displaying Chinese or  
Japanese, even though those languages use essentially the same "CJK" parts  
of Unicode.
>
> Section 3 of 2277 refers to "character data". Section 4 refers to  
> "text". E-mail addresses may be many things, including being placed in  
> text, but they are not text.

It is not <addr-spec>s one should be concerned about, but rather the  
textual stuff one finds in <display-name>s and in unstructured headers  
such as Subject.
>
> --On 28. januar 2007 20:45 +0100 Frank Ellermann  
> <[email-masked]> wrote:
>

>> the weird RFC 2231 constructs for MIME parameters.

Sure, but the whole purpose of this WG is to do away with continued use of  
RFC 2047/2231 weirdness (though for sure both will linger on for quite a  
time :-( ).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Jan 2007 11:53:01 -0000

> Header-Type field does not provide that information, it merely provides  
> a non-authoritative hint about the message type which any robust  
> processing agent needs to ignore.

Repeatedly scanning the message at every agent is passes through is not  
the panacea that people seem to imaging. It is hard work (I have provided  
a Perl script at the end of this message to indicate how hard), and harder  
than simply checking for whether 8BITMIME is required. It is conceivable  
that such scanners will be routinely written into MTAs and user agents,  
less so that they will be adequately covered in gateways and mailing list  
expanders, and most improbable that they will be incorporated into the  
numerous ad hoc scripts which people tend to write to process or generate  
email messages for all sorts of random reasons.

>> information is no longer available,... for example,  good
>> ol'
> distinguish the two file formats, including by file extension, by  
> out-of-band typing (e.g. MacOS creator code), by file location, or by  
> putting a magic number at the beginning of the file (a UTF-8 BOM  
> followed by "From " would be a good magic number for a UTF8SMTP mbox  
> file).

Mbox format is nowhere rigorously defined. People interpret it as best  
they can, and the likelyhood of any such extensions being universally  
adopted is vanishingly small and is totally beyong our control (as opposed  
to POP3/IMAP where we have proposed extensions which have a reasonable  
chance of being adopted).
> file would have a software bug that needs to be corrected.

I disagree totally. It is going to happen whether we like it or not.

----------------------------------------------------------------------------

I wrote the following script to illustrate what needs to be done in order  
to scan a message to detect, in all cases, whether it needs to be treated  
as UTF8SMTP. It is not a pretty sight. Naturally, for serious use within  
some agent it would have to be rewritten in C to achieve any sem,blance of  
efficiency. The Perl version, however, is better in order to illustrate  
clearly the steps that need to be taken.

#!/usr/local/bin/perl

use English;
use MIME::Head;

my $LONGLINE = ' ' x 1000;
$_ = $LONGLINE;	# provide a buffer long enough for most lines;
		# if it needs more, it will malloc it

#-------------------------------------------
#
# Set up $in and $out handlers; by default, STDIN and STDOUT
# (this program is designed to operate in a pipeline).
#
my ($in, $out);
if (defined $ARGV[0]) {open($in, "<$ARGV[0]")}
else {$in = STDIN}
if (defined $ARGV[1]) {open($out, ">$ARGV[1]")}
else {$out = STDOUT}

my $header = 0, $body=0; # To record whether we found an 8th bit

#-------------------------------------------
#
#   read_chunk ($bound)
#
#   reads up to the given MIME boundary, or up to EOF, checking for an 8th  
bit
#
sub read_chunk {
	my ($bound) = @_;

	my $eos_type = "EOF";	# DELIM, CLOSE or EOF

	while (<$in>) {		# will read up to next '\n'
				# which might be a long way away in
				# genuine binary attachments, but should
				# otherwise fit in 1000 octets
		if (m/[\200-\377]/o) {
			$body = 1;
		}

		if (substr($_,0,2) eq '--'
		   && m/^(--$bound(--)?[ \t]*)/) {
		    ### it was the expected boundary, and we note whether
		    ### it is a CLOSEing boundary
			$eos_type = ($2 ? 'CLOSE':'DELIM');
			return ($eos_type);
		}
	}
	return ($eos_type);
}

#-------------------------------------------
#
#   process_header ($deftype)
#
#   Reads headers, both at the top level and at the start of multipart
#   and message types. It reurns the Content-Type (defaulting to $deftype),
#   and checks the header for any 8th bit.
#
sub process_header {
	my ($deftype) = @_;

	my $head = new MIME::Head;
	$head->read($in);
	my $mime_type = $head->mime_type($deftype);
	my $bound = $head->multipart_boundary;

	if ($head->as_string =~ m/[\200-\377]/o) {
		$header = 1;
	}
	return ($mime_type, $bound);
	}

#-------------------------------------------
#
#   process_part ($oldbound, $deftype)
#
#   process_part is the heart of the whole system. A 'part' is a part of
#   a multipart, or a message type, or even the whole message. This
#   routine walks through the tree of parts, calling itself recursively
#   as needed.
#
sub process_part {
	my ($oldbound, $deftype) = @_;

	my ($mime_type, $bound) =
	    process_header($deftype);

	my ($type, $subtype) = split('/', $mime_type);
	my $eos_type;
	if ($type eq 'multipart') {
	### multipart
		# determine the default type of the parts
		my $retype =
		      $subtype eq 'digest'?'message/rfc822':'text/plain';

	    ### parse preamble
		# the preamble is not supposed to be displayed, but it
		# still gets checked
		($eos_type) = read_chunk($bound);
		die "boundary $bound not found\n" if ($eos_type eq 'EOF');
			# an expected CLOSEing boundary was never found;
			# a fatal error
		$more_parts = ($eos_type eq 'DELIM');

	    ### process parts
		while ($more_parts) {
			# the recursive call for each part within this
			# multipart
			($eos_type) = process_part($bound, $retype);
			die "boundary $bound not found\n" if ($eos_type eq 'EOF');
			$more_parts = 0 if ($eos_type eq 'CLOSE');
		}

	    ### process epilogue
		# the epilogue also is not for display, but still gets
		# checked; we are now looking for the next boundary
		# ($oldbound) withing our parent
		($eos_type) = read_chunk($oldbound);

	} elsif ($type eq 'message') {
	### message
	    ### all message types have the same structure, though only
	    ### some of them, such as message/rfc822 and message/partial
	    ### possess bodies which might need checking;
	    ### so, having already processed the header for the current
	    ### part, we now encounter the header of the message proper
	    ### - another recursive call; the mesage will be terminated
	    ### by the boundary of the parent, hence the $oldbound
	    ### parameter

		($eos_type) = process_part($oldbound, 'text.plain');

	} else {
	### simple part
	    ### this is a tip of the part tree
		($eos_type) = read_chunk($oldbound);
	}
	return ($eos_type);
}

#-------------------------------------------
#
#   main program
#
#   The two parameters represent:
#	no boundary for any parent
#	no default type (actually, it is text/plain, but process_header
#		will tell us that)
#
process_part('', '');

if ($body) {print $out "This message requires 8BITMIME\n"}
if ($header) {print $out "This message requires UTF8SMTP\n"}

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Jan 2007 12:05:35 -0000

>   header field to draft-ietf-eai-utf8headers instead.

Please sunnarize again what the UTF8-required-fields would do.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Jan 2007 12:12:34 -0000

> that UTF8 indicator is not needed.

What evidence do you have that this is commonly done? See my Perl script  
in reply to Chris which shows that it is hard work to do this properly,  
and generally speaking MTAs will be passing the whole body on untouched,  
and hence currently have little need to be aware of the MIME structure.
> header fields. If that is wanted,

That information will still be useful for handling the message sensibly  
when it arrives at its ultimate destination.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 29 Jan 2007 16:34:28 +0100

Charles Lindsey wrote:

> the whole purpose of this WG is to do away with continued use of
> RFC 2047/2231 weirdness (though for sure both will linger on for
> quite a time :-( ).

RFC 2047 and 2231 chapter 5 (= the *langtag stuff) are IMO okay.
The rest of RFC 2231 about MIME parameter values is admittedly odd.

 From my POV the purpose of this "EAI" WG is if anything the A in
EAI or IMA.  IFF that "A" part works in the experiment adding more
(probably anything else modulo Content-*) later should be no big
issue.  For that "A" part we need no language tags, and for 2047-
words we have it already.  Kari said that he has implemented the
complete 2231, not only chapter 5.

Therefore we're not forced to worry about langtags for complete
headers, the 822 7bit headers also don't need / have that.  It's
more important to define and register a "message/utf822" superset
of the pure ASCII message/rfc822 than to tackle non-"A" details.

Maybe we can agree to disagree for the moment, I'm not yet sure
about this "Header-Format" issue, I've too see a draft trying to
get away without it, and if that breaks somewhere.  But langtags
probably won't be the showstopper.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 29 Jan 2007 20:21:06 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> What evidence do you have that this is commonly done? See my Perl
> script  in reply to Chris which shows that it is hard work to do this
> properly,  and generally speaking MTAs will be passing the whole body
> on untouched,  and hence currently have little need to be aware of the
> MIME structure.

On my answer to your another question I noticed that sendmail parses
MIME structure of every mail (on default configuration).

Have you information of some other commonly used MTAs?

/ Kari Hurtta

---------CLIP------------CLAP--------------
From: Kari Hurtta <[email-masked]>
Subject: Re: Header-Type, Mime-Version (Re: [EAI] Status of utf8header draft)
Newsgroups: gmane.ietf.ima
To: [email-masked]
Date: 25 Jan 2007 16:57:30 +0200

<...>

[1]  Actually this is not completely true. If MaxMimeHeaderLength
     or MaxMimeFieldLength is set, MMME structure is always parsed.
     These options are set by default starting from sendmail 8.11.7
     Quote from RELEASE_NOTES

8.11.7/8.11.7   2003/03/29
<...>
        To provide partial protection for internal, unpatched MTAs that may be
                performing 7->8 or 8->7 bit MIME conversions, the default
                for MaxMimeHeaderLength has been changed to 2048/1024.
                Note: this does have a performance impact, and it only
                protects against frontal attacks from the outside.
                To disable the checks and return to pre-8.11.7 defaults,
                set MaxMimeHeaderLength to 0/0.

---

From: Kari Hurtta <[email-masked]>
Date: 29 Jan 2007 21:12:26 +0200

> Please sunnarize again what the UTF8-required-fields would do.

Downgrade needs able to delete headers (when it do not
know syntax of headers).  Content of these headers are 
stored to some encapsulation format (for example Downgraded
-header).

These headers may be necessary for some protocols.
That header field makes possible to sender specify that.

UTF8-required-fields sets requirements, which header fields
downgrading process must know. It says that these headers
must exists even when message is on downgraded form.

/ Kari Hurtta

----------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: [EAI] draft-ietf-eai-utf8headers-01.txt /
 draft-ietf-eai-downgrade-02.txt suggestion
Newsgroups: gmane.ietf.ima
Cc: Kari Hurtta <[email-masked]>
Date: Sat, 14 Oct 2006 14:48:07 +0300 (EEST)

draft-ietf-eai-utf8headers-01.txt:

   header-content  = "UTF8SMTP:" [FWS] content-code [FWS]
                     *( ";" parameter ) CRLF
   content-code    = "UTF8" / "ASCII" / "Downgraded"
   parameter       = "Header-Language:" [FWS] Language-List
   Language-List   = Language-Tag [FWS]
                     *("," [FWS] Language-Tag [FWS])

following was suggested:

   header-content  = "Header-Type:" [FWS] content-code [FWS]
                      *( ";" parameter ) CRLF

   ; parameter is defined on RFC 2045 

   content-code    = "UTF8SMTP" / "ASCII" / "Downgraded"

   The "header-language" parameter is defined for "Header-Type"
   header-field. Value of "header-language" parameter is defined
   as following (after possible quotation of quoted-string is decode):

   header-language-value = Language-Tag [FWS]
                     *("," [FWS] Language-Tag [FWS])

I also suggest following (almost same was also suggested):

   The "required-fields" parameter is defined for "Header-Type"
   header-field. Value of "required-fields" parameter is defined
   as following (after possible quotation of quoted-string is decode):

   required-fields-value = field-name [FWS]
                     *("," [FWS] field-name [FWS])

   ; field-name is defined on RFC 2822

draft-ietf-eai-downgrade-02.txt:

  Downgrading:
      *  Check if the mail has 'UTF8SMTP' header and its value is
         "UTF8".  Otherwise, downgrading is not required.
      *  Check each header whether it contains UTF-8 characters or not.
         If the header contains UTF-8 characters,
         +  If the header is an address header which is described in
            Section 5.1.1,
            -  Preserve the header in 'Downgraded' header.
            -  Downgrade the header defined in Section 5.1.1.
         +  The other header case, encode the header by [RFC2047] with
            UTF-8 tag.
      *  Change 'UTF8SMTP' header value to "Downgraded".

Suggested changes:

   Downgrading:
      *  Check if the mail has 'Header-Type' header field and 
         its value is "UTF8SMTP".  Otherwise, downgrading is not required.
      *  Check each header field whether it contains UTF-8 characters or not.
         If the header field contains UTF-8 characters,
         +  If the header field is an address header field which is 
            described in Section 5.1.1,
            -  Preserve the header in 'Downgraded' header field.
            -  Downgrade the header field defined in Section 5.1.1.
         +  Otherwise if syntax of header field is known:
            - Encode header elements according of [RFC2047]
              and [RFC2231]
            - If encoding is not possible, preserve the header in 
              'Downgraded' header field and remove original header field.
         -  If syntax of header field is not known
            - Preserve the header in 'Downgraded' header field and 
              remove original header field.
      * If some header field was removed, which is mentioned on
        "required-fields" parameter on 'Header-Type' header field,
        downgrading fails. 
      *  Change 'Header-Type' header field value to "Downgraded".
     
      NOTE: Address header downgrading in Section 5.1.1 may also
            remove header field. If that header field is mentioned 
            on "required-fields" parameter on 'Header-Type' header 
            field, downgrading fails. 
            
draft-ietf-eai-downgrade-02.txt:

    Upgrading:
      *  Check if the mail has 'UTF8SMTP' header and its value is
         "Downgraded".  Otherwise, upgrading is not required.
      *  Decode all 'Downgraded' headers.
         +  Decode header value field string which is [RFC2047] encoded.
         +  If the header is address header described in Section 5.1.1,
            -  Apply address header downgrading to the decoded header.
            -  Remove the header line which is the same with the
               downgraded line.
         +  Remove the 'Downgraded' header.
         +  Add decoded header to mail header.
      *  If each mail header has [RFC2047] encoded part and which
         encoding is "UTF-8", it is a downgraded header, so decode it.
      *  Change 'UTF8SMTP' header value to "UTF8".

Suggested changes:

      *  Check if the mail has 'Header-Type' header field and its value is
         "Downgraded".  Otherwise, upgrading is not required.
         +  Decode header value field string which is [RFC2047] encoded.
         +  If the header is address header field described in Section 5.1.1,
            -  Apply address header field downgrading to the decoded header.
            -  Remove the header field line which is the same with the
               downgraded line.
         +  Remove the 'Downgraded' header field.
         +  Add decoded header to mail header field.
      *  For every mail header field, which syntax is known
         +  If each mail header field has [RFC2047] or [RFC2231] encoded part 
            and which encoding is "UTF-8", it is a downgraded header field, 
            so decode it.
      *  Change 'Header-Type' header value to "UTF8SMTP".

/ Kari Hurtta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 29 Jan 2007 11:16:15 -0800

--On 29. januar 2007 11:53 +0000 Charles Lindsey <[email-masked]> 
wrote:

> a Perl script at the end of this message to indicate how hard),

In what universe does 65 lines of Perl constitute "hard"?

(I removed comment lines and blank lines before counting - a well commented 
program, btw - kudos for that!)

              Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 29 Jan 2007 11:18:50 -0800

--On 29. januar 2007 11:13 +0000 Charles Lindsey <[email-masked]> 
wrote:

> Japanese, even though those languages use essentially the same "CJK"
> parts of Unicode.

Have you read RFC 2277?

---

From: Chris Newman <[email-masked]>
Date: Mon, 29 Jan 2007 17:15:49 -0800

While this is a clever technical idea, it strikes me as over-design (something 
we've all done now and then ;-).

This sort of "reject if you don't understand" feature has a history of either 
causing interoperability problems (e.g. PKIX) or being ignored (e.g. 
Disposition-Notification-Options, RFC 3798).  It seems to me we should instead 
have a fixed set of "bounce-if-you-can't downgrade" headers listed in the base 
downgrade spec and that's it.

The primary focus of this work should be to make the UTF8SMTP -> UTF8SMTP 
experience work well and make the UTF8SMTP -> SMTP experience not break.

The UTF8SMTP -> SMTP -> UTF8SMTP experience is a tertiary case.  While it's 
important to preserve a fixed set of critical information in that case, we have 
to be careful not to go overboard optimizing for a transitional case.

                - Chris

---

From: Chris Newman <[email-masked]>
Date: Mon, 29 Jan 2007 17:34:36 -0800

Sun's MTA also parses the MIME structure.

It was very thoughtful of Charles to provide perl code that proves just how 
simple and straightforward it is to do MIME parsing.  I couldn't have supported 
my position as well as he did.  The several C MIME parsers I've written in the 
past for different purposes are longer but still straightforward.  The one 
which uses Boyer-Moore to look for MIME boundaries and mbox delimiters is a bit 
less simple; I was probably over-optimizing for performance when I wrote that 
code...

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: 30 Jan 2007 06:54:17 +0200

Harald Tveit Alvestrand <[email-masked]> writes
in gmane.ietf.ima:

> --On 27. januar 2007 11:24 +0200 Kari Hurtta
> term "upgrade" as anything but something that can optionally be
> applied by the end-recipient after final delivery.

It is still too early, if it is applied on IMAP server
before MUA sees message.

If "upgrading" is always done, DKIM (for example) is
forced to use some other (encoded) charset than UTF-8
on headers. For example convert phrases and other places
whese are signed to UTF-7 so that UTF8SMTP upgrading
do not trigger.

This affects also possible MIME header fields 
(Content-Description) on case S/MIME message.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 30 Jan 2007 07:22:56 +0200

Chris Newman <[email-masked]> writes in gmane.ietf.ima:

> While this is a clever technical idea, it strikes me as over-design
> (something we've all done now and then ;-).
> 
> This sort of "reject if you don't understand" feature has a history of
> either causing interoperability problems (e.g. PKIX) or being ignored
> (e.g. Disposition-Notification-Options, RFC 3798).  It seems to me we
> should instead have a fixed set of "bounce-if-you-can't downgrade"
> headers listed in the base downgrade spec and that's it.
> 
> The primary focus of this work should be to make the UTF8SMTP ->
> UTF8SMTP experience work well and make the UTF8SMTP -> SMTP experience
> not break.
> 
> The UTF8SMTP -> SMTP -> UTF8SMTP experience is a tertiary case.  While
> it's important to preserve a fixed set of critical information in that
> case, we have to be careful not to go overboard optimizing for a
> transitional case.
> 
>                 - Chris

No.  That required-fields is only for UTF8SMTP -> SMTP   case.

UTF8SMTP -> SMTP -> UTF8SMTP is not problem, assuming that unknown
header fields are encapsulated.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Tue, 30 Jan 2007 08:38:13 +0200 (EET)

>>> Chris Newman

Charles' code was using MIME -module, so it was not all perl code
required.

/ Kari Hurtta

---

From: Jeff Yeh <[email-masked]>
Date: Tue, 30 Jan 2007 15:28:36 +0800

+1

Jeff Yeh
-TWNIC

---

From: Martin Duerst <[email-masked]>
Date: Tue, 30 Jan 2007 12:30:21 +0900

At 19:13 07/01/26, Harald Tveit Alvestrand wrote:
>The discussion of whether or not we need a header marker for marking UTF8SMTP messages that actually use UTF-8 characters has reached a point where it seems that no new technical information is being provided.
>
>At this point, the chairs wish to see if the group is near a possible consensus on the issue.
>
>PROPOSED RESOLUTOIN:
>
>There is no significant operational or implementation benefit in having a marker in the headers of UTF8SMTP messages, and significant additional complexity. Therefore, the proposed "Header-Type" section, draft-ietf-eai-utf8headers-02 section 5, will be removed.
>
>Note that this is not a call on the proposal to have a similar marker in the SMTP protocol; that's a separate issue.
>
>POSSIBLE RESPONSES:
>

I agree. I the test implementation a student of mine is working on,
we haven't used this header (but we have generated it). We have implemented
the sending side of SMTP in an MUA. The MUA keeps its own information
on emails in draft stage or 'ready to send' stage.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 30 Jan 2007 17:18:33 -0000

I have just been looking through the code of sendmail, and it seems that
it is done in order to avoid some buffer overflows elsewhere if Content-*
headers of inordinate length are submitted. But the performance hit seems
quite large, since it resuses the code that does genuine 8to7 converions -
it bypasses the actual conversion bit, but still does a lot of stuff that
is unnecessary for the limited purpose of checking header lengths. I doubt
that other MTAs do such a complete MIME parse as a matter of routine, and
there must be more efficient ways in which sendmail could have fixed this
particular problem.

It does, of course, check for an 8th bit set anywhere in the message as it
reads it in after the DATA command (as John mentioned) for the purpose of
deciding whether 8BITMIME downgrading needs to be considered.

---

From: Kari Hurtta <[email-masked]>
Date: 30 Jan 2007 19:29:31 +0200

> deciding whether 8BITMIME downgrading needs to be considered.

Originally that was invented to protect some MUAs. Quote from
RELEASE_NOTES of sendmail:

8.10.0/8.10.0   2000/03/01
<...>
        New option MaxMimeHeaderLength which limits the size of MIME
                headers and parameters within those headers.  This option
                is intended to protect mail user agents from buffer
                overflow attacks.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 30 Jan 2007 17:33:23 -0000

>> Japanese, even though those languages use essentially the same "CJK"

Yes, the relevant bit seems to be:

    Many operations, including high quality formatting, text-to-speech
    synthesis, searching, hyphenation, spellchecking and so on benefit
    greatly from access to information about the language of a piece of
    text. [WC 3.1.1.4].

Which seems to cover the sort of situation that I mentioned.

    Protocols that transfer text MUST provide for carrying information
    about the language of that text.

    Protocols SHOULD also provide for carrying information about the
    language of names, where appropriate.

And, within headers, that would seem to apply to <display-name>s,  
<comment>s and <unstructured>s (e.g. in Subjects), all of which can freely  
use UTF8 in our proposals.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 30 Jan 2007 17:58:09 -0000

> Sun's MTA also parses the MIME structure.

I thought Sun still used sendmail. For sure, that is what comes with all  
Solaris versions <= 10.
>
> It was very thoughtful of Charles to provide perl code that proves just  
> how simple and straightforward it is to do MIME parsing.  I couldn't  
> have supported my position as well as he did.  The several C MIME  
> parsers I've written in the past for different purposes are longer but  
> still straightforward.  The one which uses Boyer-Moore to look for MIME  
> boundaries and mbox delimiters is a bit less simple; I was probably  
> over-optimizing for performance when I wrote that code...

It is simple conceptually (provided you spot all the bits that need  
attention), but it is a significant performance hit if it has to be  
repeated on every email that passes through every agent. And, as I said,  
it is unlikely to happen in people's ad hoc scripts.

---

From: [email-masked]
Date: Wed, 31 Jan 2007 21:15:10 +0900 (JST)

In downgrading, we can use 'Downgraded:' header as a downgraded marker.

--
Kazunori Fujiwara, JPRS

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 31 Jan 2007 17:08:57 -0000

> RELEASE_NOTES of sendmail:
Which is odd, because the later change to make it use that check by  
default referred to protecting MTAs, not MUAs.

BTW, I tried an experiment to see how much overhead this check cost, and I  
could not detect any difference, which tends to suggest that sendmail uses  
up an awful lot of time elsewhere (all I was doing was sending a 22MB  
attachment to localhost, and I checked with DBX that it only called the  
MIME check in 1 of the 2 cases).

---

From: Yoshiro YONEYA <[email-masked]>
Date: Thu, 1 Feb 2007 07:34:01 +0900

-- 
Yoshiro YONEYA <[email-masked]>

---

From: Yoshiro YONEYA <[email-masked]>
Date: Thu, 1 Feb 2007 07:34:01 +0900

---

From: Chris Newman <[email-masked]>
Date: Wed, 31 Jan 2007 15:14:25 -0800

Charles Lindsey wrote on 1/31/07 17:08 +0000:
> attachment to localhost, and I checked with DBX that it only called the  MIME
> check in 1 of the 2 cases).

On modern hardware, the MTA bottlenecks are either the spam/virus engine or the 
disk I/O subsystem.  The cost to scan the MIME structure (or scan/verify a DKIM 
signature) is negligible by comparison.  You'd probably see the same result 
(MIME scanning has no real cost) on any modern MTA which properly commits 
messages to disk with fsync/fdatasync or the equivalent.

                - Chris