Subject: Re: [EAI] #1485: Choice of body part for transport of UTF8SMTP messages

From: Chris Newman <[email-masked]>
Date: Thu, 31 May 2007 19:41:18 -0700

NOTE: This is my opinion as a technical contributor, not as area director.

When an SMTP server advertises "UTF8SMTP", it is saying "I accept either RFC 
822 and UTF8SMTP messages".  So the DATA command (or BDAT/BURL command) is not 
a type label, it's merely a mechanism to carry one of the supported types. 
Indeed SMTP was originally designed as a generic transfer protocol and the fact 
RFC 822 / MIME is the only format presently used is an accident of history.

What concerns me with UTF8SMTP content in a message/rfc822 body part is the use 
of an explicit type label in a way that is known to be incompatible with the 
current defined meaning for that type.  Perhaps we could get away with it if we 
could guarantee it would never ever leak to a non-UTF8SMTP system.  But this is 
a redefinition of the meaning of a standards track type label as opposed to an 
isolated extension to existing protocols.  I'm a lot more comfortable with 
message/utf8smtp as today's MIME compliant systems are expected to treat 
unknown message subtypes as equivalent to application/octet-stream, much as 
systems unfamiliar with application/zip treat it as an opaque object.

With message/utf8smtp, existing message/rfc822 systems can be vetted to make 
sure they're UTF-8 clean, address the UTF-8 security considerations and then 
add "message/utf8smtp" to the list of compound types they descend.

                - Chris

Kari Hurtta wrote on 5/27/07 6:42 PM +0300:
> Harald Tveit Alvestrand <[email-masked]> writes in gmane.ietf.ima:
>
>> Subject: [psg.com #1485]: UTF8HDR 4.6/DSN: Choice of body part for
>> transport of UTF8SMTP messages
>>
>> The choice of the body part to transport UTF8SMTP messages in a message
>> has proved controversial.
>> Suggestions include:
>>
>> - message/utf8 (inconsistent with certain statements in the MIME RFCs)
>> - application/utf8smtp (stylistically questionable)
>> - message/rfc822 (incompatible with existing definition)
>>
>> A decision needs to be made.
>
> I prefer
>     UTF8SMTP messages labeled as message/rfc822 on UTF8SMTP environment
>     and then downgraded to RFC 2822 messages when then leave UTF8SMTP
>     environment.
>
>
> On downgrading there is two different cases
>     A)   bounce/reject is available
>     B)   bounce/reject is not available
>
> Case A is when downgrading is done by UTF8SMTP gateway on smtp when envelope
> sender is not <>.
>
> Case B is when downgrading is done after delivery (for example on IMAP or
> POP server) or when downgrading is  done by UTF8SMTP gateway on smtp when
> envelope sender is <>.
>
> In case A if downgrading of attached message (with type message/rfc822)
> fails, enclosed message is rejected or bounced (NDN generated).
>
> In case B is it is just better leave out data what is not downgradeable
> (for example if there is no alternative address for some address on header)
> or on extreme case left out whole message/rfc822 body part.
>
>
> ( In case B also some other options exists, but they are better discussed
>   when downgrading document is on subject.  Also if embedded message is not
>   downgradeable, one option is do some sort encapsulation on both
>   cases A and B )
>
>
> For why I prefer message/rfc822 I have mentioned these reasons already:
>
>    When working group was decided that there is no need labeling
>    for UTF8SMTP messages, that quite clearly follow that UTF8SMTP
>    messages does not need new label (ie. message/utf8 type) when
>    they are attached to another message.
>
>    When whole message is not need labeling on UTF8SMTP case (ie no ESMTP
>    paramater), there is no need also different kind labeling if
>    message is attached to another message.
>
>    That makes "message/rfc822" just mean "mail message". And consistent
>    that "message/rfc822" is just that type what occurs inside of DATA
>    on smtp.
>
>    Actually "message/rfc822" is not mean to be only mail message,
>    but superset of it (rfc 2046):
>
>       It should be noted that, despite the use of the numbers "822", a
>       "message/rfc822" entity isn't restricted to material in strict
>       conformance to RFC822, nor are the semantics of "message/rfc822"
>       objects restricted to the semantics defined in RFC822. More
>       specifically, a "message/rfc822" message could well be a News article
>       or a MIME message.
>
>
>    If there is reason why "message/rfc822" can not be used to
>    mean any mail message on UTF8SMTP universe, same problems are
>    also on top level.
>
>    For example if there is some problems for IMAP on UTF8SMTP universe,
>    when UTF8SMTP message is attaches as "message/rfc822", same problems
>    also aply to top level message.
>
>
> This issue have subject UTF8HDR 4.6. That chapter do not mention
> MIME type of UTF8SMTP message at all. Instead that chapter says:
>
>    the object defined in [EAI-dsn] is intended to be transportable as
>    part of an ordinary [RFC2822] message.
>
> That implies that UTF8SMTP messages transported inside of messages are not
> downgraded. I prefer that they are downgraded.  That is important when
> this MIME type is just used for forwarding.
>
> On situation where UTF8SMTP messages start appear is when MUA and MSA supports
> UTF8SMTP, and just non-ascii subject is used. If embedded messages are
> downgraded when message is forwarded outside of UTF8SMTP environment, that
> does not cause surprises.   If downgrading is not done (and encapsulation is
> done instead), recipient get different situation depending was just UTF8SMTP
> MSA+MUA used.
>
> That is: End result outside of UTF8SMTP environment should be same on
> following  cases when non ascii subject is typed:
>         1) MUA sends ordinary MIME message (non UTF8SMTP message)
>         2) MUA sends UTF8SMTP message
> On both cases end result outside of UTF8SMTP environment should be ordinary
> MIME message with MIME encoded words on subject. And that should happend
> even when this message leaves UTF8SMTP environment inside of another message.
>
> Perhaps that can be say also on following way: "Fully downgradeable UTF8SMTP
> messages should be downgraded (and not encapsulated) when they leave UTF8SMTP
> environment. And that does not depend is message stand alone or is it embedded
> inside of another message."
>
>
>
> MIME type of UTF8SMTP message is mentioned on chapter 4.2 instead:
>
>    In all those header fields, Observe that such Content-Type and other
>    header fields may be found both amongst the top-level fields of a
>    message and also within multiparts; and also that a complete message
>    conforming to this document may now appear as a message/rfc822 (in
>    both cases, subject to downgrade when that is necessary)
>
>
>
> application/utf8smtp have some values on situations where message is
> not downgradeable. In that case that type can be used for encapsulation,
> but I do not recommended it for general use.
>
>
> message/utf8 is bad if  UTF8SMTP content is not downgraded (and retyped
> as message/rfc822) when message  leaves UTF8SMTP environment. Some reasons
> I have discussed on my messages with subjects
>         - Ascii user X sends mail
>         - UTF8SMTP / 8BITMIME / ASCII -universe amd message/utf8smtp
>
>
> / Kari Hurtta
>
>
>
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ima
>

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Fri, 01 Jun 2007 11:13:41 -0400

Hi.  

I am personally gone back and forth on this one, but, at this
point, find myself largely in agreement with Chris.  See below.

--On Thursday, 31 May, 2007 19:41 -0700 Chris Newman
<[email-masked]> wrote:

> When an SMTP server advertises "UTF8SMTP", it is saying "I
> accept either RFC 822 and UTF8SMTP messages".  So the DATA
> command (or BDAT/BURL command) is not a type label, it's
> merely a mechanism to carry one of the supported types. Indeed
> SMTP was originally designed as a generic transfer protocol
> and the fact RFC 822 / MIME is the only format presently used
> is an accident of history.

As a historical note on that accident, some of us tried to
preserve that "generic transfer" idea for many years.  The
notion of being able to accept an envelope-only message (i.e.,
construct headers on delivery if at all) was actually critical
to the successful working of a number of gateways, just as the
ability of accept header-only (RFC822, more or less) messages
and construct an envelope in flight was essential for others. 

The issue was especially important before DRUMS deprecated the
SMTP SEND command: many of us believed that the message data for
SEND should be sent header-free since display of the headers on
the user terminal would make the IM character of SEND
essentially useless.  Others believed that the headers should be
sent but perhaps stripped by the receiver-SMTP, with only body
content displayed.  Of course, if one sent things without
headers, and whatever passed for a mail store on the delivery
system required them, SAML and SOML became tricky... but SMTP
was designed for a "deliver to mailbox", rather than "deliver to
mail store" model, and that, too, changes things.

However, and particularly relevant to this work, the end of
SMTP-as-generic-transport came, I think, with the extension
mechanism.  If one is going to have extensions that make
assumptions about message content or that might result in
transforming it, then there really needs to be a well-understood
header format.  So sending or accepting EHLO are now effectively
an assertion of MIME-compliance.

And that leads me to...

> What concerns me with UTF8SMTP content in a message/rfc822
> body part is the use of an explicit type label in a way that
> is known to be incompatible with the current defined meaning
> for that type.  Perhaps we could get away with it if we could
> guarantee it would never ever leak to a non-UTF8SMTP system.
> But this is a redefinition of the meaning of a standards track
> type label as opposed to an isolated extension to existing
> protocols.  

More important, I think all of our experience is that an
assertion that one can guarantee that anything that appears in
message content will never leak should be met with pained
laughter.   I don't see a big problem with modifying the
presumed rule against having any message/ subtype whose content
is non-ASCII, but I think there is a sufficient expectation that
the headers of message/rfc822 is ASCII-only and non-extended
that trying to change it is bad news -- the alternate use is
just not sufficiently protected by an UTF8SMTP envelope
environment.

> I'm a lot more comfortable with message/utf8smtp
> as today's MIME compliant systems are expected to treat
> unknown message subtypes as equivalent to
> application/octet-stream, much as systems unfamiliar with
> application/zip treat it as an opaque object.
> 
> With message/utf8smtp, existing message/rfc822 systems can be
> vetted to make sure they're UTF-8 clean, address the UTF-8
> security considerations and then add "message/utf8smtp" to the
> list of compound types they descend.

Yes.  In retrospect, the strong position I've taken against
making this a subtree of application/ has to do with the opacity
of that type.  But treating it as a message/ subtype that might
fall over to application/octet-stream works for me and seems to
be a reasonable long-term, as well as short-term, mechanism.

One request/suggestion: since users will see this subtype (just
as they see, and sometimes even talk about, other subtype
names), I hope that people who are better at picking names than
I am can come up with something other than message/utf8smtp.
First, this ought to have to do with the message, rather than
how it is transported.  And, second, this has more to do with
i18n than with UTF-8 itself: I fear that message/utf8smtp will
encourage someone to start thinking about message/is8859-1smtp
or some such nonsense.  I'd hope for something that is more
along the lines of message/intl-email, but I'm not good at
actual names.

(I'm not ignoring Kari's comments about downgrading these
things.. I just want to think more about that before commenting.
If we are going to downgrade these things, they will need
special handling in the downgrading document, but maybe we can
isolate the issue there.)

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 04 Jun 2007 12:09:53 +0100

On Fri, 01 Jun 2007 03:41:18 +0100, Chris Newman <[email-masked]>
wrote:

> What concerns me with UTF8SMTP content in a message/rfc822 body part is  
> the use of an explicit type label in a way that is known to be  
> incompatible with the current defined meaning for that type.  Perhaps we  
> could get away with it if we could guarantee it would never ever leak to  
> a non-UTF8SMTP system.

Yes, that is an essential feature of any proposal to allow UTF8SMTP
content in a message/rfc822.

>  But this is a redefinition of the meaning of a standards track type  
> label as opposed to an isolated extension to existing protocols.

That might well be a valid argument were it not for the fact that we have
already changed what is allowed within multipart/* types (when within the
UTF8SMTP universe, of course). That is now solidly written into into both
the utf8headers and downgrade drafts, and noone has raised any ISSUE
against it. It leads, of course, straight to a requirement to scan the
full body of each message looking for UTF-8 in suspicious places before
you can declare that it is not a UTF8SMTP message, but we agreed on all
that at the timne when we abolished the Header-Type header field.

Now, since a message/rfc822 is rather similar in its structure to a
multipart with only one part, it follows that whatever mechanisms an
implementor has to deploy to deal with our extended multipart/* types is
readily adaptable to deal with an extended message/rfc822, and whatever
risks arise from UTF-8 leaking into the non-UTF8SMTP world are identical
in the two cases.

> I'm a lot more comfortable with message/utf8smtp as today's MIME  
> compliant systems are expected to treat unknown message subtypes as  
> equivalent to application/octet-stream, much as systems unfamiliar with  

It is not clear to me that that requirement applies to unknown message/*
types, since adhering to it produces objects that violate that "EXPRESSLY
FORBIDDEN" provision in RFC 2045.

But, first of all, we are proposing a new message/utf8smtp type without
being given any clear indication of the properties it is supposed to
possess. Everyone who advocate it seems to be working to a different model
of what it actually does. So please can we have answers to some basic
questions:

1. Is a message/utf8smtp supposed to be transportable without serious loss
of information over the existing network? If not, then it would need to be
downgraded somehow when leaving the UTF8SMTP world.

2. But if so, then what happens when it needs to be offered to a system
that does not support BITMIME? Is it:
     a) rejected (5xx response or bounce) and is that rejection silent?
     b) converted to Q-P or Base64, as if it were application/octer-stream?
     c) passed onwards with the 8bit stuff still present?
     d) have the 8bit stuff truncated to 7bit?

3. IMO, only (a) is fully standards compliant, though you appear to argue
that (b) is compliant too.

4. But, more to the point, what is the actual behaviour of currently
deployed software, since that forms the environment with which EAI will
need to interoperate when it is first deployed?

So far, there have been reported 3 systems which exhibit behaviour (c),
which between them account for a significant portion of the installed base.

There have been rumours of ancient systems that supported (d).

I have seen NO reports of systems which implement either (a) or (b)
(though I have asked for such reports).

It is really up to the people who advocate message/utf8smtp to provide
answers to these questions because, unless a significant part of the
installed base currently supports (a) or (b), or unless the answer to
Question 1 is "No", then message/utf8smtp is simply not going to work.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5