Subject: [EAI] POLL: HDR= extension on MAIL FROM

From: Harald Alvestrand <[email-masked]>
Date: Wed, 04 Apr 2007 16:13:01 +0200

As of this moment, I believe the current alternatives are on the table:

A - New SMTP argument to MAIL FROM: HDR=, with defined value HDR=UTF8

This indicates that the message being sent contains UTF-8 data in MAIL 
FROM, RCPT TO or headers (including inner body headers), which means 
that it needs downgrading before it can be passed to an ASCII-only 
recipient.

Advantage: In the case where the recieving MTA knows which recipients 
can handle UTF8SMTP mail, and the MTA does not have downgrade 
capability, it can reject recipients by replying to the RCPT TO command 
with an appropriate error code.
(It will still have to do downgrade-or-bounce if it discovers the final 
destination after accepting the RCPT TO command.)

Advantage: The MTA can decide on whether downgrading is required based 
on the SMTP argument, and can treat any 8-bit character in the headers 
as a protocol error if the flag is not given.

B - No new SMTP argument.

Advantage: Less complexity. No need for an MTA to consider whether or 
not to apply downgrading based on the error code it got back; any 
recipients rejected with a 5XX code can be considered rejected immediately.

Advantage: Less cruft for the UTF8SMTP case - cleaner end state.

There are other opinions on the relative merits of the two alternatives, 
but I believe those are the most important ones

If you have an opinion on this issue, please send a note to the mailing 
list with ONE of the following statements:

A - I support HDR=UTF8
B - I support NOT adding HDR=UTF8
C - I have an opinion different from the 2 alternatives given, which is....

This poll closes in 7 days, on April 9, 2007.

                       Harald, for the chairs

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 04 Apr 2007 16:18:51 +0200

A - I support HDR=UTF8

---

From: Yangwoo Ko <[email-masked]>
Date: Thu, 05 Apr 2007 00:28:25 +0900

> B - I support NOT adding HDR=UTF8

Support B.

This will (hopefully) result in a more cleanly internationalized email 
environment. (Next generation email developers will wonder why we should 
ALWAYS put HDR=UTF8 if we go for A)

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 04 Apr 2007 18:07:57 +0200

Yangwoo Ko wrote:

> Next generation email developers will wonder why we should
> ALWAYS put HDR=UTF8 if we go for A

That's IMO not the case, it's possible to have this scenario:

envelope:
 MAIL FROM:<[email-masked]>
 RCPT TO:<[email-masked]>
 RCPT TO:<[email-masked]>
 DATA
  From:[email-masked]
  To:[email-masked]
  ...

An ordinary message/rfc822 from an ASCII-address, to a second
ASCII address, with a blind carbon copy to your UTF-8 address.

If the 2nd address is a rfc822 mailbox the mail can be accepted
as is for 2nd@ascii (no message/utf-8, no downgrading required).

Another slightly more tricky scenario without HDR=UTF-8:

envelope:
 MAIL FROM:<[email-masked]> ALT-ADDRESS=[email-masked]
 RCPT TO:<[email-masked]>
 DATA
  From:[email-masked]
  To:[email-masked]
  ...

You used your default envelope sender address including an
ASCII alternative.   Knowing that the mail is to me you did
not send a message/utf-8, but an ordinary message/rfc822.

The receiving SMTP might try to record the Return-Path incl.
your UTF8SMTP address, thereby modifying the message/rfc822
into a message/utf-8.  But if it knows that my mailbox can't
handle message/utf-8 that would be a bad move, and accepting
the mail as is with an ASCII Return-Path: <[email-masked]>
will work.

Without a HDR=UTF-8 the mailer can assume that the DATA is
an ordinary 8BITMIME message/rfc822 (with an ASCII header),
and reject it after DATA if that turns out to be a lie.

Frank

P.S.:  I've omitted BODY=8BITMIME in the example envelopes

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Apr 2007 15:26:13 -0400

--On Wednesday, 04 April, 2007 16:13 +0200 Harald Alvestrand
<[email-masked]> wrote:

> If you have an opinion on this issue, please send a note to
> the mailing list with ONE of the following statements:
> 
> A - I support HDR=UTF8

I support B -- NOT adding an additional parameter.

My reasoning is a trifle different from your summary so, for
whatever it is worth...

(i) We know this parameter is not strictly necessary, even if it
might be a convenience in some cases.

(ii) Every additional parameter we add that isn't strictly
orthogonal to all the others creates an additional opportunity
for silly states.  That, in term, requires that we understand,
and probably specify actions for, the error conditions, and
increases the risks of interoperability problems and bugs.
>From a syntax standpoint, if we add HDR=UTF8, a mail transaction
can contain any combination of zero or more of HDR, BODY, and
ALT-ADDR.  So, for example...

	Are all of those combinations meaningful?  If they are
	not, and some of the inconsistent ones are specified,
	what do we expect to have happen?  In particular, do we
	treat HDR=UTF8 as implying BODY=8BITMIME strongly enough
	that the latter can be omitted?  Could that cause other
	problems?  Or, if we don't make that type of rule, could
	HDR without BODY take us back down the path of "Just
	Send <whatever>"?  

While I think that, given time and energy, we could answer those
questions, produce firm specifications, and expect that most
(but not all) implementations would understand and follow the
instructions well enough to get things right, it seems to me to
be a high cost and high risk for something that is not
absolutely necessary.    And...

(iii) If our target and expectation are a long-term mail
environment in which substantially everyone supports UTF8SMTP
and those who do not understand that they are running crippled
and obsolete systems, then this parameter becomes long-term
clutter with no value.  The same could be said about 8BITMIME
today and perhaps, long-term, about ALT-ADDR itself.  But the
existence of two of them is, IMO, a strong reason to not add a
third, not a justification for it.

      john

---

From: Soobok Lee <[email-masked]>
Date: Thu, 5 Apr 2007 09:15:35 +0900

> third, not a justification for it.

UTF8 is one of ASCII compatible encodings of full Unicode characters. 

KSC5601,BIG5 etc are all ASCII compatible encodings for subsets of Unicode
characters. Some mail vendors want to use "HDR=KSC5601" for regional needs.

Moreover, we can't assume there won't come any other ASCII compatible 
encoding UTF?? (other than UTF8) of full Unicode characters in the future.

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Thu, 5 Apr 2007 09:41:58 +0900

> C - I have an opinion different from the 2 alternatives given, which is....

C)  HDR= is newly defined parameter. But HDR=UTF8 is optional and may be omitted.

    Mail servers may assume HDR=UTF8 if HDR= parameter is not prenet in SMTP session.

This is for making room for other encoding than UTF8 in HDR=.

Soobok

---

From: Kari Hurtta <[email-masked]>
Date: 05 Apr 2007 08:16:15 +0300

A - I support HDR=UTF8

(however not very strongly)

   Paramater value should NOT look like charset name, therefore

              HDR=UTF8SMTP

   is better value for HDR.

( otherwise there is danger for invention of HDR=KSC560
  as Soobok Lee mentions)

( I think that in somewhere on Header-Type: label 
  thread I mentioned that I prefer ESMTP parameter. )

/ Kari Hurtta

---

From: Yangwoo Ko <[email-masked]>
Date: Thu, 05 Apr 2007 14:37:05 +0900

Really sorry for making you confused. Please blame my poor English 
rather than me :-( .

My gut feeling said that IF we are very successful (i.e. new clean 
UTF8SMTP world), lazy developers will always put HDR=UTF8 regardless of 
whether it is really needed or not because;

(1) it is harmless if there is no UTF8SMTP-incapable server en route, 
which will be very often the case.

(2) it doesn't need to implement and run codes to check whether this 
option should be added or not.

---

From: Soobok Lee <[email-masked]>
Date: Thu, 5 Apr 2007 15:36:02 +0900

>    Paramater value should NOT look like charset name, therefore

I agree with  you that UTF8SMTP is better and clearer.

This is somewhat off the topic, but, I am not UTF8 partisan. 

Some proprietary solutions may introduce HDR=KSC5601SMTP
for internal or regional use for their own *risk*/benefit. 
many email portals are using local charset encoding for 
both header and *body* in 8-bit clean format of single message
file for storage/transfer efficiency.

We don't need to try to standardize such non-standard
way of SMTP here.  Defining some "HDR=" parameter values is enough.
Let them extend parameter values for their own risk /benefits
in interoperable ways. 

Soobok

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 05 Apr 2007 12:19:19 +0100

> that it needs downgrading before it can be passed to an ASCII-only  

Not sure that is quite right. Frank has raised the possibility that there  
might be  a MAIL FROM or a RCPT TO with utf-8 address (possibly  
+ Alt-address) but a body that was 100% pure ASCII. It is arguable that  
case does not need HDR=UTF8 because the MAIL FROM or RCPT TO commands  
already show the extent to which UTF8SMTP is required for the next hop.

> as a protocol error if the flag is not given.

Ans that is the argument that was given for the Header-Type header, but it  
did not carry thew day then.

So I suppose that makes me C, because I don't think the definition is  
quite right.

But between A and B I am more or less neutral - perhaps marginally in  
favour of B, on the grounds that even if HDR-UTF-8 is not given, you can't  
be sure  until you actually look at the body (though that means you can  
only tell after DATA, which might make you decide to reject rather than do  
a last-minute downgrade).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 05 Apr 2007 17:25:02 +0200

Yangwoo Ko wrote:
 
> UTF8SMTP world), lazy developers will always put HDR=UTF8 regardless
> of whether it is really needed or not because;

> option should be added or not.

Yes, that's a plausible scenario.  OTOH if receivers reject HDR=UTF8 --
or as Kari said HDR=UTF8SMTP -- for mailboxes not accepting this type
of message ("strict" message/utf-8 or "lazy developer" message/rfc822)
the senders getting unnecessary bounces would submit bug reports... :-)

The parameter makes it easier to "opt-in" to EAI for the parts of the
world where it's not very relevant (Latin-1 languages close to ASCII),
and where users will keep their "legacy" mailboxes for some decades.

Without the parameter well-behaved MXs without "downgrade" capability
will have to reject any UTF8SMTP message after DATA as soon as there's
a single "legacy" mailbox.  Ordinary users do not upgrade unless it's
very important, or they install a new operating system.  Or mybe it's
only me (= lazy user). 

Frank

---

From: Tony Finch <[email-masked]>
Date: Thu, 5 Apr 2007 17:15:03 +0100

I don't have any particular opinion, but I wonder why the BODY= parameter
was deemed necessary if we now think a HDR= parameter is not.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH FAIR ISLE: NORTHEAST 4 OR 5, BACKING NORTHWEST 3 OR 4. ROUGH OR VERY
ROUGH. RAIN OR DRIZZLE THEN FAIR. MODERATE OR POOR BECOMING GOOD.

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Apr 2007 12:46:23 -0400

--On Thursday, 05 April, 2007 17:15 +0100 Tony Finch
<[email-masked]> wrote:

> I don't have any particular opinion, but I wonder why the
> BODY= parameter was deemed necessary if we now think a HDR=
> parameter is not.

You'd need to ask Ned -- he may remember the discussion, if
there was one.  But, if I needed to guess, I would say that we
were 

	(i) more optimistic about the ability to believe
	parameter values than we are now
	
	(ii) more optimistic about the ability to delay in-depth
	message examination and the importance of doing so than
	we are now
	
	(iii) justifiably concerned about the implications of
	the Just-send-8 trend and a need to construct a wall
	around it.

If your question is really about whether we would require that
parameter if we were doing 8BITMIME for the first time today, I
would share your doubts.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 05 Apr 2007 19:20:59 +0200

> I agree with  you that UTF8SMTP is better and clearer.

+1, as for BODY=8BITMIME, using the name of the extension.

> This is somewhat off the topic, but, I am not UTF8 partisan.

We could start a club of "legacy charset dinosaurs"... :-)

But in practice UTF-8 will be the "new ASCII", with those
partisans claiming that it already is for at least 7 years.

> Some proprietary solutions may introduce HDR=KSC5601SMTP

<shudder />  No, they won't, not on port 25, it's a horrible
idea.  I like pc-multilingual-850+euro, but not in a header.

In <org/[email-masked]> you wrote:
| Moreover, we can't assume there won't come any other ASCII
| compatible encoding UTF?? (other than UTF8) of full Unicode
| characters in the future.

It's theoretically impossible to design an "ASCII compatible"
UTF "better" than UTF-8 -- i.e. without giving up one or more
essential features of UTF-8.  Ignoring all draft-terrell-* :-)

Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Apr 2007 13:51:48 -0400

> partisans claiming that it already is for at least 7 years.

I have to agree with this.  And anyone who considers me a UTF-8
partisan, or Unicode partisan more generally, should consult the
UTC.    But, for international use, the choices are between
Unicode (in some coding), local character sets with universal
labeling and great potential for an N**2 problem, or going off
and inventing a new UCS.  Fortunately or unfortunately, there is
absolutely no evidence of energy for the third and little
evidence that, faced with similar constraints, a new effort
would produce a significantly better overall result than
Unicode. So it behooves us to accept Unicode (and, for many
purposes, UTF-8), mostly stop complaining about it, and try to
make it work.
 
> essential features of UTF-8.  Ignoring all draft-terrell-* :-)

Well, I think that depends on one's objectives.  Put
differently, some of the "essential features" of UTF-8 are
pathological in some contexts.  Punycode points to a family of
block encodings that are better for some purposes: ignoring the
specifics of the IDNA application, they use an offset into
Unicode followed by coding that is dependent on the block/region
starting with that offset.  It has much better properties than
UTF-8 if one is concerned about characters that are "high" in
the BMP or even higher.

Similarly, an essential property of UTF-8 is that ASCII
characters are coded as themselves.  Even taking that as a
given, it then marches up the Unicode spec, with the next blocks
of codes being encoded into two octets and the strings getting
longer as one goes up.  In principle, one could design a code
that would preserve this ASCII property but subtract the value
of every character above U+007F from 0xFFFF, yielding a signed
integer that was negative in the rest of the BMP and positive
otherwise.  If one then applied a UTF-8-like coding to that
integer, one would end up with a code that maintained the
essential ASCII-as-ASCII property, but was much more friendly to
non-European scripts than to European ones.

All of that said, I don't see any strong movement away from
UTF-8 and, even if there were, would see immense
interoperability advantages to converting to UTF-8 before
putting something onto or pulling it off of, the proverbial wire.

     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Apr 2007 14:29:39 -0400

--On Thursday, 05 April, 2007 15:36 +0900 Soobok Lee
<[email-masked]> wrote:

> way of SMTP here.  Defining some "HDR=" parameter values is
> enough. Let them extend parameter values for their own risk
> /benefits in interoperable ways. 

Hmm.  A few observations, partially as free advice for those
contemplating doing this...

First, one thing we keep learning, over and over again, is that
things leak.  There is almost no such thing as "internal or
regional use" that doesn't end up showing up somewhere else and
causing greater or lesser problems when it does.  This is a
strong argument for localizing all one wants to but, when
something is actually transmitted, using the global/standardized
forms and then, if necessary, re-localizing at the far end.

Second,  if one is going to invent locally-coded header forms
and put them on the wire, contrary to the advice above, there
are many reasons why it is probably better to do that with a
switch --e.g., advertising a EHLO option of
X-Korean-special-Mail and using MAIL FROM:<a@b>
X-Korean-special-Mail-- that to fuss around with fine-tuning the
argument values to standard parameters and hoping that the right
thing will happen.   Experience indicates that, if the only
valid arguments to HDR= are "UTF8" or "ASCII" (or some alternate
spelling of those), then an implementation that treats
everything as UTF-8 may simply ignore the parameter, getting
into trouble if KSC5601 is specified instead.

In any event, this is not an argument for a HDR= parameter.   If
anything, it is an argument against such a parameter because, if
one wants to make a local extension that violates the standard,
interoperability considerations suggest that it be as clear and
distinctive as possible.

best,
      john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 05 Apr 2007 22:24:29 +0200

> pathological in some contexts.

The essential features I've in mind are:

1: anywhere in a stream of octets it's always clear where you are,
   at an ASCII character, at the begin of a multi-octet UTF-8, or
   in the tail of a multi-octet UTF-8 encoding.
2: each Unicode point has only one corresponding "canonical" UTF-8
   encoding.
3: flipping a single bit destroys precisely one character.
4: the lead octet determines the length of the encoding.
5: if an octet appears to be ASCII it really is ASCII.

Encoding is (almost) straight forward modulo 64.  STD 63 decoding
is minimally harder, but still trivial.

> Punycode points to a family of block encodings that are better
> for some purposes: ignoring the specifics of the IDNA application,
> they use an offset into Unicode followed by coding that is
> dependent on the block/region starting with that offset.

Yes, but AFAIK it can't encode Unicode strings of arbitrary length,
please correct me if that's wrong, the one thing I know for sure
about punycode is that it's *not* trivial... :-)  BOCU-1 could be
an alternative, but it doesn't have property (3), flipping a bit
could destroy a complete line.

> In principle, one could design a code that would preserve this
> ASCII property but subtract the value of every character above
> U+007F from 0xFFFF, yielding a signed integer that was negative
> in the rest of the BMP and positive otherwise.  If one then
> applied a UTF-8-like coding to that integer, one would end up
> with a code that maintained the essential ASCII-as-ASCII
> property, but was much more friendly to non-European scripts
> than to European ones.

FRom u+0080 up to u+7FFF you'd still have to encode 16 bits, an
"UTF-8 like coding" would then need three 10xx xxxx tail octets,
and one 1110 0001 lead byte.  It starts to get friendlier with
u+8000 ff., but the upper part of the BMP contains lots of code
points never used in ordinary texts (like surrogates and 34 non-
characters).  I don't think that that's really "better", it's a
kind of permutation.  I'm not sure what your algorithm does with
code points in planes 1 to 15, maybe just use the original UTF-8.

> I don't see any strong movement away from UTF-8

I don't see any movement away from UTF-8 at all.  BOCU-1 didn't
make it so far, and SCSU isn't MIME compatible.  And "UTF-4" is
only a theoretical construct imitating all features of UTF-8,
replacing "ASCII" by "ASCII or u+00A0 up to u+00FF" (Latin-1).

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 05 Apr 2007 23:24:28 +0200

> "UTF-8 like coding" would then need three 10xx xxxx tail octets,

Correction:  It would use 1110 xxxx 10xx xxxx 10xx xxxx (3 octets)

And with 110x xxxx 10xx xxxx it can encode 11 bits in 2 octets,
for John's algorithm that boundary is at u+F800.  For most code
points in the BMP it would need three octets just like UTF-8.

Frank

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 10:13:32 +0900

> non-European scripts than to European ones.

<OT>
Thanks for your interesting comment. Yes. There is much room
for improvement or adjustment in future UTF8-variant  encodings 
for non-western-european characters. 

UTF8 multiplies length of strings by 50%~200%, compared with local charset
encodings.

Cyrillic/Greek/Hebrew/Arabic strings : 200 % ( 1->3 octets)
Chinese/Japanese/Korean strings (DBCS): 50 % ( 2->3 octets)

UTF8 is still not in overwhelming use in email messages worldwide, 
even though it was born 7~8 years ago.
Main use of utf8 encoding in email is a fallback encoding to encode 
non-local-charset characters in headers or body.

If you consult gmail/yahoo/hotmail personels, they will reply to you  that
they welcome local charset encodings as currently most favored encodings
over utf8 for email messages.

</OT>

My position is this : Let's go with UTF8 now, but let's leave rooms for
other encodings for future needs.

Soobok

---

From: Yangwoo Ko <[email-masked]>
Date: Fri, 06 Apr 2007 10:29:30 +0900

> other encodings for future needs.

Significant increase of network throughput and even faster drop of 
storage prices make me little impressed by the fact that UTF-8 is not 
the best solution for some languages (including ours :-) ).

In addition to this, as far as my mail experiences are concerned, 
majority of bits transmitted over email messages are from big 
attachments such as PDFs, PPTs, and ZIPs, which are not that relevant to 
character encoding issues.

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 10:37:21 +0900

On Thu, Apr 05, 2007 at 02:29:39PM -0400, John C Klensin wrote:
> thing will happen.  

Yes.  For this case, HDR= alone is not enough, and local EHLO option
like above is needed, as you point out.

>  Experience indicates that, if the only

Thank you for your advice.  

Soobok

---

From: "YAO Jiankang" <[email-masked]>
Date: Fri, 6 Apr 2007 09:52:53 +0800

----- Original Message ----- 
From: "Soobok Lee" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: "Frank Ellermann" <[email-masked]>; <[email-masked]>
Sent: Friday, April 06, 2007 9:13 AM
Subject: Re: [EAI] OT: HDR=KSC5601SMTP (was: POLL: HDR= extension on MAIL FROM)

> My position is this : Let's go with UTF8 now, 

yes. we should go with UTF8 now.

>but let's leave rooms for

if  too many encoding are used,  every system have to need a lot of encoding to communicate with other machines.
that is a bad things. this is a obstacle to communication. 
so I do not like HDR= extension.

in your examples about gmail/yahoo/hotmail , they welcome local encodings.
actually, they welcome the display of the mesage with the local characters in spite of which coding is used, this is convient to local users. this does not mean they welcome the local encoding.

so UTF-8(UNICODE) is the best tool to  gmail/yahoo/hotmail , which can communicate with any machine in the world, but can display the message with the friendly local characters.

YAO Jiankang
CNNIC

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 10:55:43 +0900

Correction:

> Cyrillic/Greek/Hebrew/Arabic strings : 100 % ( 1->2 octets) ( < U+0800)

Addition:

  Devanagari : 200 % ( 1->3 octets) ( >= U+0900)

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 11:07:48 +0900

On Fri, Apr 06, 2007 at 09:52:53AM +0800, YAO Jiankang wrote:
> if  too many encoding are used,  every system 

 I am talking about proprietary local/internal/regional *extensions*
of UTF8SMTP. Such systems should be able to talk with UTF8SMTP-only
system. So UTF8SMTP-only system have no problem with communicating
such extended systems. Burdens of converting/back-converting from/to 
UTF8 message fall only onto the extended system side. Think about 
the smooth operation bewteen non-UTF8SMTP-aware and UTF8SMTP-aware systems 
mediated by UTF8SMTP EHLO option.  It's similar in above extension.

> have to need a lot of encoding to communicate with other machines.

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 11:21:54 +0900

> the best solution for some languages (including ours :-) ).

Email portals compete with one another and always under pressure
for cost cut  to make profits. If you can, try to ask portal personels
about how much they pay for netowrk/storage in a month and about
whether they welcome 50~200% increase of network/storage needs and
about why they still stick to local charset encodings in message format
even now after utf8 is born 7~8 years ago. 

They will make best answers than me.

Soobok

---

From: Martin Duerst <[email-masked]>
Date: Fri, 06 Apr 2007 11:34:01 +0900

Yes. I agree with John's feeling, but the 'negative instead of positive'
doesn't really help much. First, between U+FFFF and U+F800, it's all
compatibility dead bodies, and we definitely do NOT want to code them
shorter. Similarly, the stuff outside the BMP is there for a very
good reason: it's used extremely rarely, and therefore doesn't really
need much compression.

What could in theory be done, but would require more complex
operations (or a redesign of the code table, and I guess we all
agree we don't want to go there) is to move the various Latin
blocks out of the area below U+0800, and move in some other
frequently used scripts, starting with Devanagari. The reason
for this is that once you have basic Latin (=ASCII) with one
byte per character, paying three bytes per character for
characters with diacritics isn't too bad; even in languages
that use diacritics heavily (Vietnamese, Polish,...), you'd
most probably still stay below 2 bytes per character on average.

You could press along this route and move out non-basic Arabic
and Syriac and open up space for a script or two more below the
U+0800 boundary, and so on, but whatever you do, there will still
be too little space to fit the currently used alphabetic scripts
all into that space.

Not that I'm proposing to do any of this, of course!

Regards,     Martin.

At 06:24 07/04/06, Frank Ellermann wrote:
>CorrectFrom [email-masked] Thu Apr 05 22:49:19 2007
Date: Fri, 06 Apr 2007 11:34:01 +0900
To: Frank Ellermann <[email-masked]>, [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: [EAI] OT: HDR=KSC5601SMTP
Cc: 

Yes. I agree with John's feeling, but the 'negative instead of positive'
doesn't really help much. First, between U+FFFF and U+F800, it's all
compatibility dead bodies, and we definitely do NOT want to code them
shorter. Similarly, the stuff outside the BMP is there for a very
good reason: it's used extremely rarely, and therefore doesn't really
need much compression.

What could in theory be done, but would require more complex
operations (or a redesign of the code table, and I guess we all
agree we don't want to go there) is to move the various Latin
blocks out of the area below U+0800, and move in some other
frequently used scripts, starting with Devanagari. The reason
for this is that once you have basic Latin (=ASCII) with one
byte per character, paying three bytes per character for
characters with diacritics isn't too bad; even in languages
that use diacritics heavily (Vietnamese, Polish,...), you'd
most probably still stay below 2 bytes per character on average.

You could press along this route and move out non-basic Arabic
and Syriac and open up space for a script or two more below the
U+0800 boundary, and so on, but whatever you do, there will still
be too little space to fit the currently used alphabetic scripts
all into that space.

Not that I'm proposing to do any of this, of course!

Regards,     Martin.

>points in the BMP it would need three octets just like UTF-8.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Fri, 06 Apr 2007 11:48:43 +0900

>> Punycode...

>please correct me if that's wrong,

I haven't checked the actual spec, which may use some limitations,
but in theory, there is no problem with encoding strings of arbitrary
length. However, in practice, there are two big showstoppers:
- Decompression requires actual string insertion, which is inefficient.
- For most compression methods, the longer the string, the better
  the compression. Not so for punycode-type compressions; the compression
  ratio is bound to be about the same after a certain point, and
  much lower than the usual compression methods (LZ(W), block sorting)
  which are based on encoding symbol sequences rather than single symbols.

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp      ion:  It would use 1110 xxxx 10xx xxxx 10xx xxxx (3 octets)

---

From: Martin Duerst <[email-masked]>
Date: Fri, 06 Apr 2007 11:48:43 +0900

Regards,   Martin.

 mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Fri, 06 Apr 2007 11:55:46 +0900

>other encodings for future needs.

As far as I understand, we are not at all requiring UTF-8 for
the body of text messages. So we are talking about an overhead
applied to just a relatively small part of the message.

Regards,     Martin.

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 12:28:59 +0900

> applied to just a relatively small part of the message.

Yes, only with header parts. 
The body part still may contain base64-encoded local-charset-encoded texts
even with UTF8SMTP headers. 

But, conventionally many webmail portals have avoided base64(and QP) encoding of 
body parts to save storage/network bandwidth (i.e.: send/store in 8bit clean ).
They store header and body part even in a single message file. 
Either adoptin of  UTF8 or base64 encoding  results in increase of network/storage
need.

Some portals may be okay with this overhead, and other may not.

Soobok

---

From: Philip Guenther <[email-masked]>
Date: Thu, 5 Apr 2007 23:26:13 -0600

...
> even with UTF8SMTP headers.

Why would they need to be base64-encoded?  Isn't the point of the existing 
8BITMIME extension to avoid that?  Indeed you then say:

> But, conventionally many webmail portals have avoided base64(and QP) 
> encoding of body parts to save storage/network bandwidth (i.e.: 
> send/store in 8bit clean ).

Why would use of UTF8SMTP suddenly prevent those systems from treating the 
body as they currently do?  UTF8SMTP doesn't ban the use of other charsets 
in the body part(s).

> They store header and body part even in a single message file. Either 
> adoptin of UTF8 or base64 encoding results in increase of 
> network/storage need.

So don't do that.  I don't recall any suggestions of banning the use 
charsets other than UTF-8 in the body of messages with headers of envelope 
addresses in the non-ASCII part of UTF-8.  Using different charsets in the 
different parts of a message is an already deployed feature of base MIME 
and is not going away with UTF8SMTP, so I don't understand why you're 
implying that use of UTF-8 in headers implies use of UTF-8 in bodies.

Philip Guenther
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Apr 2007 01:48:39 -0400

--On Friday, 06 April, 2007 10:13 +0900 Soobok Lee <[email-masked]>
wrote:

>> > It's theoretically impossible to design an "ASCII
>> > compatible" UTF "better" than UTF-8 -- i.e. without giving
>> > up one or more essential features of UTF-8.  Ignoring all
>> > draft-terrell-* :-)
>> 
>> given, it then marches up the Unicode spec, with the next
>> blocks of codes being encoded into two octets and the strings
>> getting longer as one goes up.  In principle, one could
>> design a code that would preserve this ASCII property but
>> subtract the value of every character above U+007F from
>> 0xFFFF, yielding a signed integer that was negative in the
>> rest of the BMP and positive otherwise.  If one then applied
>> a UTF-8-like coding to that integer, one would end up with a
>> code that maintained the essential ASCII-as-ASCII property,
>> but was much more friendly to non-European scripts than to
>> European ones.
> 
> <OT>
> for improvement or adjustment in future UTF8-variant
> encodings  for non-western-european characters. 
> 
> UTF8 multiplies length of strings by 50%~200%, compared with
> local charset encodings.
> 

Well...

(i) As long as those strings entirely use characters in the BMP.
If, for example, one needs characters in the U+20000 - U+2FA1D
range, UTF-8 takes us to four octets.  Now, of course, some of
those character don't appear in an DBCS character coding, so the
comparison is odd.  But it is important to remember that the
result is not always three UTF-8 octets for CJK.

(ii) While DBCS is one option, the other traditional option,
especially in Japan, has been to use code table designators
which result, as I understand it, in single byte characters plus
switching overhead.  If the latter occurs infrequently, there
may be a significant advantage even over DBCS and an even more
significant advantage over UTF-8.
 
> UTF8 is still not in overwhelming use in email messages
> worldwide,  even though it was born 7~8 years ago.
> Main use of utf8 encoding in email is a fallback encoding to
> encode  non-local-charset characters in headers or body.

I think this is really not a comment about UTF-8, but about
Unicode itself.  For the record, while you believe that to be
true, and I do too, several of the people in the Unicode
Technical Committee firmly believe that the use of Unicode is
nearly ubiquitous on the network.

> If you consult gmail/yahoo/hotmail personels, they will reply
> to you  that they welcome local charset encodings as currently
> most favored encodings over utf8 for email messages.

I think there is a different conclusion to be drawn from this,
with the understanding that it is just my opinion.  With email
and with the web, we really have two separate elements of the
transmission to consider: the transport-related material needed
to get the material to its destination and the payload --the
material itself.  
For the first -- email addresses, URLs, and so on -- there is a
rather strong case to be made that considerations of
interoperability, and specifically, making sure that things can
be delivered to or reached at the intended definition, should
dominate all other decision criteria.  That makes absolute
standardization of what goes on the wire important, important
enough that local variations become a significant threat to a
globally-interoperable network.   In the case of URLs, email
addresses, and domain names, that principle has caused some
people to argue that we should just stick with ASCII.  I don't
believe that we need to go that far (or restrict ourselves that
much), but I do think we need to understand their point of view.
In any event, for those strings, the disadvantages of UTF-8
relative to other possible codings seem to be far outweighed by
the advantages of having _one_ agreed-upon coding rather than
local choices.

For the second element, the content, local character sets have
been permitted, and methods for identifying them provided, since
the dawn of MIME.  It seems to me that is most of where the
local variations are important: the body of the text may be
large enough that octets-per-character issues make a difference,
it may be important to transmit something closer to what is used
locally, it may be possible to know (or assume) that the systems
at both ends of a transmission are compatible (and, in general,
only the ends are affected, not the transport path), and so on.

So I just don't see the issue here.

> </OT>
> 
> My position is this : Let's go with UTF8 now, but let's leave
> rooms for other encodings for future needs.

For the envelopes, headers, and transmission negotiations,
allowing for another coding form just creates more complexity
without any significant payoff unless you expect Unicode itself
to be replaced in the near future.  For the content, body parts,
and payload, provisions already exist for other encodings, so it
is a non-issue.

     john

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Apr 2007 02:00:57 -0400

--On Friday, 06 April, 2007 11:48 +0900 Martin Duerst
<[email-masked]> wrote:

>> Yes, but AFAIK it can't encode Unicode strings of arbitrary
>> length, please correct me if that's wrong,
> 
> I haven't checked the actual spec, which may use some
> limitations, but in theory, there is no problem with encoding
> strings of arbitrary length.

The reason I tried  to talk about the punycode family of methods
(bootstring encodings, I think Adam called them), was precisely
to dodge that issue.  Whether punycode is overoptimized to a
particular application or not is irrelevant.

> However, in practice, there are
> two big showstoppers: - Decompression requires actual string
> insertion, which is inefficient. - For most compression
> methods, the longer the string, the better   the compression.
> Not so for punycode-type compressions; the compression   ratio
> is bound to be about the same after a certain point, and
> much lower than the usual compression methods (LZ(W), block
> sorting)   which are based on encoding symbol sequences rather
> than single symbols.

I don't think one should really compare punycode to a
compression metnod because it isn't.   To a first order
approximation, it is a character-by-character, variable length
encoding, just like UTF-8.   The differences are (i) that its
output is an ACE (which need not be true for other members of
the family and (ii) while UTF-8 bases the encoding of every
character on the distance from 0x0000, the bootstring family
base the encoding of characters on an offset to the beginning of
a range.

For fairly large blocks of text (much larger than a domain name
or email address), I'd expect that starting with UTF-32 (UCS-4)
and applying a standard compression method would typically
produce output of shorter length and less sensitivity to
placement in the tables than any of the variable length,
discrete character encoding methods.

And, of course, what makes this discussion odd is that it would
be perfectly rational to apply one of those compression methods
to message content in either transmission or storage if the
costs of compressing and decompressing where significantly lower
than the costs of transmission or storage respectively.  And
nothing in the existing standards prohibits doing either.

     john

---

From: "Jeff Yeh" <[email-masked]>
Date: Fri, 6 Apr 2007 15:38:10 +0800

> B - I support NOT adding HDR=UTF8

Jeff Yeh

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 19:08:50 +0900

> implying that use of UTF-8 in headers implies use of UTF-8 in bodies.

Right. Body parts can contain even raw binary octets streams (of even pdf/word file).

But, many webmail portals provide end users with "view-source" or "print source" feature
(display the content of raw message file as it is) for verification purpose.
If two different charset encodings are used in the single message, they will be
shown as broken texts at some header or body parts.

If message/rfc822 type file should be strictly regardded as a stream of binary octets, 
we can ignore this need as non-technical one( aestheticaly one),  
we have no issue about this problem. But, there is a tendency among users and 
developers (like me) that message/rfc822 type had better be in a human readable and
verifiable form.

Soobok

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 06 Apr 2007 12:05:49 +0200

Soobok Lee wrote:

> system.

In theory you could say that anything happening between MUA and MSA
on the sending side, or final delivery MTA and MUA on the receiving
side, is the local business of these sides.

They could still use UTF8SMTP "over the wire" (for a short wire from
MSA to final delivery MTA), and proprietary formats on their sides.

But in practice users have more than one email service provider, and
they'd try to use the format they got from ESP A with ESP B, where
ESP B might be a provider not supporting proprietary format A.

It would be a similar mess as with those misguided users sending or
receiving a variant of the Internet Message format with a popular
calendar software, using TNEF or other proprietary inventions.

> Burdens of converting/back-converting from/to UTF8 message fall
> only onto the extended system side.

I doubt that.  It's the unlucky end users who will pay the price
in the form of expedited unsubscriptions or killfiling say "TNEF".

Or needing hours to configure their software towards a format
remotely related to what's generally accepted as Internet Message.

Or bothering an admin because they want to install a decent mail
program in addition to the builtin calendar software.

Frank

---

From: Soobok Lee <[email-masked]>
Date: Fri, 6 Apr 2007 21:13:07 +0900

> ESP B might be a provider not supporting proprietary format A.

In this case, MUA should switch message format to UTF8SMTP one
to talk with ESP B which doesn't EHLO KSC5601SMTP option.

> calendar software, using TNEF or other proprietary inventions.

MUA should *export* the message in UTF8SMTP form 
for inputs to such non-KSC5601SMTP-aware external applications.

> program in addition to the builtin calendar software.

In some cases, KSC5601SMTP may be burdensome. But, not without
benefits. 

For example, you can telnet to 25 port and send SMTP
command from KSC5601-based x-terminal directly, no need for utf8 
conversion which will be done by server side. 
Unix "mail" command or non-EAI-capable MUA or PERL SMTP library may be 
tuned to work with KSC5601SMTP with minimal cost, even without any utf8 
related coding work in some cases. Server side will take all the hassles 
under local assumption of KSC5601 use.  KSC5601SMTP concept may be a 
transitional/complementry solution for UTF8SMTP.

Programmers in Korea seldom use UTF8 in his editing/coding work, 
rather,in most case, use KSC5601 (or CP949). In Server Administration 
environment as well.

(This thread  went too far  off the topic).

Soobok

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Apr 2007 08:17:04 -0400

--On Friday, 06 April, 2007 19:08 +0900 Soobok Lee <[email-masked]>
wrote:

>> > They store header and body part even in a single message
>> > file. Either  adoptin of UTF8 or base64 encoding results in
>> > increase of  network/storage need.
>> 
>> So don't do that.  I don't recall any suggestions of banning
>> the use  charsets other than UTF-8 in the body of messages
>> with headers of envelope  addresses in the non-ASCII part of
>> UTF-8.  Using different charsets in the  different parts of a
>> message is an already deployed feature of base MIME  and is
>> not going away with UTF8SMTP, so I don't understand why
>> you're  implying that use of UTF-8 in headers implies use of
>> UTF-8 in bodies.
> 
> But, many webmail portals provide end users with "view-source"
> or "print source" feature (display the content of raw message
> file as it is) for verification purpose. If two different
> charset encodings are used in the single message, they will be
> If message/rfc822 type file should be strictly regardded as a
> stream of binary octets,  we can ignore this need as
> non-technical one( aestheticaly one),   we have no issue about
> this problem. But, there is a tendency among users and 
> developers (like me) that message/rfc822 type had better be in
> a human readable and verifiable form.

Then the relevant developers need to re-read and understand the
MIME spec because there has never been any guarantee that either
"human readable form" or "display all body parts using the same
decoders (charset, codec, etc.).

First, remember that MIME supports multimedia.  Nothing is going
to make a music or video file "human readable and verifiable" by
visual inspection.  Application/octet-stream can contain
anything at all.  And there has never been a requirement that,
if a message contains multiple textual parts, the different body
parts use the same character encoding.  Indeed, while it has not
often been used for what we originally assumed would be one of
its more interesting applications, multipart/alternative was
designed to permit several different ways to express the same
content: different scripts, different languages, and, yes,
different encodings.   And message/rfc822 can encapsulate any of
these.

Under the principle that the EAI WG is not permitted to, and
doesn't need to, reopen fundamental design principles of MIME
and the SMTP Extension model unless they are directly and
uniquely associated with internationalization, this has become
seriously off-topic.  I suggest you take it to the 822 list if
you still believe that there is a protocol problem.

    john

---

From: Tony Hansen <[email-masked]>
Date: Fri, 06 Apr 2007 12:26:49 -0400

> and payload, provisions already exist for other encodings, so it

+1

	Tony Hansen
	[email-masked]

---

From: Soobok Lee <[email-masked]>
Date: Sat, 7 Apr 2007 10:33:37 +0900

> be too little space to fit the currently used alphabetic scripts

This is a compression-flavored idea: Let's call it UTF-8E(astern)

Top frequently used 1024 CJK Han Ideo chars : cover 90% or more usage
Top frequently used 512 Hangeul Syllable chars : cover 99% or more usage
fullset of Devanagari etc and Hiragana/Katakana :  below 412 chars 

These eastern characters can be mapped into the range 0x80~0x07ff.
For this, 65536 length of forward/reverse code mapping table is needed .

Then, UTF-8E  can encode most frequently used eastern asia characters
as two octets while extended latin/arabic/cyrillic/gree will be encoded
in three octets.

UTF-8E will share the same property of UTF-8.

Soobok

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 07 Apr 2007 04:15:00 +0200

> These eastern characters can be mapped into the range 0x80~0x07ff.
[...]
> UTF-8E will share the same property of UTF-8.

If you're serious about this idea I propose to
- continue the discussion on the Unicode list
- leave u+0080 up to u+009F alone, e.g. for the XML 1.1 "NEL"
- submit the I-D as "independent draft", don't let the Unicode or
  IETF folks touch it, I think they'd kill it for various reasons.

Frank

---

From: Soobok Lee <[email-masked]>
Date: Sat, 7 Apr 2007 14:23:50 +0900

> - continue the discussion on the Unicode list

Thanks for your encouragement.

This "UTF-8E" has different binary-sort ordering than UTF-8 and UTF-32.
So if it ever come out, it won't be general use for protocol strings,
but, rather mainly for efficient encoding of unicode texts, as replacement
for local charsets and their encodings. 
Anyway, it will help promote Unicode/ISO 10646 character set in eastern
asia.

I'v done brief research for fun about how to design "UTF-8E". See attached
draft excerpts.

Under the following schemes, even greek/cyrillic/arabic/latin
don't lose so much in encoding efficiency.

If anyone have further interests, reply to me personally,
not into this list.

Soobok

--------------------------------------------------------------------------

Appendix A.  Preserved Code Ranges in U+0080~U+07FF

   2048 (0x800) - 128 (0x80) = 1920 
   
   U+0080 ~ U+009F : (32) C1 control
   U+00DF ~ U+00FF : (33) Small Latin Supp-1 Letters
      
   U+03AC ~ U+03CE : (35)  Basic Greek Small
   U+0430 ~ U+044F : (32)  Basic Russian Small 
   U+05D0 ~ U+05F4 : (36)  Hebrew based on ISO 8859-8
   U+0621 ~ U+064A : (42)  Arabic based on ISO 8859-6
   (Capital Letters removed from Latin/Gree/Cyrillic)
   
   Sub-total : 210 characters  

   1920 - 210 = 1710 characters

Appendix B.  To be moved into U+0080~U+07FF from High Area in BMP

   Devanagari : U+0904 ~ U+0939 : (54)
   Hiragana: U+3041 ~ U+3096 : (86)
   Katakana: U+30A1 ~ U+30FF : (95)
   Thai : U+E01 ~ U+0E4F ( 79 )
   
   Sub-Total : 315
   
   Top frequenctly used Hangeul Syllables  : 395 characters
   Top frequenctly used CJK Han Ideographs : 1000 characters from UNIHAN

   (each set would cover top 90% of frequently used characters )
   
   1710 characters

-------------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 07 Apr 2007 13:25:02 +0200

> They will make best answers than me.
 From http://www.gmail.com/ (login page):

    *

      *Don't throw anything away.*
      Over 2837.433624 megabytes (and counting) of free storage so
      you'll never need to delete another message.

'nuff said.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 7 Apr 2007 17:50:06 +0200

Isn't this basically a rehash of http://unicode.org/reports/tr6/?

TR6 seems to be entirely unused. My unscientific guess is that most 
implementers actually prefer the simplicity of UTF-8, no matter what 
they say they prefer.

Arnt

---

From: Chris Newman <[email-masked]>
Date: Mon, 09 Apr 2007 15:36:29 -0700

- Chris

Harald Alvestrand wrote on 4/4/07 16:13 +0200:

> This indicates that the message being sent contains UTF-8 data in MAIL FROM,
> RCPT TO or headers (including inner body headers), which means that it needs
> downgrading before it can be passed to an ASCII-only recipient.
>
> Advantage: In the case where the recieving MTA knows which recipients can
> handle UTF8SMTP mail, and the MTA does not have downgrade capability, it can
> reject recipients by replying to the RCPT TO command with an appropriate
> error code.
> Advantage: The MTA can decide on whether downgrading is required based on the
> SMTP argument, and can treat any 8-bit character in the headers as a protocol
> error if the flag is not given.
>
> Advantage: Less complexity. No need for an MTA to consider whether or not to
> apply downgrading based on the error code it got back; any recipients
> rejected with a 5XX code can be considered rejected immediately.
>
> There are other opinions on the relative merits of the two alternatives, but
> I believe those are the most important ones
>
> If you have an opinion on this issue, please send a note to the mailing list
> with ONE of the following statements:
>
> A - I support HDR=UTF8

---

From: Randall Gellens <[email-masked]>
Date: Fri, 13 Apr 2007 17:39:53 -0700

>  absolutely necessary.    And...

Yes, I agree.

>  third, not a justification for it.

In an environment in which most systems have upgraded to support 
UTF8SMTP, then ALT-ADDRESS nicely goes away.  The same can't be said 
for HDR=UTF8 (although, in fairness, if you said that the server 
advertising UTF8SMTP means that all messages are assumed to require 
such support unless either the recipient server has scanned it and 
determined it is all ascii, or the sender specifies something like 
MSG=ALL-ASCII, and there is no other MSG= value, especially, there is 
no value which means UTF8 (just the absence of the parameter) then 
this parameter could also go away).

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
I was gratified to be able to answer promptly, and I did.  I said
I didn't know."                                      --Mark Twain

---

From: Soobok Lee <[email-masked]>
Date: Sat, 14 Apr 2007 11:10:39 +0900

On Fri, Apr 13, 2007 at 05:39:53PM -0700, Randall Gellens wrote:
> for HDR=UTF8 

I agree on this last sentence from different viewpoint.  See this.

<utf8smtp submission session  from=the sender's MUA >
MAIL FROM: [email-masked] HDR=UTF8
RCPT TO:   [email-masked]
RCPT TO:   [email-masked]

DATA
From:   "display_name" <[email-masked]>
TO:     [email-masked], [email-masked]
Subject:  utf8-subject

bodytext goes here
.

</utf8smtp submission session>

The utf8smtp server tries to fork another
outgoing smtp session in order to relay to [email-masked]
as follows.

<utf8smtp relaying session>
MAIL FROM: [email-masked] HDR=UTF8
RCPT TO:   [email-masked]

DATA
From:   "display_name" <[email-masked]>
TO:     [email-masked], [email-masked]
Subject:  utf8-subject

bodytext goes here
.

</utf8smtp relaying session>

If @world.com has UTF8SMTP EHLO option (no downgrade attempted), 
and the envelope addresses are all in 7BIT ASCII,  the only
sign of UTF8SMTP header presence in this SMTP session
is "HDR=UTF8" . Without this parameter, the server
should parse  message headers to determine that.

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Sat, 14 Apr 2007 11:10:54 +0900

DATA

bodytext goes here
.

DATA

bodytext goes here
.

Soobok

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Apr 2007 07:19:59 -0400

--On Friday, 13 April, 2007 17:39 -0700 Randall Gellens
<[email-masked]> wrote:

>>  While I think that, given time and energy, we could answer
>>  those questions, produce firm specifications, and expect
>>  that most (but not all) implementations would understand and
>>  follow the instructions well enough to get things right, it
>>  seems to me to be a high cost and high risk for something
>>  that is not absolutely necessary.    And...
> 
> Yes, I agree.
> 
>> 
>>  and those who do not understand that they are running
>>  crippled and obsolete systems, then this parameter becomes
>>  long-term clutter with no value.  The same could be said
>>  about 8BITMIME today and perhaps, long-term, about ALT-ADDR
>>  itself.  But the existence of two of them is, IMO, a strong
>>  reason to not add a third, not a justification for it.
> 
> In an environment in which most systems have upgraded to
> support UTF8SMTP, then ALT-ADDRESS nicely goes away.  The same
> can't be said for HDR=UTF8 (although, in fairness, if you said
> that the server advertising UTF8SMTP means that all messages
> are assumed to require such support unless either the
> recipient server has scanned it and determined it is all
> ascii, or the sender specifies something like MSG=ALL-ASCII,
> and there is no other MSG= value, especially, there is no
> value which means UTF8 (just the absence of the parameter)
> then this parameter could also go away).

Yes, if one were to do this at all, an assertion of "all-ASCII"
would be better than one of "utf8", as you suggest.  One would
probably have to be prepared to have that assertion violated --
just as one has to have some way to deal with "high bit on"
situations in the legacy environment -- so I am still not
convinced it is worth the extra effort.

    john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 14 Apr 2007 15:23:17 +0200

> convinced it is worth the extra effort.

Note that changing the default doesn't lessen the complexity of 
implementing the parameter one iota; in both cases, there are 2 codepaths 
to maintain, forever. The only thing that is improved by changing the 
default is the optics of the final state.

                    Harald

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Apr 2007 14:44:35 -0400

--On Saturday, April 14, 2007 3:23 PM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

>>> support UTF8SMTP, then ALT-ADDRESS nicely goes away.  The
>>> same can't be said for HDR=UTF8 (although, in fairness, if
>>> you said that the server advertising UTF8SMTP means that all
>>> messages are assumed to require such support unless either
>>> the recipient server has scanned it and determined it is all
>> Yes, if one were to do this at all, an assertion of
>> "all-ASCII" would be better than one of "utf8", as you
>> suggest.  One would probably have to be prepared to have that
>> assertion violated -- just as one has to have some way to
>> deal with "high bit on" situations in the legacy environment
>> -- so I am still not convinced it is worth the extra effort.
>
> Note that changing the default doesn't lessen the complexity
> of implementing the parameter one iota; in both cases, there
> are 2 codepaths to maintain, forever. The only thing that is
> improved by changing the default is the optics of the final
> state.

Exactly.  And that is the reason I remain opposed to doing this 
at all.   If we decided to do it anyway, then the optics become 
relevant.

    john

---

From: Soobok Lee <[email-masked]>
Date: Mon, 16 Apr 2007 15:06:44 +0900

> should parse  message headers to determine that.

$s/parse message headers/parse EVERY message header it receives/;

Moreover,as we havn't touched DKIM/SMIME related header definition
and header normalization requirement here, we can't exclude future "UTF8SMTPv2" 
which may introduce more headers or more restrictions or more normalization
requirement. In that case, even "HDR=UTF8" is not enough, rather,
as Kari suggested, "HDR=UTF8SMTP" and "HDR=UTFSMTPv2" may become
better and necessary. "UTF8SMTPv2" EHLO options  as well.

"HDR=UTF8SMTP" is so clear that it  not only specifies the headers 
are in UTF8 encoding,  but also specifies that the header section abides by 
the UTF8SMTPv1 restrictions and requirements. "HDR=UTF8SMTPv2" may indicate
that the headers keep the new rules imposed by UTF8SMTP version 2.

Soobok