Subject: [EAI] UTF8SMTP mailbox on SMTP reply (Re: POLL: HDR= extension on MAIL FROM)

From: Kari Hurtta <[email-masked]>
Date: Sat, 7 Apr 2007 21:30:43 +0300 (EEST)

>>> Harald Tveit Alvestrand
> There hasn't been any suggestion that the HDR=8BIT (or 7BIT, or its 
> absence) say anything in general about what the client is able to suppport.
> 
> If you want to have response strings with UTF-8 in them, better go push the 
> LANG command extension that Alexey has been mumbling about for half a dozen 
> years.
> 
>                    Harald

I think that you misunderstand my comment. Issue was about reply codes,
which may include UTF8SMTP mailboxes on reply (and specically when SMTP comand
argument was ASCII only.)

See YAO Jiankang' answer (message is included). He answered that UTF8SMTP 
mailboxes are allowed on replies.

One example is

  EXPN root
  250 2.1.5 Ville Keränen <[email-masked]>

when SMTP cleint is not used any argument which indicates that it supports
UTF8SMTP.

Another one, which can include UTF8SMTP mailbox on reply, is VRFY command.

All reply strings are still on default language (english?) on these scenarios.

Another one are 251 and 551 response codes which include mailbox.

EHLO xxx
250-mail.example.com
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5
250-DELIVERBY
250-UTF8SMTP
250 HELP
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
251 <[email-masked]> is new address, will forward

( client crashes ?)

And another case

MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
551 <[email-masked]> is new address, please resend

( SMTP client failes to generate proper DSN for that response. )

On both cases client is sending non-UTF8SMTP message to server 
(and client does not know about UTF8SMTP)

> --On 6. april 2007 08:40 +0300 Kari Hurtta <[email-masked]> 
> wrote:
> 
> > Soobok Lee <[email-masked]> writes:
> >
> >> C)  HDR= is newly defined parameter. But HDR=UTF8 is optional and may be
> >> omitted.
> >>
> >>     Mail servers may assume HDR=UTF8 if HDR= parameter is not prenet in
> >>     SMTP session.
> >
> > Get I this correctly?
> >
> >    HDR=UTF8  implicates that SMTP _client_ supports UTF8SMTP
> >
> >    So if HDR=UTF8 assumed, it means that SMTP server assumes that all
> >    smtp clients supports UTF8SMTP
> >
> >    ( Of course _presence_ of HDR=7BIT parameter also means that smtp
> > clients      supports  UTF8SMTP )
> >
> >
> > And if SMTP client supports UTF8SMTP, smtp server may use UTF-8 on
> > it's SMTP reply strings.
> >
> > So with these assumtions, smtp server may use UTF-8 always on it's reply
> > strings even when SMTP client knows nothing about UTF8SMTP.
> >
> > (hmm. Perhaps actually it is EXPN where that need arise without that
> > argument of  command itself use UTF-8. )
> >
> > That is not very critical. It is unlikely that smtp client reply parser
> > failes to parse reponse even if there is some UTF8SMTP addresses on
> > response. (this is guess.)

/ Kari Hurtta

Subject: Re: draft-ietf-eai-smtpext-04.txt: EXPN, VRFY
From: "YAO Jiankang" <[email-masked]>
To: "Kari Hurtta" <[email-masked]>
Cc: "Wei MAO" <[email-masked]>
Date: Fri, 6 Apr 2007 15:31:16 +0800

----- Original Message ----- 
From: "Kari Hurtta" <[email-masked]>
To: "YAO Jiankang" <[email-masked]>
Cc: "Wei MAO" <[email-masked]>; "Kari Hurtta" <[email-masked]>
Sent: Friday, April 06, 2007 3:16 PM
Subject: Re: draft-ietf-eai-smtpext-04.txt: EXPN, VRFY

>>>> YAO Jiankang
>> 
>> ----- Original Message ----- 
>> From: "Kari Hurtta" <[email-masked]>
>> To: "Yao Jiankang" <[email-masked]>
>> Cc: <[email-masked]>; "Wei MAO" <[email-masked]>
>> Sent: Thursday, April 05, 2007 2:25 PM
>> Subject: Re: draft-ietf-eai-smtpext-04.txt: EXPN, VRFY
>> 
>> 
>> > 
>> > Another question: On which point server may use  UTF-8
>> > on it's reply strings.  (or should this asked on list?
>> > If it is, you are free to forward that to list. )
>> 
>> actually, only EAI capability MUA can deliver the UTF-8 stuff to SMTP server normally.
>> If MUA can send the UTF-8 stuff , it means that MUA has the EAI ability.
>> 
>> So, IMO,  in general, if SMTP server receives the UTF-8 stuff, it can return it to original sender.
> 
> I think that you msissed my question. See my example.
> 
>> > 
>> > That that server announces UTF8SMTP not yet mean that
>> > client supports UTF8SMTP.   Specially that is issue
>> > for EXPN command, I think.  (paramater of EXPN
>> > may be ASCII, but expansion may include UTF-8 -- so
>> > what server does? )
>> 
>> since paramater of EXPN will include UTF-8, the server should be updated to it.
>> John's RFC2821bis may be updated to include it.
>> currently, many email system normally deny this command due to security reason.
>> EXPN is not the key command of SMTP and it may be not necessary to have a specific text or session to emphasize it.
> 
> No.  EXPN is  expand .. it may and will list another mailbox than what is as argument
> 
> hurtta@Hurtta06k:~$ telnet localhost smtp
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> 220 Hurtta06k.keh.iki.fi ESMTP Sendmail 8.13.4/8.13.4/Debian-3ubuntu0.1; Fri, 6 Apr 2007 10:06:02 +0300; (No UCE/UBE) logging access from: localhost(OK)-localhost [127.0.0.1]
> EHLO xxx
> 250-Hurtta06k.keh.iki.fi Hello localhost [127.0.0.1], pleased to meet you
> 250-ENHANCEDSTATUSCODES
> 250-PIPELINING
> 250-EXPN
> 250-VERB
> 250-8BITMIME
> 250-SIZE
> 250-DSN
> 250-ETRN
> 250-AUTH DIGEST-MD5 CRAM-MD5
> 250-DELIVERBY
> 250 HELP
> EXPN root
> 250 2.1.5 Kari Hurtta <[email-masked]>
> QUIT
> 221 2.0.0 Hurtta06k.keh.iki.fi closing connection
> Connection closed by foreign host.
> hurtta@Hurtta06k:~$                                   
> 
> Is following allowed:
> 
> EHLO xxx
> 250-Hurtta06k.keh.iki.fi Hello localhost [127.0.0.1], pleased to meet you
> 250-ENHANCEDSTATUSCODES
> 250-PIPELINING
> 250-EXPN
> 250-VERB
> 250-8BITMIME
> 250-SIZE
> 250-DSN
> 250-ETRN
> 250-AUTH DIGEST-MD5 CRAM-MD5
> 250-DELIVERBY
> 250-UTF8SMTP
> 250 HELP
> EXPN root
> 250 2.1.5 Ville Keränen <[email-masked]>
> 
> 
> Assume that these are UTF-8 characters (this mail
> may use actually IS-8859-1 I think)

very good example.

yes, allow it.

IMO, may also allow the below comand:

 EXPN  UTF-8
 250 2.1.5 utf-8 <utf-8@utf-8>

Thanks.

 YAO Jiankang
CNNIC

> 
> 
>> Thanks you for your kind comments.
>> YAO Jiankang
>> CNNIC
>> 
>> > 
>> > There is no explicit method to enable UTF-8 for replies.
> 
> / Kari Hurtta
_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sat, 07 Apr 2007 16:25:05 -0400

--On Saturday, April 07, 2007 9:30 PM +0300 Kari Hurtta 
<[email-masked]> wrote:

>>>> Harald Tveit Alvestrand
>> There hasn't been any suggestion that the HDR=8BIT (or 7BIT,
>> or its  absence) say anything in general about what the
>> client is able to suppport.
>>
>> If you want to have response strings with UTF-8 in them,
>> better go push the  LANG command extension that Alexey has
>> been mumbling about for half a dozen  years.
>>
>>                    Harald
>
> I think that you misunderstand my comment. Issue was about
> reply codes, which may include UTF8SMTP mailboxes on reply
> (and specically when SMTP comand argument was ASCII only.)

Unless the client has already done something to clearly indicate 
that it is UTF8SMTP-capable (and, no, I don't think this 
justifies a HDR parameter or anything else), replies had 
probably best not contain non-ASCII information.

> See YAO Jiankang' answer (message is included). He answered
> that UTF8SMTP  mailboxes are allowed on replies.

I think that is wrong in general.  See the examples in 2821, 
which rarely include a mailbox name.

>   250 2.1.5 Ville Keränen <[email-masked]>

One could, in principle, do this.  But I note that it is 
independent of the UTF8SMTP work.  It would presumably require a 
separate extension, or at least something that the client would 
need to ask for/ authorize.  While I think the specifics of 
Alexey's "lang" proposal are a bad idea, what is needed here is 
much closer to that than to what is being suggested above.

> when SMTP cleint is not used any argument which indicates that
> it supports UTF8SMTP.

Unless something has been explicitly extended in this area, a 
non-ASCII reply (or a reply containing any non-ASCII character) 
violates 2821.  The situation is, IMO, much more like the DSN 
one that a situation in which a non-ASCII address can be 
arbitrarily returned.

> (and client does not know about UTF8SMTP)

And in both cases, either a separate extension is needed or the 
client needs to report that the only available address is 
non-ASCII and then, if appropriate, escape it and indicate that 
is being done.

I agree that the SMTPEXT document should probably mention these 
cases.  Briefly.

    john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 09 Apr 2007 15:31:12 +0200

--On 7. april 2007 21:30 +0300 Kari Hurtta <[email-masked]> wrote:

>> If you want to have response strings with UTF-8 in them, better go push
>> the  LANG command extension that Alexey has been mumbling about for half
>> a dozen  years.
>>
>>                    Harald
>
> which may include UTF8SMTP mailboxes on reply (and specically when SMTP
> comand argument was ASCII only.)

Thanks for the clarification. On rereading, I stand by the words of my 
response, but it was not a response to what you were suggesting.

The problem you cite is general, and troubling. Lots of SMTP responses have 
embedded mailbox names in them (in practice; whether they are necessary or 
not is beyond this particular question).

I see at least 3 different ways to address that issue in an UTF8SMTP 
context:

- Brutally suppress UTF8 mailbox names in responses (which will make VRFY 
<utf8> kind of pointless)
- Always return mailboxes in some kind of encoded form (walks straight into 
all the issues with "automatic ACE" that we've debated and discarded so 
many times)
- Have some kind of command from the client to the server saying "I am 
willing to accept UTF8 in responses".

I do not think "using an UTF8 command" is a good pick for the last one; the 
idea of having a dialogue that goes

-> EHLO
<- 250-UTF8SMTP
-> VRFY <utf8@utf8>
<- 550 Can't return an UTF8 mailbox
-> MAIL FROM:<utf8@utf8>
....
<- 250 OK
-> VRFY <utf8@utf8>
<- 200 <utf8@utf8> is fine with me

strikes me as a downright bad design.

                      Harald

---

From: John C Klensin <[email-masked]>
Date: Mon, 09 Apr 2007 12:08:02 -0400

Just on comment on your three options...

--On Monday, April 09, 2007 3:31 PM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

> I see at least 3 different ways to address that issue in an
> UTF8SMTP context:
>
> - Brutally suppress UTF8 mailbox names in responses (which
> will make VRFY <utf8> kind of pointless)
> - Always return mailboxes in some kind of encoded form (walks
> straight into all the issues with "automatic ACE" that we've
> debated and discarded so many times)

I don't know if we want to take advantage of it, but there is a 
difference.   Since the reply text is intended for people, and 
is free-form rather than standardized, one could have (leaving 
out the extended codes for clarity)

<- 250-UTF8SMTP
-> VRFY <foo@bar>
<- 550 No mailbox - <foo@bar>

Do note that there is essentially no information in the response 
here that wasn't already known to the client, so simply 
suppressing the address would not lose any information, contrary 
to your assertion above.

and also have

<- 250-UTF8SMTP
-> VRFY <foo1@bar>
<- 250 Address is actually <foo1@bar1>
      ## this and situations like it are the worst case,
      ## because there actually is new information in the
      ## reply
-> VRFY <utf8@utf8>
<- 250-Address is a UTF8SMTP mailbox, encoded
<- 250 according to RFC9999 as <uncode-escape(utf8)@ACE(utf8)>

Also consider the third, even more difficult, case

<- VRFY <foo2@bar>
<- 250-Address is a UTF8SMTP mailbox, encoded
<- 250 according to RFC9999 as 
<uncode-escape(utf8-a)@ACE(utf8-b)>

This is not wildly attractive, but it is not rocket science and 
does not cross the line into the discussions we have repeatedly 
had about encoded forms.

Otherwise, I see little option other than either an explicit 
assertion from the client that UTF8SMTP stuff is in use.  That 
probably is not the controversial HDR argument, since the 
headers might not affected at all (note that
   EHLO
   VRFY
   QUIT
is a perfectly valid sequence)
But might take us toward VRF8 and EXP8.  I'd rather not try to 
do the latter in this WG if we can avoid it, but only because 
VRFY and EXPN, more than any other SMTP command that has not 
been deprecated completely, are in need of a modernizing 
redesign: to create new commands without doing that would be a 
shame.

Of course, one could also have
   VRFY <foo@bar> You-can-send-back-utf8-replies-if-you-want-to
(probably spelled in a more reasonable way) and optionally apply 
that argument to any of MAIL, RCPT, VRFY, or EXPN, possibly 
using it to set a flag so it didn't need to be invoked more than 
once.  I'd rather avoid it, but it seems a _much_ better 
solution than something whose semantics get involved with what 
goes on in the message.

> - Have some kind of command from the client to the server
> saying "I am willing to accept UTF8 in responses".

The last suggestions above sort of start down that path, of 
course.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 09 Apr 2007 20:37:44 +0200

John C Klensin wrote:
 
> I see little option other than either an explicit assertion
> from the client that UTF8SMTP stuff is in use.  That

> redesign: to create new commands without doing that would be a

Okay, but the UTF8SMTP draft has to mention this or any similar
choice somewhere.  VRF8 and EXP8 would be really ugly.  Maybe
the 2821bis crowd can deprecate EXPN while they're at it, I'm
not sure what 2821bis-01 3.5.2 actually means wrt EXPN.  Is it
some kind of "SHOULD implement and immediately disable it" ?

Maybe we could add a command USE8 enabling UTF8SMTP addresses
in responses, valid until the next EHLO, HELO, or RSET ?

> The last suggestions above sort of start down that path, of

What's the alternative for say 251 + 551, new response codes
in the direction of "I'd like to tell you more but can't" ?

Frank

---

From: Chris Newman <[email-masked]>
Date: Mon, 09 Apr 2007 16:09:43 -0700

Harald Tveit Alvestrand wrote on 4/9/07 15:31 +0200:
> I see at least 3 different ways to address that issue in an UTF8SMTP context:
>
> all the issues with "automatic ACE" that we've debated and discarded so many
> times)

Another option is to defer the issue of SMTP status response encoding to:

  <http://www.ietf.org/internet-drafts/draft-melnikov-smtp-lang-06.txt>

If the client negotiates the LANG extension, then the server is free to provide 
utf8@utf8 in the status text.

                - Chris

---

From: John C Klensin <[email-masked]>
Date: Mon, 09 Apr 2007 19:40:17 -0400

--On Monday, 09 April, 2007 16:09 -0700 Chris Newman
<[email-masked]> wrote:

> If the client negotiates the LANG extension, then the server
> is free to provide utf8@utf8 in the status text.

Except that the non-ASCII SMTP status problem needs to be solved
and at least a few of us believe that Alexey's draft violates a
fundamental architectural principle of the Internet.

   john

---

From: Tony Finch <[email-masked]>
Date: Tue, 10 Apr 2007 01:39:42 +0100

> fundamental architectural principle of the Internet.

If you are referring to the RFC 2277 protocol/text distinction, then I
think it is very difficult to argue that the textual parts of SMTP replies
are protocol not intended to be read by humans - since everyone uses them
for human readable text. Enhanced Status Codes do not provide an adequate
foundation for internationalization since they do not cover the complete
range of possible reasons for rejection and the range they do cover is not
covered in enough detail. These inadequacies cannot be fixed since new
ESCs for existing parts of the protocol are not interoperable - older
software won't have textual explanations for them - and anti-spam tactics
mean there's a never-ending supply of new reasons for rejecting email.
Therefore the ESC registry is only useful for co-ordinating ESC
allocations for new protocol extensions.

In practice I usually configure my systems to produce a brief error
message and a URL, such as a DNS blacklist lookup URL, or a link to
further documentation. In principle HTTP can then do some of the i18n
heavy lifting - though this is not a solution for UTF8 in SMTP replies.

Alexey's draft has the same architectural model as the LANG command in FTP
(RFC 2640) and proposed for POP and IMAP.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE: NORTH OR NORTHEAST 2 OR 3, OCCASIONALLY 4. SLIGHT OR MODERATE. FOG
PATCHES. MODERATE OR GOOD, OCCASIONALLY VERY POOR.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 10 Apr 2007 12:23:11 +0100

On Mon, 09 Apr 2007 17:08:02 +0100, John C Klensin <[email-masked]> wrote:

> <- 250 according to RFC9999 as <uncode-escape(utf8-a)@ACE(utf8-b)>

And that situation is even more likely to arise with the EXPN command.

{Slightly off topic, but can someone please explain to me why VRFY and  
EXPN are so often deprecated?}

However, there is a crude but possibly adequate solution to this problem,  
and that is that the server should "just send 8bit (i.e. UTF-8)". This  
will often make more sense to the client than a message saying, in ASCII,  
"sorry, I cannot answer because I am not allowed to use 8bit". Even if the  
client is reading in ISO-8859-X, UTF-8 will show up as obviously garbled  
characters, which the user will interpret as "seems to be some charset  
inknown to me". And most likely the same effect if the text gets truncated  
to 7bit. Eitherway, what is conveyed to the client is "it is possible that  
you are not going to be able to read this".

BTW, RFC3977 (the latest NNTP draft) officially specified UTF-8 as the  
default charset, not that it is expected to occur much unless and until  
Netnews adopts EAI or something like it. However, that does mean that NNTP  
servers are now allowed to send UTF-8 responses, and older NNTP clients  
will have to do the best they can with such responses - essentially as I  
have indicated above.

Alternatively, if we really must have a means for a client to say "I will  
accept UTF-8 responses", then Alexey's LANG command (even without  
parameters) would be a good way to indicate that - even if sent to a  
system that did not advertixe LANGUAGE it would do little harm, and make  
things no worse than any other solution.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Apr 2007 09:27:18 -0400

> some kind of "SHOULD implement and immediately disable it" ?

Closer to "while EXPN is often bad news in the general case,
there may be exceptions in which it is very useful   In
particular, if the user or system making the EXPN request is
known an authenticated via some mechanism, the considerations
may be quite different from those if that is not the case.".
If you think 2821bis needs text to make that more explicit,
please suggest it on that list and provide the text.

> in responses, valid until the next EHLO, HELO, or RSET ?

That would be one approach.

> in the direction of "I'd like to tell you more but can't" ?

For reasons that I hope 2821 explains adequately, including the
simple observation that alias and forwarding processing are
rarely done these days until long after the message has been
accepted, 251 and 551 have not been widely supported for many,
many years.  I believe I argued during the DRUMS period that
they should be deprecated entirely but, if I did, I lost the
argument.   Again, there may be special circumstances but,
speaking personally, it will be hard to convince me to lose
sleep over message syntax associated with those codes.  But, if
one wanted to do it, some appropriate text, followed by an
escaped form of the address (see my other note on this subject)
would presumably be an alternative.

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Apr 2007 09:40:39 -0400

--On Tuesday, 10 April, 2007 01:39 +0100 Tony Finch
<[email-masked]> wrote:

>> Except that the non-ASCII SMTP status problem needs to be
>> solved and at least a few of us believe that Alexey's draft
>> violates a fundamental architectural principle of the
>> Internet.
> 
> If you are referring to the RFC 2277 protocol/text
> distinction,

I was not.  I was referring to the much older idea that we
should establish single "line" format and forms and let systems
at the end points convert to and from them rather than creating
the N**2 problem of everyone being expected to understand
everyone else's forms (or, in this case, languages and character
sets).

>...
> Enhanced Status Codes do not provide an adequate
> foundation for internationalization since they do not cover
> the complete range of possible reasons for rejection and the
> range they do cover is not covered in enough detail. These
> inadequacies cannot be fixed since new ESCs for existing parts
> of the protocol are not interoperable - older software won't
> have textual explanations for them - and anti-spam tactics
> mean there's a never-ending supply of new reasons for
> rejecting email. Therefore the ESC registry is only useful for
> co-ordinating ESC allocations for new protocol extensions.

If you are correct, then enhanced status codes are a failure and
should be pulled of the standards track and junked:  providing
the adequate foundation to which you refer was a large part of
the justification for doing those codes in the first place.
Note, in that regard, the second sentence of Section 2 of RFC
1893:

	"These status codes are intended to be used for media
	and language independent status reporting."

So I await the proposal to deprecate the things and, ideally, to
replace them with something that will accomplish that goal.

> In practice I usually configure my systems to produce a brief
> error message and a URL, such as a DNS blacklist lookup URL,
> or a link to further documentation. In principle HTTP can then
> do some of the i18n heavy lifting - though this is not a
> solution for UTF8 in SMTP replies.
> 
> Alexey's draft has the same architectural model as the LANG
> command in FTP (RFC 2640) and proposed for POP and IMAP.

IMO, 2640 is badly broken for other reasons.  Trying to write
that up and fix it generated a collection of prerequisite work.
And I have almost the same objections to the POP and IMAP
proposals that I do to the SMTP one.   However, SMTP is more
difficult for the usual other reason: FTP, POP, and IMAP all
require a direct client-server connection (in FTP's case, for
the command channel) to operate.  Consequently, any negotiations
about replies are between the client and the server that will
produce them.  Because of the relay issues, an SMTP negotiation
about the language and form of responses is much more tentative
and tenuous.  Certainly it will work sometimes, but the other
cases a problematic.

     john

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 10 Apr 2007 20:33:32 +0200

> heavy lifting - though this is not a solution for UTF8 in SMTP replies.

In a nutshell one of the reasons why there are no I18N considerations
in RFC 4408:  A http-URL is much better, or at least good enough.

> Alexey's draft has the same architectural model as the LANG command
> in FTP (RFC 2640) and proposed for POP and IMAP.

FTP etc. replies aren't copied verbatim into a DSN.  The DSN has its
own charset, and the mailer creating a DSN has no idea which charset
an I18N reply code might use.  If Alexey manages to specify that it
MUST be UTF-8 if it's not ASCII, then it's still in a language I don't
know (as original sender, in no direct relationship with the mailer
forced to create the DSN).

Frank