Subject: [EAI] 5738bis

From: Timo Sirainen <[email-masked]>
Date: Tue, 11 Sep 2012 15:47:53 +0300

>   IMAP servers that advertise support for "UTF8=ACCEPT" or "UTF8=ONLY"
>   MUST reject an APPEND command that includes any 8-bit in the message
>   headers with a "NO" response, when IMAP clients do not issue "ENABLE
>   UTF8=ACCEPT" or "ENABLE UTF8=ONLY".

This requires changing behavior of servers' existing behavior when it encounters such data. Yes, such messages shouldn't exist, but they do. I have tens of messages from Hotmail with 8bit subject headers in my INBOX. If I wanted to copy them to another IMAP server via IMAP protocol, this spec would require the destination server to fail the operation tens of times, making it highly annoying. (I doubt (m)any IMAP clients would attempt to convert 8bit headers to 7bit).

Similarly I wouldn't want my server to reject the APPEND when the 8bit header contained non-UTF8 input.

>   The "UTF8=ONLY" capability permits an IMAP server to advertise that
>   it does not permit the international mailbox name convention
>   (modified UTF-7),

This doesn't really sound like what "design rationale" section talks about UTF8=ONLY. Isn't the idea simply to behave as if client has done ENABLE UTF8=ACCEPT (which may or may not happen to work with current clients)?

> servers that
>   may be accessed by the same user with different clients or methods
>   (e.g., POP or webmail systems in addition to IMAP or IMAP clients
>   with different capabilities) will need to exert extreme care to be
>   sure that UIDVALIDITY behaves as the user would expect

I wonder what exactly that means. I guess UIDVALIDITY means something more than the UIDVALIDITY value specifically. If client A downloads UID 5 downgraded and client B downloads it non-downgraded, the messages are different but that probably isn't a problem?

> The
>   "ENABLE UTF8=ACCEPT" command MUST only be used in the authenticated
>   state.  (Note that the "UTF8=ONLY" capability described in 
> Section 6
> 
>   implies the "UTF8=ACCEPT" capability.

So client should use ENABLE UTF8=ACCEPT instead of ENABLE UTF8=ONLY? .. except this kind of sneaks in:

> when IMAP clients do not issue "ENABLE
>   UTF8=ACCEPT" or "ENABLE UTF8=ONLY".

Probably should be removed from here? Or shouldn't it even be simply ENABLE UTF8?

> When an IMAP server uses a mailbox format that supports UTF-8 headers
>   and it permits selection or examination of that mailbox without the
>   "UTF8" parameter

This document doesn't talk about UTF8 parameter anywhere else.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Sep 2012 21:17:18 +0100

Hi Timo,

> Similarly I wouldn't want my server to reject the APPEND when the 8bit header contained non-UTF8 input.
Hmm. Good point. I wish you brought this up earlier. Would changing the 
MUST to SHOULD address your concern or would you like to see further 
changes to this text? If the latter, can you suggest new text?

> This doesn't really sound like what "design rationale" section talks about UTF8=ONLY. Isn't the idea simply to behave as if client has done ENABLE UTF8=ACCEPT (which may or may not happen to work w…

No, I think the idea is to advertise that modified UTF-7 is not supported.

> I wonder what exactly that means. I guess UIDVALIDITY means something more than the UIDVALIDITY value specifically. If client A downloads UID 5 downgraded and client B downloads it non-downgraded,…
Correct. This text is trying to talk about the case of a client A 
downloading a downgraded version, then being upgraded and still thinking 
that it has the authoritative version of the message.
We don't want to require IMAP servers to use different UIDVALIDITY 
values for non-EAI-aware and EAI-aware clients.

> So client should use ENABLE UTF8=ACCEPT instead of ENABLE UTF8=ONLY? .. except this kind of sneaks in:
No, I think this is trying to say that either one can be used.

So maybe the following change can clarify:

In Section 3, change the 3rd sentence in the 2nd paragraph to read:

OLD:
    Note that the "UTF8=ONLY" capability described in Section 6
    implies the "UTF8=ACCEPT" capability.  See additional information in
    that section.)

NEW:
    Note that the "UTF8=ONLY" capability described in Section 6
    implies the "UTF8=ACCEPT" capability and thus "ENABLE UTF8=ONLY" can
    be used instead of "ENABLE UTF8=ACCEPT".

> Probably should be removed from here? Or shouldn't it even be simply ENABLE UTF8?

I think it would be simpler to keep parity with IMAP CAPABILITY names.

> This document doesn't talk about UTF8 parameter anywhere else.
Well spotted. The document used to use UTF8 parameter to SELECT/EXAMINE, 
but this was removed. Now there is no other way of signaling the server 
that the client supports UTF8=ACCEPT/ONLY other than using ENABLE.
So I propose the document should be changed to say:

In Section 8 change the first sentence of the first paragraph to read:

OLD:
    When an IMAP server uses a mailbox format that supports UTF-8 headers
    and it permits selection or examination of that mailbox without the
    "UTF8" parameter, it is the responsibility of the server to comply
    with the IMAP4rev1 base specification [RFC3501] and [RFC5322] with
    respect to all header information transmitted over the wire.

NEW:
    When an IMAP server uses a mailbox format that supports UTF-8 headers
    and it permits selection or examination of that mailbox without issuing
    "ENABLE UTF8=ACCEPT" or "ENABLE UTF8=ONLY" first, it is the 
responsibility of the server to comply
    with the IMAP4rev1 base specification [RFC3501] and [RFC5322] with
    respect to all header information transmitted over the wire.

I think this also means that that the following change should be done:

In Section 3, 2nd paragraph, 1st sentence:

OLD:
    A client MUST use the "ENABLE" command (defined in [RFC5161]) with
    the "UTF8=ACCEPT" option (defined in Section 4 below) to indicate to
    the server that the client accepts UTF-8 in quoted-strings.

NEW:
    A client MUST use the "ENABLE" command (defined in [RFC5161]) with
    the "UTF8=ACCEPT" option (defined in Section 4 below) to indicate to
    the server that the client accepts UTF-8 in quoted-strings and
    supports UTF8=ACCEPT extension.

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 15 Sep 2012 00:37:15 +0300

On 14.9.2012, at 23.17, Alexey Melnikov wrote:

> Hmm. Good point. I wish you brought this up earlier. Would changing the MUST to SHOULD address your concern or would you like to see further changes to this text? If the latter, can you suggest new text?

How about changing the "MUST reject" to "MUST reject or translate to 7-bit", although that requires rewriting the whole paragraph.. Something like:

If IMAP client does not issue "ENABLE UTF8=ACCEPT" or "ENABLE UTF8=ONLY"
and it sends an APPEND command that includes any 8-bit in the message
headers, the server MUST either reject the APPEND command with "NO"
response or alternatively translate the 8-bit data to 7-bit in
implementation-defined way.

Also I think there should be a paragraph:

If IMAP client does issue "ENABLE UTF8=ACCEPT" or "ENABLE UTF8=ONLY" and it
sends an APPEND command that includes invalid UTF-8 data in the message
headers, the server MUST reject the APPEND with "NO" response. Old messages may
contain non-UTF8 headers, so when a client copies messages from another store
to IMAP server, it SHOULD verify that it is sending only valid UTF8 headers.

Non-UTF8 data in message bodies is allowed, I think?

> No, I think the idea is to advertise that modified UTF-7 is not supported.

If that's the only reason for UTF8=ONLY it seems a rather wasteful capability. A server that doesn't bother to implement mUTF7, but does bother to implement other kinds backwards compatibility seems rather unlikely to me.

The text in design rationale section talks that in future everyone would be using UTF8 capable clients and servers and all the old compatibility code could be removed. Wouldn't it make much more sense for UTF8=ONLY to mean that the server has no more such compatibility code? (Basically my idea about assuming client has done ENABLE UTF8=ACCEPT.)

> Correct. This text is trying to talk about the case of a client A downloading a downgraded version, then being upgraded and still thinking that it has the authoritative version of the message.
> We don't want to require IMAP servers to use different UIDVALIDITY values for non-EAI-aware and EAI-aware clients.

If the UIDVALIDITY and the UID stays the same, the upgraded client can still think that its cached message is the authoritative version, right? Unless this text is intended for client developers to discard cache in such situations?

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 15 Sep 2012 19:35:45 +0300

Maybe it should also have a sentence about using UTF8 in NAMESPACE reply. It could contain UTF8 prefixes, and also when LANGUAGE extension is enabled the TRANSLATION could contain UTF8.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Sep 2012 21:55:05 +0100

Hi Timo,

> implementation-defined way.

Seems sensible.

> to IMAP server, it SHOULD verify that it is sending only valid UTF8 headers.

Ok.

> Non-UTF8 data in message bodies is allowed, I think?

Yes. Can be binary.

> If that's the only reason for UTF8=ONLY it seems a rather wasteful capability. A server that doesn't bother to implement mUTF7, but does bother to implement other kinds backwards compatibility seem…

I think the idea was that everybody would implement UTF8=ONLY and UTF8=ACCEPT goes away and can be obsoleted.
> The text in design rationale section talks that in future everyone would be using UTF8 capable clients and servers and all the old compatibility code could be removed. Wouldn't it make much more se…

Can you suggest some text, I am not yet sure what exactly you are suggesting.

> If the UIDVALIDITY and the UID stays the same, the upgraded client can still think that its cached message is the authoritative version, right?

Exactly.

> Unless this text is intended for client developers to discard cache in such situations?

Yes, maybe it should be clearer about that.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Sep 2012 21:55:05 +0100

Hi Timo,

Seems sensible.

Ok.

Yes. Can be binary.

Exactly.

---

From: "Jiankang YAO" <[email-masked]>
Date: Mon, 17 Sep 2012 16:58:56 +0800

----- Original Message ----- 
From: "Timo Sirainen" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, September 11, 2012 8:47 PM
Subject: [EAI] 5738bis
...
> I wonder what exactly that means. I guess UIDVALIDITY means something more than the UIDVALIDITY value specifically. If client A downloads UID 5 downgraded and client B downloads it non-downgraded,…

for the UIDVALIDITY issue, the wg did have a discussion thread named with "[EAI] Status summary on SimpleDowngrade"
http://www.ietf.org/mail-archive/web/ima/current/msg04793.html

then the wg has the following proposed text about 5738bis

http://www.ietf.org/mail-archive/web/ima/current/msg04820.html

so the proposed text appear in the current version of 5738bis.

Jiankang Yao

---

From: "Jiankang YAO" <[email-masked]>
Date: Mon, 17 Sep 2012 17:30:35 +0800

----- Original Message ----- 
From: "Timo Sirainen" <[email-masked]>
To: "Alexey Melnikov" <[email-masked]>
Cc: "IETF EAI" <[email-masked]>
Sent: Saturday, September 15, 2012 5:37 AM
Subject: Re: [EAI] 5738bis

> implementation-defined way.

The wg decides assumes that assumes that the IMAP server will be
   operating in a fully internationalized environment, and that all downgrade material goes to section 7.
so I propose to change your text "alternatively translate the 8-bit data to 7-bit in
 implementation-defined way" to "alternatively do it based on the discussed in Section 7 below"

Jiankang Yao

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 17 Sep 2012 15:17:09 +0300

> implementation-defined way" to "alternatively do it based on the discussed in Section 7 below"

I think that only talks about how to do downgrading of proper UTF8 mails to 7bit mails? I'm more worried about how to handle old 8bit mails that are not necessarily UTF8 at all.

---

From: Timo Sirainen <[email-masked]>
Date: Sun, 30 Sep 2012 16:26:29 +0300

OK, finally feeling energetic enough today to answer this mail :)

> Can you suggest some text, I am not yet sure what exactly you are suggesting.

I'm hoping to avoid having to implement the message downgrading. Has anyone tried various IMAP clients what they do if they are already being fed UTF8 headers and LIST reply? My guess is that they will handle it pretty well. So I'm thinking that the differences between UTF8=ACCEPT vs. UTF8=ONLY are:

 * APPEND: UTF8=ACCEPT requires UTF8 parameter for UTF8 headers, UTF8=ONLY doesn't. Downgrading not needed, since APPEND can simply fail with NO.

 * FETCH BODY[], BODY[HEADER], etc: Non-UTF8 enabled client is tricky, but with UTF8=ONLY the headers can be sent as-is always.

 * FETCH BODY/BODYSTRUCTURE/ENVELOPE: Semi-easy to convert everything(?) for non-UTF8 enabled client. With UTF8=ONLY just send as-is.

 * LIST: Trivial to convert mailbox names to mUTF7 for non-UTF8 enabled clients. With UTF8=ONLY always send as UTF8.

>  it does not permit the international mailbox name convention

to:

The "UTF8=ONLY" capability permits an IMAP server to advertise that it does not support legacy clients anymore, but behaves as if the client had issued ENABLE UTF8=ONLY command immediately after connect. This also means that when a client detects the UTF8=ONLY capability, there is no need for it to even send the ENABLE UTF8=ONLY command.