Subject: Re: [lemonade] Expert Review: annotatemore (like WGLC)

From: Dan Karp <[email-masked]>
Date: Mon, 2 Apr 2007 21:31:14 -0700 (PDT)

> http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt

Comments interspersed btween snippets of the draft:

   Changes from -10 to -11:
   7.   Added match type and collation identifier to the LIST-EXTENDED
        selection option.
   8.   Made support for IMAP-I18N a requirement.

   Text comparisons may be done as part of the GETMETADATA or LIST-
   EXTENDED commands.  As a result, support for the COMPARATOR
   [I-D.ietf-imapext-i18n] extension is REQUIRED.

GETMETADATA doesn't actually do text comparisons, right?  The BNF
doesn't seem to tie in the "matchtype" or "collation" productions.

But more to the point, I believe that the intention was that METADATA
should be a simple, baseline extenstion that will be required by a wide
range of other extensions.  If that's the case, *PLEASE* reconsider the
wisdom of adding the LT, LE, GT, and GE matchers (and thus the
COMPARATOR requirement) to this draft.  It doesn't apear to be core
functionality for METADATA, and since COMPARATOR appears nontrivial to
implement this may not only prevent server authors from implementing
METADATA but also keep us from impelenting any future extension that
requires METADATA.

   A user can only set and retrieve private or shared annotations on a
   mailbox which exists and is returned to them via a LIST or LSUB
   command, irrespective of whether they have read or write access to
   the actual message content of the mailbox.  If the client attempts to
   set or retrieve annotations on other mailboxes, the server MUST
   respond with a NO response.

I can understand the logic behind requiring LIST/LSUB access for reading
public mailbox annotations and for setting private annotations.  But
that seems to be a very low bar for setting public mailbox annotations.
Adding a new ACL permission for setting annotations would make this
extension more complex and would also introduce a dependency on the ACL
extension, but perhaps requiring [READ-WRITE] access to the mailbox
might be reasonable?

   The new SETMETADATA command and the METADATA response each have a
   mailbox name argument, indicating that the annotations being referred
   to are attached to the specified mailbox.  An empty string can be
   used for the mailbox name to signify server annotations.

What rights are required to set public and private annotations on the
server?

   This command sets the specified list of entries by adding or
   replacing the specified attributes with the values provided, on the
   specified existing mailboxes.  Clients can use NIL for values of
   attributes it wants to remove from entries.  The server MAY return a
   METADATA response containing the updated annotation data.

In this case, the server may return a METADATA response that doesn't
contain the updated annotation data or a LIST response that does,
correct?

   This extension adds the GETMETADATA command.  This allows clients to
   retrieve server annotations.  Mailbox annotations are retrieved via
   the extended LIST command described in the next section.

   This extension adds the SETMETADATA command.  This allows clients to
   set annotations.

It's a little incongruous that GETMETADATA is used only to retrieve
server annotations, but SETMETADATA can be used to set both server and
mailbox annotations.  I understand that the mailbox-level function of
GETMETADATA has been passed to the LIST-EXTENDED feature, but generally
it's clearer when a GETFOO and SETFOO pair cover the same target space.

   If the METADATA extension is present, support for shared annotations
   is REQUIRED, whilst support for private annotations is OPTIONAL.
   This recognizes the fact that support for private annotations may
   introduce significantly more complexity to a server in terms of
   tracking ownership of the annotations, how quota is determined for
   users based on their own annotations etc.  Clients that support the
   METADATA extension MUST handle both shared and private annotations.

Is there a mechanism for informing clients that a server does not
support private annotations?  Or is a client expected to determine this
from METADATA NOPRIVATE response codes on a SETMETADATA NO response?

If the latter is the case, won't client implementations end up allowing
users one attempt to set private annotations and only then inform them
that the functionality is not supported by the server?  (This would
probably occur once per IMAP session...)

   If the server is unable to set an annotation because the size of its
   value is too large, the server MUST return a tagged NO response with
   a "[METADATA TOOBIG]" response code.

Likewise, is there a mechanism for a server to tell its clients the
maximum annotation length it supports?  Or is it up to the client to
determine this from METADATA TOOBIG error code responses?

   Entry and attribute names are case-sensitive.

   Entry and attribute names MUST NOT contain asterisk ("*") or percent
   ("%") characters and MUST NOT contain non-ASCII characters or the NUL
   octet.  Invalid entry or attribute names result in a BAD response in
   any IMAP commands where they are used.

   Use of control or punctuation characters in entry and attribute names
   is strongly discouraged.

It's probably cleaner in the long run to use the BNF to explicitly
specify a limited set of permitted characters for entry and attribute
names.  Constraining them to something like "atom" plus the delimiters
is probably not a bad idea.

Given that these names are already constrained to ASCII, I'd favor
eliminating the case-sensitivity requirement.  It differs from other
IMAP commands and extensions' general consistent overall case-
insensitivity (e.g. RFC 3501 keywords).

   value
      A string or binary data representing the value of the annotation.
      To delete an annotation, the client can store "NIL" into the
      value.  The content represented by the string is determined by the
      Content-Type used to register the entry (see Section 6.1 for entry
      registration templates).  Where applicable, the registered
      Content-Type MUST include a charset parameter.  Text values SHOULD
      use the utf-8 [RFC3629] character set.

Is there any reason not to mandate UTF-8 for text values?  We seem to be
moving towards standardizing on UTF-8 in IMAP, and this seems a good
opportunity to draw a line.

   /sort
      Defines the default sort criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      sorting is not required.  Note that the presence of this attribute
      does not imply that the server supports the IMAP SORT
      [I-D.ietf-imapext-sort] extension - servers can choose to support
      or not support SORT independently of this extension.  In the
      absence of the SORT extension, clients can choose to interpret
      this entry as suggesting a client-side sort ordering of the
      corresponding mailbox.

   /thread
      Defines the default thread criteria [I-D.ietf-imapext-sort] to use
      when first displaying the mailbox contents to the user, or NIL if
      threading is not required.  If both sort and thread are not NIL,
      then threading should take precedence over sorting.  Note that the
      presence of this attribute does not imply that the server supports
      the IMAP THREAD [I-D.ietf-imapext-sort] extension - servers can
      choose to support or not support THREAD independently of this
      extension.  In the absence of the THREAD extension, clients can
      choose to interpret this entry as a client-side thread ordering of
      the corresponding mailbox.

These registered annotations are odd ducks.  They seem most appropriate
as private annotations on a mailbox, but in practice they may be
confusing to the end user because sorting and threading preferences are
often client-specific, not user-specific.

For instance, a user may prefer to read their mail nonthreaded on the
desktop, but on a mobile device they may want to make use of the reduced
visual footprint of a threaded display.  Likewise, more likely than not
a user won't expect their sort preferences to automatically be
transferred between their devices and clients.

   Mailboxes or the server as a whole may have zero or more annotations
   associated with them.  An annotation contains a uniquely named entry
   with a set of uniquely named attributes, each of which has a value.
   Annotations can be added to mailboxes when a mailbox name is provided
   as the first argument to the new command, or to the server as a whole
   when the empty string is provided as the first argument to the new
   command.

Just a nitpick here, but you should probably refer to SETMETADATA
command by name since (a) it hasn't been introduced at this point in the
document and (b) there is more than one new command added by the draft.

   Each annotation is an entry that has a hierarchical name, with each
   component of the name separated by a slash ("/").  An entry name MUST
   NOT contain two consecutive "/" characters and MUST NOT end with a
   "/" character.

   Each entry is made up of a set of attributes.  Each attribute has a
   hierarchical name, with each component of the name separated by a
   period (".").  An attribute name MUST NOT contain two consecutive "."
   characters and MUST NOT end with a "." character.

My last comment is personal preference: I fear that this may be a bit of
overkill for folder annotation.  I understand the need for this type of
flexible syntax for ACAP's purposes, but the use cases for IMAP appear
significantly more limited in scope.  In accordance with the KISS
philosophy, is there any chance I might be able to convince everyone to
consider shedding a layer of hierarchy here?

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 09 Apr 2007 20:07:40 +0100

>Comments interspersed btween snippets of the draft:
Hi Dan,

>doesn't seem to tie in the "matchtype" or "collation" productions.
Check the <list-select-metadata>, it references both.

>implement
Can you please elaborate on this?
(Note that the COMPARATOR extension is about to switch to 
i;unicode-casemap collation (as mandatory to implement), which is much 
easier to implement than i;basic)

> this may not only prevent server authors from implementing
>extension,
Dependency in what sense?

> but perhaps requiring [READ-WRITE] access to the mailbox
>might be reasonable?
>
>
None. But maybe this should be stated explicitly.

>it's clearer when a GETFOO and SETFOO pair cover the same target space.
Yes, but the alternative is to have 2 ways of getting annotations, which 
is worse.

>from METADATA NOPRIVATE response codes on a SETMETADATA NO response?
Currently the latter.

>that the functionality is not supported by the server?
>
Clients can try to workaround this by trying to set an annotation to a 
fake value.
Yes, this doesn't look very nice, but works.

Defining a new capability wouldn't work, because in general case 
NOPRIVATE can be namespace or even mailbox-specific.

We can add a new \NoPrivate per mailbox attribute to return this in LIST?

>  (This would probably occur once per IMAP session...)
>
>
The latter.
Also note, that the server might return TOOBIG because it simply ran out 
of storage space, so advertising the maximum might not be that helpful 
to the client.

>Given that these names are already constrained to ASCII,
>
I don't think this is the case! Annotation names can contain UTF-8.

> I'd favor eliminating the case-sensitivity requirement.  It differs from other
>Is there any reason not to mandate UTF-8 for text values?
>
Values can also be binary and not every binary value is a valid UTF-8.

>  We seem to be
"Values for annotations published on Standard track MUST use charset=UTF-8"?
This still allows vendors to do whatever they want with their private 
annotations.

>transferred between their devices and clients.
I would let other people comment on this.

>document and (b) there is more than one new command added by the draft.
Agreed.

>consider shedding a layer of hierarchy here?
This is not going to be a problem in practice. IMHO, new attributes are 
unlikely to be standardized.
IMHO, the current model of value.priv + value.shared per annotation 
should work for 99% cases.

---

From: Dan Karp <[email-masked]>
Date: Mon, 16 Apr 2007 09:14:57 -0700 (PDT)

I'd really like to implement LEMONADE.  LEMONADE consists of several
IMAP extensions which require METADATA, and the current METADATA draft
in turn requires LIST-EXTENDED and COMPARATOR.  Unfortunately, my server
implementation is not compatible with COMPARATOR, in particular the
requirement to support i;octet in SEARCH.  I'm hoping that there's a way
to come to a compromise whereby METADATA can return to not requiring
COMPARATOR while still maintaining all its core functionality.  I'd hate
for such a small thing to be a roadblock for my and others' LEMONADE
implementations.

Right now, METADATA uses COMPARATOR to specify LIST-EXTENDED selection
constraints.  The GETMETADATA command doesn't appear to support these
constraints at present (though this may be an oversight):

    getmetadata       = "GETMETADATA" SP entries SP attribs
    entries           = entry-match /
                        "(" entry-match *(SP entry-match) ")"
    entry-match       = list-mailbox
    attribs           = attrib / "(" attrib *(SP attrib) ")"
    attrib            = astring

I've looked through all the entries defined by the base METADATA draft
as well as those defined by several of the LEMONADE drafts that metion
METADATA (/comment, /motd, /admin, /sort, /thread, /check, /checkperiod,
/convert/..., etc.).  None of these appear to benefit in a meaningful
way from the LIST-EXTENDED match/collation feature, so I'm gathering
that the inclusion of COMPARATOR is intended as more of a nice-to-have
with an eye towards accommodating possible future needs rather than a
real core METADATA feature.  If this is the case -- and towards the end
of making METADATA a workable foundation extension that every server can
implement -- I'd like to request that the LIST-EXTENDED METADATA
selection option type be removed from the METADATA draft.  If in the
future there is a new METADATA entry that would substantially benefit
from using COMPARATOR with LIST-EXTENDED, it should be possible to
reintroduce this functionality (or a subset thereof) in the same RFC
that registers the new entry.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 18:34:46 +0200

Dan Karp writes:
> Unfortunately, my server implementation is not compatible with 
> COMPARATOR, in particular the requirement to support i;octet in 
> SEARCH.

Why not?

Arnt

---

From: Dan Karp <[email-masked]>
Date: Mon, 16 Apr 2007 09:41:33 -0700 (PDT)

> > COMPARATOR, in particular the requirement to support i;octet in 

Searchable content is effectively transcoded to UTF-16.  The original content needed for "i;octet" is available only by serially opening, reading, and reparsing each message in the mailbox (stored 1 file per message), which is sufficiently nonperformant that we can't permit it.

---

From: Dan Karp <[email-masked]>
Date: Mon, 16 Apr 2007 10:03:51 -0700 (PDT)

> > Searchable content is effectively transcoded to UTF-16.  The
> > original content needed for "i;octet" is available only by serially
> > opening, reading, and reparsing each message in the mailbox (stored 1
> > file per message), which is sufficiently nonperformant that we can't
> > permit it.
> 
> Do you support full substring searches as standard SEARCH command
> requires? In Zimbra's feature list I see "Ability to search using a
> prefix plus a wildcard", which seems like a "no" answer. In that case
> you're already noncompliant.

*That* is a red herring, and I believe that you know it.  Can we please
redirect the discussion back to the core issue, which is that

  a) COMPARATOR is a reasonably heavy extension to implement
  b) METADATA is meant to be a reasonably light extension to implement
  c) METADATA is being positioned as a prerequisite for many new
extensions
  d) No METADATA entries appear able to make use of the COMPARATOR
functionality

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 16 Apr 2007 19:53:10 +0300

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 19:08:37 +0200

Dan Karp writes:
> Searchable content is effectively transcoded to UTF-16.  The original 
> content needed for "i;octet" is available only by serially opening, 
> reading, and reparsing each message in the mailbox (stored 1 file per 
> message), which is sufficiently nonperformant that we can't permit 
> it.

Makes sense. (We do something similar, although in our case it the 
problem is mitigated by other circumstances.)

I'm wondering whether the i;octet requirement has outlived its 
rationale. I know why it needed to be there in the early drafts (other 
collations fell back to i;octet for some input), but I don't know 
whether it needs to any more...?

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 19:12:25 +0200

>   a) COMPARATOR is a reasonably heavy extension to implement

I don't think COMPARATOR ought to be that. Various other extensions 
require it, and it seems wrong for me that COMPARATOR should be heavier 
than its users. Tail wagging dog.

Tell me what makes COMPATOR heavy?

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Apr 2007 10:15:04 -0700 (PDT)

On Mon, 16 Apr 2007, Timo Sirainen wrote:
>> message), which is sufficiently nonperformant that we can't permit it.

True.  I'm not sure what "search using a prefix plus a wildcard" is 
supposed to mean in IMAP searching, which is of substrings only.

For collations in the IMAP context, UTF-16 is problematic for i;octet, 
i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

I suggest that you use either UTF-8 or UCS-4.  The attraction of UTF-16 is 
to save memory space and a mostly-constant width if most characters are in 
the BMP (thus effectively the same as UCS-2 handling), but at the cost of 
MUCH more complex handling of non-BMP characters.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Dan Karp <[email-masked]>
Date: Mon, 16 Apr 2007 10:19:17 -0700 (PDT)

> require it, and it seems wrong for me that COMPARATOR should be
> heavier than its users. Tail wagging dog.
> 
> Tell me what makes COMPARATOR heavy?

IMAP (via SEARCH) essentially requires that message content be
searchable in some intermediate transcoded form (we chose UTF-16).
COMPARATOR (via "i;octet") requires that message content be searchable
in its original form pace content-encoding.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 19:20:05 +0200

> For collations in the IMAP context, UTF-16 is problematic for i;octet,

yes.

> i;ascii-casemap, and i;unicode-casemap.  I'm not sure about i;basic.

None of these - the problem was raised about a year ago by an RFC 4790 
reviewer and we solved it.

Using UTF-16 or UTF-8 instead of a sensible unicode encoding may make it 
more difficult to implement e.g. i;ascii-casemap, but not especially. 
Every kind of processing is more difficult in UTF-* than in UCS-*, IMO, 
and most collations are just specific examples of this general rules.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 16 Apr 2007 20:43:33 +0300

> whether it needs to any more...?

The only place I see i;octet required is i18n draft, so I guess it's
still possible to change it?

I would also want to see i;octet requirement gone. Actually I'd go even
a bit further. Is there a reason why a client should be able to force a
server to use i;ascii-casemap if i;unicode-casemap is also available? I
can't think of any, but it would reduce optimization possibilities for
server implementations.

Is the i;basic mentioned in i18n draft the same as i;octet?

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 16 Apr 2007 18:51:40 +0100

>Is the i;basic mentioned in i18n draft the same as i;octet?
No. i;basic is defined in draft-gulbrandsen-collation-basic-01.txt.
Note that the mandatory-to-implement in i18n draft is going to be 
i;unicode-casemap.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Apr 2007 11:00:47 -0700 (PDT)

IMHO:

i;ascii-casemap is important because there are legacy implementations that 
implement it.  Specifically, i;ascii-casemap *is* effectively the 
comparator of IMAP since its creation in 1986.  I believe that most modern 
IMAP server implementations do some version of i;unicode-casemap instead 
of i;ascii-casemap, and there is no way for the client to know with a 
server that does not otherwise support COMPARATOR.

I can't imagine that a client would choose i;ascii-casemap instead of 
i;unicode-casemap.

AFAIK, i;octet was documented for completeness.

I would be alright with i;unicode-casemap as being mandatory-to-implement 
for COMPARATOR, and optional-to-implement (albeit with a SHOULD) instead 
of i;ascii-casemap for non-COMPARATOR servers.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 21:35:41 +0200

Timo Sirainen writes:
>>  rationale. I know why it needed to be there in the early drafts 
>>  (other collations fell back to i;octet for some input), but I don't 
>>  know whether it needs to any more...?
>

That's the stated requirement. There may be an actual requirement 
elsewhere. SORT comes to mind.

> I would also want to see i;octet requirement gone. Actually I'd go 
> even a bit further. Is there a reason why a client should be able to 
> force a server to use i;ascii-casemap if i;unicode-casemap is also 
> available?

Yes, but they're odd and weird.

> I can't think of any, but it would reduce optimization possibilities 
> for server implementations.

i;ascii-casemap is the same as i;unicode-casemap for the intersection of 
their definition set, which would give you these optimisations back. 
(IIRC, and it's late and...)

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 16 Apr 2007 21:38:51 +0200

> AFAIK, i;octet was documented for completeness.

FWIW: 2244 specified it, I don't know why, although I'm sure the reason 
is in an archive somewhere. 4790 defined a registry and inserted the 
preexisting collations.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Apr 2007 12:44:33 -0700 (Pacific Daylight Time)

On Mon, 16 Apr 2007, Arnt Gulbrandsen wrote:
>> The only place I see i;octet required is i18n draft, so I guess it's still 
>> possible to change it?
> That's the stated requirement. There may be an actual requirement elsewhere. 
> SORT comes to mind.

No. SORT would use i;unicode-casemap or (older servers) i;ascii-casemap. 
I always assumed that i;octet was there as a baseline for documentation 
purposes.  I never implemen ted it.

>> Is there a reason why a client should be able to force a 
>> server to use i;ascii-casemap if i;unicode-casemap is also available?

Examples, please?  My software did i;ascii-casemap, but only in old 
versions as an interim measure.  Newer versions are i;unicode-casemap 
only.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 16 Apr 2007 22:54:09 +0300

On 16.4.2007, at 22.35, Arnt Gulbrandsen wrote:

>> even a bit further. Is there a reason why a client should be able  
>> to force a server to use i;ascii-casemap if i;unicode-casemap is  
>> also available?
>
>> I can't think of any, but it would reduce optimization  
>> possibilities for server implementations.
>
> i;ascii-casemap is the same as i;unicode-casemap for the  
> intersection of their definition set, which would give you these  
> optimisations back. (IIRC, and it's late and...)

I'm just worried that some clients (or users) would select the  
i;ascii-casemap even when they could (or should) use i;unicode-casemap.

Non-ASCII characters with i;ascii-casemap are different, so if the  
server optimized i;unicode-casemap searches and a client did a non- 
ASCII search with i;ascii-casemap, the server would have to verify  
the unicode-optimized matches by searching through the messages to  
see if the non-ASCII casing matches exactly or not. Probably not that  
big of a deal, but it would be pretty pointless extra code.
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Mon, 16 Apr 2007 22:59:09 +0100

On Mon Apr 16 20:38:51 2007, Arnt Gulbrandsen wrote:
> FWIW: 2244 specified it, I don't know why, although I'm sure the 
> reason is in an archive somewhere. 4790 defined a registry and 
> inserted the preexisting collations.

ACAP deals in arbitrary binary data which *might* be treatable as 
text - the server has no way of knowing. "i;octet" is remarkably 
useful at doing exact matches, whether text or binary, and was the 
only comparator capable of doing case-sensitive matching on text.

In IMAP, this is the only use, and there are undoubtedly better ways 
of doing this.

(I do feel the need, sometimes, for case sensitivity - as today, when 
searching for messages with the subject line including "OMA").

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Apr 2007 16:04:24 -0700 (Pacific Daylight Time)

On Mon, 16 Apr 2007, Dave Cridland wrote:
> ACAP deals in arbitrary binary data which *might* be treatable as text - the 
> server has no way of knowing. "i;octet" is remarkably useful at doing exact 
> matches, whether text or binary, and was the only comparator capable of doing 
> case-sensitive matching on text.
> In IMAP, this is the only use, and there are undoubtedly better ways of doing 
> this.

I don't understand.  Everything in IMAP is case-insensitive by definition. 
There is, AFAIK, no need or use for i;octet anywhere in IMAP.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 17 Apr 2007 10:38:31 +0200

Mark Crispin writes:
> No. SORT would use i;unicode-casemap or (older servers) 
> i;ascii-casemap. I always assumed that i;octet was there as a 
> baseline for documentation purposes.  I never implemen ted it.

SORT requires a complete sort, doesn't it? Nothing other than i;octet 
provides that any more.

i;unicode-casemap contains a sentence which might be construed as "and 
use i;octet from this point on". I'll try to suggest a alternative 
wording there, something that reads naturally for an UCS-4 
implementation without i;octet.

>>> Is there a reason why a client should be able to force a server to 
>>> use i;ascii-casemap if i;unicode-casemap is also available?
> Examples, please?

Any test suite. A test client wishing to test everything required by 
lemonade would append some carefully constructed messages and use 
search on them with i;ascii-casemap. An IMAP client which connects to n 
IMAP servers and executes the same commands on each, noting any 
differences between the server responses.

I thought of something rather different last night, can't remember what 
it was. But odd and weird - nothing a MUA would do.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 17 Apr 2007 10:43:54 +0200

Dan Karp writes:
> COMPARATOR (via "i;octet") requires that message content be 
> searchable in its original form pace content-encoding.

Only i;octet does that... we did a lot of work for RFC4790 to allow your 
kind of implementation, so I think it should be reasonable that 
COMPARATOR allows it too.

As far as I can see, the mention of i;octet in 
draft-crispin-collation-unicasemap is the only substantial requirement 
on i;octet, and it's weak. The one in draft-ietf-imapext-i18n is a 
leftover - something I forgot to delete when its rationale disappeared.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Tue, 17 Apr 2007 10:11:21 +0100

On Tue Apr 17 09:38:31 2007, Arnt Gulbrandsen wrote:
> i;unicode-casemap contains a sentence which might be construed as 
> "and use i;octet from this point on". I'll try to suggest a 
> alternative wording there, something that reads naturally for an 
> UCS-4 implementation without i;octet.

Well, to do "i;octet" would mean encoding to UTF-8 first.

However, you get the same result if you perform a dictionary-like 
ordering over UCS-4, as far as I'm aware, so it's not a problem for 
collations, and equality tests work fine too.

It's things like substring matching which cause problems, since you 
can search for a substring matching partial UTF-8 sequences in 
principle, however, if you've already called those out as illegal 
input at a protocol level (or "super-comparator" level in this case), 
even that's blocked.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 17 Apr 2007 10:20:20 +0100

On Tue Apr 17 00:04:24 2007, Mark Crispin wrote:
>> In IMAP, this is the only use, and there are undoubtedly better 
>> ways of doing this.
> 
> I don't understand.  Everything in IMAP is case-insensitive by 
> definition. There is, AFAIK, no need or use for i;octet anywhere in 
> IMAP.

As I said, I was searching for OMA in subject lines rather a lot 
yesterday - if I were able to have specified a case-sensitive match, 
I would have done, since the sequence "oma" appears rather a lot in 
spurious results. (Such as "Domain", "Automaton", and rather 
interestingly, "to make" and "from a" on several implementations).

I don't think it's quite right to state that "everything in IMAP is 
case-insensitive", since it seems intuitive to me that subject lines 
and other human-orientated text is demonstratively case-sensitive; it 
merely may be more useful to search it case-insensitively for much of 
the time.

(This is not a demand for an "i;unicode-not-casemap" or anything, 
merely an observation.)

Dave.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Apr 2007 06:30:14 -0700

On Tue, 17 Apr 2007, Dave Cridland wrote:
> case-insensitive", since it seems intuitive to me that subject lines and 
> other human-orientated text is demonstratively case-sensitive; it merely may 
> be more useful to search it case-insensitively for much of the time.

Take a time machine back 21 years and tell me that I am wrong for 
specifying that everything in IMAP is case-insensitive.  I would have told 
you back then that you are wrong, and I see no good cause to reverse that 
choice today.

-- Mark --

---

From: Dave Cridland <[email-masked]>
Date: Tue, 17 Apr 2007 16:20:06 +0100

On Tue Apr 17 14:30:14 2007, Mark Crispin wrote:
>> I don't think it's quite right to state that "everything in IMAP 
>> is case-insensitive", since it seems intuitive to me that subject 
>> lines and other human-orientated text is demonstratively 
>> case-sensitive; it merely may be more useful to search it 
>> case-insensitively for much of the time.
> 
> specifying that everything in IMAP is case-insensitive.  I would 
> have told you back then that you are wrong, and I see no good cause 
> to reverse that choice today.

NO, ON FURTHER CONSIDERATION YOU'RE ABSOLUTELY RIGHT. CASE IS NEVER 
INDICATIVE OF ANYTHING, THEREFORE IT MAKES NO SENSE TO SEARCH FOR IT 
IN ANY INSTANCE. ALSO, ALL EMAIL SHOULD BE PRESENTED ON PROPER 80 
COLUMN DISPLAYS, WITH NICE GREEN TEXT ON A PALE GREY SMEARED 
BACKGROUND YOU'RE ENCOURAGED TO CALL BLACK.

Seriously, though, while I'd readily agree that case should be 
irrelevant in IMAP protocol elements - we've already had that 
argument, in fact - I don't see that the same applies to 
human-created, human-directed text, such as subject fields, where the 
paragraph above does show that case conveys information.

It seems reasonable that we wouldn't want to mandate that clients 
should be prevented from searching case-sensitively. Similarly, many 
IMAP servers, by stripping away spaces and punctuation before the 
search test, can harm the usability of searches.

Not always. Just sometimes. I don't think that's a contentious 
statement, and I suspect I've worded this badly if you're disagreeing 
with me this strongly.

Dave.

---

From: Dan Karp <[email-masked]>
Date: Tue, 17 Apr 2007 08:52:18 -0700 (PDT)

> Only i;octet does that... we did a lot of work for RFC4790 to allow
> your kind of implementation, so I think it should be reasonable that 

i;ascii-casemap is problematic as well, for roughly the same reasons:

   Its equality, ordering, and substring operations are as for i;octet,
   except that at first, the lower-case letters (octet values 97-122) in
   each input string are changed to upper case (octet values 65-90).

Servers wishing to optimize i;unicode-casemap may well store only a
searchable upper case version of the text, which will sort incorrectly
under i;ascii-casemap.  COMPARATOR requires servers to implement both
i;ascii-casemap and i;octet (though i;octet sounds like it may be on the
way out).  If your aim is to make baseline COMPARATOR support fairly
lightweight for IMAP servers, you may be constrained to only requiring
i;unicode-casemap.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 17 Apr 2007 18:45:19 +0200

Dan Karp writes:
>    Its equality, ordering, and substring operations are as for 
>    i;octet, except that at first, the lower-case letters (octet 
>    values 97-122) in each input string are changed to upper case 
>    (octet values 65-90).
>
> searchable upper case version of the text, which will sort 
> incorrectly under i;ascii-casemap.

Good point.

I wanted to permit implementations of i;ascii-casemap to do whatever 
comes easiest outside US-ASCII. However, the final RFC doesn't reflect 
my viewpoint in this respect. I don't remember why and don't feel like 
searching my archive. I certainly wasn't aware of the argument you 
bring up now.

Do you think this is a signficant issue? It doesn't affect searching to 
a noticeable degree, and its impact on sorting is O(size of result 
set), not O(size of mailbox).

> If your aim is to make baseline COMPARATOR support fairly lightweight 
> for IMAP servers, you may be constrained to only requiring 
> i;unicode-casemap.

Mumble. Given how many IMAP servers actually implement i;ascii-casemap 
and nothing else, I'm not happy doing that. Make some other people 
agree with you so Pete will say "WG consensus".

Arnt

---

From: Dan Karp <[email-masked]>
Date: Tue, 17 Apr 2007 10:06:04 -0700 (PDT)

> Do you think this is a signficant issue? It doesn't affect searching
> to a noticeable degree, and its impact on sorting is O(size of result 

Actually, yeah, it does impact search.  If you search for
"M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that to
match "M<O-umlaut>TLEY CR<U-umlaut>E".  If you've only stored an
upcased version of the text, you can't tell the case on the original
and thus you can't determine if the non-ASCII accented characters
match under i;octet.

> > If your aim is to make baseline COMPARATOR support fairly
> > lightweight for IMAP servers, you may be constrained to only
> > requiring i;unicode-casemap.
> 

Is this true?  Do these servers really implement i;ascii-casemap as
defined in RFC 4790?  I'd expect any RFC 3501-compliant server that
knows about more charsets than just US-ASCII to be supporting
i;unicode-casemap and *not* i;ascii-casemap because of case-insensitive
"SEARCH CHARSET".

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 17 Apr 2007 20:25:36 +0200

Dan Karp writes:
>>  comes easiest outside US-ASCII. However, the final RFC doesn't 
>>  reflect my viewpoint in this respect. I don't remember why and 
>>  don't feel like searching my archive. I certainly wasn't aware of 
>>  the argument you bring up now.
>>
>>  to a noticeable degree, and its impact on sorting is O(size of 
>>  result set), not O(size of mailbox).
>
> "M<o-umlaut>tley Cr<u-umlaut>e", i;ascii-casemap doesn't allow that 
> to match "M<O-umlaut>TLEY CR<U-umlaut>E". If you've only stored an 

By the time you get that far, you've eliminated almost all messages not 
in the actual result set, so opening and rechecking isn't so awfully 
expensive.

>>  Mumble. Given how many IMAP servers actually implement 
>>  i;ascii-casemap and nothing else, I'm not happy doing that. Make 
>>  some other people agree with you so Pete will say "WG consensus".
>
> i;unicode-casemap and *not* i;ascii-casemap because of 
> case-insensitive "SEARCH CHARSET".

Build some consensus please. I don't want to argue today - I'm in a mood 
to help little old ladies halfway across the street. Being civil is a 
strain.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 17 Apr 2007 21:54:36 +0300

On Tue, 2007-04-17 at 20:25 +0200, Arnt Gulbrandsen wrote:
> Build some consensus please.

My server implements only i;ascii-casemap currently, but that's just
because I've been busy getting it stable and no-one's complained. I'm
going to move to i;unicode-casemap soon, and I don't see a need for
i;ascii-casemap. I'd think anyone who is going to implement COMPARATOR
won't see i;unicode-casemap requirement as a problem.

---

From: Dan Karp <[email-masked]>
Date: Wed, 18 Apr 2007 21:19:04 -0700 (PDT)

> Build some consensus please.

I chatted with Arnt, and he thinks it'd be worthwhile to canvas the
troops and get some hard data on what various servers actually implement
for their default SEARCH comparator.  I've thrown together a little
5-question survey which I'll send out under separate cover.