Subject: [lemonade] Context update

From: Dave Cridland <[email-masked]>
Date: Mon, 14 May 2007 20:00:11 +0100

I've just submitted update to my IMAP CONTEXT draft.

I've put it up at 
http://dave.cridland.net/draft-cridland-imap-context-02.txt until 
it's announced.

The major change is that update contexts can now be explicitly freed, 
and there's a response code to allow servers to refuse to honour them.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: <[email-masked]>
Date: Tue, 15 May 2007 15:24:00 +0300

Hi Dave,
Comments on -02:
Section 2.
1. "might overwhelm the storage available for results on even relatively high-end desktop machines."
In a comment vs. -01 I proposed to get rid of "relatively high-end desktop machines". It's still there, but what You call today realtively high-end might be the extreme low-end tomorrow... If You insist, keep it - but I do not think it adds much value to the draft.

Section 3.3.1
2. "In some cases, the server MAY refuse to provide updates, such as if an internal limit on the number of update contexts is reached."
I do not like the "MAY". It suggests that a server can refuse updates whenever it feels like it. A "SHOULD NOT" would be better.
Could we change this something like:
"The server SHOULD NOT refuse to provide updates unless it is running out of resources, or, some other internal limiting factor has been reached."
Addition of something like this might be beneficial:
"In order to receive updates in the future, the client MAY issue the FREECONTEXT command and perform the search."

Section 3.3.5
3. "The server MAY free any resource associated with a context so disabled."
Fuzzy/incomplete(?) sentence. Also, if all associated resources are dropped, how can I use PARTIAL after FREECONTEXT? According to paragraph one it is possible: "When a client no longer wishes to receive updates, it may issue the FREECONTEXT command, which will prevent all updates to the contexts named in the arguments from being transmitted by the server." <- this says only updates will cease, so the client could still request PARTIAL.
I suggest the sentence to be completed with something like this:
"The server MAY free any resource associated with a context and the client MUST NOT use the freed context any further during the current IMAP session."

Other than that, it look Ok to me, thank You for the update.

Best regards: Zoltán Ördögh
E-mail: zoltan dot ordogh at nokia dot com
Phone: +358 50 386 0566

>-----Original Message-----
>From: ext Dave Cridland [mailto:[email-masked]] 
>Sent: 14 May, 2007 22:00
>To: Enhancements to Internet email to support diverse service 
>enivronments; Messaging Organization
>Subject: [lemonade] Context update
>
>http://dave.cridland.net/draft-cridland-imap-context-02.txt 
>until it's announced.
>
>The major change is that update contexts can now be explicitly 
>freed, and there's a response code to allow servers to refuse 
>to honour them.
>
>Dave.
>--

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 17 May 2007 18:48:58 +0300

> http://dave.cridland.net/draft-cridland-imap-context-02.txt until 

I'm planning on implementing virtual mailboxes to my server, but I think
CONTEXT won't be integrated with that feature. CONTEXT seems like it's
useful for temporary views of mailboxes, while actual virtual mailboxes
are more permanent and often generated from multiple physical mailboxes.
For example CONTEXT mentions virtual Trash, but people usually want the
Trash to contain \Deleted messages from all the mailboxes, not just one.

Also since CONTEXT requires that the mailbox is selected and so all the
flag changes, expunges etc. are sent to clients, this doesn't really
give server the possibility of optimizing the search views the same way
as separate virtual mailboxes would.

I could still implement CONTEXT, but I think saying virtual mailboxes
could be implemented with it is a bit misleading. At least I was
expecting CONTEXT to be quite different.

I read the draft now for the first time without knowing anything about
CONTEXT before that. Some comments:

3.1:

"Such servers also accept three additional return options, and provide
three new result data items, and no new responses."

I've no idea what return options or result data items are at this point.
After reading far enough down I finally figure out it's talking about
ESEARCH. So mentioning [ESEARCH] already in the first sentence would be
helpful.

3.2:

"The return option CONTEXT SHOULD be used by a client to indicate that
subsequent use of the criteria are likely."

"the search criteria" would have helped me to understand this more
easily.

3.3.2:

   The client MUST process (and the server MUST generate) ADDTO and
   REMOVEFROM return data items in order, including those within a
   single ESEARCH response.

What order?

3.3.3:

   The ADDTO return data item MAY include several new results to be
   inserted - therefore it is imporant to note that the positions
   included in a single ADDTO return data item contain positions before
   the shifting due to other new results take place.

I can't parse that sentence. I'm guessing it means that if there are
multiple results the shifting needs to be done after processing the next
result. Or exactly the opposite.

   If the position is non-zero, the new results in the update are to be

What if it's zero? I don't think I understand this position thing at all
since it can even be ignored by clients. Or is it related to SORT?

3.3.4:

       S: * ESEARCH (TAG "B01") UID REMOVEFROM 32768
       C: B03 NOOP
       S: * EXPUNGE 23762
       S: B03 OK Nothing done.

"* 23762 EXPUNGE". And maybe a note that the REMOVEFROM MUST be sent
before the EXPUNGE (at least with sequences).

3.3.5:

   The server MAY free any resource associated with a context so
   disabled.

?

3.4:

   Where a PARTIAL search return option references results which do not
   exist, by using a range which starts or ends higher than the COUNT of
   results, then the server returns those results which are in the set.
   This yields a PARTIAL return data item which has, as payload, the
   original range and a potentially missing set of results which may be
   shorter than the extent of the range.

This paragraph is somewhat weird. It's not necessary to do COUNT for a
partial SEARCH, so mentioning COUNT in there isn't probably right. And
if the mailbox had changed since the COUNT was returned, the range does
exist. Unless that's what the last sentence tries to say with "missing
set of results"?

Also the section doesn't mention anything about problems if the mailbox
changes. So:

C: A03 UID SEARCH RETURN (PARTIAL 1:500) UNDELETED UNKEYWORD $Junk
..
C: A04 UID SEARCH RETURN (PARTIAL 501:1000) UNDELETED UNKEYWORD $Junk
S: * 1 FETCH (FLAGS $Junk)
..

Now this most likely means that the original 501th result was actually
skipped, because the first message had $Junk keyword set and the partial
result set got moved by one. Right?

---

From: <[email-masked]>
Date: Fri, 18 May 2007 11:55:55 +0300

Hi Dave,
there is another problem related to one of the comments sent by Timo.
Please find it below Timo's comment.

>From: ext Timo Sirainen [mailto:[email-masked]] 
>Sent: 17 May, 2007 18:49
>To: Dave Cridland
>Cc: Enhancements to Internet email to support diverse service 
>Subject: [lemonade] Re: [MORG] Context update
>
>3.3.3:
>
>I can't parse that sentence. I'm guessing it means that if 
>there are multiple results the shifting needs to be done after 
>processing the next result. Or exactly the opposite.
>
>What if it's zero? I don't think I understand this position 
>thing at all since it can even be ignored by clients. Or is it 
>related to SORT?

The contents of the ADDTO data item is not sorted, therefore there is a problem when there are some items to be inserted one after another immediately.
Let me elaborate using an example.
I have made a search, and I have 10 results. My current "window" is over results 1-5, like this:
1 0xAAAA
2 0xBBBB
3 0xCCCC
4 0xDDDD
5 0xEEEE
Now, the server sends an update, where ADDTO says that the client needs to insert these to the list:
3 0xBBCC
3 0xBCCC
3 0XBBBC
Now, it's pretty cleat that the result on position 3 shifts to position 6, but how does the client know which result one will occupy positions 3, 4, 5 in the updated result list (since the contents of the ADDTO data item are not sorted)?
My suggestion would be to sort ADDTO data items by position.

---

From: Dave Cridland <[email-masked]>
Date: Fri, 18 May 2007 13:33:30 +0100

On Fri May 18 09:55:55 2007, [email-masked] wrote:
> The contents of the ADDTO data item is not sorted, therefore there 
> is a problem when there are some items to be inserted one after 
> another immediately.

Yes, that's missing - the new results are inserted as a block, in the 
order they appear.

> Now, it's pretty cleat that the result on position 3 shifts to 
> position 6, but how does the client know which result one will 
> occupy positions 3, 4, 5 in the updated result list (since the 
> contents of the ADDTO data item are not sorted)?

Yes, or rather, by the order specified (or in mailbox order, for 
searches).

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Fri, 18 May 2007 14:24:01 +0100

> Trash to contain \Deleted messages from all the mailboxes, not just 
Right, although it certainly helps with the non-trash view. The 
problem with modelling the output of what one might call a volatile 
search - one which matches on mutable properties of a message, such 
as the \Deleted flag - as a mailbox is that you'd be removing and 
reinserting messages from a mailbox rather a lot.

> as separate virtual mailboxes would.
Sure, but in practise, I'm not convinced this is actually an issue to 
be worried about - you're unlikely to see much in the way of flag 
changes and expunges in most cases. And if that is a problem for 
clients, they can use NOTIFY to prevent flag changes from being sent, 
at least. (They can also prevent expunges, although that's less 
suitable for many clients).

> expecting CONTEXT to be quite different.
It allows for implementing simple views. In a lot of cases, that's 
all you need. It's certainly what I use most.

> ESEARCH. So mentioning [ESEARCH] already in the first sentence 
Right.

> "the search criteria" would have helped me to understand this more
Okay.

>    REMOVEFROM return data items in order, including those within a
I'll change "order" to "the order they appear".

> result. Or exactly the opposite.
Yes, that's confusing. :-)

> since it can even be ignored by clients. Or is it related to SORT?
This is related to SORT, which is now going back into the document, 
so a lot of this will make more sense. SORT related ADDTOs cannot 
have a zero position.

> before the EXPUNGE (at least with sequences).
Very good point. I'd completely forgotten that.

>    The server MAY free any resource associated with a context so
Servers aren't obliged to remove local state for a context that's 
freed, primarily because the client is still allowed to perform 
PARTIAL requests from it. Servers just aren't allowed to send updates 
anymore.

> exist. Unless that's what the last sentence tries to say with 
Yes, that's it.

> result set got moved by one. Right?
Yes, it does - this is why clients almost certainly want to use 
PARTIAL in one of two ways:

1) For snapshot views of part of a SEARCH/SORT result. A typical 
use-case might be a webmail, where you don't have a particularly 
long-running connection.

2) With UPDATE - in which case you won't "miss" the result.

The thing you don't want to be doing is trying to walk the result 
without UPDATE - you are indeed very likely to miss results as the 
mailbox changes.

I should probably add something about this.

Dave.

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 20 May 2007 13:25:15 +0100

> PARTIAL requests from it. Servers just aren't allowed to send updates 

What you say was not obvious to me when I've reviewed your draft. I 
suggest you clarify this in the next revision.