Subject: Re: [lemonade] Treat as WGLC: draft-cridland-imap-context-01.txt

From: Alexey Melnikov <[email-masked]>
Date: Fri, 11 May 2007 19:29:23 +0100

Alexey Melnikov wrote:

> Dave Cridland wrote:
>
>> Network Working Group                                        D. Cridland
>> Internet-Draft                                                   C. King
>> Expires: October 27, 2007                                  Isode Limited
>>                                                          April 25, 2007
>>
>>
>>                           Contexts for IMAP4
>>                      draft-cridland-imap-context
>
> Work Group Last Call closes on 11 May 2007.
>
> If you chose to send private comments, please don't forget to CC them 
> to me, so that I can keep track of issues in the document.

The WGLC is ending today. If you've reviewed the document and found no 
issues, please let me know.

So far it seems that everybody who reviewed the document is happy with 
this. If this continue to be the case, I will request publication of the 
document.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 11 May 2007 20:11:55 +0100

Some minor comments on -01:

3.3.2. REMOVEFROM Return Data Item

   C: B03 SEARCH RETURN () 1:* ALL

I think "UPDATE ALL" is missing from the ().
Also, the command should be UID SEARCH, as the second ESEARCH response 
below includes the UID tag:

   S: * ESEARCH (TAG "B03") ALL 1:49152
   S: * ESEARCH (TAG "B01") UID REMOVEFROM 32768
   S: B03 OK Search completed.

In section 4:
  ret-data-removefrom = "REMOVEFROM" sequence-set
       ;; <sequence-set> from [IMAP]

SP is missing before the sequence-set

---

From: <[email-masked]>
Date: Mon, 14 May 2007 13:13:19 +0300

Hi Dave,
here are my comments:

Section 2.
"   Although the basic SEARCH command defined in [IMAP], as enhanced by
   [ESEARCH], is relatively compact in its representation, this
   reduction only saves a certain amount of data, and huge mailboxes can
   overwhelm the storage available for results on even relatively high-
   end desktop machines."
1. "as enhanced by" -> "and enhanced by" ?
2. "this reduction only saves" -> "the reduction saves only" ?
3. "and huge mailboxes can" -> "therefore huge mailboxes might" ?
4. "for results on even relatively high-end desktop machines." -> "for results. This extension reduces the amount of data to be sent by providing a windowing facility for the search results." ?

Section 3.3
5. So far the extension is not clear about the notification used (IDLE is shown in the examples). Does this extension require IDLE or NOTIFY? If so, which? I would suggest updating all examples showing IDLE to NOTIFY instead and require NOTIFY as mandatory.
6. "The search return option UPDATE, if used by a client, causes the server to ..."
Should we make it mandatory? At least I would not want to see a client that supports notifications (IDLE/NOTIFY will be in all LEMONADE clients anyway) but cannot coop with search result updates. Server and client will get seriously out of sync without notifications, so when the client requests a next window, some results might be displayed again (when new results have been inserted before the current window) and some might be missed as well (when obsolete results have been removed before the current window).

Section 3.3.1
7. What exactly does the ADDTO Return Data Item contain? It seems to me that the client will only get position and UID. Does it mean that the client will have to fetch all data (sender, subject, - well, the header) for each email "inserted"?
Also, I did not find description, about how the addition works. Here is my assumption, please consider adding something like this: The new result is to be inserted at the given position; meaning that the new result will occupy the indicated position and all results starting from that position are shifted to the next position. The client MUST updated the position numbers of the shifted results. The ADDTO Return Data Item MAY include several new results to be insterted - therefore it is imporant to note that the positions included in a single ADDTO Return Data Item contain positions before the shifting due to other new results take place. (If the recieved ADDTO new results were sorted descending, by position, it would be very easy). Also, something similar to REMOVEFROM (3.3.2) should be added.
8. "The results are specified as UIDs or message numbers, ..."
What does 'results' mean here? My guess would be the search results on the client (and not the new results to be added). Would it be possible to use another term for the new/obsolete (see 3.3.2 as well) results?

Section 3.3.2 (3.3.1 is relevant as well).
9. "Servers SHOULD sort the results in order to use the sequence-set syntax as efficiently as possible."
Sort by ... (what? UID, date, sender, subject, priority, etc)? (Sorry, I am not familiar with SORT). Should we make this mandatory (that the list MUST be sorted)?
10. Are the ADDTO/REMOVEFROM Return Data Items to be sorted as well? (There was nothing about this in 3.3.1 or 3.3.2.)
11. "There is no requirement on servers to avoid issuing REMOVEFROM return data at any particular moment, in particular this is distinct from EXPUNGE responses."
What does it mean in practice? Would it be better to state the server must (should?) send update on EXPUNGE (or on new email arrival -> section 3.3.1), should (may) on any other event (of course only when an email is involved in our current search)?
12. Looking at the questions above I think these should be a general description (that is valid for both ADDTO and REMOVEFROM) about the updates in section 3.3. I mean some things that are described in 3.3.2 are not described in 3.3.1.

Section 3.4
13. "The subset of results are returned in sequence-set syntax, and servers SHOULD order results from a SEARCH for maximum efficiency."
This is is an amost identical duplicate of the sentence in 3.3.2 paragraph 1. Should one be removed (I understand UPDATE and PARTIAL are both optional, but still...)
Same questions regarding this question - see comment #9.

Section A.1.
14. CONTEXT is a replacement for VFOLDER. VFOLDER had the advantage of re-using virtual folders: I could easily create new virtual folders from virtual folders, and altering one virtual folder in the "path" would alter all "child" virtual folders. It appears (or, I missed something) that it is no longer possible using this extension.

Section A.2.
15. This is useful, but it would be nice to tell that it is also possible to "leave out" trashed messaged from the inbox - by using an inverted search for \Deleted.

Section A.3.
16. "It is entirely possible to simultaneously have two or more UPDATE searches in operation."
No PARTIAL?

Additonal question that I did not find an answer for:
What happens when I search without sorting, get a list of results, start windowing, cannot find what I am looking for so I decide to sort the results (or, change the basis for sorting)? I get REMOVEFROM removing everything from my view and I get an ADDTO to re-fill my view? Will I see blinking (most likely all positions and UIDs will be messed up). What happens in the same case if I do not support UPDATE (I proposed to make it mandatory above, but it is not yet mandatory).

Thank You - and sorry about the delay with the review. Hopefully still in time for the interim...

Best regards: Zoltán Ördögh
E-mail: zoltan dot ordogh at nokia dot com
Phone: +358 50 386 0566

>-----Original Message-----
>From: ext Dave Cridland [mailto:[email-masked]] 
>Sent: 25 April, 2007 16:15
>To: [email-masked]
>Cc: Enhancements to Internet email to support diverse service 
>enivronments
>Subject: [lemonade] Updated Draft: draft-cridland-imap-context
>
>This is draft-cridland-imap-context-01.txt
>
>Thanks.
>--
>Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
>  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
>  - http://dave.cridland.net/
>Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
>

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 May 2007 17:22:41 +0100

Hi Zoltan,
Thanks for the detailed comments.
Some comments below.

[email-masked] wrote:

>Should we make it mandatory?
>
Mandatory for clients to use?

> At least I would not want to see a client that supports notifications (IDLE/NOTIFY will be in all LEMONADE clients anyway) but cannot coop with search result updates. Server and client will get seriously out of sync without notifications, so when the client requests a next window, some results might be displayed again (when new results have been inserted before the current window) and some might be missed as well (when obsolete results have been removed before the current window).
>
>Section 3.3.1
Yes.

>Also, I did not find description, about how the addition works. Here is my assumption, please consider adding something like this: The new result is to be inserted at the given position; meaning that the new result will occupy the indicated position and all results starting from that position are shifted to the next position. The client MUST updated the position numbers of the shifted results. The ADDTO Return Data Item MAY include several new results to be insterted - therefore it is imporant to note that the positions included in a single ADDTO Return Data Item contain positions before the shifting due to other new results take place.
>
You observation is important, this needs to be described in the document.
 [...]

>14. CONTEXT is a replacement for VFOLDER. VFOLDER had the advantage of re-using virtual folders: I could easily create new virtual folders from virtual folders, and altering one virtual folder in th…
This is still possible by constructing a SEARCH criterion which 
logically ANDs two or more SEARCH criteria.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 14 May 2007 19:38:37 +0100

> here are my comments:
Thanks for the review.

> 4. "for results on even relatively high-end desktop machines." -> 
> "for results. This extension reduces the amount of data to be sent 
> by providing a windowing facility for the search results." ?
> 
> 
I've used some of this.

> Section 3.3
> 5. So far the extension is not clear about the notification used 
> (IDLE is shown in the examples). Does this extension require IDLE 
> or NOTIFY? If so, which? I would suggest updating all examples 
> showing IDLE to NOTIFY instead and require NOTIFY as mandatory.

I've stripped IDLE from the examples, and made it clear that neither 
IDLE nor NOTIFY is required. (Although in practise, I'm expecting 
servers to send updates when they'd usually send unsolicited data, 
and therefore take into account IDLE, NOTIFY, etc).

> Should we make it mandatory? At least I would not want to see a 
> client that supports notifications (IDLE/NOTIFY will be in all 
> LEMONADE clients anyway) but cannot coop with search result 
> updates. Server and client will get seriously out of sync without 
> notifications, so when the client requests a next window, some 
> results might be displayed again (when new results have been 
> inserted before the current window) and some might be missed as 
> well (when obsolete results have been removed before the current 
> window).
> 
> 
Yes, clients which don't use UPDATE, but do use PARTIAL, may well get 
confused, depending on what they're trying to do.

> Section 3.3.1
> 7. What exactly does the ADDTO Return Data Item contain? It seems 
> to me that the client will only get position and UID. Does it mean 
> that the client will have to fetch all data (sender, subject, - 
> well, the header) for each email "inserted"?

Yes.

> Also, I did not find description, about how the addition works. 
> Here is my assumption, please consider adding something like this: 
> The new result is to be inserted at the given position; meaning 
> that the new result will occupy the indicated position and all 
> results starting from that position are shifted to the next 
> position. The client MUST updated the position numbers of the 
> shifted results. The ADDTO Return Data Item MAY include several new 
> results to be insterted - therefore it is imporant to note that the 
> positions included in a single ADDTO Return Data Item contain 
> positions before the shifting due to other new results take place. 
> (If the recieved ADDTO new results were sorted descending, by 
> position, it would be very easy). Also, something similar to 
> REMOVEFROM (3.3.2) should be added.

Used more or less verbatim, thanks.

> What does 'results' mean here? My guess would be the search results 
> on the client (and not the new results to be added). Would it be 
> possible to use another term for the new/obsolete (see 3.3.2 as 
> well) results?
> 
> 
I've tried to make this clearer.

> 9. "Servers SHOULD sort the results in order to use the 
> sequence-set syntax as efficiently as possible."
> Sort by ... (what? UID, date, sender, subject, priority, etc)? 
> (Sorry, I am not familiar with SORT). Should we make this mandatory 
> (that the list MUST be sorted)?

The protocol is unaffected by sorting the results, but the 
representation is smaller if servers do. Typically, the server will 
issue SEARCH results in the natural order of the mailbox, which will 
then be easy to collapse into a compact set-syntax representation.

> 10. Are the ADDTO/REMOVEFROM Return Data Items to be sorted as 
> well? (There was nothing about this in 3.3.1 or 3.3.2.)

REMOVEFROM should be, ADDTO shouldn't - but for SEARCHes, you might 
as well. (It doesn't stipulate that ADDTO result updates should be 
sorted because it'd break SORT results).

> 11. "There is no requirement on servers to avoid issuing REMOVEFROM 
> return data at any particular moment, in particular this is 
> distinct from EXPUNGE responses."
> What does it mean in practice? Would it be better to state the 
> server must (should?) send update on EXPUNGE (or on new email 
> arrival -> section 3.3.1), should (may) on any other event (of 
> course only when an email is involved in our current search)?

I added a huge great paragraph on this. Basically, servers can send a 
REMOVEFROM whenever they like, as with any response - with the 
exception of EXPUNGE, which is special. I also added an appendix 
noting that it's possible to use the extension to gain immediate 
EXPUNGE notifications whilst not losing message sequence numbering.

> 12. Looking at the questions above I think these should be a 
> general description (that is valid for both ADDTO and REMOVEFROM) 
> about the updates in section 3.3. I mean some things that are 
> described in 3.3.2 are not described in 3.3.1.
> 
> 
Added.

> Section 3.4
> 13. "The subset of results are returned in sequence-set syntax, and 
> servers SHOULD order results from a SEARCH for maximum efficiency."
> This is is an amost identical duplicate of the sentence in 3.3.2 
> paragraph 1. Should one be removed (I understand UPDATE and PARTIAL 
> are both optional, but still...)
Yes, it's partly for the same reason as above, but also because 
SEARCH results need to be ordered consistently for PARTIAL to be 
useful for virtual scrollbar techniques. I've seperated these 
requirements into a MUST and a SHOULD.

> Section A.1.
> 14. CONTEXT is a replacement for VFOLDER.

No, it isn't. It's an alternate mechanism for achieving similar 
results. Arnt (and others) will tell you, quite rightly, that VFOLDER 
is optimized for different cases, and covers different requirements.

>  VFOLDER had the advantage of re-using virtual folders: I could 
> easily create new virtual folders from virtual folders, and 
> altering one virtual folder in the "path" would alter all "child" 
> virtual folders. It appears (or, I missed something) that it is no 
> longer possible using this extension.
> 
> 
Not directly. The mechanism one might use for doing this is Alexey's 
named filter draft, which would allow you to build composite 
user-defined search keys. I have no idea what the desired result of 
changing one used by an update context would be.

However, the simplest method would be to drop the context (now 
possible with a FREECONTEXT command) and recreate it with the new 
desired search program.

> Section A.2.
> 15. This is useful, but it would be nice to tell that it is also 
> possible to "leave out" trashed messaged from the inbox - by using 
> an inverted search for \Deleted.
> 
> 
I think it does - if you've a suggestion for how to make this 
clearer, I'd welcome it.

> UPDATE searches in operation."
That too, but the emphasis here is that there's no upper limit on 
update contexts. (Servers can, in -02, enforce them if they wish).

> What happens when I search without sorting, get a list of results, 
> start windowing, cannot find what I am looking for so I decide to 
> sort the results (or, change the basis for sorting)? I get 
> REMOVEFROM removing everything from my view and I get an ADDTO to 
> re-fill my view? Will I see blinking (most likely all positions and 
> UIDs will be messed up). What happens in the same case if I do not 
> support UPDATE (I proposed to make it mandatory above, but it is 
> not yet mandatory).
> 
> 
No changes to the definition of a context are possible, you'd have to 
drop and recreate.

> still in time for the interim...
Shouldn't be a problem, I'll post the update to the list as well, 
shortly. (Just finishing it off now).

Dave.

---

From: <[email-masked]>
Date: Tue, 15 May 2007 10:44:57 +0300

Hi Alexey,
further comments below.
(I removed the comments that I am OK with.)

>Mandatory for clients to use?

For all LEMONADE clients and servers. Reasoning below. I know that the draft is independent of LEMONADE, therefore it is not very convenient to enforce it in this draft.

>> At least I would not want to see a client that supports 
>notifications (IDLE/NOTIFY will be in all LEMONADE clients 
>anyway) but cannot coop with search result updates. Server and 
>client will get seriously out of sync without notifications, 
>so when the client requests a next window, some results might 
>be displayed again (when new results have been inserted before 
>the current window) and some might be missed as well (when 
>obsolete results have been removed before the current window).

>>Section A.1.
>>14. CONTEXT is a replacement for VFOLDER. VFOLDER had the 
>advantage of re-using virtual folders: I could easily create 
>new virtual folders from virtual folders, and altering one 
>virtual folder in the "path" would alter all "child" virtual 
>folders. It appears (or, I missed something) that it is no 
>This is still possible by constructing a SEARCH criterion 
>which logically ANDs two or more SEARCH criteria.

Yes, but the entire path would not exist, would it?

---

From: <[email-masked]>
Date: Tue, 15 May 2007 10:53:28 +0300

Hi Dave,
thanks for the fixes.
I will check the new draft later on today hopefully.
*rushing to a meeting*

>> Should we make it mandatory? At least I would not want to 
>see a client 
>> that supports notifications (IDLE/NOTIFY will be in all LEMONADE 
>> clients anyway) but cannot coop with search result updates. 
>Server and 
>> client will get seriously out of sync without notifications, so when 
>> the client requests a next window, some results might be displayed 
>> again (when new results have been inserted before the 
>current window) 
>> and some might be missed as well (when obsolete results have been 
>> removed before the current window).
>> 
>> 

Could we add a requiement at least (PARTIAL requires UPDATE)? Alexey sent His response on this (well, the response was a question), please consult with Him.

>then be easy to collapse into a compact set-syntax representation.

Ah, I thought that server would sort the results the way the client wants it. I guess I should familiarize myself with SORT.

-cut- I will sent another mail starting from comment#11 - gotta go.

---

From: <[email-masked]>
Date: Tue, 15 May 2007 11:48:32 +0300

Hi Dave,
Here are my repsonses to Your comments starting from #11 (which is OK, so it's missing from the mail below).

>changing one used by an update context would be.

This draft You are referring to is not part of LEMONADE, is it? Is this draft published somewhere?

>possible with a FREECONTEXT command) and recreate it with the new 

Erm, is this in the new draft? If so, then there is no need to repond; I will see it when I get there.

>I think it does - if you've a suggestion for how to make this 

I will check the new draft first.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 15 May 2007 10:53:08 +0100

> repond; I will see it when I get there.
Not explicitly - do you think it needs an explicit mention? 
Currently, there's simply no other way of changing the search program 
used for a context. Only Alexey's draft would provide a mechanism for 
this.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 15 May 2007 10:53:07 +0100

On Tue May 15 08:53:28 2007, [email-masked] wrote:
> >> 6. "The search return option UPDATE, if used by a client, causes 
> the >> server to ..."
> >> Should we make it mandatory? At least I would not want to >see a 
> client >> that supports notifications (IDLE/NOTIFY will be in all 
> LEMONADE >> clients anyway) but cannot coop with search result 
> updates. >Server and >> client will get seriously out of sync 
> without notifications, so when >> the client requests a next 
> window, some results might be displayed >> again (when new results 
> have been inserted before the >current window) >> and some might be 
> missed as well (when obsolete results have been >> removed before 
> the current window).
> >> >> >Yes, clients which don't use UPDATE, but do use PARTIAL, may 
> well get >confused, depending on what they're trying to do.
> 
> Could we add a requiement at least (PARTIAL requires UPDATE)? 
> Alexey sent His response on this (well, the response was a 
> question), please consult with Him.
> 
> 
It's a requirement for servers - if a server has "CONTEXT" or 
"CONTEXTS=", it has to do UPDATE. (Although in principle it could 
refuse to honour the requests, or something, but it's possible to 
mangle any specification into uselessness if one tries hard enough.)

> >> 9. "Servers SHOULD sort the results in order to use the >> 
> >> (Sorry, I am not familiar with SORT). Should we make this 
> mandatory >> (that the list MUST be sorted)?
> >
> >representation is smaller if servers do. Typically, the server 
> will >issue SEARCH results in the natural order of the mailbox, 
> which will >then be easy to collapse into a compact set-syntax 
> representation.
> 
> Ah, I thought that server would sort the results the way the client 
> wants it. I guess I should familiarize myself with SORT.
> 
> 
SORT isn't addressed by CONTEXT. The design is intended to handle it, 
and this specification was originally written to include it - now 
I've merely tried to make it work with a future CONTEXT=SORT 
extension.

Dave.

---

From: <[email-masked]>
Date: Tue, 15 May 2007 15:21:28 +0300

>Sent: 15 May, 2007 12:53
>To: Ordogh Zoltan (Nokia-TP-MSW/Tampere)
>Subject: RE: [lemonade] Updated Draft: draft-cridland-imap-context
>
>> Erm, is this in the new draft? If so, then there is no need 
>to repond; 
>> I will see it when I get there.
>> 
>> 
>Currently, there's simply no other way of changing the search 
>program used for a context. Only Alexey's draft would provide 
>a mechanism for this.

I do not have a strong opinion on this one. It would avoid questions
like mine ;-)

---

From: <[email-masked]>
Date: Tue, 15 May 2007 15:23:52 +0300

>> updates. >Server and >> client will get seriously out of 
>sync without 
>> notifications, so when >> the client requests a next window, some 
>> results might be displayed >> again (when new results have been 
>> inserted before the >current window) >> and some might be missed as 
>> well (when obsolete results have been >> removed before the current 
>> window).
>"CONTEXTS=", it has to do UPDATE. (Although in principle it 
>could refuse to honour the requests, or something, but it's 
>possible to mangle any specification into uselessness if one 
>tries hard enough.)

Ok, then I guess I can answer Alexey's question here and say: yes, for
the clients supporting this extension, too.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 16 May 2007 10:53:30 +0100

[email-masked] wrote:

>This draft You are referring to is not part of LEMONADE, is it?
>
I think this is an open issue.

My personal opinion is that my draft should be included in the Lemonade 
Profile Bis, because it is a missing piece of the whole solution.

>Is this draft published somewhere?
>  
>
draft-melnikov-imapext-filters-01.txt

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 20 May 2007 13:04:30 +0100

[email-masked] wrote:

>For all LEMONADE clients and servers. Reasoning below.
>
It would not be possible to have such a requirement on clients, as 
clients are free to implement anything they want. The document can only 
give an advice on the best way to achieve things.

But you are right about the requirement on servers. The draft doesn't 
say that servers are required to implement UPDATE. I also think that the 
document needs to say that every server MUST be able to support at least 
X contexts per connection. I suggest X=2.

> I know that the draft is independent of LEMONADE, therefore it is not very convenient to enforce it in this draft.
>  
>
Agreed.

>Yes, but the entire path would not exist, would it?
I don't think that a change in an intermediate virtual folder is a 
sufficiently frequent event, so I don't think there is a need to 
optimize for it.

As Dave already pointed out, one might be able to use my FILTER draft. 
It adds a new FILTER SEARCH key, that gets expanded when the SEARCH 
criteria gets evaluated. If the client changes one of the FILTERs, it 
can recreate the whole context.
(I can show a protocol example, if this is not sufficiently clear).