Subject: Re: [lemonade] draft-gulbrandsen-imap-notify-01 use cases

From: Randall Gellens <[email-masked]>
Date: Wed, 10 Jan 2007 17:08:23 -0800

Sorry for the delay in replying (catching up with backlogged stuff).

At 11:41 AM +0100 11/23/06, Arnt Gulbrandsen wrote:

>  Martin Konold writes:
>>  (*) a typical use case is a CRM system with a subfolder for each 
>> customer. This can result easily into many thousand folders though 
>> most of the time nothing actually changes.

>  The NOTIFY command handles that nicely, just one wildcard and 
> hardly any responses, but the client also has to send 1000 STATUS 
> commands immediately after starting NOTIFY, so for a two-hour IMAP 
> connection that's 1001 commands and 2006 responses in order to 
> notify the client of 4 events.

Why does the client have to send STATUS commands with NOTIFY?  From 
reading draft-gulbrandsen-imap-notify-02.txt, it looked to me that 
the client would get STATUS responses without ever sending STATUS 
commands.

>  For IDLEPLUS that approach works less well - it would switch to SMS 
> notifications for 2.3 seconds every time the client sends an IMAP 
> command. Very unpredictable for the user. It would also be possible 
> to give IDLEPLUS transaction/checkpoint ability, client identity 
> and a notification spool, so the client could say "IDLEPLUS 
> 1111111232" to identify itself, receive its spooled events and wait 
> for more, and "NOOPPLUS 423142342" to tell the server that all 
> events up to number 423142342 have been received.

Is this still needed even though the notifications include the MODSEQ 
(as we agreed to in Dallas, and is in the draft)?  With the MODSEQ, 
clients can determine if they've processed the event or not.

At 1:22 PM +0100 11/23/06, Arnt Gulbrandsen wrote:
>  If condstore's modseq is global, we could give notify a modseq 
> argument and it would send immediate STATUS responses for each 
> maillbox whose highestmodseq is larger... but that won't work if 
> condstore "next modseq" variable is per-mailbox, which I believe 
> 4551 allows.

My understanding is that each mailbox has its own set of MODSEQ 
values, but even so, the combination of mailbox name and MODSEQ 
should suffice (as long as UIDVALIDITY hasn't changed, I guess).

Is there a case where a client could think its mailbox cache 
synchronized with the server but really it isn't?  Perhaps if the 
client missed a notification but received a subsequent one, and 
stored the MODSEQ of the one it did get, it would never know that it 
missed the earlier one?  I thought this wouldn't happen, because when 
the client sees the unsolicited STATUS response saying there is a new 
message in mailbox FOO, the client (at some point) resynchronizes FOO 
as normal.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
As you know, any depiction of nasal mucus brings with it problems for
our sales division.
    --ABC's Broadcast Standards & Practices board, to producers of the
    cartoon "Bump in the Night"

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 11 Jan 2007 10:35:35 +0100

Randall Gellens writes:
>>  The NOTIFY command handles that nicely, just one wildcard and hardly 
>>  any responses, but the client also has to send 1000 STATUS commands 
>>  immediately after starting NOTIFY, so for a two-hour IMAP 

The STATUS commands are necessary (in this particular case at least) to 
be notified of changes that happened while the client was offline.

> clients can determine if they've processed the event or not.

Yes. But Martin suggest that clients should use a dedicated connection 
for IDLEPLUS (ie. connect, login, IDLEPLUS and just sit there), which 
permits a simpler and better solution to this problem (tag SP 
"IDLEPLUS" CRLF *("STILLIDLE" CRLF) "DONE" CRLF in ABNF).

> should suffice (as long as UIDVALIDITY hasn't changed, I guess).

Dave shot this possibility down quite thoroughly in a jabber 
conversation. There are tricky issues to do with transaction commit 
order. It basically won't work.

> message in mailbox FOO, the client (at some point) resynchronizes FOO 

Dave explained this very well a few weeks ago. I'm too tired now to find 
the right words. Dave?

Arnt

---

From: Lingyan Wu <[email-masked]>
Date: Mon, 22 Jan 2007 20:40:13 +0800

----- Original Message ----- 
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Cc: "Randall Gellens" <[email-masked]>
Sent: Thursday, January 11, 2007 5:35 PM
Subject: Re: [lemonade] draft-gulbrandsen-imap-notify-01 use cases

>>>> customer. This can result easily into many thousand folders though most 
>>>> of the time nothing actually changes.
>>>
>>> immediately after starting NOTIFY, so for a two-hour IMAP connection 
>>> that's 1001 commands and 2006 responses in order to notify the client of 
>>> 4 events.
>>
>> reading draft-gulbrandsen-imap-notify-02.txt, it looked to me that the 
>> client would get STATUS responses without ever sending STATUS commands.
>
> The STATUS commands are necessary (in this particular case at least) to be 
> notified of changes that happened while the client was offline.
>

Two small methods to save bandwidth as follows:

1, If STATUS commands (which are sent after starting NOTIFY) contain 
UIDVALIDITY and HIGHESTMODSEQ (catched in client) of related mailboxes , 
after receiving the STATUS commands the server can determine which mailboxes 
are out of sync and send STATUS(HIGHESTMODSEQ) responses of the mailboxes to 
client.
E.g. The client sends 1000 STATUS commands containing UIDVALIDITY and 
HIGHESTMODSEQ immediately after starting NOTIFY, the server determines and 
sends only a few STATUS(HIGHESTMODSEQ) to client for those mailboxes out of 
sync.

2, After receiving Notify command, the server sends unsolicited  STATUS 
responses for all of the mailboxes to the client.

Maybe absurd! Any guidance?

>>>  immediately after starting NOTIFY

>>> command. Very unpredictable for the user. It would also be possible to 
>>> give IDLEPLUS transaction/checkpoint ability, client identity and a 
>>> notification spool, so the client could say "IDLEPLUS 1111111232" to 
>>> identify itself, receive its spooled events and wait for more, and 
>>> "NOOPPLUS 423142342" to tell the server that all events up to number 
>>> 423142342 have been received.
>>
>> Is this still needed even though the notifications include the MODSEQ (as 
>> we agreed to in Dallas, and is in the draft)?  With the MODSEQ, clients 
>> can determine if they've processed the event or not.
>
> Yes. But Martin suggest that clients should use a dedicated connection for 
> IDLEPLUS (ie. connect, login, IDLEPLUS and just sit there), which permits 
> a simpler and better solution to this problem (tag SP "IDLEPLUS" CRLF 
> *("STILLIDLE" CRLF) "DONE" CRLF in ABNF).
>
>>>  If condstore's modseq is global, we could give notify a modseq argument 
>>> and it would send immediate STATUS responses for each maillbox whose 
>>> highestmodseq is larger... but that won't work if condstore "next 
>>> modseq" variable is per-mailbox, which I believe 4551 allows.
>>
>> My understanding is that each mailbox has its own set of MODSEQ values, 
>> but even so, the combination of mailbox name and MODSEQ should suffice 
>> (as long as UIDVALIDITY hasn't changed, I guess).
>
> Dave shot this possibility down quite thoroughly in a jabber conversation. 
> There are tricky issues to do with transaction commit order. It basically 
> won't work.
>
>> Is there a case where a client could think its mailbox cache synchronized 
>> with the server but really it isn't?  Perhaps if the client missed a 
>> notification but received a subsequent one, and stored the MODSEQ of the 
>> one it did get, it would never know that it missed the earlier one?  I 
>> thought this wouldn't happen, because when the client sees the 
>> unsolicited STATUS response saying there is a new message in mailbox FOO, 
>> the client (at some point) resynchronizes FOO as normal.
>