Subject: Re: [lemonade] I-D ACTION:draft-ietf-lemonade-reconnect-client-02.txt

From: Randall Gellens <[email-masked]>
Date: Thu, 11 Jan 2007 18:12:47 -0800

In Section 5, it says:

    The client MUST
    cache the value from HIGHESTMODSEQ OK response code received on
    mailbox opening and update it whenever the server sends MODSEQ FETCH
    data items or HIGHESTMODSEQ response code.

I think the client shouldn't immediately cache the value received on 
mailbox open, but instead use that to determine if it needs to 
resynchronize.  Only after resynchronizing can it store the 
HIGHESTMODSEQ.  Otherwise it might miss changes.  It may be better to 
say that the client "sets aside" the value returned on mailbox open, 
and after resynchronizing, updates its cache with this value.  That 
way, if the connection is lost the client won't lose changes.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Democracy is a beautiful thing, except for that part about letting
just any old yokel vote. --Anonymous, Age 10

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 29 Jan 2007 14:05:32 +0000

> way, if the connection is lost the client won't lose changes.

Randy, you are absolutely right. What you describe was the intent. I've 
reworded the quoted text to read as follows:

        The client would use the value from the HIGHESTMODSEQ OK 
response code received
        on mailbox opening to determine if it needs to resynchronize. 
Once the synchronization is
        complete it MUST cache the received value (unless the mailbox 
UIDVALIDITY value has changed; See below).
        The client MUST update its copy of the HIGHESTMODSEQ value 
whenever the server
        sends MODSEQ FETCH data items or a subsequent HIGHESTMODSEQ OK 
response code.

However Dave Cridland pointed out another problem with this text (which 
is also present in the CONDSTORE document). It is easier to describe the 
problem by example:

 C: A142 SELECT INBOX (QRESYNC (3857529045 8 1:198))
 S: * 172 EXISTS
 S: * 1 RECENT
 S: * OK [UNSEEN 12] Message 12 is first unseen
 S: * OK [UIDVALIDITY 3857529045] UIDs valid
 S: * OK [UIDNEXT 201] Predicted next UID
 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
 S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
 S: * OK [HIGHESTMODSEQ 20010715194045007]
 S: * 2 FETCH (UID 6 MODSEQ (9) FLAGS (\Deleted))
 S: * 5 FETCH (UID 9 MODSEQ (10) FLAGS ($NoJunk $AutoJunk $MDNSent))
 S: * 6 FETCH (UID 11 MODSEQ (10) FLAGS ($NoJunk $AutoJunk))
 S: * 7 FETCH (UID 12 MODSEQ (10) FLAGS ($NoJunk $AutoJunk))
    ...
 S: * VANISHED 1:5,7:8,10,13:15
 S: A142 OK [READ-WRITE] SELECT completed

Now imagine that the client loses connectivity after receiving the line 
starting with "* 6 FETCH (UID 11". Based on the instructions mentioned 
above, the client would already update its cached HIGHESTMODSEQ value to 
be 10, however it hasn't received the change for the message with UID 
12. If upon reconnect the client issues "SELECT INBOX (QRESYNC 
(3857529045 12 1:198))" it will miss the changed for the message with 
UID 12.

So, a correctly written client has to do one of the two things:
1). It can update its cached HIGHESTMODSEQ value with FETCH MODSEQ 
values, but if it detects a disconnect, it must use <HIGHESTMODSEQ>-1 in 
the subsequent SELECT (QRESYNC) command.

2). The client doesn't updates its cached HIGHESTMODSEQ value with a 
FETCH MODSEQ value until
a). it receives another FETCH MODSEQ value bigger than the current one.
b). it receives the tagged response for the SELECT/EXAMINE command.
**

---

From: Dave Cridland <[email-masked]>
Date: Mon, 29 Jan 2007 15:04:24 +0000

On Mon Jan 29 14:05:32 2007, Alexey Melnikov wrote:
> values, but if it detects a disconnect, it must use 
> <HIGHESTMODSEQ>-1 in the subsequent SELECT (QRESYNC) command.
> 
> 
Nope. This doesn't work if the client crashes, only if the connection 
severs or the server dies.

> 2). The client doesn't updates its cached HIGHESTMODSEQ value with 
> a FETCH MODSEQ value until

Yes, this one works.

I believe the best way of solving both would be to revise CONDSTORE 
to add an untagged response indicating that all updates up to and 
including a certain MODSEQ have been sent. This is basically the 
strategy that ACAP uses with CONTEXTs, and it works fine.

Because CONDSTORE already exists, and has some deployment (on servers 
and clients), we'll have to use an untagged OK with a response code 
to manage it. In principle, we could just re-use HIGHESTMODSEQ for 
this. This would mean we would have to move where that response code 
occurs in the SELECT return for QRESYNC, of course.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 29 Jan 2007 17:42:30 +0100

> severs or the server dies.

Using x-1 for all observed (rather than declared) HIGHESTMODSEQ values 
works, is simple, and costs approximately nothing. Approximately one 
extra flag fetch per connection teardown, no extra roundtrips.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Mon, 29 Jan 2007 16:55:11 +0000

On Mon Jan 29 16:42:30 2007, Arnt Gulbrandsen wrote:
>>> 1). It can update its cached HIGHESTMODSEQ value with FETCH 
>>> MODSEQ values, but if it detects a disconnect, it must use 
>> Nope. This doesn't work if the client crashes, only if the 
>> connection severs or the server dies.
> 
> Using x-1 for all observed (rather than declared) HIGHESTMODSEQ 
> values works, is simple, and costs approximately nothing. 
> Approximately one extra flag fetch per connection teardown, no 
> extra roundtrips.

Yes, it works, and is a reasonable thing to do.

My point against this was the "if it detects a disconnect".

Dave.

---

From: Philip Guenther <[email-masked]>
Date: Mon, 29 Jan 2007 11:25:58 -0700

On Mon, 29 Jan 2007, Alexey Melnikov wrote:
...
> 1). It can update its cached HIGHESTMODSEQ value with FETCH MODSEQ values, 
> but if it detects a disconnect, it must use <HIGHESTMODSEQ>-1 in the 
> subsequent SELECT (QRESYNC) command.
>
> 2). The client doesn't updates its cached HIGHESTMODSEQ value with a FETCH 
> MODSEQ value until

I can't seem to find any requirement for a server to sort the FETCH 
responses by their MODSEQ before returning them, so I don't see how (1) or 
(2a) can be reliable.  Do we expect connections to drop during resync 
often enough to be worth handling it specially?  (Note that a connection 
that drops before the first FETCH response to the SELECT makes no 
progress, period, while a connection that makes it to the tagged response 
always does.)

Philip Guenther

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 29 Jan 2007 23:30:23 +0100

Philip Guenther writes:
> responses by their MODSEQ before returning them, so I don't see how 
> (1) or (2a) can be reliable.

If you respond to EXISTS by issuing 'FETCH ... MODSEQ' for the new 
messages, the MODSEQs you see will be sorted.

Arnt

---

From: Philip Guenther <[email-masked]>
Date: Mon, 29 Jan 2007 15:46:09 -0700

On Mon, 29 Jan 2007, Arnt Gulbrandsen wrote:

>> (2a) can be reliable.
> If you respond to EXISTS by issuing 'FETCH ... MODSEQ' for the new messages, 
> the MODSEQs you see will be sorted.

So, let's say I have a mailbox selected and the following occurs:
1) two messages are injected
2) some other session sets the $MDNSent flag on the first of the two

I then send a NOOP and get an EXISTS showing that there are two new 
messages.  You're saying that if I now do "tag FETCH N:N+1 (FLAGS MODSEQ)" 
then the server MUST return the FETCH response for message N+1 before 
message N.  Can you please point to the MUST in RFC 4551 that says that?

Philip Guenther

---

From: Lingyan Wu <[email-masked]>
Date: Tue, 30 Jan 2007 10:47:57 +0800

Yes. I can give you another example if I don't misunderstand you.

Example 12 in RFC4551:
      C: s100 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 12345)

      S: * 1 FETCH (UID 4 MODSEQ (65402) FLAGS (\Seen))

      S: * 2 FETCH (UID 6 MODSEQ (75403) FLAGS (\Deleted))

      S: * 4 FETCH (UID 8 MODSEQ (29738) FLAGS ($NoJunk $AutoJunk

           $MDNSent))

      S: s100 OK FETCH completed

The Fetch Responses are not sorted according to MODSEQ but according to Sequence Number.

I divide the synchronization flow into two stages just as described in the following figure.

  The figure is just an example. The first box of the second stage can be replaced by "a new email arrives in the mailbox."

  Description: 
  The first stage: When do synchronization after determining that the HIGHESTMODSEQ of the mailbox in client catch is less than the value in server, FETCH Responses are usually not sorted according to MODSEQ just as the example 12 in RFC4551, so the client should not update its cached HIGHESMODSEQ value with FETCH MODSEQ values but update it according to the value from HIGHESTMODSEQ OK response code only after synchronizing.
  The second stage: While connection, after synchronizing, when some events happen in server, the server can send unsolicited FETCH responses including MODSEQ to client which are sorted according to MODSEQ (according to the time the event happens), so the client can update its cached HIGHESMODSEQ value with FETCH MODSEQ values.
BTW. I'm very sorry that in this email I didn't use format=flowed in order to insert the figure in the text directly.

Best Regards,

Lingyan

  ----- Original Message ----- 
  From: Philip Guenther 
  To: Arnt Gulbrandsen 
  Cc: [email-masked] ; Alexey Melnikov 
  Sent: Tuesday, January 30, 2007 6:46 AM
  Subject: Re: [lemonade] I-D ACTION:draft-ietf-lemonade-reconnect-client-02.txt

  Philip Guenther

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 30 Jan 2007 10:57:36 +0100

Philip Guenther writes:
> messages.  You're saying that if I now do "tag FETCH N:N+1 (FLAGS 
> MODSEQ)" then the server MUST return the FETCH response for message 
> N+1 before message N.  Can you please point to the MUST in RFC 4551 
> that says that?

No, you're right.

Arnt

---

From: "Ben Last" <[email-masked]>
Date: Wed, 31 Jan 2007 09:18:27 -0000

Sorry for the slightly delayed reply (written on a train).  Mobile clients are not the easiest things on which to review drafts :) 

>From: Alexey Melnikov [mailto:[email-masked]] 
>However Dave Cridland pointed out another problem with this text (which is also
>present in the CONDSTORE document). It is easier to describe the problem by example:
>... excellent illustration of the problem snipped ...
>1). It can update its cached HIGHESTMODSEQ value with FETCH MODSEQ values, but if it detects a
>disconnect, it must use <HIGHESTMODSEQ>-1 in the subsequent SELECT (QRESYNC) command.
>2). The client doesn't updates its cached HIGHESTMODSEQ value with a FETCH MODSEQ value
>until a). it receives another FETCH MODSEQ value bigger than the current one.

>strategy that ACAP uses with CONTEXTs, and it works fine.

But surely the client can make that assumption when it sees the tagged response?

Ah - I see that Philip Guenther has identified that there is no requirement for MODSEQ responses to be sorted/grouped by the server, so that argues even more strongly for option (2) above.

Regards
Ben

R&D Manager
EMCC Software Ltd

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 31 Jan 2007 13:49:14 +0000

Philip Guenther wrote:

>> values, but if it detects a disconnect, it must use <HIGHESTMODSEQ>-1 
>> in the subsequent SELECT (QRESYNC) command.
>>
> responses by their MODSEQ before returning them,

There is no such requirement.

> so I don't see how (1)

> or (2a) can be reliable.

Ah, you are right. The server can send "* <n1> FETCH (MODSEQ 5 ...)" 
first, followed by "* <n2> FETCH (MODSEQ 3 ...)". If the client only 
receives the first response and detects disconnect, it can't 
resynchronize starting from modseq 4 upon reconnect, or it will miss 
changes to the message <n2>.

So basically the client can't update its copy of HIGHESTMODSEQ with 
FETCH MODSEQ until after it sees the tagged response for the command.

> Do we expect connections to drop during resync often enough to be 
> worth handling it specially?

Maybe we don't need to extend CONDSTORE to handle this specially, but I 
think it is our duty to properly document how clients can cope.

> (Note that a connection that drops before the first FETCH response to 
> the SELECT makes no progress, period, while a connection that makes it 
> to the tagged response always does.)