Subject: [lemonade] Comments on draft-ietf-lemonade-reconnect-client-04.txt

From: Mike Zraly <[email-masked]>
Date: Thu, 03 May 2007 10:46:17 -0400

When processing a SELECT ... (QRESYNC ...) command, what is
the correct behavior of the IMAP server if the server DOES
support persistent storage of mod-sequence values, but only
retains a limited number of records of EXPUNGEd messages?

For example, let's say the server retains at most some maximum
number of expunged records, each containing a mod-sequence
value and an associated uid value.  Let's also say that for
each mailbox the server retains the highest mod-sequence number
of all the expunged messages for which the server does not
retain an expunged record.  Call this value MAXUNAVAILABLE.

If a client issues a SELECT/QRESYNC command with a cached
mod-seq value less than or equal to MAXUNAVAILABLE, then
the server will know that it cannot provide a list of all
the EXPUNGEd messages.  (It would be able to provide a list
of changed messages however, by scanning through all the
messages in a mailbox and selecting messages with a large
enough MODSEQ field.)

However, in this case the client would be incorrect in assuming
that the server does not maintain MODSEQ values for each
message or a HIGHESTMODSEQ value for each mailbox.  In
particular, the client should still be able to issue
conditional STORE commands using the UNCHANGEDSINCE modifier.
So returning a NOMODSEQ response would be the wrong thing to do
here (see section 3.1.2 in RFC 4551).

I suppose in this case the server could increment the
mailbox's UIDVALIDITY to force the client to resynchronize
some other way without implying that MODSEQ values are
not being maintained for messages in the mailbox, but
that would force OTHER clients to resynchronize too.

It seems to me that there should be an alternate response
for this case, perhaps called MISSINGEXPUNGED or something
like that, to indicate that there may be messages in the
client's cache whose EXPUNGE operations are no longer recorded
by the server, but that the mailbox DOES support MODSEQ in
general and conditional STORE operations in particular.

It is not clear to me if, given a MISSINGEXPUNGED response,
the server should still return untagged FETCH responses for
new and modified messages and an untagged VANISHED response
for the EXPUNGEd messages it knows about.

- Mike Zraly

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: "Ben Last" <[email-masked]>
Date: Fri, 4 May 2007 08:08:42 +0100

>From: Mike Zraly [mailto:[email-masked]] 
>It seems to me that there should be an alternate response for this case, perhaps called MISSINGEXPUNGED or something like that...
But what could a client do when it receives such a response (other than a state-comparison resync)?  Isn't this pretty much equivalent (in terms of what it means to the client) to the untagged RESYNC login response that was in the P-IMAP spec?  That is, a response that means "hello client, the only sensible way for you to remain in sync is to do a state-comparison resync".

I guess I'm arguing for explicit being better than implicit: if the server knows that it has insufficient stored history to replay modification events to the client, perhaps an explicit "let's start over" is better.

Regards
Ben

Ben Last
R&D Manager
EMCC Software

---

From: Dan Karp <[email-masked]>
Date: Fri, 4 May 2007 00:39:15 -0700 (PDT)

> >It seems to me that there should be an alternate response for this
> > case, perhaps called MISSINGEXPUNGED or something like that...
> 
> But what could a client do when it receives such a response (other
> than a state-comparison resync)?  Isn't this pretty much equivalent
> (in terms of what it means to the client) to the untagged RESYNC login
> response that was in the P-IMAP spec?  That is, a response that means
> "hello client, the only sensible way for you to remain in sync is to
> do a state-comparison resync".

If the server doesn't have enough persisted state to answer "what's VANISHED since last sync", it can just downgrade from the functionality described in section 4.3 to the functionality described in section 4.1.  The situations requiring such a downgrade should be rare, but the server will still be fully compliant:

   Strictly speaking, a server implementation that doesn't remember
   modsequences associated with expunged messages can be considered
   compliant with this specification.  Such implementations return all
   expunged messages specified in the UID set of the UID FETCH
   (VANISHED) command every time, without paying attention to the
   specified CHANGEDSINCE modsequence.  Such implementations are
   discouraged, as they can end up returning VANISHED responses bigger
   than the result of a UID SEARCH command for the same UID set.

There shouldn't be a need for a resync.

---

From: "Ben Last" <[email-masked]>
Date: Fri, 4 May 2007 08:48:27 +0100

>From: Dan Karp [mailto:[email-masked]] 

>If the server doesn't have enough persisted state to answer "what's VANISHED since last sync", it can just downgrade from the
>functionality described in section 4.3 to the functionality described in section 4.1.
Thanks, Dan.  I guess that's a partial-state-comparison sync by default, in that I assume a server without expunge history will report as VANISHED any UIDs for which there are no messages in the UID set.  However, from a client point of view, it's "remove all UIDs that aren't confirmed in this list" rather than "remove all messages that are reported as EXPUNGEd in this list", which has some implications for design (ie, one removes on successful completion of a command rather than as responses are processed).

Regards
Ben

Ben Last
R&D Manager
EMCC Software

---

From: Mike Zraly <[email-masked]>
Date: Fri, 04 May 2007 09:53:26 -0400

Dan Karp wrote:
> If the server doesn't have enough persisted state to answer
> "what's VANISHED since last sync", it can just downgrade from
> the functionality described in section 4.3 to the functionality
> described in section 4.1.  The situations requiring such a
> downgrade should be rare, but the server will still be fully
> compliant:
> 

True.

What I am trying to work my head around is how the server should
handle the case where the expunged message queue described in
section 4.3 has been truncated, i.e. there are expired expunged
messages.

In this case, I think there is value in having the server return
BOTH the set of known changed messages (via FETCH and VANISHED
responses) AND an indication that the VANISHED response is known
to be incomplete.

Given this information, plus the number of messages in the mailbox
on the server as reported by the EXISTS response, a client could
decide the most efficient way to resynchronize the messages not
affected by the reported changes.

If the number of messages reported by the EXISTS response is less
than the number of messages in the client's cache not affected by
the FETCH and VANISHED responses, then it would make sense for the
client to ignore the changes, discard its cache, and repopulate
it directly from the server (via FETCH 1:* ...).

OTOH if the number of messages reported by the EXISTS response is
greater than the number of messages in the client's cache not affected
by the FETCH and VANISHED responses, then it would make sense for the
client to apply the changes, then ask the server only about the messages
in its cache unaffected by the FETCH and VANISHED responses.

To save the client for applying changes to its cache that it will
immediately discard, the server could send untagged VANISHEDINCOMPLETE
(what I called MISSINGEXPUNGED in my first email) and TOTALCHANGES
responses before sending the FETCH and VANISHED responses.

The VANISHEDINCOMPLETE response would tell the client that there is at
least one message in its cache that was expunged but that the server
no longer has a record of it.

The TOTALCHANGES response would tell the client how many messages in
its cache will be affected by the forthcoming FETCH and VANISHED
responses.

When a client receives both VANISHEDINCOMPLETE and TOTALCHANGES
responses, it can subtract the number of changes from the size
of its cache to determine the number of messages in the cache
not affected by the FETCH and VANISHED responses.  The client
can then compare this number with the value returned by the EXISTS
response to decide how to detect expunges of messages in that set,
as described above.

The interaction might look like this:

C: A02 SELECT INBOX (QRESYNC (67890007 90060115194045000))
S: * 314 EXISTS
S: * 15 RECENT
S: * OK [UIDVALIDITY 67890007] UIDVALIDITY
S: * OK [UIDNEXT 567] Predicted next UID
S: * OK [HIGHESTMODSEQ 90060115205545359]
S: * OK [UNSEEN 7] There are some unseen messages in the mailbox
S: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
S: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft
         \Deleted \Seen \*)] Permanent flags
S: * OK [VANISHEDINCOMPLETE] some expunged message data has expired
S: * OK [TOTALCHANGES 547] changed messages plus 
S: * 49 FETCH (UID 117 FLAGS (\Seen \Answered))
S: * 50 FETCH (UID 119 FLAGS (\Draft $MDNSent))
S: ...
S: * 100 FETCH (UID 541 FLAGS (\Seen $Forwarded))
S: * VANISHED 41,43:116,118,120:211,214:540
S: A02 OK [READ-WRITE] mailbox selected

Note that it would be best for the client if the TOTALCHANGES
response were to precede the FETCH and VANISHED responses.
However, some servers may not want to collect all the changes
and count them before emitting the FETCH  and VANISHED responses,
so it probably makes more sense to say that servers SHOULD emit
TOTALCHANGES before the FETCH and VANISHED responses rather
than that they MUST do so.  Of course, if the TOTALCHANGES
response is only emitted after the FETCH and VANISHED responses,
then it doesn't really add any extra information.

If the TOTALCHANGES response is emitted prior to the FETCH and
VANISHED responses, then I would say VANISHEDINCOMPLETE MUST also
be emitted prior to the FETCH and VANISHED responses.  If the
TOTALCHANGES response is not emitted until after the FETCH and
VANISHED responses, then I would say VANISHEDINCOMPLETE MAY be
emitted after the FETCH and VANISHED responses.

Reasonable?

- Mike

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 05 May 2007 14:58:17 +0100

>> There shouldn't be a need for a resync.

Hi Mike,

> responses) 

Correct so far.

> AND an indication that the VANISHED response is known
> to be incomplete.

This part is wrong. If the server knows that it doesn't have information 
about some expunged messages, the server should always return them as 
expunged (after checking that they are in the UID list specified by the 
client). After all, there is no harm in reporting a message expunged 
long time ago as a "recently expunged message". This might cause some 
extra traffic, but the client must be able to handle it anyway.

[...]

---

From: Mike Zraly <[email-masked]>
Date: Mon, 07 May 2007 09:16:21 -0400

> extra traffic, but the client must be able to handle it anyway.

Agreed that if the client *does* specify the UID list then it is
easy and correct for a server to declare that any UID in the list
which the server does not know about is expunged.

But what if the client doesn't specify the UID list?  The set of
known UIDs is an optional parameter, per section 3.1.

Hmm, later on in section 3.1 the draft says that the server should
treat an unspecified known UID list as "1:*".  I suppose this could
work, depending on how "*" is interpreted.  If "*" is interpreted
the same ways as it is for the UID command (RFC 3501 section 6.4.8),
i.e. to mean the highest numbered UID in the mailbox, then this will
not work in the case where there are expunged messages with higher
IMAP UIDs (and there are enough of them that the server does not
have a record of each one).

If on the other hand the "*" in this context is interpreted as
meaning the largest UID value that was ever assigned to a message
in the mailbox, then this would work.  Generally I suppose this
value would be one less than a mailbox's UIDNEXT value.

- Mike

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 07 May 2007 21:35:50 +0100

> value would be one less than a mailbox's UIDNEXT value.

I will clarify that "*" references the latter in this case.

Thanks,
Alexey