Subject: [lemonade] Comments on draft-ietf-lemonade-reconnect-client-06

From: Eric Rescorla <[email-masked]>
Date: Thu, 04 Oct 2007 15:28:11 -0700

$Id: draft-ietf-lemonade-reconnect-client-06-rev.txt,v 1.1 2007/10/04 22:25:53 ekr Exp $

OVERALL
This document describes an extension to IMAP to provide faster
synchronization between client and server. As far as I can
tell, the optimizations are:

- Removing one round trip needed to discover which messages
  have been expunged.
- A more compact representation of the list of expunged
  messages

I have some skepticism about the importance of these optimizations.
The document does not come with any performance measurements,
and 1 RTT really isn't that much. In particular, I wonder if
VANISHED really saves that much bandwidth over EXPUNGED if
compression is in use. I don't know what standard is being
used to decide whether optimizations of this type are worthwhile.

I found this document fairly hard to read. I understand that it's
a delta to IMAP and requires quite a bit of knowledge of IMAP
to understand, but I think it could have been written to be
more clear about how it changes IMAP's behavior in each case.
In particular, the examples would be improved by always
having New and Old and having some sort of indicator of exactly
which PDUs have changed and why.

DETAILED COMMENTS
S 2.
This would be improved by some overall diagram of the new and
old behavior and some measurement, even an ad hoc one, of
the performance improvement.

S 3.1.

   Conceptually, the client provides a small sample of sequence numbers
   for which it knows the corresponding UIDs.  The server then compares
   each sequence number and UID pair the client provides with the
   current state of the mailbox.  If a pair matches, then the client
   knows of any expunges up to, and including, the message, and thus
   will not include that range in the VANISHED response, even if the
   "mod-sequence-value" provided by the client is too old for the server
   to have data of when those messages were expunged.

This is probably my ignorance of IMAP, but how can this happen? Why
doesn't the client have a mod-sequence-value corresponds to these
UIDs?

      S: * VANISHED (EARLIER) 1:2,4:5,7:8,10:11,13:14 [...]
         29998:29999,30001:30002,30004:30005,30007:30008

This [...] hides the data you're optimizing away, right? This
would help if it were called out more clearly.

S 3.3, 3.4, 3.5.
These would all benefit from a statement of how they differ from
3501, rather than just stating new rules.

   If the server is capable of storing modification sequences for the
   selected mailbox, it MUST increment the per-mailbox mod-sequence if
   at least one message was permanently removed due to the execution of
   the EXPUNGE command.  For each permanently removed message the server
   MUST remember the incremented mod-sequence and corresponding UID.  If
   at least one message got expunged, the server MUST send the updated
   per-mailbox modification sequence using the HIGHESTMODSEQ response
   code (defined in [CONDSTORE]) in the tagged OK response.

So, this is repeated in all three sections. That seems less than
optimal.

Rather than refing 3501, it would probably be good to point out
why the message #s are as they are in these examples, due to
auto-decrement.

S 3.6.
   The VANISHED response has two forms.  The first form contains the
   EARLIER tag, which signifies that the response was caused by a UID
   FETCH (VANISHED) or a SELECT/EXAMINE (QRESYNC) command.  This
   response is sent if the UID set parameter to the UID FETCH (VANISHED)
   command includes UIDs of messages that are no longer in the mailbox.
   When the client sees a VANISHED EARLIER response it MUST NOT
   decrement message sequence numbers for each successive message in the
   mailbox.

   The second form doesn't contain the EARLIER tag and is described
   below.  Once a client has used "(VANISHED)" with a UID FETCH or
   "(QRESYNC)" with SELECT/EXAMINE command, the server SHOULD use the
   VANISHED response without the EARLIER tag instead of the EXPUNGE
   response.  The server SHOULD continue using VANISHED in lieu of
   EXPUNGE for the duration of the connection.  In particular this
   affects the EXPUNGE [RFC3501] and UID EXPUNGE [UIDPLUS] commands, as
   well as messages expunged in other connections.  Such VANISHED
   response MUST NOT contain the EARLIER tag.

This is pretty unclear to the non-IMAP expert. Could you explain
in english what this is trying to accomplish in the document,
not just specify the protocol mechanics.

In the example, swap before and after. Also, it would be good
to show an example of (EARLIER).

S 4.1.
   Strictly speaking, a server implementation that doesn't remember
   modsequences associated with expunged messages can be considered
   compliant with this specification.  Such implementations return all
   expunged messages specified in the UID set of the UID FETCH
   (VANISHED) command every time, without paying attention to the
   specified CHANGEDSINCE modsequence.  Such implementations are
   discouraged, as they can end up returning VANISHED responses bigger
   than the result of a UID SEARCH command for the same UID set.

Isn't this inconsistent with:

   If the server is capable of storing modification sequences for the
   selected mailbox, it MUST increment the per-mailbox mod-sequence if
   at least one message was permanently removed due to the execution of
   the EXPUNGE command.  For each permanently removed message the server
   MUST remember the incremented mod-sequence and corresponding UID.  If
   at least one message got expunged, the server MUST send the updated
   per-mailbox modification sequence using the HIGHESTMODSEQ response
   code (defined in [CONDSTORE]) in the tagged OK response.

If not, why not?

S 5.
   The client MUST also take note of any MODSEQ FETCH data items
   received from the server.  Whenever the client receives a tagged
   response to a command, it calculates the highest value among all
   MODSEQ FETCH data items received since the last tagged response.  If
   this value is bigger than the client's copy of the HIGHESTMODSEQ
   value, then the client MUST use this value as its new HIGHESTMODSEQ
   value.

So, I probably misunderstand something, but my read of 4551 made
it seem like you could do a MODSEQ FETCH that would not return
all the metadata for every message. In that case, wouldn't this
procedure risk you having a modseq that is higher than some
messages you haven't examined yet? What am I missing.

I don't see any new security issues in this document.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 05 Oct 2007 12:18:20 +0100

Hi Eric,
Thank you for your comments.

(Today is about the worst time for me to reply to your comments, as I am 
going on holidays tomorrow.)

>- Removing one round trip needed to discover which messages
Yes, if compared to the case when the client/server also implement the 
CONDSTORE extension (RFC 4551).
(Actually it removed 2 round trips per mailbox synchronization.)

When compared to RFC 3501, this extension can potentially provides huge 
bandwidth saving. If a client wants to synchronize flag changes in a 
mailbox, the client needs to fetch flags for *all* mailboxes. For a 
30,000 message mailbox that I currently have is quite painful over a 
slow link.

>- A more compact representation of the list of expunged
Correct.

>I have some skepticism about the importance of these optimizations.
See above.

>used to decide whether optimizations of this type are worthwhile.
I will let Dave comment on this.

Just a couple of extra comments:
1). As I mentioned above, this extension saves 2 round trips per mailbox 
synchronization. A user might have multiple mailboxes.
2). This is mostly useful for mobile clients which can experience 
frequent connection loss.

>which PDUs have changed and why.
Right.

>the performance improvement.
Ok. An older version of this document had some numbers, but people 
complained about "irrelevant text".
See section 2.1 of 
<http://tools.ietf.org/id/draft-ietf-lemonade-reconnect-07.txt>
(Note that the protocol has changed substantially since, but the basic 
observation is still correct.)

>doesn't the client have a mod-sequence-value corresponds to these
I will let Dave explain this.

>      S: * VANISHED (EARLIER) 1:2,4:5,7:8,10:11,13:14 [...]
Yes. I will add a sentence on this.

>3501, rather than just stating new rules.
(Actually, 3.5 updates UID EXPUNGE which was defined in RFC 4315.)

Right. I believe some people wanted to see sections replacing the old 
definitions, as opposed to just pointing out the difference from RFC 3501.

The following text in section 2 summarizes the difference:

   This document puts additional requirements on a server implementing
   the [CONDSTORE] extension.  Each mailbox that supports persistent
   storage of mod-sequences, i.e., for which the server has sent a
   HIGHESTMODSEQ untagged OK response code on a successful SELECT/
   EXAMINE, MUST increment the per-mailbox mod-sequence when one or more
   messages are expunged due to EXPUNGE, UID EXPUNGE or CLOSE; the
   server MUST associate the incremented mod-sequence with the UIDs of
   the expunged messages.

I can try to add a sentence to each one of 3.3, 3.4 and 3.5 to clarify that.

>why the message #s are as they are in these examples, due to
I've added a clarifying sentence to sections 3.3 and 3.5. I don't think 
section 3.4 needs to change.

>not just specify the protocol mechanics.
Basically the VANISHED response is used for 2 purposes: to report UIDs 
of messages expunged earlier and to report UIDs of messages expunged now.
The difference between the two is that in the former case the client 
doesn't need to decrement the number of messages in the mailbox, while 
in the latter case it must.
The former can be distinguished from the latter by presence of the 
"(EARLIER)" label.

>to show an example of (EARLIER).
Ok.

>   code (defined in [CONDSTORE]) in the tagged OK response.
No, there is no inconsistency:

"a server implementation that doesn't remember modsequences" == "a 
server is incapable of storing modsequences".
The second paragraph you quoted is conditional on server's ability to 
store modsequences. This behaviour is optional for servers.

>all the metadata for every message.
>
Yes, if the client issues a FETCH MODSEQ for a subset of messages. 
However the previous paragraph in the same section requires the client 
to perform a full synchronization.

>In that case, wouldn't this
The intent of this paragraph is to talk about unsolicited FETCH MODSEQ 
returned by the server *after* a full synchronization is complete (the 
previous paragraph in the same section). So the situation you describe 
can't happen, because the server is required to send all unsolicited 
FETCH MODSEQ responses.
The beginning of this paragraph is ambiguous, so I suggest that it 
should be clarified by changing the first quoted sentence to read:

After completing full synchronization, the client MUST also take note of 
any MODSEQ FETCH data items received from the server.

>I don't see any new security issues in this document.
Regards,
Alexey

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 05 Oct 2007 07:53:21 -0700

At Fri, 05 Oct 2007 12:18:20 +0100,
> 30,000 message mailbox that I currently have is quite painful over a 

I don't think it makes sense to compare this to 3501 without 4551,
since 4551 is already an RFC. The question is whether this document
should be advanced. What's the optimization compared
to 4551? Also, you say it's "quite painful" now. How much less
painful is it with this document. How about if compression is 
used. This seems like something
where measurements would be nice. 

> complained about "irrelevant text".

Maybe I'm missing something, but those comparisons are to 3501,
not CONDSTORE, right?

> definitions, as opposed to just pointing out the difference from RFC 3501.

As a wordsmithing issue, I would recommend adding a single clarificatory
master section with these replacement definitions as subsections.

> The former can be distinguished from the latter by presence of the 

OK. That could be clearer in the main text.

> store modsequences. This behaviour is optional for servers.

OK. I misread this. Thanks.

> to perform a full synchronization.

OK.

> any MODSEQ FETCH data items received from the server.

WFM.

-Ekr

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Oct 2007 08:10:23 -0700 (PDT)

> where measurements would be nice.

+1.

Assuming that a typical FLAGS response is about 40 octets, then fetching 
all flags is 1,200,000 octets.  This may be a bit slow over a 2.5G 
wireless network, but "quite painful" overstates the case somewhat.

Also, this is only an issue for "disconnected" clients that insist upon 
synchronizing a local copy with the server.  True "online" clients don't 
have this issue, and don't care about flags in messages that the user 
isn't viewing.

Being an online client kind of guy, I see the problem as a flaw in the 
disconnected client model, especially as the whole idea of reconnect seems 
to be to make disconnected have similar real-time characteristics as 
online.  I sympathize, but only to a limited degree.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 05 Oct 2007 18:13:20 +0100

Eric Rescorla wrote:

>to 4551?
CONDSTORE can significantly reduce bandwidth in some cases. It doesn't 
reduce the number of roundtrips.
QRESYNC reduces number of roundtrips when compared to CONDSTORE or base 
IMAP. It can also reduce bandwidth in some cases (on top of the 
bandwidth reduction given by CONDSTORE).

>Also, you say it's "quite painful" now. How much less
>used. This seems like something where measurements would be nice.
>
Yes. Dave will answer this ;-).

>Maybe I'm missing something, but those comparisons are to 3501,
Yes.