Subject: Re: [lemonade] URLFETCH Partial Fetches

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 2 Dec 2004 09:00:31 +0100

Mark Crispin writes:
> Unless more people speak up, I'm afraid that this idea will end up in 
> the trash can of "good ideas that can be done, but are not good 
> enough to bother doing."

IMAP surely has enough functionality that clients don't use already.

Arnt

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Dec 2004 08:51:48 -0800 (PST)

On Thu, 2 Dec 2004, Arnt Gulbrandsen wrote:
>> Unless more people speak up, I'm afraid that this idea will end up in the 
>> trash can of "good ideas that can be done, but are not good enough to 
>> bother doing."

Such as?  Pine uses almost everything that IMAP provides.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 2 Dec 2004 18:10:13 +0100

Mark Crispin writes:
>>> Unless more people speak up, I'm afraid that this idea will end up 
>>> in the trash can of "good ideas that can be done, but are not good 
>>> enough to bother doing."

I know one other client that does that, too. Two's not a crowd.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Dec 2004 09:32:52 -0800 (PST)

> I know one other client that does that, too. Two's not a crowd.

I know two other clients that do that.  We're now up to four.

Four is a crowd if you exclude POP clients that babble IMAP protocol.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 2 Dec 2004 19:28:00 +0100

> I know two other clients that do that.  We're now up to four.

You know four clients that use a lot of base IMAP and a smattering of 
extensions? Wow. Which? Do they use all of the \recent flag, 
subscribe/lsub and fetch header.fields/bodystructure? AUTH=DIGEST-MD5?

> Four is a crowd if you exclude POP clients that babble IMAP protocol.

Maybe I'm overly cynical tonight, but I sometimes have the impression 
that for some extensions, server implementations outnumber clients 
ones.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Dec 2004 11:24:07 -0800 (PST)

On Thu, 2 Dec 2004, Arnt Gulbrandsen wrote:
> extensions? Wow. Which? Do they use all of the \recent flag, subscribe/lsub 
> and fetch header.fields/bodystructure?

Yes.  I'll let the authors speak up, if they wish.

I'll note that all clients based upon the c-client library do this.  I 
only counted this as one, but in fact there are several out there.

> AUTH=DIGEST-MD5?

I don't know.  The only reason why I haven't implemented it has been lack 
of time; it's certainly superior to CRAM-MD5 although I think that the 
most of the problems in CRAM-MD5 are overblown.

> Maybe I'm overly cynical tonight, but I sometimes have the impression that 
> for some extensions, server implementations outnumber clients ones.

Which ones?

There are certainly proposals for extensions which were of limited value, 
but of the ones which reached standards track only ACL, QUOTA, and UIDPLUS 
stand out as having problems which hamper their general deployment.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 3 Dec 2004 05:51:03 +0100

Mark Crispin writes:
>> subscribe/lsub and fetch header.fields/bodystructure?
>
> Yes.

That pleases me.

>> AUTH=DIGEST-MD5?
>
> I don't know.  The only reason why I haven't implemented it has been 
> lack of time; ...

Isn't that another way of saying "DIGEST-MD5 is too much work for the 
benefit it brings"? ;)

>> that for some extensions, server implementations outnumber clients 

REFERRAL, say. ANNOTATE/ANNOTATEMORE.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Dec 2004 21:55:37 -0800 (PST)

> REFERRAL, say. ANNOTATE/ANNOTATEMORE.

Referrals are implemented and used.  I can't speak for annotations.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 3 Dec 2004 07:20:45 +0100

> Referrals are implemented and used.  I can't speak for annotations.

Tell more?

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Dec 2004 22:44:18 -0800 (PST)

>> Referrals are implemented and used.  I can't speak for annotations.

Cyrus Murder clusters use referrals.  We spent some time making sure that 
Pine supported them correctly.

I believe that Exchange and Outlook also use them.

-- Mark --

---

From: Steve Hole <[email-masked]>
Date: Fri, 3 Dec 2004 09:49:36 -0700

> subscribe/lsub and fetch header.fields/bodystructure? AUTH=DIGEST-MD5?

All of the above actually, both in our GUI client and the mailbox workflow
toolset.

Almost all clients use the subscribe/lsub support.    Not using 
bodystructure is simply bad programming.

Cheers.

---
Steve Hole
Chief Technical Officer - Electronic Billing and Payment Systems
ACI Worldwide

Email: [email-masked]
Phone: 780 424 4922

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 05 Dec 2004 18:31:57 +0000

Mark Crispin wrote:

>> that for some extensions, server implementations outnumber clients ones.
>
> Which ones?
>
> There are certainly proposals for extensions which were of limited 
> value, but of the ones which reached standards track only ACL, QUOTA, 
> and UIDPLUS stand out as having problems which hamper their general 
> deployment.

I am curious to know what are the problems with UIDPLUS?

---

From: Mark Crispin <[email-masked]>
Date: Sun, 5 Dec 2004 11:41:07 -0800 (PST)

> I am curious to know what are the problems with UIDPLUS?

Good question, Alexey!

I've been taking a long hard look at UIDPLUS lately, as I may have to 
implement it.

UIDPLUS bundles two orthogonal functionalities: the UID EXPUNGE command, 
and the APPENDUID and COPYUID responses.  There is no implementation 
dependendency between the two; they should not be in the same capability.

But the worst part is that APPENDUID and COPYUID are poorly thought out, 
especially in light of the interaction with MULTIAPPEND.

There should have been a single response for all cases, not the current 
situation where very similar, but not quite the same, responses are used. 
In some mail stores, certain cases of COPY end up having to be implemented 
as APPEND internally; this creates an added implementation burden to 
generate the correct COPYUID.

When multiple messages are involved, APPENDUID/COPYUID suffer from the 
flaw that a set<->set mapping needs to be calculated; the excuse being 
that if monotonically increasing UIDs are assigned then the response is 
less chatty than alternatives.  It's also an invitation to bugs.  There 
should be an explicit source->destination mapping for each message, not 
"match this set with that set".

In some mail stores, it can be very expensive to return APPENDUID/COPYUID 
data, *especially* in those cases where the IMAP server vendor does not 
control the delivery mechanism.  In fact, UIDPLUS could be called the 
FCTGTMSFASS (Force Customers To Get Their Mail Software From A Single 
Source) extension to IMAP.  Since IMAP requires strictly ascending UIDs, 
it is impossible to assign a UID to a newly APPENDed or COPYed message 
without assurance than any new mail delivered to the mailbox has also had 
a UID assigned.

Hence FCTGTMSFASS; if the mail delivery agent is from another vendor that 
assurance may be absent and the IMAP server has to fix things.  This, in 
turn, requires a sort of read/write open that assigns UIDs but doesn't 
affect \Recent flags or the RECENT count in the next open.  That also 
means that you can't use "no UID was assigned when mailbox opened" to mean 
\Recent -- the \Recent flag has to be maintained independently.

All this is not cheap, although with a lot of extra programming much of 
the extra cost can be ameliorated.  Among other things, this means 
additional mailbox metadata in order to avoid taking the worst-case fixup 
path each time.

Better would have been to have two new commands that would return new 
responses (instead of response codes).  This would save servers the work 
if the client does not care (the majority of clients!), and also would 
have allowed much more flexibility in the response syntax.

In conclusion, I believe that UIDPLUS:
  1) bundles together things that did not need to be bundled;
  2) creates server burdens that often are unnecessary;
  3) has a incredibly kludgy design in order to avoid adding new commands
     and responses.
In some ways, the third sin is the worst of all, and caused the other two 
sins.

I've often criticized ACL, but it is well-designed.  The problem with is 
that ACL is a *very* hard problem.  The difficulty is *far* worse when you 
attempt to export external legacy facilities.  Similar comments can be 
made about certain other troublesome extensions (e.g. QUOTA).

UIDPLUS, on the other hand, stands out as expediency over good design.

I fear that it's much too late to fix it now.  We're stuck with the wart. 
But we can learn from the mistake, and avoid similar mistakes in the 
future.

That I think is the important point to move forward.  What seems to be 
simple and expedient today can have long-term deleterious consequences in 
the future.  It's often difficult, and frustrating!!, to be delayed in 
discussions on how to do things right when an attractive shortcut seems to 
exist.  It's even more frustrating when deadlines are faced and/or needs 
are unmet.

But the results of doing it right are worth it.

-- Mark --