Subject: [lemonade] problems implementing RESETKEY GENURLAUTH and URLFETCH

From: Peter Coates <[email-masked]>
Date: Fri, 07 Apr 2006 08:55:11 -0700

I have problems with all three of these commands as they currently exist.
These are surmountable problems, but it is very ugly.
 
The problem stems from the architecture of our server as deployed in large
installations.
 
users and their mail stores are distributed over a number of servers.  users
(generally) connect to their home server.  If they select a folder which is
on another server, the commands are proxied across to the hosting server.
All commands are either executed locally (the normal case) or proxied in
their entirety.  There are some cases where the local server issues a
command to a remote server.  There is no case where a command is partially
proxied.
 
GENURLAUTH and URLFETCH can carry several URLs.  these URLs can be
unrelated.  this requires that parts of the command be proxied.  this is
pretty ugly  I don't suppose that it is possible to put in a restriction
that the URLs in these commands be restricted so that they all must refer to
the same user/mailbox...  It would be nice.
 
RESETKEY poses a problem of another kind.
 
the RESETKEY mailbox form of the command is not a problem.  The bare
RESETKEY command is.  It may require the local server to send the RESETKEY
to all the other hosts to reset their keys.  That is not very pretty.  I
suspect I have even less chance of persuading people that the the bare
RESETKEY is a poor idea.
 
Peter
 
Just trying to make my own life easier.
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 10 Apr 2006 17:03:36 -0700 (Pacific Daylight Time)

Hello Peter.

I am confused by your messages.  Is the client expected to access the 
other servers (that is, does your server issue referrals)?

If your server does not issue referrals, then from the user's (and 
client's) point of view there is just one server.  What happens internally 
in your server farm is not a concern of the user, the client, or the 
protocol.

I think that the ability to forward multiple messages, from perhaps 
multiple sources, is a benefit.  I would like to hear a better argument 
for removing this capability than "it inconveniences one server 
implementation's way of distributing mailboxes across different servers."

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Peter Coates <[email-masked]>
Date: Mon, 10 Apr 2006 17:19:25 -0700

Our server spreads users over multiple back ends, and if joe shares any of
his folders with sue, then the server sue is directed to will have to proxy
some requests to joe's machine.  This is an entirely reasonable approach.

This can be made to work reasonably with all IMAP commands to date, and
resetkey, and urlfetch can also be made to work.

Genurlauth is nastier because it works in two phases, a validation phase and
an operation phase.  For the internal mechanism it can be made to work by
proxying the commands across during the validation phase and holding the
results until the operational phase.  This is OK because the internal
mechanism has no side effects.

But it is very ugly.

It is not clear to me what is actually gained by clients by being able to
collect together in one command a set of URLs that are only slightly
related.  You would not expect the command

1 URLFETCH imap://[email-masked]/inbox/;uid=10...
Imap:/[email-masked]/inbox/;uid=42...

To return anything really useful.

Why is it so imperative that  

1 URLFETCH imap://[email-masked]/inbox/;uid=10...
Imap:/[email-masked]/inbox/;uid=42...

Work?

And similarly for genurlauth

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Mark Crispin
Sent: April 10, 2006 17:04
To: Peter Coates
Cc: [email-masked]
Subject: Re: [lemonade] problems implementing RESETKEY GENURLAUTH and
URLFETCH

Hello Peter.

I am confused by your messages.  Is the client expected to access the other
servers (that is, does your server issue referrals)?

I think that the ability to forward multiple messages, from perhaps multiple
sources, is a benefit.  I would like to hear a better argument for removing
this capability than "it inconveniences one server implementation's way of
distributing mailboxes across different servers."

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Mon, 10 Apr 2006 17:28:00 -0700 (Pacific Daylight Time)

> some requests to joe's machine.  This is an entirely reasonable approach.

And thus IMAP has a mechanism, called "referrals", to support this.  You 
seem to have chosen not to use referrals.

> You would not expect the command
>

That's because the domain identified in the URL is the domain of the IMAP 
server.  Thus it can not work unless xyz.com and abc.com are nicknames for 
the same machine.

> Imap:/[email-masked]/inbox/;uid=42...

That's the wrong question.  Why shouldn't it work?

How does that differ from

1 URLFETCH imap://[email-masked]/inbox/;uid=10... imap:/[email-masked]/vacation/;uid=42...

Why shouldn't that be allowed?

> And similarly for genurlauth

If my client needs multiple URLs generated, why should it have to go 
through multiple RTTs to do it?

-- Mark --

---

From: Tony Finch <[email-masked]>
Date: Tue, 11 Apr 2006 02:10:34 +0100

On Mon, 10 Apr 2006, Mark Crispin wrote:
> And thus IMAP has a mechanism, called "referrals", to support this.  You seem
> to have chosen not to use referrals.

That seems to be a common design choice - cf. the Cyrus Murder and the
Perdition proxy.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BERWICK ON TWEED TO WHITBY: WEST OR SOUTHWEST 5 OR 6, PERHAPS INCREASING 7
LATER IN NORTH. RAIN AT FIRST. MAINLY GOOD. SLIGHT OR MODERATE.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 10 Apr 2006 21:28:01 -0400

> That seems to be a common design choice - cf. the Cyrus Murder and the

Cyrus Murder will issue referrals iff the client has shown that it can 
handle them (it sends either an RLIST or RLSUB)

Do any clients other than Pine support mailbox referrals?

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: Peter Coates <[email-masked]>
Date: Mon, 10 Apr 2006 18:58:23 -0700

If we used referrals, the story with GENURLAUTH and URLFETCH would be worse.
These are commands that are not open to being referred.  They have to be
proxied piecemeal.  The discussion here is leading me to believe that there
is a problem with the design of these commands.  Bit of a shame that as I
have done the (seriously ugly) work to do this proxying.  On the other hand,
the cost of a GENURLAUTH command with multiple URLs which happen to be
hosted on multiple hosts is potentially much higher than a client would
expect.  That is the sort of surprise that ought not to be.

Peter 

-----Original Message-----
From: Ken Murchison [mailto:[email-masked]] 
Sent: April 10, 2006 18:28
To: Tony Finch
Cc: Mark Crispin; [email-masked]; Peter Coates
Subject: Re: [lemonade] problems implementing RESETKEY GENURLAUTH and
URLFETCH

Tony Finch wrote:
>>> Our server spreads users over multiple back ends, and if joe shares 
>>> any of his folders with sue, then the server sue is directed to will 
>>> have to proxy some requests to joe's machine.  This is an entirely
reasonable approach.
>> And thus IMAP has a mechanism, called "referrals", to support this.  
>> You seem to have chosen not to use referrals.
>

---

From: Dave Cridland <[email-masked]>
Date: Tue, 11 Apr 2006 08:53:21 +0100

On Tue Apr 11 02:28:01 2006, Ken Murchison wrote:
> Cyrus Murder will issue referrals iff the client has shown that it 
> can handle them (it sends either an RLIST or RLSUB)
> 
> 
Or a LIST (REMOTE) - if the client supports LISTEXT, which mine does.

> Do any clients other than Pine support mailbox referrals?

Yes, Polymer does, although I've only tested this against the public 
Cyrus IMAP Murder. I'm very tempted, however, to remove the support, 
or at least make it off by default, because:

a) The latency increase is huge - remote mailboxes cause an 
appreciable delay in the client due to having to setup a new IMAP 
connection - in addition, you lose state if you're referred by a 
SELECT, although CONDSTORE helps with this.

b) The complexity issues are huge, since there's a sudden distinction 
between the server a URL points to and the actual connection that 
needs to be made. For a client (like mine) that uses URLs 
intensively, this is a bit of a pain, especially when I need to 
persist URLs (for instance, as bookmarks. Stored in ACAP of course).

c) It's impossible to copy messages between folders without 
downloading, and re-uploading them.

I strongly suspect that whilst referrals are good protocol design, 
this is an area where practical concerns rule in favour of the proxy 
design - not only can you immediately support more clients, but 
various complexities vanish.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Mark Crispin <[email-masked]>
Date: Tue, 11 Apr 2006 08:31:36 -0700

On Tue, 11 Apr 2006, Dave Cridland wrote:
> Yes, Polymer does, although I've only tested this against the public Cyrus 
> IMAP Murder. I'm very tempted, however, to remove the support, or at least 
> make it off by default

Note though that there are two types of referrals; a referral which takes 
place even though the server will proxy, and one that takes place when the 
server will give an error.  Of course, the latter happens most often with 
login referrals.

-- Mark --

---

From: Randall Gellens <[email-masked]>
Date: Tue, 11 Apr 2006 14:09:50 -0700

At 5:28 PM -0700 4/10/06, Mark Crispin wrote:

>  If my client needs multiple URLs generated, why should it have to 
> go through multiple RTTs to do it?

Separating each URL into its own command requires a round-trip for 
each because the URLAUTH response is untagged?
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
It is much easier to suggest solutions
when you know nothing about the problem.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 12 Apr 2006 14:15:11 +0100

> each because the URLAUTH response is untagged?

But surely
* GENURLAUTH <url1> <url2>

is the same as

* GENURLAUTH <url1>
* GENURLAUTH <url2>

or

* GENURLAUTH <url2>
* GENURLAUTH <url1>
?
So the IMAP server can treat a single GENURLAUTH command with multiple 
URLs as multiple GENURLAUTH commands with one URL.

The client can always match a received URL against an URL it requested 
to sign.

---

From: Randall Gellens <[email-masked]>
Date: Wed, 12 Apr 2006 16:37:20 -0700

At 2:15 PM +0100 4/12/06, Alexey Melnikov wrote:

>>>   If my client needs multiple URLs generated, why should it have 
>>> to go through multiple RTTs to do it?
>>

That's what I thought, which is why I asked.  So even if the client 
had to send multiple commands, it can pipeline them.

>  So the IMAP server can treat a single GENURLAUTH command with 
> multiple URLs as multiple GENURLAUTH commands with one URL.

In the case where the one IMAP server is proxying for additional 
back-end servers, it would turn the one command into several, then 
intercept the multiple tagged OKs, and send back one OK after the 
proxied commands completed?

>
>  The client can always match a received URL against an URL it 
> requested to sign.

Good point.

-------------- Randomly-selected tag: ---------------
You see, wire telegraph is a kind of very, very long cat.  You
pull his tail in New York and his head is meowing in Los Angeles.
Do you understand this?  And radio operates in exactly the same
way: you send signals here, they receive them there.  The only
difference is that there is no cat.            --Albert Einstein