Subject: [lemonade] IMAP COMPOSE extension

From: Pete Resnick <[email-masked]>
Date: Mon, 20 Oct 2003 17:08:32 -0500

[Bcc'ed to IMAPEXT; please followup to LEMONADE]

I didn't have time last week to finish an I-D before the deadline, 
but I figured I should at least start the conversation on the list 
before Minneapolis.

When talking about IMAP Push vs. IMAP Pull for "forward without 
download" in Vienna, a lot of us thought that in either case, it 
would be handy to have an an IMAP COMPOSE command. The idea here 
would be COMPOSE would be pretty much like APPEND (with the mailbox 
in which to append, an optional flag list and optional date/time 
string), except that instead of a message literal, it would take as 
parameters any number of literals and "message part identifiers". All 
of the literals and the message parts would be appended together in 
the destination mailbox as one big message. The client would be 
responsible for providing the appropriate MIME boundaries in the 
literals.

Having a composed message on the IMAP server would work for either 
IMAP push or IMAP pull, and this would obviously be useful for more 
than just "forward without download".

However, I still have some issues with the exact nature of this 
command. Specifically, my big question is how to deal with the 
"message part identifiers". A message part identifier would clearly 
include a sequence number or UID followed by message data item names 
or a macro, just like FETCH. However, the downside to making it look 
exactly like the FETCH parameters is that you can only compose a 
message made up of parts that are contained in the currently selected 
mailbox. Maybe that is sufficient given the application for which 
LEMONADE wants this, but it's clearly a limitation.

The other choice is that the message part identifier would also 
include a mailbox name. We could come up with some syntax for that or 
just use a relative IMAP URL. Of course, the upside is that you'd be 
able to collect together messages in different parts of the IMAP 
store. But there are a couple of really big downsides:

1. It breaks the assumption that you only ever have one mailbox 
selected. In effect, you'd be required on the server to access 
multiple mailboxes in the course of a single command execution. Maybe 
that's OK to require for this extension; maybe not.

2. If it is allowed to be other than a relative URL to *this* IMAP 
server, suddenly you get into the business of doing all of the IMAP 
client work that folks were so worried about when this discussion 
started.

3. Allowing it to be a URL might start us down the slippery slope of 
allowing *any* kind of URL, not just IMAP. The results of that are 
left as an exercise to the reader.

Having the more general syntax (i.e., something that includes the 
mailbox) might be nice for folks who want to do fancier things in the 
future. We could allow any server to say "NO" if given something with 
another mailbox (or server, or scheme). Or maybe we don't want to do 
this at all.

Feedback desired. I'll float a straw-man proposal to the list (which 
obviously won't make it as an I-D for Minneapolis) as soon as I can.

pr
-- 
Pete Resnick <http://wwww.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 20 Oct 2003 15:32:30 -0700 (Pacific Daylight Time)

Pete -

It seems to me that the URLAUTH mechanism which Chris and I have proposed
answers most (if not all) of the problems which you are having with an
IMAP COMPOSE extension.  Given URLAUTH, I think that most (if not all) of
the use/need for a COMPOSE extension in IMAP is overtaken by events; or at
least IMAP COMPOSE degenerates into just APPENDing a message template to a
mailbox and then sending a URLAUTH URI that references that message.

Please refer to the latest I-D for URLAUTH:
	draft-crispin-imap-urlauth-03.txt

In particular, note that the URLAUTH mechanism allows collection of
messages from different parts of the IMAP store, and even from different
IMAP stores.  Even better, an agent referencing a URLAUTH URI doesn't have
to know all that much about IMAP.

I noted your objection about the slippery slope of allowing any kind of
URL as opposed to just IMAP.  I wonder how well any proposal is going to
prevent this from happening, short of banning any use of URLs; and if the
attempt to do so doesn't ultimately doom that proposal.

Although in the short term, a limit to IMAP URLs seems reasonable, I think
that in the long term other types of URLs (e.g. FTP URLs) will be
desirable.  That's the beauty of the BURL approach, and URLAUTH fits in
with that quite nicely.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Pete Resnick <[email-masked]>
Date: Tue, 21 Oct 2003 01:24:06 -0500

On 10/20/03 at 3:32 PM -0700, Mark Crispin wrote:

>It seems to me that the URLAUTH mechanism which Chris and I have 
>proposed answers most (if not all) of the problems which you are 
>having with an IMAP COMPOSE extension.

So are you saying that URLFETCH is an acceptable general mechanism 
and such an operation is reasonable, even in the context of COMPOSE? 
The main problem I was having was with the notion (I believe 
mentioned to me by Chris) that requesting parts from non-selected 
mailboxes would be problematic for many server implementations. If 
that's not an issue, then I will plow ahead with COMPOSE using IMAP 
URLs for the referenced parts.

>Given URLAUTH, I think that most (if not all) of the use/need for a 
>COMPOSE extension in IMAP is overtaken by events;

I'm not sure I know what you mean by "overtaken by events".

>or at least IMAP COMPOSE degenerates into just APPENDing a message 
>template to a mailbox and then sending a URLAUTH URI that references 
>that message.

Well, sort of. COMPOSE as I see it is simply APPENDing a template 
message, where that template is a concatenation of message literals 
and something equivalent to the result of URLFETCHs (if we decide 
that using URLs is OK). The nice thing is that it gives the 
responsibility for building that message (i.e., putting in the MIME 
boundaries, etc.) to the client. Then, that composed message can 
handed off to whatever submit mechanism we choose.

As was discussed in Vienna (see the minutes), there was widespread 
support for a compose mechanism being separate from the submission 
mechanism. Without going into great detail, the idea was that no 
matter what method for submit we choose, it would most logical and 
efficient to have the composition take place in IMAP. This was viewed 
as not only a good compromise, but as a way to get by the knot of the 
push vs. pull debate by reducing the question to simply the 
submission mechanism.

Anyway, I see COMPOSE as a good thing to add to IMAP, with or without 
this particular problem to solve.

>messages from different parts of the IMAP store, and even from 
>different IMAP stores.  Even better, an agent referencing a URLAUTH 
>URI doesn't have to know all that much about IMAP.

Well, here I think you're talking about the BURL mechanism that 
allows this, not URLAUTH per se. And I believe most everyone in the 
room, including Chris (please correct me if I'm wrong, Chris) agreed 
that having a composition mechanism in IMAP was the better way to 
approach the problem at hand.

>I noted your objection about the slippery slope of allowing any kind 
>of URL as opposed to just IMAP.  I wonder how well any proposal is 
>going to prevent this from happening, short of banning any use of 
>URLs; and if the attempt to do so doesn't ultimately doom that 
>proposal.

Well, like I said, I'm not sure whether the URL approach is going to 
be acceptable for COMPOSE. If it isn't, I don't think it dooms the 
proposal; we can always use a BURL type mechanism in the future for 
the more complicated cases. But it seemed to most of us in the room 
that the simplest way to approach the problem at hand was to do the 
compose in IMAP and then use either URLAUTH/submit-server mechanism 
or an IMAP submit mechanism afterwards.

pr

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2003 12:37:37 +0200

Pete Resnick writes:
> The client would be responsible for providing the appropriate MIME 
> boundaries in the literals.

Why?

The server can pick boundaries that don't collide with the content. All 
the client can do is pick boundaries that have low probability of doing 
so.

...
> include a mailbox name.
...
> that's OK to require for this extension; maybe not.

Barry Leiba's SEARCHM has the same property. Noone implemented it 
(sadly), but noone complained about that property either.

IMNSHO, a client that's smart enough to gather together parts from 
different mailboxes in the first place is also smart enough to gather 
them using a sequence of COPY, COMPOSE, STORE +FLAGS \DELETED and 
EXPUNGE.

--Arnt

---

From: Mark Crispin <[email-masked]>
Date: Tue, 21 Oct 2003 07:07:33 -0700 (PDT)

On Tue, 21 Oct 2003, Pete Resnick wrote:
> mailboxes would be problematic for many server implementations.

The whole point of the URLAUTH/URLFETCH mechanism is to minimize dealing
with IMAP.  There is no need for a composer to have a selected mailbox,
much less worry about whether a particular URI is in a selected mailbox.

In fact, it leads to the obvious question of why there are such extreme
efforts to make this strange IMAP protocol do composition, which all you
have to do with IMAP is grab the data and go away.

> handed off to whatever submit mechanism we choose.

Why not just use APPEND, with *no* special composition requirements on the
IMAP server?  Then every IMAPrev1 server in the world today works.

> push vs. pull debate by reducing the question to simply the

Some important people could not go to Vienna.  I don't think that we can
consider anything said there to be final, although it is a consideration.

> this particular problem to solve.

I do not.  I see it as a disaster -- effectively an attempt to make IMAP
into the IETF answer to Exchange and other all-inclusive proprietary
systems.

As an answer to Exchange, IMAP fails miserably.  IMAP was focused on
solving one problem of email -- message access -- and nothing else.  I
don't like giving ammo to the clowns who argue for a POP4 on the grounds
that "IMAP is a bloated mess that's becoming more bloated."

Yet that is where I see the addition of orthogonal functionality heading.
URLAUTH/URLFETCH fits in will with the concept of message access, and
seems to be a worthwhile extension for other reasons.  The practice of
using APPEND to upload a template barely fits in because APPEND is already
there.  Adding an explicit COMPOSE command, or otherwise mandating that
an IMAP server do compose-assembly, does not.

By tying compose-assembly to IMAP, this means that any use of any other
access mechanism as a data source for a composed message must either use
both IMAP and that other mechanism, or all the compose-assembly work in
IMAP must be duplicated (either in that mechanism or in an independent
protocol).  Think NNTP.  Or suppose that POP4 does happen.

OK, I admit it.  This *is* a slippery slope argument.  Being a slippery
slope argument doesn't mean that it is an invalid argument.

Actually, I think that there should be a URLAUTH mechanism in POP3; and I
probably should add appropriate text to the URLAUTH document to extend
POP3 as well.  Although the thought of POP3 clients being obliged to use
IMAP for composition *is* a delicious irony,......

> allows this, not URLAUTH per se.

I'm talking about an external mechanism, such as BURL.  The exact nature
of the external mechanism is a separate issue.  It doesn't have to be
BURL, although BURL seems to be a reasonable offering.

> And I believe most everyone in the

If Chris has abandoned BURL in favor of compose-assembly in IMAP, that is
news to me.

> or an IMAP submit mechanism afterwards.

I advocate taking the long view, and doing the right thing now; rather
than be sidetracked on an interim quick-and-dirty "easy" answer (that some
of us don't think will be so "quick" or "easy", but will be dirty).

-- Mark --

---

From: Nathaniel Borenstein <[email-masked]>
Date: Tue, 21 Oct 2003 10:58:10 -0400

I've been lurking on this topic without any dog in the hunt, but I 
think I've been following things long enough to be willing to admit to 
being totally confused.  Why are we doing this in the first place?  It 
would be very helpful to me if someone could step back and clearly 
[re-]state the problem we are trying to solve with IMAP COMPOSE.

Is there in fact a general agreement about goals?  Two people with 
completely different goals who are trying to collaborate on building a 
single artifact need to at least understand each other's goals.  
Otherwise they will frequently view each other as bozos.

My apologies if I've missed something totally obvious.  -- nathaniel

---

From: Pete Resnick <[email-masked]>
Date: Tue, 21 Oct 2003 13:49:08 -0500

On 10/21/03 at 11:35 AM -0400, Cyrus Daboo wrote:

>Note that APPEND does not require selected state - i.e. you can 
>append to any mailbox not just the one that may be currently 
>selected in that session.

My concern wasn't about the destination mailbox; that I understood 
was independent of selected state. My concern was the *source* 
mailboxes for the included parts.

>What I would prefer to see here is something more like an 'expand' 
>action rather than a 'compose' action (multiple items that are put 
>together to form the final message) as you described. By that I mean 
>the client submits a fully formed rfc2822 MIME message that uses 
>message/external-body (or some new message/xxx type as appropriate) 
>with URL (or a new 'IMAP' type) references and the server is then 
>instructed to 'expand' the external references as it appends the 
>message.

I thought about this (and I think I discussed it with several folks 
in Vienna). The downside to expand is that you then have a message 
with "expandable parts". That means that I can send you an 
"unexpanded message" and perhaps invoke some unexpected shenanigans 
in your server, perhaps getting it to expand things upon forwarding 
my message that you didn't expect. Also, EXPAND makes this operation 
depend on examining the internals of a message in the message store, 
which is one of the things that Mark especially objected to earlier 
with regard to other proposals. I think it's an interesting idea, but 
I think COMPOSE is safer and easier to implement for IMAP servers.

>Expand rather than compose would also make handling of those types 
>of messages easier on the client-side. Often a user does not compose 
>and send a message at the same time. They might 'partially' compose 
>it, save it for later, then work on it again and send it out later. 
>By not using the 'expand' option for APPEND until the final 
>composition is done, the overhead of the expand/compose only occurs 
>once during the lifetime of a draft message that gets 
>saved/rewritten many times.

That makes it easier (overhead wise) on the server side, not the 
client side. And though I agree that an implementation might go ahead 
and copy the included part into the new message immediately, there is 
nothing to say that the server can't just keep a pointer to the 
included part and only expand it when the message is FETCHed.

>Also, consider how to deal with such messages in disconnected mode. 
>Again it would be much easier for a client to use the 'expand' 
>method in disconnected mode as it could cache the complete 
>unexpanded message and then sync that up to the server using 
>append+expand. That would be harder to do using 'compose' as the 
>disconnected client would have to keep track of the separate 
>elements of the compose message items whilst disconnected.

I don't really see a big difference. The client is going to have some 
local representation of the message with the included parts. That 
could be some sort of MIME entity with special pointers to included 
parts, or it could be something else. I guess in the EXPAND case, the 
client can use an identical representation locally to the template it 
uploads to the server. In COMPOSE, the client would change syntax 
(literals and URLs), but I really don't see that as a big deal. And 
again, I'm uninclined to have "expandable" MIME entities floating 
around.

pr

---

From: Pete Resnick <[email-masked]>
Date: Tue, 21 Oct 2003 13:54:43 -0500

On 10/21/03 at 10:58 AM -0400, Nathaniel Borenstein wrote:

>think I've been following things long enough to be willing to admit 
>to being totally confused.  Why are we doing this in the first 
>place?  It would be very helpful to me if someone could step back 
>and clearly [re-]state the problem we are trying to solve with IMAP 
>COMPOSE.

<http://www.ietf.org/internet-drafts/draft-ietf-lemonade-submit-00.txt>

That document describes the main problem. In summary, the basic 
problem we're trying to solve is, "How can I forward this huge 
message (or part of a message) that's on my IMAP server without 
having to download it to my client and re-upload it to my SMTP/submit 
server?" One part of that issue is (1) "How do I collect together the 
things I want to put in the outgoing message?" and the other is (2) 
"How do I get that outgoing message to my SMTP/submit server?". 
COMPOSE is designed to address question #1.

pr

---

From: Mark Crispin <[email-masked]>
Date: Tue, 21 Oct 2003 16:27:19 -0700 (Pacific Daylight Time)

On Tue, 21 Oct 2003, Pete Resnick wrote:
> The downside to expand is that you then have a message
> with regard to other proposals.

If the expansion is *NOT* done in the IMAP server, but instead in a SUBMIT
server which obtains the pre-expanded message via URLFETCH, then there is
nothing at all wrong with expand from the IMAP point of view.

The only reason why any problem comes up is in the insistance of doing it
in the IMAP server.

> >it, save it for later, then work on it again and send it out later.

Indeed, if IMAP is used strictly as a place to leave drafts, with
composition/expansion done elsewhere, this works very well.

> client side.

That means that you would have an explicit requirement that the server
store the unexpanded form.  And you have to have separate commands to
fetch expanded and fetch unexpanded.

Otherwise, the client will end up fetching the expanded version.

> >append+expand.

There are similar benefits if expand was a function of the submit server
and not of the IMAP server.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 21 Oct 2003 18:35:22 -0700 (Pacific Daylight Time)

On Tue, 21 Oct 2003, Cyrus Daboo wrote:
> The only problem with that is in dealing with the sent-mail copy, which the
> user may or may not want to appear expanded on their IMAP server. The
> submit/expand server would have to have the ability to do the sent-mail
> copy too, or else the expand has to be on the IMAP server.

That's a good point.  I agree that a user may want his sent-mail to be
either way (expanded or non-expanded), and that this is something that any
message assembly mechanism (whether compose on an IMAP server, or expand
on a submit server) must address.

Fortunately, there are already mechanisms in place that a submit server
can use to do this.  Compose in IMAP does not have such mechanisms,
although I guess that it would not be too difficult to add those.

-- Mark --

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 21 Oct 2003 11:35:26 -0400

Hi Pete,

--On Monday, October 20, 2003 17:08 -0500 Pete Resnick 
<[email-masked]> wrote:

| However, I still have some issues with the exact nature of this command.
| Specifically, my big question is how to deal with the "message part
| identifiers". A message part identifier would clearly include a sequence
| number or UID followed by message data item names or a macro, just like
| FETCH. However, the downside to making it look exactly like the FETCH
| parameters is that you can only compose a message made up of parts that
| are contained in the currently selected mailbox. Maybe that is sufficient

                       ^^^^^^^^^^^^^^^^^^^^^^^^^^

| given the application for which LEMONADE wants this, but it's clearly a
| limitation.

Note that APPEND does not require selected state - i.e. you can append to 
any mailbox not just the one that may be currently selected in that 
session. That's something we frequently do. In fact it can be desirable to 
do the APPEND in a background session so that it does not interfere with 
the user working on the mailbox in the foreground. i.e. you will almost 
certainly have to have a reference to a mailbox rather than be restricted 
to the currently selected one.

What I would prefer to see here is something more like an 'expand' action 
rather than a 'compose' action (multiple items that are put together to 
form the final message) as you described. By that I mean the client submits 
a fully formed rfc2822 MIME message that uses message/external-body (or 
some new message/xxx type as appropriate) with URL (or a new 'IMAP' type) 
references and the server is then instructed to 'expand' the external 
references as it appends the message. If we did it that way, then a similar 
procedure could be easily used with SMTP submission to have an SMTP server 
do the 'expansion'.

Expand rather than compose would also make handling of those types of 
messages easier on the client-side. Often a user does not compose and send 
a message at the same time. They might 'partially' compose it, save it for 
later, then work on it again and send it out later. By not using the 
'expand' option for APPEND until the final composition is done, the 
overhead of the expand/compose only occurs once during the lifetime of a 
draft message that gets saved/rewritten many times.

Also, consider how to deal with such messages in disconnected mode. Again 
it would be much easier for a client to use the 'expand' method in 
disconnected mode as it could cache the complete unexpanded message and 
then sync that up to the server using append+expand. That would be harder 
to do using 'compose' as the disconnected client would have to keep track 
of the separate elements of the compose message items whilst disconnected.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 21 Oct 2003 19:38:14 -0400

Hi Mark,

--On Tuesday, October 21, 2003 16:27 -0700 Mark Crispin 
<[email-masked]> wrote:

|> > Also, consider how to deal with such messages in disconnected mode.
|> > Again it would be much easier for a client to use the 'expand'
|> > method in disconnected mode as it could cache the complete
|> > unexpanded message and then sync that up to the server using
|> > append+expand.
|
| There are similar benefits if expand was a function of the submit server
| and not of the IMAP server.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 29 Oct 2003 18:11:54 +0100

Cyrus Daboo writes:
> The only problem with that is in dealing with the sent-mail copy, 
> which the user may or may not want to appear expanded on their IMAP 
> server. The submit/expand server would have to have the ability to do 
> the sent-mail copy too, or else the expand has to be on the IMAP 
> server.

Not necessarily. The client can tell the submit server "here, expand and 
deliver to these addresse" and subsequently "here, deliver to me".

There's a window in which things may go wrong, between the ends of the 
two DATA commands. But the added complexity to handle one DATA command 
in two ways also offers scope for problems.

Also, sending twice wastes some bandwidth. Not much. The reason it's not 
much is that submit/expand is being used.

--Arnt