Subject: [lemonade] CATENATE and drafts

From: Pete Resnick <[email-masked]>
Date: Sat, 10 Apr 2004 16:59:30 -0500

Talking to Alexey on another topic (more on that in a different 
message), we talked about how to make CATENATE draft-friendly. One of 
the things I suggested (which I believe I've mentioned before) is:

- When CATENATE'ing, the client adds a field to the CATENATE'd part 
indicating that it has been included from elsewhere.

- When the client goes back to edit the draft, it first gets the 
BODYSTRUCTURE, and then retrieves only those parts not marked as 
above.

- When the client is done editing, it re-CATENATE's, just as it did above.

The question is, how should they be marked? It would be most 
desirable if the marker contained some description of the CATENATE'd 
part since the client doesn't want to have to store local data to 
find where the part came from. Something like Content-Location comes 
to mind since it can hold the URL and will be returned by 
BODYSTRUCTURE. The downside of that (or whatever is chosen) is that 
it will be sent with the message unless the client downloads them 
again and re-CATENATE's without the markers. (I'm not sure, but I 
would think that exposing information about the user's IMAP mailbox 
structure would be a security issue.) I thought about encrypting it, 
like MD5'ing against something, but aside from ever increasing pain 
for the client, I couldn't think of something easy to encrypt against 
that would stay stable. And of course an annotation would be perfect 
for this, but I don't think we want to depend on that. (Or maybe we 
just say that CATENATE is draft-unfriendly for the client unless you 
have ANNOTATE. Ick.)

Anybody have a suggestion for stashing this piece of data?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Sat, 10 Apr 2004 16:19:59 -0700 (PDT)

On Sat, 10 Apr 2004, Pete Resnick wrote:
> The question is, how should they be marked? It would be most desirable if the 
> marker contained some description of the CATENATE'd part since the client 
> doesn't want to have to store local data to find where the part came from.

Although we probably are going to use CATENATE instead of Embedded URI for 
Push, I recommend that the chosen mechanism have Embedded URI in mind.

Putting on my "IMAP architecture hat", and discussing the "IMAP way" of 
doing it...:

My suggestion is *NOT* a Content-Location, since that overloads what that 
header is used for.  Rather, to maximize IMAP friendliness, it should use 
a Content-Type parameter.  The idea here is that a MUA picking up a draft 
would pick up the ENVELOPE, BODYSTRUCTURE, and body part(s) rather than 
picking up a raw RFC 2822 message.  The matter of final assembly for 
CATENATE and removal of Embedded URIs then becomes trivial.

To further highlight that this is a draft and should not be sent as-is, 
perhaps there should be some other mandatory blurdybloop for drafts which 
make it unsuitable for direct sending.  CATENATE should check for that 
blurdybloop, and either barf if it sees it or remove it (and the 
Content-Type parameter suggested above).

Anyway, this is how IMAP was intended to be used.  I fear that these notes 
may be a bit sketchy for some people, but hopefully at least the general 
idea comes through.  The whole point is to work with IMAP, rather than 
fighting it.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Pete Resnick <[email-masked]>
Date: Sat, 10 Apr 2004 18:29:34 -0500

On 4/10/04 at 4:19 PM -0700, Mark Crispin wrote:

>>desirable if the marker contained some description of the 
>>CATENATE'd part since the client doesn't want to have to store 
>>local data to find where the part came from.
>
>Although we probably are going to use CATENATE instead of Embedded 
>URI for Push, I recommend that the chosen mechanism have Embedded 
>URI in mind.

So, you are effectively suggesting what I talked about a little later 
in my message:

>>The downside of that (or whatever is chosen) is that it will be 
>>sent with the message unless the client downloads them again and 
>>re-CATENATE's without the markers.

So, you're saying that when there is a draft on the server, it 
shouldn't be sendable, but rather the client should be required to 
download that draft and CATENATE it back, replacing the marked up 
parts (in your description, with embedded URI). Again, this increases 
pain for the client.

The question I asked (which you didn't answer) was: What if I *want* 
the draft to be a sendable message? Is there a reasonable way to do 
that.

pr

---

From: Mark Crispin <[email-masked]>
Date: Sat, 10 Apr 2004 17:52:42 -0700 (PDT)

On Sat, 10 Apr 2004, Pete Resnick wrote:
> So, you're saying that when there is a draft on the server, it shouldn't be 
> sendable, but rather the client should be required to download that draft and 
> CATENATE it back, replacing the marked up parts (in your description, with 
> embedded URI). Again, this increases pain for the client.

Not necessarily.  Let's say we define some sort of standard Embedded URI 
(or equivalent) format for drafts.  All that is required is that CATENATE 
be savvy of that standard format.

This is actually an assemble command.  I don't know if it's worth making 
one command do both, or if we have two commands: CATENATE (and assemble) 
from sources specified by the client, and ASSEMBLE a draft.

> The question I asked (which you didn't answer) was: What if I *want* the 
> draft to be a sendable message? Is there a reasonable way to do that.

IMHO, just upload the draft (via CATENATE or APPEND) as a message instead 
of a draft.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Sun, 11 Apr 2004 11:12:14 -0500

On 4/10/04 at 5:52 PM -0700, Mark Crispin wrote:

>Let's say we define some sort of standard Embedded URI (or 
>equivalent) format for drafts.
>[...]
>This is actually an assemble command.

This is the other issue that I mentioned Alexey and I were discussing 
earlier: an EXPAND (or as you put it, ASSEMBLE) command. I think this 
is an interesting approach, but here are the concerns I've expressed 
about it before:

1. The skeleton message (Embedded URI or equivalent) is going to need 
some sort of indication in the message that a particular part is to 
be expanded. I'm very afraid of the situation where someone sends the 
user a message with an Embedded URI hiding in it, the user replies, 
does the ASSEMBLE/EXPAND, and exposes some part of his mail store to 
the attacker.

2. It creates two mechanisms to build a message: You either do it on 
the fly with CATENATE, or you do it on a draft with ASSEMBLE/EXPAND. 
We could say that we would *only* have ASSEMBLE/EXPAND, but that 
means that the act of building a message (even if you were not going 
to have a draft) is always a two-step process: First APPEND the 
skeleton, then ASSEMBLE/EXPAND. Maybe that's not a big deal.

3. ASSEMBLE/EXPAND requires that the IMAP server parse the message. I 
thought that this was actually something that Mark had objected to in 
the past as well, but maybe I am mistaken.

>>The question I asked (which you didn't answer) was: What if I 
>>*want* the draft to be a sendable message? Is there a reasonable 
>>way to do that?
>
>IMHO, just upload the draft (via CATENATE or APPEND) as a message 
>instead of a draft.

I'm obviously not being clear, so let me try this again:

I want CATENATE to result in a sendable message on the server. 
However, I want to allow the client to retrieve that message from the 
server for further editing without having to download the included 
parts. I also want the client to be able to discover the URI of the 
included part when it retrieves the message for editing. And finally, 
I'd prefer not to expose information about the structure of my IMAP 
store to a recipient of the message. So the question I'm asking is:

Is there a way to do the above without using annotations on the included parts?

pr

---

From: Mark Crispin <[email-masked]>
Date: Sun, 11 Apr 2004 11:51:08 -0700 (PDT)

On Sun, 11 Apr 2004, Pete Resnick wrote:
> 1. The skeleton message (Embedded URI or equivalent) is going to need some 
> sort of indication in the message that a particular part is to be expanded. 
> I'm very afraid of the situation where someone sends the user a message with 
> an Embedded URI hiding in it, the user replies, does the ASSEMBLE/EXPAND, and 
> exposes some part of his mail store to the attacker.

I agree that the design and specification for this needs to address the 
possibility of this attack in its security considerations, as well as the 
measures to overcome it.

This attack is not specific to the Embedded URI idea; it occurs whenever 
data of arbitrary origin is parsed and processed as anything other than 
raw text.  All of HTTP is one big security hole of this type.

> 2. It creates two mechanisms to build a message: You either do it on the fly 
> with CATENATE, or you do it on a draft with ASSEMBLE/EXPAND. We could say 
> that we would *only* have ASSEMBLE/EXPAND, but that means that the act of 
> building a message (even if you were not going to have a draft) is always a 
> two-step process: First APPEND the skeleton, then ASSEMBLE/EXPAND.

I think that client needs are sufficient diverse that a "one shoe fits 
all" solution is probably less satisfactory.

There are actually at least four ways to send a message:
  1) transmit directly to submit server.  Any saved copy is send
     independently.
  2) APPEND saved copy to IMAP server, then submit saved copy (whether push
     or pull)
  3) APPEND draft with pointers to IMAP server, ASSEMBLE draft into saved
     copy, then submit saved copy (whether push or pull)
  4) CATENATE message parts to IMAP server as saved copy, then submit saved
     copy (whether push or pull).

> 3. ASSEMBLE/EXPAND requires that the IMAP server parse the message. I thought 
> that this was actually something that Mark had objected to in the past as 
> well, but maybe I am mistaken.

I am opposed to it.  But some bad things are worse than other bad things.

I'm not happy about it, but convincing arguments have been made about the 
need to do message composition on an IMAP server.  However, those 
arguments do not support a choice of push vs. pull unless one side linds 
itself to reality and refuses to accept message composition.

With all this in mind, CATENATE is alright as a first order implementation 
of composition.  For some clients it is all that is needed.  But it has 
limitations.  Hence the need for ASSEMBLE for a more advanced facility.

I have not given up on Embedded URI and submit-server based assembly as a 
long-term solution.  I remained convinced that this is the superior 
solution.  However, a well-designed IMAP-server based assembly capability 
does not preclude this, and has sufficient utility (the fcc/saved copy 
problem) that it's worth doing.

So, my concern is that IMAP based assembly be well-designed.

I think that both CATENATE and ASSEMBLE are useful, and by focusing on 
doing the task each was designed to do the result is likely to be better 
than trying to make one command do both.

> I want CATENATE to result in a sendable message on the server.

As I envision it, CATENATE will always result in a sendable message on the 
server.  Normally, a client would APPEND drafts with pointers (Embedded 
URI or whatever) in a well-known format, and ASSEMBLE the draft.

I see CATENATE's purpose as being for a one-shot, with no drafts.  In this 
case, CATENATE is one RTT and has fewer error conditions than 
APPEND+ASSEMBLE.  Although strictly speaking CATENATE is an "unnecessary" 
command in this view, the circumstance is common enough and the benefit 
large enough that I think that it's worth it to have the shortcut of 
CATENATE.

In other words:
  . CATENATE makes a saved copy, not a draft.
  . drafts with pointers are not directly sendable, at least not until we
    have a submit server functionality to do this (the Pull + Embedded URI
    variant).  Drafts must be ASSEMBLEd first.

I would not think of storing drafts and saved copies in the same mailbox, 
and I don't know of any client which does (perhaps there is, but I don't 
know of one).  The risk of confusion, especially when multiple clients are 
involved, is too great.

This suggests that ASSEMBLE needs to take a target mailbox argument ala 
COPY, and must return a UID of the assembled message.

I think that you envision it differently.  You see CATENATE as somehow 
storing implicit Embedded URIs that don't show up if the message is pulled 
by (or pushed to) a submit server.  There then needs to be some procedure 
to obtain the draft as a framework + pointers.

To do this, you will need an additional command (UNCATENATE?) to split the 
message back into CATENATE arguments.

> However, I 
> want to allow the client to retrieve that message from the server for further 
> editing without having to download the included parts. I also want the client 
> to be able to discover the URI of the included part when it retrieves the 
> message for editing. And finally, I'd prefer not to expose information about 
> the structure of my IMAP store to a recipient of the message. So the question 
> I'm asking is:
>

In my suggested model there's no reasonable way to do this, but 
fortunately the question is more or less moot.

In your suggested model, you need an UNCATENATE functionality.

Either one will work.  I think that my model ultimately is better, because 
it goes to greater effort to keep things separate.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Sun, 11 Apr 2004 16:15:04 -0500

On 4/11/04 at 11:51 AM -0700, Mark Crispin wrote:

>You see CATENATE as somehow storing implicit Embedded URIs that 
>don't show up if the message is pulled by (or pushed to) a submit 
>server.

No, actually that's not how I see CATENATE. I see CATENATE as 
creating a message which the server cannot distinguish from the same 
message made by the client downloading and APPEND'ing all of the 
parts. What I want is for the client to be able to distinguish the 
parts without the server (in FETCH or COPY) being able to see them.

This can easily be done with ANNOTATE. The client stores the URI in 
the annotation saying where it got the part from. When it goes to 
retrieve the message for further editing, it grabs the annotations 
and can decide which parts it does and does not need to retrieve. I'm 
looking for a way to do this without ANNOTATE. As I asked before:

>>However, I want to allow the client to retrieve that message from 
>>the server for further editing without having to download the 
>>included parts. I also want the client to be able to discover the 
>>URI of the included part when it retrieves the message for editing. 
>>And finally, I'd prefer not to expose information about the 
>>structure of my IMAP store to a recipient of the message. So the 
>>question I'm asking is:
>>

I understand that something similar can be done with embedded URI and 
an ASSEMBLE/EXPAND command. That's got a different set of problems 
and a different set of side effects. The server needs a lot more 
knowledge and does lot more of the work.

I understand that something similar can be done by having the server 
keep track of this and having an UNCATENATE command. That again makes 
the server have a lot more knowledge of the message and makes the 
server do a lot more work. (And introduces *yet another* new command 
for the client to deal with.)

Really, would anyone like to take a shot at answering the question I 
asked? The answer might be "There's no way to do it that doesn't run 
into the same security problems or additional knowledge on the part 
of the server that other solutions face." But, please, can someone 
answer the question I asked?

pr

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 13 Apr 2004 15:58:58 +0200

> answer the question I asked?

Well, I can't answer your question, but sometimes an approximate 
solution will do...

In this case, perhaps CATENATE could generate a hash of each included 
body-part, and record that body part in a MIME header? A smart client 
could look at the hash and know whether it already has downloaded the 
same body-part elsewhere.

Content-MD5 is one possibility. Content-ID using a specified ID format 
based on SHA-1 is another. Adding something to Content-Type that will 
be included in FETCH BODYSTRUCTURE is a third.

(IMO, this has no chance to work interoperably unless there's a test suite.)

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 13 Apr 2004 16:12:14 +0200

> I want CATENATE to result in a sendable message on the server.

Why, exactly?

> parts.

The client needs to download those parts it wishes to edit or display, 
and which it doesn't have. That seems orthogonal to whether those parts 
are original to that message or were glued it from some other message.

> I also want the client to be able to discover the URI of the included 
> part when it retrieves the message for editing. And finally, I'd 
> prefer not to expose information about the structure of my IMAP store 
> to a recipient of the message. So the question I'm asking is:
>

Perhaps using the hashes I mentioned. When the client sees MD-5/SHA-1 
value xyz and has seen that before, it can remember where it saw xyz.

Arnt