Subject: RE: [lemonade] Re: IMAP as a submit server

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Date: Tue, 10 Jun 2003 12:01:27 -0500

Mark, 

- What happens if the body is deleted off the IMAP server before the external component is resolved by the submit server?  How is this race condition resolved?   (use case: User forwards big message and then deletes the original)  

Greg V.

-----Original Message-----
From: Mark Crispin [mailto:[email-masked]]
Sent: Monday, June 09, 2003 7:21 PM
To: [email-masked]
Subject: Re: [lemonade] Re: IMAP as a submit server

On Mon, 9 Jun 2003, Doug Royer wrote:
> I really do not understand your points below (I hope I did not cut
> out any thing important). It seems to me as if your points are
> in conflict with each other.

I hope that I can explain.

> If I give a submit server my authentication credentials, then it is
> in effect an IMAP client - with FULL IMAP access anyway.

I am not proposing that that be done; or rather, I am not proposing that
that be done when talking to an enhanced IMAP server.  If you have to use
a non-enhanced IMAP server, you have no choice but to do that.

> Are you
> proposing that each submission be given a unique authentication so
> that the IMAP server knows to give 'this' and only 'this' message
> or meassage-body-part to 'that' IMAP client?

Not exactly.  Each IMAP URL (assuming that URLs are the means to convey
the tokens) in the submission has its own unique authentication that gives
"this" and only "this" mesage or message-body-part.

The basic idea is that the token is something that only someone with
access to the data can generate.  So, the token needs to contain the
necessary data (user, mailbox name, message UID and part specifier) and
some signature which the IMAP server will recognize.

One way for the client to build this token is to ask the IMAP server for
it (after all, it has to be in communication with the IMAP server to know
that the data is there!).  So, instead of doing:
	tag UID FETCH 122239 BODY[1.3]
it would do something like:
	tag UID GENTOKEN 122239 BODY[1.3]
and insert the result into the message being submitted.  You can imagine
adding things like expiration times, or tokens that only the submit server
can use, etc.

The submit server would use a SASL authenticator to get at it.  In other
words, it would all be done via the AUTHENTICATE command.

> In order for the IMAP server to know that only this MIME body part
> should be given to 'this' submit server for the purpose of doing
> an in-line expansion would require the IMAP client tell the IMAP server
> each and every time that 'this' MIME object can be shared.
> This looks like a lot of round trips and complication to me.

Not if we make it possible to coalesce requests, such as:
	tag UID GENTOKEN (122239 BODY[1.3] BODY[1.4])(121232 BODY[3.2])

> It seems to me that if an IMAP server were also a submit server
> then it would reduce any cross-vendor submit/imap-server security
> holes. And it would reduce the ACLs (or whatever is used) control
> complexities which again I think would reduce the security holes.

I don't see how you came to that conclusion.

By making an IMAP server do submit you're putting all your security eggs
into one monolithic basket.  Crack any egg and everything is lost.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 10:41:36 -0700 (Pacific Daylight Time)

On Tue, 10 Jun 2003, Vaudreuil, Greg M (Greg) wrote:
> - What happens if the body is deleted off the IMAP server before the
> external component is resolved by the submit server?  How is this race
> condition resolved?  (use case: User forwards big message and then
> deletes the original)

IMAP-as-submit has the same race.

More specifically, the same thing that happens if the body is deleted from
the IMAP server between the time that the IMAP-as-submit client gets the
idea to make the request and the request is actually processed.  IMAP is
asynchronous, and there can be multiple simultaneous IMAP sessions.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Tue, 10 Jun 2003 12:48:01 -0500

On 6/10/03 at 10:41 AM -0700, Mark Crispin wrote:

>>- What happens if the body is deleted off the IMAP server before 
>>the external component is resolved by the submit server?  How is 
>>this race condition resolved?  (use case: User forwards big message 
>>and then deletes the original)
>
>More specifically, the same thing that happens if the body is 
>deleted from the IMAP server between the time that the 
>IMAP-as-submit client gets the idea to make the request and the 
>request is actually processed.  IMAP is asynchronous, and there can 
>be multiple simultaneous IMAP sessions.

But an IMAP server could note the reference to the body from the 
to-be-sent message and not "really" delete that body until the 
reference count gets to zero. Or it could immediately copy the part 
upon upload of the to-be-sent message if it really had no other 
choice. But the submit server has no such knowledge and therefore is 
SOL if the body part disappears asynchronously.
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Rick Block <[email-masked]>
Date: Tue, 10 Jun 2003 12:25:33 -0600

Does the "IMAP as submit protocol" proposal imply that clients
using it will use two different submit protocols, IMAP for
messages containing bits of others and SMTP for messages
originated in toto by the client?

It seems to me either the answer to this question is:

yes, in which case I would think the client developers would
not be very happy (two different protocols for submitting
messages, based on the context of the composition?)

or

no, in which case it would seem the IMAP "submit" capabilities
must be a superset of SMTP (for example, it would seem
IMAP submit would have to support the client side of quota
if not lemonade's quota by media)

In addition, what shall an IMAP server do if an SMTP submit
it's been requested to do (via some new IMAP operation) fails,
or will the IMAP server be required to do the resultant SMTP
submit synchronously so any SMTP error can be conveyed back
to the original client with an IMAP response?

I understand the appeal of an IMAP submit mechanism, but it
seems this effectively requires completely merging IMAP and SMTP.
Perhaps we should be talking about an entirely new protocol that
really is a combination of IMAP and SMTP and could be implemented
by a server that effectively proxies independent (regular) IMAP
and SMTP sessions.

Rick Block
Avaya Inc.

---

From: Doug Royer <[email-masked]>
Date: Tue, 10 Jun 2003 12:29:27 -0600

> asynchronous, and there can be multiple simultaneous IMAP sessions.

Currently that problem already exists in IMAP servers. It has nothing
to do with if IMAP is also a submit server. Currently one session could
delete a message while another session uploads it. I'll bet all IMAP
servers have a instance counting or message locking code to deal with
this problem already.

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 12:46:59 -0700 (Pacific Daylight Time)

On Tue, 10 Jun 2003, Pete Resnick wrote:
> choice.

No it can't.  You're making the assumption that the deletion is occuring
in the same IMAP server session as the one doing the submit.  You can't
make that assumption.

Nor can you assume that separate IMAP server sessions can communicate this
information to each other.  There is nothing in the IMAP specification
that requires this.

There is nothing that you can do to prevent the timing race; refer to RFC
2180 for some of the gory details.  All you can do is make it less likely.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 12:51:58 -0700 (Pacific Daylight Time)

> seems this effectively requires completely merging IMAP and SMTP.

I agree, and especially with the earlier observation that IMAP would have
to be a superset.

> by a server that effectively proxies independent (regular) IMAP

This is a possible solution, and actually becomes feasible if it is
specifically oriented towards mobile devices (IMHO it is pointless for
larger devices).

One very desirable aspect of this is that any piece of it (IMAP, SMTP, or
this new protocol) can be replaced without harm to the other two. Or, if
it turns out that a new generation of mobile devices needs something more
powerful, that another proxy protocol can be invented for those; once
again with no harm to IMAP or SMTP.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 12:54:33 -0700 (Pacific Daylight Time)

On Tue, 10 Jun 2003, Doug Royer wrote:
> delete a message while another session uploads it.

Correct.

> I'll bet all IMAP

Not correct, unfortunately (OK, *highly* unfortunately).  Refer to RFC
2180.

Some servers have this (mine does), but some others do not.

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Tue, 10 Jun 2003 14:30:28 -0600

> in the same IMAP server session as the one doing the submit.  You can't

You pointed out in a similar response that your IMAP server does in fact
do locking or instance counting to solve these  exact issues across
multiple sessions. Why if it happens to be called 'submit' and
not 'fetch' would it make any difference to the locking?

> 2180 for some of the gory details.  All you can do is make it less likely.

There is no reason that a submit server could not add the requirement
that the server address this issue. Connecting to a separate server
and trying to maintain a lock or instance count across separate
servers seems much more complex than across sessions in the same
server.

--

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 13:47:02 -0700 (Pacific Daylight Time)

On Tue, 10 Jun 2003, Doug Royer wrote:
> multiple sessions.

Be careful about reading too much into what I said.  My server does the
"ghosting" as described by RFC 2180.  However, that "ghosting" is only
guaranteed for sequence-number style access; whereas a submit server is
almost certainly going to do UID style access.

With UID style access, you don't can't count on there being any "ghosts";
you get holes.

> Why if it happens to be called 'submit' and

SUBMIT vs. FETCH does not matter.  Sequence number vs. UID matters.

Since SUBMIT is likely to use UIDs, you can't assume from one command to
the next that a message won't disappear.  The specification is quite firm
on that point.  Nor is there any aspect of IMAP that "locking" could hinge
on when UIDs are used.

> that the server address this issue.

That's easier said than done.  It isn't just *adding* a requirement to
SUBMIT; it is *repealing* a requirement in IMAP.

> Connecting to a separate server

Let's make this simple; there is no "lock" now that you can rely upon in
IMAP.

The best that there is in IMAP today is that, when sequence numbers are
used, an expunge can't come through except at certain commands.  If you
are lucky enough to use a server that "ghosts" *AND* if you use sequence
numbers instead of UIDs (meaning that you have to keep those sequence
numbers current -- you can't rely on a static value like UIDs), then you
can assume that the message won't go away between the last fetch and the
submit.  That's a lot of conditionals, and I don't think that you'll like
the implications if you have to implement it.

Frankly, if you want this kind of lock, you're better off using POP3 than
IMAP.  Unlike IMAP, POP3 gives you those guarantees.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Tue, 10 Jun 2003 16:34:16 -0500

On 6/10/03 at 12:46 PM -0700, Mark Crispin wrote:

>No it can't.  You're making the assumption that the deletion is 
>occuring in the same IMAP server session as the one doing the 
>submit.  You can't make that assumption.

I am making no such assumption.

>Nor can you assume that separate IMAP server sessions can 
>communicate this information to each other.  There is nothing in the 
>IMAP specification that requires this.

But there is nothing in the IMAP specification that prevents it, and 
a server which would support such functionality would likely want to 
support this feature.

>There is nothing that you can do to prevent the timing race; refer 
>to RFC 2180 for some of the gory details.  All you can do is make it 
>less likely.

There is nothing preventing a server from implementing record locking 
on the bodies, and nothing preventing it from implementing reference 
counts or copies.

pr

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 14:50:52 -0700 (Pacific Daylight Time)

> a server which would support such functionality would likely want to

To the contrary; the IMAP specification is emphatic on the circumstances
in which an unsolicited EXPUNGE can occur.  You want to stop that form of
communication, and you need a way to communicate that, and it needs to
interoperate with existing software.

The only protection -- and even that is limited (see RFC 2180) -- is if
you use sequence numbers instead of UIDs and do NOT use any command other
than FETCH (not UID FETCH), STORE (not UID STORE), and SEARCH (not UID
SEARCH).  The proposed IMAP SUBMIT command could also have that
prohibition, although it's only necessary if you want to do multiple
SUBMITs.  However, this doesn't help if you use UIDs.

> on the bodies, and nothing preventing it from implementing reference

To accomplish this, there needs to be a new IMAP extension which creates
those semantics.  Since IMAP servers don't have clairvoyance, such an
extension is also needed to identify the areas to be locked.  There also
needs to be some means to unlock.

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Tue, 10 Jun 2003 17:09:50 -0600

> needs to be some means to unlock.

With IMAP as the submit program, it simply has to not physically remove
any object until all operations on the object are complete. Currently
if I delete an object then try to access it with another session, I get
an error. I do not see anything is needed in the protocol, it would
all be new text in a new spec saying - do that.

With another non-IMAP program as the submit server, yes, but that
adds another round trip call from the client to the IMAP server
- 'lock this object'.

Perhaps a proxy service that sits on top of both IMAP and an SMTP
server that has been somewhat proposed is the solution. But clearly one
client application communicating to two servers to perform one unified act
of forwarding a message is going to be a much more complicated
protocol.

--

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 16:47:57 -0700 (Pacific Daylight Time)

On Tue, 10 Jun 2003, Doug Royer wrote:
> an error.

How is "it" supposed to know when "all operations on the object are
complete"?

> I do not see anything is needed in the protocol, it would

Consider the following situation:

Client A and client B have simultaneous sessions to the same mailbox.
Client A identifies some text that it wants to forward.  In between the
time that it identifies that text, and issues the SUBMIT command, client B
expunges the text.

To prevent this, client A must instruct the IMAP server, via a locking
command, not to permit expunges of the data; and subsequently it must
release the lock when it no longer needs that data.

Unless aided by a human behind both client A and B knowing "I shouldn't do
that", client B does not have clairvoyance to know not to do the harmful
thing.  Nor does client B's IMAP server know what client A is up to, and
vice versa.

You could try to prohibit expunge in shared sessions.  Try convincing the
enterprises that a cell phone email client is worth breaking all their
Outlook users.

> adds another round trip call from the client to the IMAP server

Sorry, you need the "lock this object" no matter what you do if you are
concerned about this problem.

> server that has been somewhat proposed is the solution.

If we can settle upon this as the course of action, let's declare peace
and go home.  From here, it looks like a win-win situation for everybody.
Win-win is always good.

> But clearly one

The complexity is there no matter what you do.  But the proxy service idea
has the advantage of moving the complexity away from the mobile device.

The proxy has an additional advantage in that its protocol can be tailored
explicitly for mobile device needs without having to do battle with the
SMTP and IMAP protocol guys.  That is a good thing from everybody's
perspective.

It also eliminates the locking question, solves SMTP and IMAP chattiness,
and solves the multiple-source problem.

Sounds like everybody should be happy.  Can we all sign off on this and
drink beer together?

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Tue, 10 Jun 2003 18:33:05 -0600

> How is "it" supposed to know when "all operations on the object are

Simple reader/writer locks - that is the exact scenario they
were designed to solve. Nothing complex needed. And no protocol
changes needed.

Always do a write lock on an object if you are modifying/deleting.
Always do a read lock on an object if you are not modifying/deleting.
Allow only one writer lock per object.
Allow multiple reader locks per object.
All reader locks MUST wait for any writer lock to be done.
A writer lock MUST wait if there are any reader locks.
All reader locks MUST be prepared for no-longer-exists error.

No modifications to the protocol needed. It is all code.

--

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 17:53:00 -0700 (Pacific Daylight Time)

> were designed to solve. Nothing complex needed. And no protocol

How are these locks to be applied without protocol changes?

> Always do a write lock on an object if you are modifying/deleting.

What is a "write lock" in the context of RFC 3501?

> Always do a read lock on an object if you are not modifying/deleting.

What is a "read lock" in the context of RFC 3501?

> All reader locks MUST be prepared for no-longer-exists error.

I know how locks work.  I've been using these sorts of locks for 25 years.
I also invented IMAP, and edited RFC 3501.

Please do not treat me like a ignorant person.

> No modifications to the protocol needed. It is all code.

I suspect that you have not studied RFC 3501 very carefully.  If you had,
you would have observed that:
 1) There may be multiple IMAP sessions operating on the same mailbox.
 2) IMAP sessions consist of multiple IMAP commands.
 3) Each IMAP command is an atomic operation.
 4) Between any two IMAP commands in a session, commands in another
     session -- on the same mailbox -- may take place.

IMAP is unlike POP3, which only permits a single session on a mailbox.
Consequently, there is *no* lock of the sort that you envision, nor is
there any way for there to be such a lock.

IMAP, by its definition, has no such locks; nor can such locking be done.
It is expressly authorized in the protocol to do simultaneous read/write
access.  Clients exploit this capability.

If you feel that it's a deficiency that IMAP lacks such locks, you may be
right.  But saying "it's just programming" doesn't make it so.  If you
want locks in IMAP, it has to be added at the protocol level.

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Tue, 10 Jun 2003 19:08:20 -0600

> How are these locks to be applied without protocol changes?

The need to share objects that can be deleted out from under
you is not unique to IMAP. It is an server implementaiton detail.
And it does not need to be in any existing IMAP or IMAP related
RFC to be discussed on this list.

This is not the IMAP mailing list. This is a mailing list
to discuss among other things ways of solving a problem for
clients that have somewhat low speed and expensive connections.

One option is to extend IMAP to allow it to submit. I am also
sure that you will not see this discussion or my comments
in an existing IMAP or IMAP related RFC.

 > ...

--

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 18:23:35 -0700 (Pacific Daylight Time)

> you is not unique to IMAP. It is an server implementaiton detail.

Sorry, you are wrong.

It is part of the IMAP protocol specification that shared objects can be
deleted out from under you.

Refer to RFC 2180, section 4.1, and RFC 3501, section 5.2.  Pay careful
attention to text related to the EXPUNGE response.

> RFC to be discussed on this list.

The contents of IMAP and IMAP related RFCs are very relevant when you are
talking about *violating* the protocol without making a change to the
protocol to allow this violation.

In the case of locking, you are talking about adding new assumptions which
current do not exist, and deleting other assumptions that do exist.  Yet
you seem to think that this can be done without changing the protocol.

> clients that have somewhat low speed and expensive connections.

If you expect IMAP to be part of that solution, then you have to consider
the characteristics of IMAP.  You can't pretend that these considerations
don't exist.

It's too late to remove unsolicited EXPUNGE responses from the IMAP
protocol.  The time to do that was about 18 years ago.  It's there now,
clients use it, and users depend upon it.

> in an existing IMAP or IMAP related RFC.

If you expect IMAP to provide any form of locking, it will have to be in
an IMAP related RFC.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 18:36:37 -0700 (Pacific Daylight Time)

> deleted out from under you.

Not only is it a part of the protocol, it is a *feature* of the protocol.

Like other features, it is sometimes a good thing and sometimes a bad
thing.  The important thing is that it *is*.

We have a timing race, in which data may disappear between the time that
it is identified for submission and when the submission entity actually
gets around to fetching it.

One possible response is to ignore the problem.  That isn't as
unreasonable as it sounds.  It will almost never happen except in the case
of user error.  There is ample precedent for "we aren't going to bend over
backwards to stop someone, who is determined to lose, from losing."

Another possible response is to create a mechanism to solve the problem.
That mechanism will require the introduction of locking semantics into the
IMAP protocol; at a minimum, it is necessary to have a means to disable
shared expunge, and subsequently to re-enable it.

Yet another possible response is to do an end run around the problem, by
using a proxy server.

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Wed, 11 Jun 2003 14:32:18 -0600

> you seem to think that this can be done without changing the protocol.

It can, I did it for several of my customeres in the past. Yet
no one has a clue because it is totally in the implementation.

And YES, we are talking about updating something, perhaps some
IMAP implementation considerations section, and I am arguing not
the over the wire protocol.

--