Subject: RE: [lemonade] Life on the List, and Vienna

From: "Jonathan T. Hawkins" <[email-masked]>
Date: Mon, 09 Jun 2003 10:29:16 -0700

Hi Mark,

I work with Chris Newman.  He is extremely busy working on our next
release. I'll ping him for a response.

I agree with your points. SMTP delivery to an INBOX would be more
reasonable in that it provides a distributed architecture in a "store
and forward" design.  Also, MTAs are typically located closer to
end-users, where as the Mail Stores are locked up in a central data
center.  Most end-users shouldn't that the message isn't immediately in
their OUTBOX. A few second (at most) propagation delay should suffice.

Also.. I almost violently agree with the idea of an IMAP data delivery
enhancements.  Today's existing IMAP could be better enhanced for large
media objects over bandwidth constrained networks.  On a mobile mail
user agent device, it is absolutely painful to pull down media content
over the air, repackage, and then resubmit back over the air.  If I
could just forward the message, it would be a much better user
experience.

At the moment, I'm not quite sure which is more desirable: IMAP delivery
(many folks against it) or IMAP URL ACLs (complicated).  The URL
ACL-based IMAP access presents some complications in access managment
for a site.  The ACL aging policy might be something worth looking at.

Another option would be to extend SMTP to reference an IMAP URL?  In
this manner, the server-side Authenticated SMTP session could provide
the SASL mechanism to retrieve the IMAP data.

Jonathan

Date: Sun, 8 Jun 2003 18:44:04 -0700 (PDT)
From: Mark Crispin <[email-masked]>
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
cc: Dave Crocker <[email-masked]>,
   Eric Burger <[email-masked]>,
   "IETF LEMONADE (E-mail)" <[email-masked]>
Subject: RE: [lemonade] Life on the List, and Vienna

On Sun, 8 Jun 2003, Vaudreuil, Greg M (Greg) wrote:
> I (with Greg White) plan to have a revised future delivery submission
> draft, reflecting the agreed upon changes at the last IETF...  No IMAP

> outbox, keep within SMTP-submit framework, other edits.

I think that a successful IMAP-aware delivery-submission mechanism will
have the following characteristics:

1) Keep IMAP and POP out of the MTA and MSA business.

2) Allow the submitted message to reference data available upon multiple

   IMAP or POP servers.

3) Not depend upon having unlimited access to the user's IMAP or POP
data.

The people who argued at the last IETF for an IMAP outbox focused on
action item (3) -- which I agree is important -- but chose to disregard
(1) (in spite of this principle appearing in both the IMAP and POP RFCs)

and completely failed to consider (2).

It is very tempting to "just have an outbox in IMAP" that, being the
IMAP
server, can grab all the needed data as local disk access.  However,
besides being a red flag for IMAP implementors, it ignores existing
server
facilities in which the user's data isn't on a single IMAP server.

I recommend a mechanism which I brought up briefly (albeit was shouted
down due to a misunderstanding of what I was going to say) at the last
IETF:
        We need a mechanism (ala IMAP URL) which, when presented
        to an IMAP server, would deliver a particular data item
        from a particular mailbox owned by a particular user
        *without* providing access to any other data in the
        mailbox.

This mechanism would be used by the submission server (which some of us
want to be a different machine from the IMAP servers!) to obtain the
data from the IMAP server.

A possible mechanism is via SASL.  Details of how such a mechanism would
work, what security mechanisms it would have, etc. to be filled in (e.g.
by experts such as Chris Newman, who may already be working on this).

The amount of effort to do this is no more than the effort to do an
outbox
in IMAP; and I suspect that it would be substantially less.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Randall Gellens <[email-masked]>
Date: Mon, 9 Jun 2003 14:24:51 -0400

>  the SASL mechanism to retrieve the IMAP data.

We had a lot of discussion on this in San Francisco.  Essentially, 
there are two camps:
	- Add submission to IMAP
		- Use IMAP Outbox or Drafts folder
		- Store desired Submit (port 587) envelope in annotation
		- Can include references to IMAP parts
		- References to IMAP parts are expanded on submission
		> Requires Submit server to trust IMAP server, or adding 
Submission to IMAP server
		> Need to be careful that all enhancements are done by using 
Submit mechanisms, otherwise we end up defining two parallel 
submission mechanisms, one for Port 587 and one for IMAP.

	- Add IMAP (and maybe other) URLs to Submission (port 587)
		- Allow arbitrary content to be incorporated into messages 
without first downloading by client
		> Requires IMAP server to trust Submit server

I apologize if I misstated either camp.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Arguments are to be avoided; they are always vulgar and often convincing.                                     --Oscar Wilde
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 11:32:35 -0700 (PDT)

On Mon, 9 Jun 2003, Jonathan T. Hawkins wrote:
> On a mobile mail

I think that everybody agrees on this point.

> (many folks against it) or IMAP URL ACLs (complicated).

I think that the complexity is the same with both; however the latter is
more functional.  No matter how you look at it, you need a mechanism to
have "insert such-and-such here" tokens in a message.

> the SASL mechanism to retrieve the IMAP data.

Something in this line is the right track, although obviously refinements
and functional enhancements are necessary.

-- Mark --

---

From: "Jonathan T. Hawkins" <[email-masked]>
Date: Mon, 09 Jun 2003 11:46:16 -0700

> I think that everybody agrees on this point.

Ok.. great

> have "insert such-and-such here" tokens in a message.

I'll sit back and let my accomplished collegues figure out this one.

> and functional enhancements are necessary.

I'd like to test the waters with this idea.  I'll put together a draft.

Cheers..

Jonathan

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 12:00:26 -0700 (PDT)

> 		> Requires IMAP server to trust Submit server

You missed an important insight developed in the second camp:
	It is NOT necessary for the IMAP server to trust the
	Submit server if we create a mechanism (URL or otherwise)
	that permits the Submit server to obtain the data it needs
	without granting it unnecessary access.

If this mechanism is based upon IMAP URLs, then we have something that we
use today, with *all* IMAP servers.  With traditional IMAP servers, the
mobile client would generate a traditional IMAP URL which would require
trust of the submit server.  With enhanced IMAP servers, the client would
generate a new-type IMAP URL that doesn't require trust.

Mobile clients will benefit because they can be deployed without requiring
a change to the IMAP server.  A mobile client that requires a special IMAP
server will face a deployment barrier.  An IT department that refuses to
offer the special IMAP server will effectively block the mobile clients.

A large (and growing number) of deployed mobile clients will serve to
convince IMAP server vendors to support the URL extensions so trust is not
required.  Assuming that the security issues in the URL extensions can be
resolved (and I think that they can), it will be almost a no-brainer to
add them to an IMAP server.

Another important detail about the second camp is that it does not depend
upon all the data for submit being on one IMAP server.  We have facilities
today in which such a dependency would be fatal.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Mon, 9 Jun 2003 15:36:47 -0500

>>		- Use IMAP Outbox or Drafts folder

Yup.

>  >		- Store desired Submit (port 587) envelope in annotation

Or simply include that information in the IMAP command. As yet undetermined.

>  >		- Can include references to IMAP parts

Yup.

>  >		- References to IMAP parts are expanded on submission

Or might simply have some special parameter to an external-body which 
says "expand me if you submit me."

Another advantage is, "Does not require a submit server to know 
anything about MIME structure". Currently, an IMAP server needs to be 
able to parse MIME messages anyway. Current submit servers are often 
strictly 822 entities, and sometimes not even that smart. I don't see 
why we would dump more smarts into a submit server rather than 
including a single additional piece of functionality into an IMAP 
server.

>  >		> Requires Submit server to trust IMAP server, or 
>adding Submission to IMAP server

Yup, though having the submit server trust anything is often a matter 
of "is accessible within the firewall" in many environments today.

>  >		> Need to be careful that all enhancements are done 
>by using Submit mechanisms, otherwise we end up defining two 
>parallel submission mechanisms, one for Port 587 and one for IMAP.

This last one I think is a misrepresentation. It assumes that there 
are to be mechanisms in common. If (as I think about it) these are 
two completely separate services, then there is no reason to worry 
about any parallelisms.

>  >		- Allow arbitrary content to be incorporated into 
>messages without first downloading by client

As I said above, this also requires the submit server to learn about 
MIME structure, as well as learn IMAP access.

>generate a new-type IMAP URL that doesn't require trust.

This is the one that scares the daylights out of me. Right now, my 
IMAP server has all of my mail data and access to my IMAP server 
means access to all of that data. Every time I use IMAP, I *must* 
authenticate by means of an AUTH mechanism. Right now, my SMTP/submit 
server has *no* data, and access to my SMTP/submit server means only 
that I can submit mail, not that I can access any data. Often, an 
ISP's SMTP/submit server does not even need authentication passed to 
it (because it is based on allowing any machine with the correct IP 
address to send without further authentication).

The suggestion for the submit model is to invent a mechanism that has 
access to my IMAP store. You want to be able to grant it access to 
the particular MIME parts needed to compose this message, and then 
only given it access for a small period of time. (I'm inclined not to 
let my ISP's submit server have access for one second more than it 
needs to in order to get the message.) I am very worried about the 
overhead to authenticate the submit server. I am very worried about 
something getting it's mitts on the token I'm going to have to pass 
to the submit server that it's going to have to pass to my IMAP 
server. I am even worried about the fact that I have to pass *any* 
information to my submit server to allow it to get stuff from my IMAP 
server. I don't want my submit server to have *any* information that 
will allow it to access parts of my IMAP store. This sounds like a 
big gaping security hole.

In the case of using IMAP to do the submission, I will already have 
an authenticated session with the IMAP server where *I* communicate 
all of the relevant authentication information directly to my IMAP 
server in a secure manner. Then my IMAP session will have all of the 
appropriate permissions to all of the relevant portions of my IMAP 
store. The IMAP server will be the thing accessing all of the 
assorted parts and collecting them into a message to pass to the 
SMTP/submit server. Now, the only piece of authentication information 
passed from server to server (if it is even necessary) is the one 
time access for my IMAP server on the submit server to send a 
message. This seems *much* more secure, and in most environments, 
giving the IMAP server free-reign to do submissions without an 
authentication token is already allowed.

>a change to the IMAP server.

But it will a require a *huge* change to the security model.

>A mobile client that requires a special IMAP

The reality is that the main focus of this particular service is 
going to require a number of changes to the IMAP server, so I don't 
see this as any significant issue.

>required.

I am not even convinced as a user that I would want such support in 
my IMAP server due to the security risks.

>Assuming that the security issues in the URL extensions can be 
>resolved (and I think that they can), it will be almost a no-brainer 
>to add them to an IMAP server.

I am far less sanguine about the security considerations.

>Another important detail about the second camp is that it does not 
>depend upon all the data for submit being on one IMAP server.

No, it only requires that all the data for submit be accessible from 
a single authenticated IMAP session.

>1) Keep IMAP and POP out of the MTA and MSA business.

is an important characteristic of such a system. I would say:

- Keep SMTP out of the mail store business.

is a much more logical principle, and one I think I can justify.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Randall Gellens <[email-masked]>
Date: Mon, 9 Jun 2003 16:48:42 -0400

>  generate a new-type IMAP URL that doesn't require trust.

Thanks for pointing this out, Mark.  I'd forgotten about this.  Can 
you elaborate on this?  I've been meaning to submit a draft outlining 
the two approaches, and would like to include this.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Reporter, n.:
A writer who guesses his way to the truth and dispels it with a
tempest of words.     --Ambrose Bierce, "The Devil's Dictionary"
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Randall Gellens <[email-masked]>
Date: Mon, 9 Jun 2003 17:06:06 -0400

At 3:36 PM -0500 6/9/03, Pete Resnick wrote:

>>   >		> Need to be careful that all enhancements are done by 
>> using Submit mechanisms, otherwise we end up defining two parallel 

I'm trying to represent both approaches equally and honestly.

Can these remain two separate services?  It seems to me that they 
aren't going to remain separate, or if they do, it will be because we 
end up creating a two-class submission world, where most IMAP clients 
use this new IMAP-based mechanism exclusively, and other clients 
(including POP and those that only send mail) use the SMTP/Submit 
mechanism.  Presumably, we'd end up with some extensions only being 
in one of the two submission mechanisms even though they might be 
useful in the other.  I really don't want that to happen.

>  No, it only requires that all the data for submit be accessible 
> from a single authenticated IMAP session.

I read the comment as saying that you could include information from 
an FTP or HTTP URL, for example, besides IMAP.

-------------- Randomly-selected tag: ---------------
The shepherd drives the wolf from the sheep's throat, for which the
sheep thanks the shepherd as his liberator, while the wolf denounces
him for the same act....  Plainly the sheep and the wolf are not
agreed upon a definition of liberty.               --Abraham Lincoln
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 14:21:27 -0700 (Pacific Daylight Time)

> useful in the other.  I really don't want that to happen.

That will inevitably happen if submit is put into IMAP.

> an FTP or HTTP URL, for example, besides IMAP.

Your reading is correct; you can include information from other kinds of
URLs.

Nevertheless, the requirement that all the data for submit be accessible
form a single authenticated IMAP session effectively requires that all the
data for submit be on one IMAP server (leaving aside silliness like
requiring that IMAP servers have built-in IMAP clients).

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 15:22:39 -0700 (Pacific Daylight Time)

On Mon, 9 Jun 2003, Pete Resnick wrote:
> anything about MIME structure".

A submit service is already required to understand headers and MIME, since
otherwise it won't be able to perform the transforms required by 8-bit.

This "advantage" therefore does not buy anything.

> Currently, an IMAP server needs to be
> able to parse MIME messages anyway.

An IMAP server is required to parse MIME as specified for an MUA.  A
submit service has substantial additional requirements which IMAP does not
have.

> Current submit servers are often
> strictly 822 entities, and sometimes not even that smart.

A strictly 822 entity does not offer 8-bit, which I was hearing was an
important requirement for lemonade.

I see no reason why to dump substantial additional requirements into IMAP
servers in order to avoid upgrading submit servers that are already
provably inadequate for the task.

> I don't see

If a submit server does 8-bit, either the smarts are there or it is
non-compliant.

> two completely separate services, then there is no reason to worry

Huh??  You are claiming that, in spite of the fact that there are going to
be two means of doing submission, that there will not be any parallelisms
between the two???

> means access to all of that data.

Yes.  And how would you like an IMAP server with all that power being made
to do processing and reformatting of submit requests?  This opens a vast
hole that you can drive a truck through!

> only given it access for a small period of time.

Yes.  This would not be required, but would be highly desirable.  The
submit process is denied access to any other data.

> I am very worried about the
> overhead to authenticate the submit server.

This is minimal; the token itself carries that.

What about the overhead placed upon the IMAP server to process submits?

> I am very worried about

This is easily solved.  It isn't as if we don't have the intelligence to
work how a way of generating tokens that only a particular entity can use.

Aren't you worried about the IMAP server being compelled to massage data
in ways that no IMAP server has ever done before?

> I am even worried about the fact that I have to pass *any*

So, to solve this security hole, you propose to pass *all* of your
information to your submit server to allow it to get stuff from your IMAP
server.  You propose to provide your submit server with information that
will grant it *full* access to your entire IMAP store -- because you want
the IMAP server to *be* the submit server!

> SMTP/submit server.

This in effect makes the IMAP server be the submit server.  Put another
way, a separate "SMTP/submit server" in this model is a completely
extraneous step with no apparent purpose.

> But it will a require a *huge* change to the security model.

Not at all.  As far as the client is concerned, it's just URLs.

> see this as any significant issue.

Wrong.

It can be deployed with *no* changes to any IMAP server, albeit with the
disadvantage that you'll have to give IMAP server authorization
credentials to the submit server.

The only enhancements to the IMAP server would be: (1) a command to
generate the token (URL or otherwise), and a SASL mechanism to return the
data given the token.

> my IMAP server due to the security risks.

I am not convinced as an IMAP server implementor that I would want submit
code (not to mention SMTP client code and outgoing SMTP connections) in
my IMAP server due to the security risks.

> I am far less sanguine about the security considerations.

I am horrified by the security considerations of making an IMAP server do
submit.

> a single authenticated IMAP session.

I debunked this statement in another message.

> is an important characteristic of such a system.

An IMAP server currently does nothing more than access data from a mail
store, and export that data in various ways to an IMAP client.

Your proposal would, for the first time, require that an IMAP server be in
the business of building RFC 2822/MIME compliant mesages from input which
is not strictly compliant, including assembly of those messages with data
from the mail store.  It would also require that the IMAP server contain
an SMTP client, to pass on the message to the MTA.

There is also another security hole in your model that I just realized.
Any submit request from the mobile client which contains incoming message
data could contain IMAP-submit pointers.  Oops.  The client would have to
be extremely careful to scan for this, and the server would also have to
have to be careful.

This isn't a problem with my counter-proposal, since the pointers would
carry access-authorization data.  Even if a bad guy tricked you into
sending along a pointer of his choosing, the pointer won't work.

> is a much more logical principle, and one I think I can justify.

SMTP is not doing mail store.  Nor is submit.  It is merely using
the existing external-body mechanism to obtain data from a mail store.

-- Mark --

---

From: Doug Royer <[email-masked]>
Date: Mon, 09 Jun 2003 16:47:37 -0600

I really do not understand your points below (I hope I did not cut
out any thing important). It seems to me as if your points are
in conflict with each other.

If I give a submit server my authentication credentials, then it is
in effect an IMAP client - with FULL IMAP access anyway. Are you
proposing that each submission be given a unique authentication so
that the IMAP server knows to give 'this' and only 'this' message
or meassage-body-part to 'that' IMAP client?

In order for the IMAP server to know that only this MIME body part
should be given to 'this' submit server for the purpose of doing
an in-line expansion would require the IMAP client tell the IMAP server
each and every time that 'this' MIME object can be shared.
This looks like a lot of round trips and complication to me.

It seems to me that if an IMAP server were also a submit server
then it would reduce any cross-vendor submit/imap-server security
holes. And it would reduce the ACLs (or whatever is used) control
complexities which again I think would reduce the security holes.

Mark Crispin wrote:

> ...
> ...
 > ...
> ...
> ...

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 17:21:19 -0700 (Pacific Daylight Time)

> in conflict with each other.

I hope that I can explain.

> in effect an IMAP client - with FULL IMAP access anyway.

I am not proposing that that be done; or rather, I am not proposing that
that be done when talking to an enhanced IMAP server.  If you have to use
a non-enhanced IMAP server, you have no choice but to do that.

> Are you

Not exactly.  Each IMAP URL (assuming that URLs are the means to convey
the tokens) in the submission has its own unique authentication that gives
"this" and only "this" mesage or message-body-part.

The basic idea is that the token is something that only someone with
access to the data can generate.  So, the token needs to contain the
necessary data (user, mailbox name, message UID and part specifier) and
some signature which the IMAP server will recognize.

One way for the client to build this token is to ask the IMAP server for
it (after all, it has to be in communication with the IMAP server to know
that the data is there!).  So, instead of doing:
	tag UID FETCH 122239 BODY[1.3]
it would do something like:
	tag UID GENTOKEN 122239 BODY[1.3]
and insert the result into the message being submitted.  You can imagine
adding things like expiration times, or tokens that only the submit server
can use, etc.

The submit server would use a SASL authenticator to get at it.  In other
words, it would all be done via the AUTHENTICATE command.

> This looks like a lot of round trips and complication to me.

Not if we make it possible to coalesce requests, such as:
	tag UID GENTOKEN (122239 BODY[1.3] BODY[1.4])(121232 BODY[3.2])

> complexities which again I think would reduce the security holes.

I don't see how you came to that conclusion.

By making an IMAP server do submit you're putting all your security eggs
into one monolithic basket.  Crack any egg and everything is lost.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Mon, 9 Jun 2003 23:07:26 -0500

On 6/9/03 at 3:22 PM -0700, Mark Crispin wrote:

>A submit service is already required to understand headers and MIME, 
>since otherwise it won't be able to perform the transforms required 
>by 8-bit.

1. There is a very big difference between being able to understand 
CTE's, things of type "multipart", and things of type "message" 
versus being able to generate an appropriately formatted MIME message 
from URLs and external-body parts. There is already plenty of code in 
the IMAP implementations I know about to get through structures much 
more complicated than any current submit server has to deal with.

2. Many folks do (and the proposed IMAP servers could) use submit 
servers that don't understand MIME. That is to say, many folks use 
plain old (non-ESMTP) SMTP servers to submit on port 25. Sure, that 
might mean your stuck with 7bit, or you might be stuck with an ESMTP 
server that cannot down-convert 8-bit (perfectly legal according to 
the 8BITMIME spec, and perfectly sensible in an environment where 
there will likely only ever be 8-bit), but it means that you can use 
a very stupid SMTP server without having it understand MIME structure.

>This "advantage" therefore does not buy anything.

Nonsense. You can use completely off-the-shelf SMTP servers for the 
submit end. My contention is that folks in "mobile client world" 
(read: 3G*, etc.) won't be able to use an off-the-shelf IMAP server 
anyway.

>I see no reason why to dump substantial additional requirements into 
>IMAP servers in order to avoid upgrading submit servers that are 
>already provably inadequate for the task.

The "task" in this case would be to accept (via SMTP) and forward an 
already formed MIME message. I would say that almost any SMTP server 
is up to that task.

>If a submit server does 8-bit, either the smarts are there or it is 

No, that's simply wrong. See RFC 2476 and RFC 1652. You can accept 
8-bit without guaranteeing down-convert.

>>>Need to be careful that all enhancements are done by using Submit 
>>>mechanisms, otherwise we end up defining two parallel submission 
>>>mechanisms, one for Port 587 and one for IMAP.
>>
>Huh??  You are claiming that, in spite of the fact that there are 
>going to be two means of doing submission, that there will not be 
>any parallelisms between the two???

No, I am simply claiming that adding submit to IMAP does not 
necessarily entail any new features for submit servers, and that any 
additions to submit servers can occur independently of anything going 
on with IMAP.

>Yes. And how would you like an IMAP server with all that power being 
>made to do processing and reformatting of submit requests? This 
>opens a vast hole that you can drive a truck through!

I'm sorry, I simply don't see the hole. Is it that because this IMAP 
server can now open a socket *to* port 25 (or 587), it can suddenly 
do something so vastly more horrible with my data than could be done 
before? And this is somehow supposed to be *worse* than my SMTP 
server being given access to port 143?

>>The suggestion for the submit model is to invent a mechanism that 
>>has access to my IMAP store. You want to be able to grant it access 
>>to the particular MIME parts needed to compose this message, and 
>>then only given it access for a small period of time.
>
>Yes.  This would not be required, but would be highly desirable.

I can't see the security considerations section of that document not 
requiring it.

>>I am very worried about the overhead to authenticate the submit server.
>

And what about my IMAP server which requires Kerberos tokens? Or 
requires other challenge-response kinds of tokens? Won't there be 
some significant overhead there? Or are SMTP servers are going to be 
given "special" (i.e., easier) access to the IMAP server?

>What about the overhead placed upon the IMAP server to process submits?

- The amount of overhead to *access* the various parts is a wash in 
either case.

-The authentication step is a wash if (in IMAP submit) the SMTP 
server requires such a token and (in SMTP gather submit) the submit 
server only has to authenticate for a single part. If there are 
multiple parts, or the SMTP server doesn't require additional 
authentication, then the IMAP submit method is better.

- The overhead to do the proposed collection in comparison to the 
overall overhead for other operations on a submit server is rather 
large. The overhead to do submission in comparison to the rest of 
IMAP operations is relatively lower. Remember, the IMAP server is in 
the habit of (and likely tuned for) doing database-query-like hits on 
the message store. Most submit servers I know about are lump-and-dump 
kinds of i/o machines.

>>I am very worried about something getting it's mitts on the token 
>>I'm going to have to pass to the submit server that it's going to 
>>have to pass to my IMAP server.
>
>This is easily solved.  It isn't as if we don't have the 
>intelligence to work how a way of generating tokens that only a 
>particular entity can use.

I am not so confident that (a) we can design this in a rock-solid way 
*and* (b) people will not screw up the implementations of it. I would 
certainly like to see a design before I would call it "easily solved".

>Aren't you worried about the IMAP server being compelled to massage 
>data in ways that no IMAP server has ever done before?

Of course I'm worried about that. But it is a worry of whether the 
IMAP server can handle the load and whether it can do it without 
incorrectly formatting the data. That's different from my worry about 
privacy and security of my data. The latter is much more worrisome.

>>information to my submit server to allow it to get stuff from my 
>>IMAP server. I don't want my submit server to have *any* 
>>information that will allow it to access parts of my IMAP store. 
>>This sounds like a big gaping security hole.
>
>information to your submit server to allow it to get stuff from your 
>IMAP server.  You propose to provide your submit server with 
>information that will grant it *full* access to your entire IMAP 
>store -- because you want the IMAP server to *be* the submit server!

No. Not at all. I want my IMAP server to collect together data that 
it already has access to and, within an authenticated session, send 
it (upon my command) through a single pipe to a SMTP server. The IMAP 
server and the submission server are two separate entities connected 
by that single port 25 (or 587) pipe. In the other proposal, the 
submission server has repeated access to my IMAP store without having 
me be in an authenticated session on the IMAP server.

>>server in a secure manner. Then my IMAP session will have all of 
>>the appropriate permissions to all of the relevant portions of my 
>>IMAP store. The IMAP server will be the thing accessing all of the 
>This in effect makes the IMAP server be the submit server. Put 
>another way, a separate "SMTP/submit server" in this model is a 
>completely extraneous step with no apparent purpose.

No, the separation in the model is key. It means that there is a 
single pipe to the SMTP service out of my IMAP server. My IMAP server 
doesn't need to be concerned with the rest of the SMTP service 
(queuing, MX records, retries, etc.) and my SMTP service doesn't have 
to be concerned with the internals of my IMAP mail store. My IMAP 
submit service effectively becomes a special kind of "collected 
FETCH" which outputs it's response to an SMTP pipe. It is, IMO, much 
closer to what an IMAP server does now than the "collect-from-IMAP" 
mechanism is to what an SMTP server does.

Of course, you could put the two together if you like, but the key 
for me is that they don't *need* to be intimately aware of each other 
as is the case with the other mechanism.

>>>Mobile clients will benefit because they can be deployed without 
>>>requiring a change to the IMAP server.
>>

But it's allowing access to data *without* having an authenticated 
session. This *is* a huge change to the IMAP security model.

>It can be deployed with *no* changes to any IMAP server, albeit with 
>the disadvantage that you'll have to give IMAP server authorization 

1. What I initially meant by the above was that the consumers of 
LEMONADE are going to want other changes in their IMAP servers 
anyway, so there is no particular advantage in having the ability to 
use off-the-shelf IMAP servers.

2. It is completely unacceptable to be handing server authorization 
credentials to the submit server, so an off-the-shelf IMAP server is 
already unacceptable for this service.

On the other hand, an off-the-shelf SMTP server *can* be used for the 
submission end in my model.

>I am not convinced as an IMAP server implementor that I would want 
>submit code (not to mention SMTP client code and outgoing SMTP 
>connections) in my IMAP server due to the security risks.

I think I have given pretty plausible security risk scenarios for the 
other case. I have yet to hear a convincing case for why the threat 
model is *worse* in this case. Please explain.

>I am horrified by the security considerations of making an IMAP 
>server do submit.

And repeating it doesn't make it any more true. Please explain.

>I debunked this statement in another message.

Calling something "debunked" does not make it so. It's a lousy 
debating tactic meant to make people who aren't paying attention 
think a good argument has been made.

Of course, crossing servers does mean that you need some sort of 
"IMAP store" sharing mechanism or some way to forward commands to 
another IMAP server (what Mark derided as "silliness like requiring 
that IMAP servers have built-in IMAP clients"), but those mechanisms 
do exist.

>An IMAP server currently does nothing more than access data from a 
>mail store, and export that data in various ways to an IMAP client.
>
>Your proposal would, for the first time, require that an IMAP server 
>be in the business of building RFC 2822/MIME compliant mesages from 
>input which is not strictly compliant,

"Not strictly compliant"? Explain please.

>including assembly of those messages with data from the mail store.

Yup. Seems to me much simpler code to put into an IMAP server than to 
put into a submit server.

>It would also require that the IMAP server contain an SMTP client, 
>to pass on the message to the MTA.

Something that even the simplest POP or IMAP client has as part of 
its code. We're not talking about rocket science here.

Again, what's the horrible impact here of not "keeping IMAP out of 
the MTA business"?

>There is also another security hole in your model that I just 
>realized. Any submit request from the mobile client which contains 
>incoming message data could contain IMAP-submit pointers.  Oops. 
>The client would have to be extremely careful to scan for this, and 
>the server would also have to have to be careful.

Or we could simply make the "auto-expand" part of this only work in 
the "Draft" or other special "Composition" mailbox. Not exactly a 
hard thing to avoid.

>SMTP is not doing mail store.  Nor is submit.

Sure it is. It's implementing an IMAP client. That seems like a much 
more radical departure than an IMAP server implementing an SMTP 
client.

pr

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jun 2003 22:28:37 -0700 (PDT)

On Mon, 9 Jun 2003, Pete Resnick wrote:
> before?

The whole is that a large and very complicated process (a message
composer), not to mention an SMTP client, neither of which have anything
to do with message access, are now running on the IMAP server machine with
unlimited access to mailbox data.

> And this is somehow supposed to be *worse* than my SMTP

Yes, it is; because the SMTP server won't have unlimited access.

> I can't see the security considerations section of that document not

Why?  It only matters if the IMAP server does not trust the submit server.
There will be many environments in which such trust will exist.

> Or are SMTP servers are going to be

I don't understand how you could have missed this.  That is *exactly* what
is being proposed.  These tokens will carry their own access authorization
credentials.

> -The authentication step is a wash

There is no "authentication step".

> IMAP operations is relatively lower.

You are disregarding the overhead of actually building the message; this
is something that is fundamentally different than anything else that an
IMAP server does.  A submit server, on the other hand, already has to know
how to build messages.

> certainly like to see a design before I would call it "easily solved".

Are you really saying that we don't know how to sign a piece of text in a
way that, when we subsequently look at it, we can tell whether or not that
text was signed by us?

> it (upon my command) through a single pipe to a SMTP server.

You just echoed exactly what I said.

You propose to move the function of a submit (as opposed to SMTP)  server
to the IMAP server.

> single pipe to the SMTP service out of my IMAP server.

So how do you propose sending a message which contains data from multiple
IMAP servers with your single pipe?

> My IMAP server

So why do you want it to have an SMTP client?

> My IMAP
> FETCH" which outputs it's response to an SMTP pipe.

What you call a "collected FETCH" is an 2822/MIME formatted message.
Building such a thing requires an extensive body of software which is not
in IMAP servers today.

> session. This *is* a huge change to the IMAP security model.

Not at all, because the tokens contain their own authentication.

I find it fascinating that you are lecturing me about the IMAP security
model.  The idea of a consolidated one-shot form of access (even UDP
based) is something that had been part of the architecture from the
beginning; this is just the first time that a need has come up for it.

> use off-the-shelf IMAP servers.

Oh?  So you're saying that you want to build mobile devices that won't
work with the installed base?

Are you trying to build a useful product, or is this something useless
like the cell phone web browsers?

> already unacceptable for this service.

It is unacceptable in *some* environments, but not in others.  There are
many environments, e.g. enterprise, in which such trust relationships
exist.

> submission end in my model.

So, in order to have a no-op SMTP server, you want to make huge changes in
IMAP?

And what if the text isn't in IMAP?  Suppose it's on a POP3 server?  Or an
NNTP server?

> model is *worse* in this case. Please explain.

Composing a message requires buffers.  The composer has to do substantial
reformatting in order to ensure that the text complies with SMTP.  The
reformatting can result in text that is substantially larger than the
source.  The composer also has to be capable of negotiating SMTP, and of
handling the error responses (which also require buffers).

All of these are places in which a single faulty consideration can cause a
buffer overflow.

> And repeating it doesn't make it any more true. Please explain.

Unlike you, I've written an IMAP server.  Multiple ones in fact.

A fundamental part of security is isolation of function.  An IMAP server
already has a great deal to do.  You are proposing a completely orthogonal
function to IMAP.  IMAP servers currently have *nothing* in the way of
code to build 2822/MIME texts, nor do they have an SMTP client.

I also have to consider your comments in light of your past statements
with regard to IMAP.  I remember quite a bit of criticism by you to the
effect that IMAP has "too much" functionality compared to POP.

> think a good argument has been made.

Saying it isn't so doesn't mean that it isn't so.

Repeating the same refuted points doesn't cancel the refutation.

Two can play at this game.

Note that I am not claiming that your model won't work.  I am claiming
that your model is functionally limited (no way to incorporate data from
other sources), puts a substantial added burden on IMAP servers with
potential security implications, and damages the IMAP architecture (I as
the IMAP architect should be considered authoritative on that poin).

You on the other hand are going to great efforts to try to claim that my
model won't work, can't work, and shouldn't work.  For some reason, you
very much want to force this on IMAP.  You are refusing to consider an
alternative?

Why?  You are getting the strongest possible pushback of "we don't want to
do it this way."  Why not consider another way?  Why does this offend you?

I can tell you why your proposal offends us; it violates the fundamental
scope of IMAP.  And at least two of the major IMAP server implementors are
terrified of its implications.

I haven't heard of any submit server implementor mentioning such fears.
To the extent that there *are* submit server implementors, they're
wondering "how is this different from SMTP"?  Well, we now have something
that will make submit very much different.

> that IMAP servers have built-in IMAP clients"), but those mechanisms

Where?  Name them.

I have mail at ndcms.cac.washington.edu and panda.com.  There is no trust
relationship between the two.  There is no common password between the
two.

How do I post a message that incorporates data from both servers in your
model?

> "Not strictly compliant"? Explain please.

By definition, a submit server can accept messages that lack such things
as Date:, From:, Message-ID:, etc.

> put into a submit server.

At least two IMAP server developers don't agree with that assessment.

> its code. We're not talking about rocket science here.

There is a difference between a client and a server.  It really doesn't
matter all that much in the overall scheme of things if a client hangs or
crashes or gets a buffer overflow.

It matters a great deal if that happens to a server.  Server code is much
more sensitive.

Now, when you have message composition done by a server, that means that
that sensitive code has to be somewhere.  You seem to want it to be in an
IMAP server.

We, observing that the state of submit (as opposed to SMTP) is rather
embryonic, believe that it belongs in the submit server.  Not only that,
but we feel that the submit server is going to have that capability sooner
or later in any case.

So, the obvious question is, why duplicate the same thing in IMAP?  Is it
as a temporary hack, to avoid having to think about building a submit
server (such as was done XSND or whatever it was called was in qpopper)?

Why bloat IMAP into such an unwieldy mess that it collapses under its own
weight?

> Again, what's the horrible impact here of not "keeping IMAP out of

Why, after 20 or so years of opposition by both the POP and IMAP
communities against being in the MTA business, are we still discussing
this?

This is like the way the Democrats get bonds for some frivilous thing
passed by the voters; just keep on putting the damn thing on the ballot
over and over again until the "correct" vote happens.

Can't we move on?

We don't need this to accomplish the requirement.

> the "Draft" or other special "Composition" mailbox. Not exactly a

That doesn't fix the problem.

Consider "reply including replied-to text".  Consider that text being
HTML.

> more radical departure than an IMAP server implementing an SMTP

It also opens the possibility of the submit (not SMTP -- why do you keep
confusing the two?) server having a POP client, or an NNTP client, or a
FTP client, or an HTTP client, or a Gopher client, or a whatever new comes
up client.

This is a great benefit, with substantial expansion in the future.  What's
more, much of the infrastructure already exists in MIME.

Why should we want to restrict this capability to IMAP, and even worse to
one single IMAP server?

Are you claiming that message submission will never want to use other
protocols?  How much are you willing to wager?

Let's do the message submission right the first time, and not leave the
wreakage of quick-and-dirty hacks behind in other protocols.

Similarly, let's do mobile device email right the first time.  We should
be able to learn from the mistakes of others (such as cell phone web
browsers) and not repeat the same mistakes.  Maybe we'll make new
mistakes, but at least they won't be the same mistakes.

-- Mark --

---

From: Lawrence Greenfield <[email-masked]>
Date: Tue, 10 Jun 2003 09:15:31 -0400

> beginning; this is just the first time that a need has come up for it.

It is!?!? That's really big news to me and I thought I had a pretty good 
understanding of the IMAP architecture.

IMAP seems to be ill-suited for one-shot access. You can't change 
authentication or authorization in midsession, meaning a new TCP session 
will need to be built for each access. The obvious granularity of access 
control, the mailbox, isn't fine grained enough to allow for the sort of 
fetches we're talking about here. Since it's difficult to do partial 
failures, it's not clear how to fail a FETCH that touches one authorized 
piece of information and one non-authorized piece of information.

In contrast, something like HTTP is well-suited for this style of access.

Larry

PS Mark, I think I agree with you, but please confine your technical 
messages to technical subjects and avoid politics.
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Pete Resnick <[email-masked]>
Date: Tue, 10 Jun 2003 12:28:20 -0500

[Mark: I am trying very hard to keep this from being personal and 
rather make these arguments solely on technical grounds. You seem to 
be doing everything to do exactly the opposite. Where I can, I have 
cut out the personal attacks you made in the text below. Please try 
and avoid the personalization in the future.]

On 6/9/03 at 10:28 PM -0700, Mark Crispin wrote:

>>I'm sorry, I simply don't see the hole.
>
>composer), not to mention an SMTP client, neither of which have 
>anything to do with message access, are now running on the IMAP 
>server machine with unlimited access to mailbox data.

- How does the message composer constitute a security hole?

- Having the SMTP client means that anything that already has access 
to the server can send its data through SMTP. How is this a 
significantly larger security hole than having IMAP access to the 
server? And how is it significantly larger compared to giving an 
entirely new process (i.e., the submit server) a completely different 
avenue to get data from the server?

>>>>You want to be able to grant it access to the particular MIME 
>>>>parts needed to compose this message, and then only given it 
>>>>access for a small period of time.
>>I can't see the security considerations section of that document 
>>not requiring it.
>
>Why?  It only matters if the IMAP server does not trust the submit 
>server. There will be many environments in which such trust will 
>exist.

We have been repeatedly beat up for having security considerations 
sections which allow for "insecure behavior so long as there is 
out-of-band trust". I don't think such a thing is going to fly 
(though perhaps it will be implemented that way.)

>>Or are SMTP servers are going to be given "special" (i.e., easier) 
>>access to the IMAP server?
>
>That is *exactly* what is being proposed.  These tokens will carry 
>their own access authorization credentials.

So, the authorization for SMTP servers is going to be *weaker* than 
the authorization for users of the system. That seems like a bad plan.

[WRT overhead]
>There is no "authentication step".

Of course I meant "authorization". Sorry.

>You are disregarding the overhead of actually building the message; 
>this is something that is fundamentally different than anything else 
>that an IMAP server does.  A submit server, on the other hand, 
>already has to know how to build messages.

Unless this is simply referring to the ability to fix up the 
top-level headers of a message, many submit servers know *nothing* 
about how to build messages, so this would be entirely new code 
(along with the ability to retrieve parts from the IMAP server). 
Furthermore, most submit server never do build messages (since they 
are often built correctly by submission clients anyway), hence the 
overhead differential (which is what this whole section is talking 
about) would be much greater.

>>I am not so confident that (a) we can design this in a rock-solid 
>>way *and* (b) people will not screw up the implementations of it. I 
>>would certainly like to see a design before I would call it "easily 
>>solved".
>
>Are you really saying that we don't know how to sign a piece of text 
>in a way that, when we subsequently look at it, we can tell whether 
>or not that text was signed by us?

It is much more involved than that. I have to sign something that the 
server needs to keep track of that the server can verify when it is 
handed this piece of text by a third party. I certainly see how it 
*could* be done. What I'm not sure of is where all of the security 
risks are with such a scheme. It is much clearer to me what the risks 
are from giving an IMAP server access to an SMTP port.

>So how do you propose sending a message which contains data from 
>multiple IMAP servers with your single pipe?

If, by "multiple IMAP servers" you mean "multiple IMAP authorized 
sessions", you cannot send the data through a single pipe unless you 
transfer the data between servers. I have never claimed that this 
proposal is more general. However, since the general use case that's 
folks have in mind is the "forward without download" case, I am 
willing to forgo that additional functionality in favor of what I 
view as a simpler model and better security.

>>My IMAP server doesn't need to be concerned with the rest of the SMTP service
>

Somewhere in either proposal there is going to have to be an 
interface between IMAP and SMTP. For the "SMTP-server-does-the-work" 
proposal, it's when the SMTP server accesses the IMAP server. For the 
"IMAP-server-does-the-work" proposal, it's where the IMAP server uses 
it's SMTP client. Am I missing something in this question?

>Building such a thing requires an extensive body of software which 
>is not in IMAP servers today.

I will leave it to others to judge whether the "extensive body of 
software" will be significantly larger for IMAP servers or for SMTP 
servers.

>>...the consumers of LEMONADE are going to want other changes in 
>>their IMAP servers anyway, so there is no particular advantage in 
>>having the ability to use off-the-shelf IMAP servers.
>
>Oh?  So you're saying that you want to build mobile devices that 
>won't work with the installed base?

Well, this is a completely different topic, but the answer is 
"maybe". This is the whole issue of point 3 in the charter. All of 
the interactions between such servers and clients will be perfectly 
legal IMAP, but the clients will not have to support some optional 
behavior in servers that current clients must account for.

>Are you trying to build a useful product, or is this something 
>useless like the cell phone web browsers?

Please try to discuss and understand the issue before resorting to 
purposeful baiting like the above.

>So, in order to have a no-op SMTP server...

No, not a no-op SMTP server. It has to support all of the *SMTP* 
functionality of routing, queuing, forwarding, etc. (unlike a simple 
SMTP submission client), but we don't have to (in the language of the 
below) "make huge changes to SMTP."

>you want to make huge changes in IMAP?

It is yet to be determined how huge the changes are. And please 
separate out "changes to the IMAP protocol" from "changes to any 
particular implementation of an IMAP server".

>And what if the text isn't in IMAP?  Suppose it's on a POP3 server? 
>Or an NNTP server?

Absolutely, those are not addressed by the proposal. I am inclined 
not to boil the ocean, especially when the requirement we are really 
trying to address is "forward without download."

>>I have yet to hear a convincing case for why the threat model is 
>>*worse* in this case. Please explain.
>
>Composing a message requires buffers.  The composer has to do 
>substantial reformatting in order to ensure that the text complies 
>with SMTP.  The reformatting can result in text that is 
>substantially larger than the source.  The composer also has to be 
>capable of negotiating SMTP, and of handling the error responses 
>(which also require buffers).
>
>All of these are places in which a single faulty consideration can 
>cause a buffer overflow.

And how is this in any way different in the other proposal? Aren't 
*those* security risks identical?

>A fundamental part of security is isolation of function.

Exactly right. There are three functions which have to be accomplished here:

1. Grab a bunch of parts
2. Take those parts and fit them into a template of an 822/MIME message
3. Send that message out through SMTP

An IMAP server already has #1, so sharing that function with  a 
submit server is worsening the isolation situation. Neither service 
has #2, and I would suggest that keeping it with the data retrieval 
engine is better than moving it over to the delivery end of things. 
So it really comes down to #3 being the one place where you have an 
de-isolation of function, and I think that's a pretty contained case.

>I also have to consider your comments in light of your past 
>statements with regard to IMAP.  I remember quite a bit of criticism 
>by you to the effect that IMAP has "too much" functionality compared 
>to POP.

[My apologies to the rest of you for popping out of "de-personalize" mode.]

Your silly characterization of my opinions of IMAP aside (of which 
you have scant little real knowledge), please *don't* try to predict 
what my statements mean in light of what you *think* my opinion of 
IMAP is. Let's have a technical discussion here, not some really bad 
attempt to divine each other's motives. Assume I mean what I say.

>Note that I am not claiming that your model won't work.  I am 
>claiming that your model is functionally limited (no way to 
>incorporate data from other sources)

A point with which I agree.

>puts a substantial added burden on IMAP servers

A point which I think is open for debate.

>with potential security implications

A point with which I complete disagree (insofar as we do a 
comparative analysis of the security risks of each model).

>and damages the IMAP architecture

I have now repeatedly asked for explanation of this supposed 
architectural damage and its impacts. I am still waiting for such an 
explanation.

>You on the other hand are going to great efforts to try to claim 
>that my model won't work, can't work, and shouldn't work.  For some 
>reason, you very much want to force this on IMAP.  You are refusing 
>to consider an alternative?

This is utter nonsense. I have never argued that the other model 
won't work. (I am quite confident it will.) I have only argued that 
it is fraught with security considerations that I don't think have 
been adequately addressed, that it is a much more complicated model 
that has many more changes to the architecture of a submission 
service than it does to an IMAP service, and that it is therefore not 
worth pursuing.

>Why?  You are getting the strongest possible pushback of "we don't 
>want to do it this way."

No, I am getting the strongest possible pushback that *you* don't 
want to do it this way. I have seen other feedback giving preferences 
for each model. (Please, just make the arguments. Don't try to 
justify them by trying to say that there is some collective "we". 
That's just hand-waving.)

>Why not consider another way?

I think that my extensive argument here gives ample evidence that I 
have given the other proposal a great deal of consideration.

>Why does this offend you?

It does not in the least offend me. I have serious technical concerns about it.

>I can tell you why your proposal offends us; it violates the 
>fundamental scope of IMAP.  And at least two of the major IMAP 
>server implementors are terrified of its implications.

I'm sorry, but who is this other IMAP implementor who is "terrified" 
and "offended". I find nothing on the list with such an indication.

>>>Your proposal would, for the first time, require that an IMAP 
>>>server be in the business of building RFC 2822/MIME compliant 
>>>mesages from input which is not strictly compliant,
>By definition, a submit server can accept messages that lack such 
>things as Date:, From:, Message-ID:, etc.

And where would the IMAP server encounter such input in this 
scenario? Are you simply saying that the IMAP server might have to 
fix up the date and unqualified domain names?

>>Yup. Seems to me much simpler code to put into an IMAP server than 
>>to put into a submit server.
>

Well, at least one. I'm waiting to hear from another.

>Now, when you have message composition done by a server, that means 
>that that sensitive code has to be somewhere.  You seem to want it 
>to be in an IMAP server.

Yup.

>We...

You...

>...observing that the state of submit (as opposed to SMTP) is rather 
>embryonic, believe that it belongs in the submit server.

So your belief is that we should introduce an IMAP client and a 
message composition function into the submit service because it is a 
*less* mature and tested service? (Again, we're talking protocol 
here, not particular implementations.) I'd like some more explanation 
of why that makes it a good idea.

>Not only that, but we feel that the submit server is going to have 
>that capability sooner or later in any case.

So the bet is that submit servers are going to have to do full 
message composition anyway. An interesting wager, but I'm not 
convinced this is the direction things are going.

>So, the obvious question is, why duplicate the same thing in IMAP? 
>Is it as a temporary hack, to avoid having to think about building a 
>submit server (such as was done XSND or whatever it was called was 
>in qpopper)?

Instead of characterizing it as a "quick hack", let's try it in a 
less loaded way:

Is the desire to have a composition facility built on top of IMAP for 
the limited functionality of composing a message containing parts in 
the message store, or is the desire to have a full-blown composition 
facility that allows the composition of parts from various places? As 
I said above, I think the former is straightforward and the latter is 
trying to boil the ocean.

>Why bloat IMAP into such an unwieldy mess that it collapses under 
>its own weight?

Why resort to unsubstantiated hyperbole?

>communities against being in the MTA business, are we still 
>discussing this?

Leaving POP out of this (since it doesn't have a facility for "upload 
this message to my server"), I'll again ask for an answer instead of 
the histrionics about the Democratic party.

>We don't need this to accomplish the requirement.

No, it's not necessary. It is sufficient. Why is it an architectural problem?

>>>The client would have to be extremely careful to scan for this, 
>>>and the server would also have to have to be careful.
>Consider "reply including replied-to text".  Consider that text being HTML.

I don't get it. Please explain.

pr

---

From: Mark Crispin <[email-masked]>
Date: Tue, 10 Jun 2003 12:38:12 -0700 (Pacific Daylight Time)

> - How does the message composer constitute a security hole?

A message composer, by its nature, has complex buffer management; it is,
in effect, a text editor.  Text in mail can expand (e.g. due to quoting)
to a much larger size than the original.

SMTP clients also have the issue of text that expands to a much larger
size than the original.  There is an additional complication in dealing
with SMTP server responses.

There are ample opportunities for buffer overflows in such a entity.  All
that is needed is for one of these to be missed.  In some IMAP servers,
that would just give access to all the user's mail; in others, it will
give access to all mail.

> avenue to get data from the server?

First, we're not talking about SMTP; we're talking about submit.  The
current state of the art of submit uses SMTP, but that does not mean that
the two should be confused.

It is a larger security hole to have submit have unlimited access to all
the user's data (and potentially all data for all users) than it is for
submit to be restricted to specific pieces of the user's data.

When submit is part of IMAP, the same entity is both the message composer
and the gatekeeper that purports to restrict the message composer.

> (though perhaps it will be implemented that way.)

That is why we are proposing a way to do things securely, and strongly
encouraging that way.  That doesn't mean that it can not be shortcut when
appropriate.

One of the most common questions that I get is "How do I turn off
mandatory SSL/TLS between my web-based IMAP client and the IMAP server?"
And since the connection is going on a secure channel (and often
localhost), this is not an unreasonable request.

> the authorization for users of the system. That seems like a bad plan.

How would it be weaker?  You are not authenticating the submit client as a
particular user with arbitrary data access; you are only authenticating
its access to a particular datum.  Only an entity which has undergone the
authentication with arbitrary data access is capable of creating the token
for particular-datum access.

This is stronger than sending the text over SMTP, and remember that's the
ultimate fate of that text.

> (along with the ability to retrieve parts from the IMAP server).

Submit would also be entirely new code in an IMAP server.

The difference is that it's appropriate for submit code to be added to a
submit server.  An IMAP server has currently nothing to do with submit.

We're not talking about SMTP vs. IMAP servers.  We're talking about submit
vs. IMAP servers.

> about) would be much greater.

IMAP servers never build messages either.  What overhead differential?

> I am

So, what happens when after we've gone to the effort to do all this
submit-in-IMAP work, and then it turns out that the "simpler model" isn't
good enough?

> it's SMTP client. Am I missing something in this question?

I reiterate: this is not SMTP.  It is submit.

In "submit-server-does-the-work", the submit server knows how to
access IMAP and other sources.

> purposeful baiting like the above.

I'm serious.  This has nothing to do with the company where you work; but
it has everything to do with a related case of mobile devices doing what
is expedient rather than what is right.

I've used (or rather tried to use) several cell phone web browsers, and
they all fall short of their promise (to put it mildly).  Once the
"gee-whiz" aspect fades, it dawns on the user that it's cheaper, faster,
and more reliable to use the cell phone as a modem for a laptop to run IE.

I very much would like to see mobile devices fufill their promise rather
than be expensive toys.  I want to see mobile devices become practical
enterprise devices.

> particular implementation of an IMAP server".

See above about enterprise devices.  If this isn't widely-adopted, it
becomes restricted to one particular vendor or service provider.  That
isn't an enterprise device.

> trying to address is "forward without download."

I don't see this as boiling the ocean.  It's what people are already doing
today.

> >[Composition in IMAP security holes]

No.  Unlike a compromised IMAP server, a compromised submit server would
not have access to any of the other user's data, much less any other
user's data.

This isolation is of critical importance to many people.

> submit server is worsening the isolation situation.

No, it makes it better because the submit server loses access to the parts
it doesn't have a right to have.

> Neither service
> has #2

Correct.

> attempt to divine each other's motives. Assume I mean what I say.

I am not trying to divine your motives.

I am asking you to reconcile your past statements about how IMAP was too
complex and had too much functionality, with your advocacy today of a
proposal to add substantial new complexity and functionality to IMAP.

I would have thought that you would be aggressively on my side on this,
saying "enough is enough, don't add anything new to IMAP."  I was hoping
that this would be the case; you were a person that I was counting on.  I
am bewildered to find that it isn't.

> architectural damage and its impacts. I am still waiting for such an

IMAP is focused on accessing and processing message data.  This, for this
first time, is an operation that is focused on editing and transmitting
data.

The goal of this proposal seems to be to make IMAP be a server-based MUA
(in competition with webmail) fufilling all client needs, rather than an
MUA component.

The component nature of IMAP is desirable.  What goes into an MUA evolves
over time; and by making IMAP be a monolith that necessarily ties IMAP to
a particular state of evolution.

> I have only argued that
> been adequately addressed,

That is unfair.  It is a given that whatever type of token mechanism is
used has to satisfy a checklist of security requirements.  You're
complaining that the checklist hasn't been addressed, when the checklist
doesn't exist yet!

Let's build the checklist first, and then see if there is an answer that
addresses them.  If there isn't, then the proposal collapses.

> that it is a much more complicated model

More complicated, yes, but not much more complicated.

> service than it does to an IMAP service

How can you say this?  An IMAP service has none of the aspects of message
composition and assembly.

You seem to be reacting to the idea to use MESSAGE/EXTERNAL-BODY and
saying "well, this requires MIME parsing, which IMAP has".

But what about if instead, it was defined as 0xff is a magic tag.  Two of
them means one 0xff in the message.  Otherwise, it's followed by a pointer
to "insert text using this pointer here".

At this point, IMAP and submit are on an even playing field.

Now, I'm not saying that 0xff magic tags are a good idea.  In fact, that's
a wretched idea.  It's just an example.

> scenario?

In submitted messages.  That's part of how submit differs from SMTP.

> Are you simply saying that the IMAP server might have to

It may also have to do 8->7 bit conversions, wrap excessively long lines,
and other things to render a submitted message into something that is
acceptable to RFC 2822, MIME, and SMTP.

> *less* mature and tested service?

Not really even an IMAP client.  The access would use SASL; so the client
would only need to know enough about IMAP to negotiate SASL with it.

> trying to boil the ocean.

These are the correct questions to ask:

Is the latter going to happen?  Why or why not?

Will the IETF declare IMAP to be the one true message submission protocol,
and that all future work on message submission is to be done in IMAP?

Is it desirable to have multiple submission protocols in the IETF suite?
If so, how will functional enhancements be coordinated in the multiple
submission protocols?

What happens when users of mobile device email want to incorporate text
from a web page, or from an NNTP server?

What does a user who uses multiple IMAP servers (I am such a user) do?

How do you answer the objections that you are proposing a short-term
facility with acknowledged limitations?

> I don't get it. Please explain.

The user does a reply.  The reply includes replied-to text, which is
interspersed in the reply under composition (thus the mobile device has
that text in its buffer).  The replied-to text contains evil instructions
to the submit server which are hidden in HTML using one of the well-known
tricks.

With an independent submit server, those evil instructions can't
accomplish much; the submit server only does submit and has no access to
data other than what it was specifically granted.  With a submit-in-IMAP
server, those evil instructions have full access to the user's entire mail
store.

There will need to be a major review taken of all aspects of
submit-in-IMAP to ensure that there is no path by which a bad guy's evil
instructions can be passed to a submit-in-IMAP server.  Since people do
things such as reply including replied-to text, this leaves a number of
potential paths open.

The bad guys are out there.  We have reports of cell phones being cracked
via evil SMS messages (of all things!).  We have to assume that they are
going to be far more creative than any of us in dreaming up ways to
subvert whatever facility we build.

Submit, no matter how it is done, will be a new facility.  In one
proposal, submit will have open and unlimited access to all of a user's
mail (and potentially, depending upon the nature of the server, to all
users' mail).  In the other proposal, submit runs in an isolated box, with
no access to any data other than what it was specifically granted.

Yet, the proposal which gives this new facility unlimited access is called
"secure", and the proposal which isolates this new facilty to no more data
than it needs to perform its task is called "boiling the ocean."  Beats me
how that makes sense.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 11 Jun 2003 17:05:18 -0500

On 6/10/03 at 12:38 PM -0700, Mark Crispin wrote:

>A message composer, by its nature, has complex buffer management; it 
>is, in effect, a text editor.  Text in mail can expand (e.g. due to 
>quoting) to a much larger size than the original.

Leaving the rest of this debate until later, I think we've got a 
basic level of not understanding each other here, because I don't see 
any complex buffer management here that IMAP doesn't already do. This 
misunderstanding may be because no one has ever written down the 
proposal for IMAP-based compose/submit. So let me, like Chris, give a 
sketch of what I'm thinking about:

There are two different potential mecahnisms we're discussion: "Send 
this message" or "Compose and send this message".

I don't think there's any misunderstanding about how "Send this 
message" works: I give a command to the IMAP server which includes a 
particular message from the IMAP store to send and a list of 
recipients (in a more complex setup with ESMTP options), and the IMAP 
server connects to a submit server (port 587 or port 25 as it likes), 
sends the SMTP commands to send the message, and sends the message 
itself (unchanged) in the DATA portion. There is definitely 
discussion to be had about the syntax of such a command (e.g., how to 
represent the ESMTP options if you desired them) as well as the 
requirements on the IMAP server (e.g., whether there is a requirement 
to connect to an SMTP server that can handle 8BITMIME to avoid 
transcoding issues), but I think everyone here has a basic 
understanding of what this feature entails.

It's the "Compose and send this message" that I think there is some 
confusion on. So here's what I see:

A client APPEND's a message *template* to the IMAP store. It marks it 
in a way to indicate that it is a template. (Perhaps we can use 
\Draft for this, but I'll leave that open for now.)

A message template will contain the entire message (headers and all) 
that the client wants to send, and may be a simple text message, or a 
complex multipart message. However, the template will not contain any 
body parts of the message (including, potentially, the top-level 
contents) that the client wishes to retrieve from other messages in 
the IMAP store. To include a body part from another message (either a 
full message or a subpart of a multipart message), the client makes 
the Content-Type of that part "message/external-body" which includes 
the pointer to the body to be included.

The client then issues a command similar (identical?) to the "Send 
this message" command above. The IMAP server connects to the submit 
server, sends the appropriate commands, and then (because it is 
indicated as a template) sends the message template one part at a 
time. If it encounters a part *in the message template* which has any 
Content-Type other than "message/external-body", it sends that body 
part of the template (including it's headers) as-is. If the IMAP 
server finds a body part with a Content-Type of 
"message/external-body" *in the template* which points to a message 
body on the IMAP server, it sends the encapsulated headers of the 
"message/external-body" followed by the contents of the IMAP message 
body indicated by the external-body.

The IMAP server does not need to examine the contents of the body 
parts it is retrieving from elsewhere in the IMAP store; it fetches 
them normally, redirecting the output to the submit server rather 
than to the requesting client. The IMAP server does not recurse 
through the referenced body parts looking for more external-body 
parts.

The actual sending would likely complete asynchronously.

There are a few caveats here:

- If the client wishes to include an external-body part in the 
template message which it actually wants sent as an external-body 
(i.e., not replaced with some other contents), we will need a 
mechanism for the client to mark that part of the template as such. 
Perhaps a parameter to the content-type?

- I'm not entirely sure of some semantics of some kinds of 
external-body references. For example, if the top-level Content-Type 
of the template message is "message/external-body", which headers get 
replaced by the encapsulated headers? Is it only the Content-* 
headers or all of them? If the latter, the top-level headers of the 
message are only dummies, and the actual "To:", "From:", etc. fields 
would actually be in the encapsulated headers. We have to know this 
to make the operation clear.

- The client is going to have to be responsible for setting the 
appropriate CTE's for the embedded parts. It should know this when it 
goes to include the referenced part (on the assumption that it's 
going to have to at least look at the headers in order to include 
it), but it's something that a client is going to have to be aware of.

- I haven't noted where the "reference count" business we were 
talking about in another thread gets included. I'm guessing that upon 
issuing the command, the server might scan the message for references 
and up the reference count on each immediately, failing immediately 
if any part has been deleted in the interim, but again, this may 
still need some hashing out.

This scheme doesn't do any "text editing" that Mark's message seemed 
to be concerned about. It doesn't have the functionality of 
external-body references to things *not* on the IMAP server.  It's 
pretty bare bones. But it does allow you the functionality of taking 
a message (or message template) which you have on the server and 
sending it. You don't have to worry about uploading it twice, once to 
the submit server and once to your "outbox" archive, or waiting for 
the submit server to deposit it back into your archive. You can 
compose it at one time, put it on your server, and not send it 
immediately (or delete it if you decide later to not send it at all). 
And you don't have your submit server given access to your IMAP store 
if you don't want to design your network that way.

Of course, doing this does not preclude also creating a full-fledged 
submit server as Chris and Mark have suggested where external-bodies 
to things *not* on the IMAP server could be included. But that 
service (which I view as much more heavyweight and much more 
complicated to implement) isn't required to get meet the requirements 
of LEMONADE, and I think this is the cleaner path to addressing those 
requirements.

pr

---

From: Mark Crispin <[email-masked]>
Date: Wed, 11 Jun 2003 16:19:27 -0700 (Pacific Daylight Time)

On Wed, 11 Jun 2003, Pete Resnick wrote:
> So let me, like Chris, give a

Thank you.  This is more or less what I expected you were talking about,
albeit a bit more complex in that you have a separate "send this message"
vs. "compose and send this message" with an earlier appended template.  A
simplified version would be a single "compose and send" which takes the
template as an argument (thus all messages being sent are templates).

You've outlined a rather complex procedure that *something* must do, no
matter which proposal ("do it in IMAP, do it in submit service, do it in a
proxy") is undertaken.

Specifically, *something* must:
 . deal with ESMTP in its full glory, including 8BITMIME issues.
 . add missing headers to the template; commonly, submit clients expect
    the submit server to provide Date, From, Message-ID, and similar
    niceties.  I expect that a mobile client, especially one with
    limited internal resources, would do the same.
 . recognize MESSAGE/EXTERNAL-BODY MIME parts, resolve them, and rewrite
    those parts using the resolved contents.
 . in some way arrange for the data referenced by these parts not to be
    deleted until the send has completed.

The desire to put this into the IMAP server is apparently based upon two
observations:
 1) IMAP already servers know how to parse MIME
 2) it is assumed that the external-body data will be on the same IMAP
     server

I dispute that second point.  I think that you conceed that this is an
failing of the "do it in IMAP" model.  Although I consider this to be a
fatal flaw, there is no concensus on that point.

There is also a implied presumption that it is easier to do a local disk
access to get the data than it is to use any protocol to get the data.

As to the first point, I personally do not believe that "IMAP servers
already know how to parse MIME" is a big advantage.  I know how to do all
four components of the above task, and I think that parsing MIME is one of
the easiest parts.  Perhaps that is just me, and everybody else thinks
that parsing MIME is very hard.

> to be concerned about.

By "text editing", I was referring to the addition of missing headers and
the rewrite of portions of the message.  RFC 2822/SMTP is pretty nasty in
that you generally need a buffer that is at least twice the size of the
input string, plus three bytes, to deal with worst case scenarios.

I've seen several buffer overflow attacks on RFC 2822/SMTP generating
code, and have spent a substantial amount of time dealing with it.  The
notion of having RFC 2822/SMTP generating code in an IMAP server gives me
the willies -- and from recent unhappy experience I know a lot of what to
look for!  It's what I don't yet know that worries me.

> You don't have to worry about uploading it twice, once to
> the submit server and once to your "outbox" archive,

I am willing to stipulate that this should be a requirement of whatever
solution is desired.

> or waiting for
> the submit server to deposit it back into your archive.

I agree that this is desirable, and perhaps even should be a requirement.
But do you really want the template, or the fully-resolved message in the
archive?  I think that the latter is more desirable, but that's just me.

> You can

OK, so you want the ability to have drafts stored on the server instead of
on the device?  We can make that be a requirement.

> if you don't want to design your network that way.

Here, we have a disconnect.

I define submit server as "the entity that accepts the template from the
client, assembles it as a complete message, and passes the message to the
MTA."

I think that you define submit server as "the first-level SMTP server that
does MTA queuing."  [Is this correct?]

Using my definition, the submit server has full access to the user's
entire IMAP store (your proposal) or only access to the authorized pieces
(my model).

Using your definition, the submit server either has no access other than
to the fully-assembled message (your proposal) or only access to the
authorized pieces (my model).

Now, if we have an IMAP/SMTP proxy which talks directly to the mobile
device (and then to the IMAP and SMTP servers) that does all this work, we
have the following:
 . the first-level SMTP server that does MTA queuing has no access other
    than to the fully-assembled message
 . the IMAP server does not do assembly or passing to the MTA
 . the proxy has only acess to the authorized pieces

We can also stipulate that fcc and resolution of locking issues be part of
the proxy's responsibility.

> to things *not* on the IMAP server could be included.

Suppose that instead of a full-fledged submit server, we back off to a
the more limited IMAP/SMTP proxy server.  It *could* grow to be the
full-fledged thing, but it doesn't have to.

I also point out that that the proxy server can be specified (as part of
its design) to talk in the most ideal way to the mobile device, without
any of the chattiness or RTTs of IMAP and SMTP.

Another beauty is that, from the mobile device point of view, there is
nothing "heavyweight" about this.  If anything, the mobile device should
see a simpler world.

I think that if we all give just a little bit, we ought to be able to
settle upon this as a win-win situation for everybody.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 11 Jun 2003 21:06:15 -0500

Let me answer one later question right up front:

>does MTA queuing."  [Is this correct?]

Right. In the model I've described, there is an 2821 or 2476 server 
that is going to accept the message from the IMAP server and forward 
it along. That could running on the same machine as the IMAP server, 
but I see it as a separate functional unit in this model.

Perhaps we can skip the term "submit server" and instead agree to 
different terms here:

1. Outgoing SMTP server or "OSMTP" server. The OSMTP server does MTA 
queuing. Some OSMTP servers are 2821 servers; some are 2476 servers.

2. The composition server is the entity that accepts the template 
from the client, assembles it into a complete message, and passes the 
message to the MTA.

And I think we agree on the definition of:

3. The IMAP server.

The questions are "Where does 2 go?" and "What is 2 responsible to 
do?" My proposal puts 2 into 3 (call it "IMAP compose"). Chris's 
proposal puts 2 into 1 (call it "OSMTP compose"). There's been the 
suggestion that 2 could also stand alone (call it "Proxy compose"). 
What 2 has to do still has some open issues. Are we OK to there?

Now, getting on to the rest of the message:

>A simplified version would be a single "compose and send" which 
>takes the template as an argument (thus all messages being sent are 
>templates).

Sure. In addition, you could add an argument which says "Don't expand 
*any* external-body parts" to give you full functionality.

>  . deal with ESMTP in its full glory, including 8BITMIME issues.

Right, though the OSMTP server might deal with some of that already 
by being a 2476 server.

>     limited internal resources, would do the same.

Interesting question. Perhaps they do need to be dealt with, perhaps 
not. I know Date used to be a serious problem for resource-limited 
clients (especially time zones), but I don't know if that's an issue 
anymore. From (or Sender)  are the more interesting ones. We need to 
nail down whether this is a requirement or not.

>     those parts using the resolved contents.

Absolutely. This is the big ticket item.

>     deleted until the send has completed.

Sure.

There are other potential requirements mentioned below. Let's bring 
them up here:

. store the template for later submission
. store "what was sent" in the IMAP store (i.e., the outbox), where 
"what was sent" might be the template or might be the composed 
message.

>The desire to put this into the IMAP server is apparently based upon 
>two observations:
>  2) it is assumed that the external-body data will be on the same IMAP server

Let's add:
3) Users are always going to identify themselves to the IMAP server; 
in some current environments, users don't identify themselves to any 
other (e.g., OSMTP) servers. (This is the security benefit.)
4) IMAP servers are good places for storage of user data (where OSMTP 
servers generally are not).
5) IMAP servers are going to have to store "what was sent" anyway, so 
starting the template off on the IMAP server is efficient.

>I dispute that second point.  I think that you conceed that this is 
>an failing of the "do it in IMAP" model.  Although I consider this 
>to be a fatal flaw, there is no concensus on that point.

Agreed that we disagree. I think the most likely usage model is 
"forward without download".

>There is also a implied presumption that it is easier to do a local 
>disk access to get the data than it is to use any protocol to get 
>the data.

There's more to it than just easier. An entity that has local control 
of the data has many more tools at its disposal to do things like 
locking, reference counting, copying, etc. If you use a protocol, you 
need protocol bits for each of these, and you have to fully 
understand the semantics for each of them. That can be a tricky 
business. So I would add this to the benefits list as:

6) Local disk access has operational benefits.

>As to the first point, I personally do not believe that "IMAP 
>servers already know how to parse MIME" is a big advantage.  I know 
>how to do all four components of the above task, and I think that 
>parsing MIME is one of the easiest parts.  Perhaps that is just me, 
>and everybody else thinks that parsing MIME is very hard.

Think about the following (and I'm not claiming the answers here 
clearly support my position):

- How big is your MIME parsing code in relation to the rest of the 
IMAP system? How big would it be in relation to the rest of a 
present-day OSMTP system?

- How much of your MIME parsing code assumes quick random access to 
storage or large memory buffers?

>By "text editing", I was referring to the addition of missing 
>headers and the rewrite of portions of the message. [Tricky to do; 
>possible buffer overflow issues.]

Addition of headers is one thing and is an open issue. If we need to 
do it, deciding what entity does it is a real concern. Other sorts of 
rewrite I don't think is as much of an issue in this system. IMAP is 
passing around legal MIME parts already. The only buffer issues would 
be if you were doing further CTE transcodings, which I *don't* 
suggest is done on the composition end of things. Is there some other 
function that I'm missing?

>But do you really want the template, or the fully-resolved message 
>in the archive?  I think that the latter is more desirable, but 
>that's just me.

If it were me, and I had an IMAP server with a nice database-like 
store, upon getting the template I'd resolve the pointer to the 
external-body, incremement the reference count on the object, and 
store the pointer to the object with the template. I wouldn't want to 
fully resolve (i.e., keep a separate copy of all of the data) if I 
didn't have to.

>OK, so you want the ability to have drafts stored on the server 
>instead of on the device?  We can make that be a requirement.

Indeed, a separate item that's been on the what-some-folks-would-like 
list is "future delivery with cancel". In the IMAP compose model, you 
could add a queuing time to the command and knock off that item as 
well.

>>And you don't have your submit server given access to your IMAP 
>>store if you don't want to design your network that way.
>

Right. So let me rephrase with the definitions I gave at the beginning:

- In the IMAP compose model, the composition server has local disk 
access to the entire IMAP store. The OSMTP server has protocol access 
only to the fully-assembled message.

- In the OSMTP compose model, the composition server and OSMTP server 
have protocol access to the authorized pieces of the IMAP store and 
the fully-assembled message.

- In the Proxy compose model, the composition server has protocol 
access to only the authorized parts. The OSMTP server has protocol 
access to only the fully-assembled message.

In all cases, the IMAP server (obviously) has local disk access to 
the entire IMAP store.

I think that makes the security implications as well as the protocol 
implications pretty clear.

>Suppose that instead of a full-fledged submit server, we back off to 
>a the more limited IMAP/SMTP proxy server.  It *could* grow to be 
>the full-fledged thing, but it doesn't have to.

I'm not sure I understand where the "limited" comes in. What is the 
difference between the limited one and the full-fledged one?

>I also point out that that the proxy server can be specified (as 
>part of its design) to talk in the most ideal way to the mobile 
>device, without any of the chattiness or RTTs of IMAP and SMTP.

Agreed, though the mobile device is required to talk to IMAP at some 
point anyway (obviously for accessing mail, but also to decide what's 
going to go into the template and, in the Proxy model, to get the 
tokens for the desired parts).

>Another beauty is that, from the mobile device point of view, there 
>is nothing "heavyweight" about this.  If anything, the mobile device 
>should see a simpler world.

I'm not clear on why. Can you describe more what you mean?

>I think that if we all give just a little bit, we ought to be able 
>to settle upon this as a win-win situation for everybody.

I still see a downside to the Proxy compose (since it has many of the 
same disadvantages of the OSMTP compose) and an upside to the IMAP 
compose, especially in relation to the 6 observations above. But I'm 
ready to listen to the upside of the proxy if you can answer a few of 
the above questions. And as I said, doing IMAP compose doesn't 
preclude doing either OSMTP compose or Proxy compose.

pr

---

From: Mark Crispin <[email-masked]>
Date: Wed, 11 Jun 2003 20:31:27 -0700 (Pacific Daylight Time)

> What 2 has to do still has some open issues. Are we OK to there?

OK, but I really don't like the idea of the name "OSMTP compose", because
it ties "SMTP" and all of its baggage.  Since you think that the term
"submit server" has baggage, how about we call it "post server compose"
and hopefully avoid anyone's baggage?

But I'll use "OSMTP compose" in the remainder of this response.

> I know Date used to be a serious problem for resource-limited
> anymore.

Do CDMA phones know about timezones?  I don't think that GSM or TDMA
phones do.

> "what was sent" might be the template or might be the composed

Both of these are possible with any of the proposals.  It's more of a work
item for OSMTP compose than the other two; but let's leave that point with
a stipulate that OSMTP compose will be required to address this in a
satisfactory fashion if that is the chosen proposal.

> other (e.g., OSMTP) servers. (This is the security benefit.)

I believe this to be a vanishing issue.

> starting the template off on the IMAP server is efficient.

I don't consider these two points to be particularly important in the
decision, because post and storage are different functions.  But if you
want to phrase this as a requirement for "only one upload from the mobile
client" we can go with that.

> An entity that has local control
> locking, reference counting, copying, etc.

[Slight shuffle in your text for brevity]

I remain unconvinced about locking/reference counting.  The locking hole
is between the last access by the mobile client and the post.  You need
some sort of assist to prevent a simultaneous session from expunging the
data.  The most likely way to do that assist would be by protocol
negotiation of a lock.

Another way would be to have the server keep track of items mentioned in
active drafts, and prohibit their expunge.  There's user interface ("why
can't I expunge?") and implementation complexity issues.

> clearly support my position):

Fair questions

> IMAP system?

Quite small.  My entire MIME parser consists of three routines: parse
content header, parse content text, and parse parameter.  The RFC 2822
code is about three times the size.

> How big would it be in relation to the rest of a

It depends upon whether or not it has 8BITMIME support, doesn't it?  I
don't have C language SMTP server code which I wrote, but my C language
SMTP client code is twice the size of the C language MIME code.

Looking at my old PDP-10 assembly language code (where I have both client
and server), I see that my SMTP client code is about 1/4 the size of my
SMTP server code, and that SMTP server code didn't do any munging of the
message contents.  Too bad I don't have any PDP-10 assembly language MIME
code, but my PDP-10 RFC 822 code is smaller(!) than the C code.

We're into "lies, damn lies, and statistics" at this point; comparing C to
assembly language in applications with vastly different functional sets.

> storage or large memory buffers?

It doesn't.  It would *like* to run that way, but it doesn't have to.
Credit having to build PC Pine in 640K DOS (days all of us are happy to
forget).

> passing around legal MIME parts already.

Yes, but only insofar as it blats out data from the message.  It doesn't
have to do any thought about how to format that data.

> The only buffer issues would
> suggest is done on the composition end of things.

CTE is actually fairly easy.  In the case of BINARY->BASE64 the exact
output size is known via calculation.  In the case of BASE64->BINARY and
QP->8BIT the resulting output is usually smaller and never larger than the
source; you can do it by overwriting the source buffer.  Only the 8BIT->QP
encoding involves having to figure a worse case.

> difference between the limited one and the full-fledged one?

One difference would be that the intended client for this thing is a
mobile device.  It is not a full-fledged submit server that something like
Pine or Eudora would want to use.

> tokens for the desired parts).

Not at all for the tokens (the proxy would do all that) and not
necessarily for the other stuff.  I know that "compressed IMAP" is
something that a lot of people talk about; why can't this proxy be a front
end for IMAP?

> I'm not clear on why. Can you describe more what you mean?

This is that "front end" idea I alluded to above.

Since the proxy server is the mobile device's window on the world, it can
talk to the mobile device exactly as the mobile device wants.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Thu, 12 Jun 2003 00:09:06 -0500

On 6/11/03 at 8:31 PM -0700, Mark Crispin wrote:

>["IMAP compose", "OSMTP compose", and "Proxy compose"]
>
>OK, but I really don't like the idea of the name "OSMTP compose", 
>because it ties "SMTP" and all of its baggage.

Gotcha. But it's necessarily (at least) an "MTA compose", right? 
Because the issue here is that the MTA and the composition server 
live in the same space. Am I mischaracterizing?

>Do CDMA phones know about timezones?

I'm pretty sure they do. (I was looking at some API's that have GMT, 
and I know of some stuff that is definitely in local time, so I think 
it's in there someplace.)

>I don't think that GSM or TDMA phones do.

I'm not sure, but I'm sure someone in this group knows.

>Both of these are possible with any of the proposals.  It's more of 
>a work item for OSMTP compose than the other two

For the "store the template for later submission", do you suppose 
that the Proxy compose will store on the IMAP server or in it's own 
storage?

>>An entity that has local control of the data has many more tools at 
>>its disposal to do things like locking, reference counting, 
>>copying, etc.
>
>I remain unconvinced about locking/reference counting.  The locking 
>hole is between the last access by the mobile client and the post. 
>You need some sort of assist to prevent a simultaneous session from 
>expunging the data.  The most likely way to do that assist would be 
>by protocol negotiation of a lock.

The other choice is to do a reference count, where no lock is 
required. Say you reference a body in another message. Its reference 
count starts at 1. You attempt to increment reference count on the 
object. That could fail if the expunge already happened (same as the 
lock attempt), but if it succeeds, the reference count is 2. Then, 
the simultaneous expunge simply decrements the reference count on the 
object, leaving it with a reference count of 1 and not deleted. The 
expunge "succeeds", but you still have a copy of the object. Of 
course, this requires a storage model where the references to the 
objects can be independent, but this isn't exactly a new way of 
thinking.

>Another way would be to have the server keep track of items 
>mentioned in active drafts, and prohibit their expunge.

That seems bad.

>>How big would [the MIME parsing code] be in relation to the rest of 
>>a present-day OSMTP system?
>
>It depends upon whether or not it has 8BITMIME support, doesn't it?

To some extent. Remember that there's a difference between parsing a 
MIME message to retrieve an individual part and converting CTEs: I'm 
pretty sure converting CTEs can be done in a single pass with a stack 
for multipart separators. I'm betting your MIME parsing code doesn't 
work that way; at least, I'll bet you keep pointers into different 
parts of the message for random access.

>We're into "lies, damn lies, and statistics" at this point;

Agreed. But my guess is that doing the kind of MIME manipulation 
required for this would be a relatively small part of an IMAP 
server's code, and a rather more significant part of an MTAs code, 
with or without 8BITMIME.

>It doesn't.  It would *like* to run that way, but it doesn't have to.

Well, right. But I assume that there's plenty of "do this if you 
don't have to suffer through small buffers and serial access" because 
it's more efficient, and that it doesn't run "ideally" if it has to 
suffer through those things.

>>Other sorts of rewrite I don't think is as much of an issue in this 
>>system. IMAP is passing around legal MIME parts already.
>
>Yes, but only insofar as it blats out data from the message.  It 
>doesn't have to do any thought about how to format that data.
>[...]
>CTE is actually fairly easy. [Discussion of output buffers.]

In fact, most CTE stuff can be streamed and not pre-buffered. But my 
original question still stands: Other than adding headers, what 
additional kind of processing that it doesn't do now is an IMAP 
server going to have to do that is going to require it to do lots of 
buffer manipulations during composition?

>>the mobile device is required to talk to IMAP at some point anyway 
>>(obviously for accessing mail, but also to decide what's going to 
>>go into the template and, in the Proxy model, to get the tokens for 
>>the desired parts).
>
>something that a lot of people talk about; why can't this proxy be a 
>front end for IMAP?
>[...]
>Since the proxy server is the mobile device's window on the world, 
>it can talk to the mobile device exactly as the mobile device wants.

Oh. Sorry. I missed this in the earlier mail. You're suggesting that 
the Proxy talks to the IMAP server and the MTA server for *all* of 
the mobile devices operations (at least for composition, and perhaps 
for everything). In effect, the Proxy *is* an IMAP client *and* an 
MTA client, and a compose server (and potentially a mail access 
server) for the mobile device.

Well, that's an interesting idea, but unlike the other two models, it 
has one really big downside: This is an entirely new service with an 
entirely new set of code (caveat that there will obviously be a bunch 
of reuse from IMAP and SMTP clients) and most importantly an entirely 
new protocol. I don't think LEMONADE should be taking on something 
like that.

pr

---

From: Mark Crispin <[email-masked]>
Date: Thu, 12 Jun 2003 16:57:24 -0700 (Pacific Daylight Time)

> live in the same space. Am I mischaracterizing?

I don't feel that either composition or submission belong in the MTA space
(and certainly not in the MUA space with IMAP).  However, for the purposes
of this discussion this disagreement may be academic.

> that the Proxy compose will store on the IMAP server or in it's own

That is either "to be decided" or left to the implementation.  Unless
there is a requirement that the template be accessible to IMAP clients
which do not use the proxy facility, I don't think it matters either way.

If there is such a requirement, then that needs to be stipulated.

> >mentioned in active drafts, and prohibit their expunge.

I agree with you on "that seems bad", but in effect what you describe in
the paragraph which begins with "The other choice..." is an implementation
method of that bad thing.

That is, the reference count would be implemented by an active draft
containing a reference, and would be decremented by deactivating the draft
(either deleting it, or sending it).

You seem to envision some kind of dichotomy between "reference count" and
"lock".  Since any lock would presumably be a shared lock, this suggests a
reference count.  I see these as being the same.

Are you thinking about a lock in the POP3 sense, which I could refer to as
an exclusive lock?  That kind of lock wouldn't be particularly useful
here, other than as a way to block the acquisition of a shared lock.

One other point.  A database for a mail store has been a personal favorite
idea of mine for a long time (from IMAP's inception, in fact).  Some IMAP
server implementations do such a thing.  However, from a server management
point of view, there is *strong* pushback against a database mail store.
The reasons are outside of the scope of this discussion or LEMONADE (it'd
make a great bar topic), other than to caution that the world is not ready
to define IMAP as requiring a database.

> I'm

Such pointers are certainly generated as part of the MIME parse, but
they're not kept in the message store.  It's actually *much* easier to
recurse over nested parts than to do it single-pass with a stack,
especially if you want to be strictly compliant with MIME.

As for the ability to use pointers and do single-pass CTE conversion, you
can thank me for sticking to my guns 12 years ago and demanding that MIME
forbid nested encodings.  That was not a popular position.

> But my guess is that doing the kind of MIME manipulation

We'll just have to disagree on this point.  I think that you've
overestimating what an IMAP server does, and underestimating what an
8BITMIME compliant SMTP server does.  But we're both speculating at this
point.

> buffer manipulations during composition?

Header building (both RFC 2822 and MIME) and compliance with SMTP
line-length restrictions come immediately to mind.  Nor can you assume
"it's in my IMAP server, so it already complies and I can just copy it
as-is"  because your mail software may allow non-compliant text (such as
excessive line lengths) or mistagged texts, but it has an obligation not
to send such.

> You're suggesting that

I'm suggesting this as one possible alternative.  I don't want to put all
my eggs in one basket.

> new protocol. I don't think LEMONADE should be taking on something

Unless there's a 100% show-stopper, let's keep in on the table.

I can't help but think that something in this vein would produce much more
satisfactory results for all concerned than forcing mobile devices to deal
with chatty IMAP and forcing IMAP to offer services to mobile devices that
are outside of its current scope.

I know that compressed IMAP is an important issue for many people in the
mobile device community.  To me, this practically screams for a proxy type
solution.

-- Mark --