Subject: Re: [lemonade] IMAP push/pull plan

From: Randall Gellens <[email-masked]>
Date: Thu, 22 Jan 2004 19:01:56 -0800

Here is a long-delayed response to the flurry of emails that were 
sent in late December and early January.  I've replied to everything 
in one long email, with significant snippage throughout.  Because the 
snippage was so great, I didn't use ellipsis or "[snip]".  Instead, 
I'm only including the direct quotation to which I'm replying.

----

At 11:29 AM -0800 12/23/03, Ted Hardie wrote:

>   For the overall architecture to have both authentication
>   and authorization, the pull model must actually use some form of
>   submission in which the sender is authenticated to the SMTP/submit server.
>   Though this is growing, individually authenticated submission is
>   still not the norm (IP-based allow lists seem to be) and those that
>   seem to be seen are of the pop-before-smtp type that limit deployment
>   in other ways.  While I can see the value of a split 
> authentication/authorization
>   model, I am quite concerned about a model that will have no authentication
>   at all in the common case.

I think the use of pop-before-smtp is fading off, as most of the 
popular clients support some form of SMTP AUTH, as do popular 
SMTP/Submit servers.  For a few years now, some servers by default 
enable the message submission port (587) and require authentication 
on it.

The use of IP-based authorization for message submission from within 
an organization's firewall is still widespread, for a variety of 
reasons, including the lack of perceived need for anything stronger. 
For access from outside, I see increasing use of SMTP AUTH.

However, even in the case of unauthenticated submission, I'm not sure 
a pull model would be introducing any new risks: in the case of a 
forged email, the submitter would still need a valid pawn ticket to 
access the message or message fragment to be included.  An attacker 
could potentially email data to himself using a stolen pawn ticket, 
but this risk depends on the pawn ticket (which has protections 
against it) and doesn't matter if the attacker authenticates to the 
Submit server or not (except that authenticating may help pinpoint 
the identity of the attacker, assuming the credentials are valid). An 
attacker could send forged mail, but this isn't affected by the 
inclusion of data from an IMAP server or not.

>   It is possible, of course, that we can make use of an 
> authentication mechanism
>   in the submit server a basic requirement of this protocol.  This 
> adds, however, to the
>   deployment hurdle we must overcome.

I don't think it adds much of a burden, given the widespread 
deployment of SMTP AUTH capable clients and servers.

>   As it stands, the historical tendency of IMAP development to put 
> complexity into the client rather than the server means that the 
> newly capable "SMTP/Submit Pull" servers will have to implement a 
> fair amount of code to act as IMAP clients. They will also have to 
> implement the code required to handle pawn tickets, as well as 
> their own form of authentication.

The "historical tendency of IMAP development to put complexity into 
the client rather than the server" is one the hurdles to IMAP use on 
wireless devices that we (lemonade) are trying to address.

I think SMTP AUTH support is already fairly widespread; servers 
shouldn't have to implement "their own form of authentication".

>   Making the locus of change for our work the SMTP/submit servers means we are
>   either at the mercy of or constraining other work related to spam abatement,
>   internationalization, etc.  Without very careful coordination, we 
> could end up
>   with a protocol that will see little or no deployment, and I don't 
> think any of
>   us wants to waste our time.

Even if we select a pull mode, I don't see it being at the mercy of, 
nor constraining, other SPAM-related work.  I think the Submit 
changes are likely to be orthogonal.

>   The addition of SMTP client functionality to an IMAP server
>   seems to me far easier than the addition of IMAP client 
> functionality to an SMTP/Submit
>   server.

I wouldn't say "far easier".  Since we're talking about adding a very 
limited IMAP client functionality to a submit server, as compared to 
a conceptually unlimited submit functionality to an IMAP server, I 
see the efforts as being roughly equal, with a risk of feature creep 
making the IMAP work much larger.

----

At 4:43 PM -0800 12/23/03, Mark Crispin wrote:

>   SUBMIT implies considerable glue between IMAP and SMTP, which for all its
>   translation still winds up on the client.  For example, note the untagged
>   SUBMIT response that returns SMTP commands and responses.  It also lacks
>   support for SMTP service extensions (although it requires DSN), meaning
>   that if a service extension is needed the client is out of luck.

Actually, the SUBMIT command has a mechanism for SMTP/Submit 
extensions.  One of my concerns with adding submission to IMAP is the 
risk of creating two separate, and even worse, potentially divergent 
mechanisms, for message submission.  In order to guard against the 
latter, and attempt to isolate the IMAP server from knowledge of 
SMTP/Submit extensions, I used annotations to allow the client to 
specify arbitrary SMTP/Submit extensions.

>   Both commands betray "do the minimum to address the task at hand" and "put
>   all the complexity in the IMAP server" thinking.

The SUBMIT command does try to keep the IMAP server from needing to 
know anything about specific SMTP/Submit extensions.

----

At 3:06 PM -0800 12/29/03, Ted Hardie wrote:

>   I believe that Randy's draft could be a bit simpler, but that it 
> demonstrates the fundamentals
>   of the approach).

Part of the complexity is the use of annotations, which is a device 
to allow SUBMIT to be used with SMTP/Submit extensions without 
requiring an IMAP-specific facility or knowledge of said extensions.

>   And while we're talking about the dangers of the easy path, I'd 
> like to point
>   out the danger in the Pull model:  that Pull-based composition at 
> the SMTP/Submit
>   server seems likely to degenerate into Pulls from HTTP (or POP) 
> document stores instead
>   of IMAP stores.

I can see a risk of this with either Push or Pull.  At least with 
Pull, the mechanism to do so is more logical.  I'd support a 
prohibition in the initial version of either model to limit its use 
to IMAP data stores, so that we can focus on the immediate task at 
hand.  If and when either mechanism needs to access other kind of 
data, it can then be done as an extension.  This can keep us from 
getting distracted by the security issues of non-IMAP URLs.

----

At 10:03 AM -0800 12/30/03, Mark Crispin wrote:

>   The other basic reason is the prospect of having to track all future
>   developments in SMTP in IMAP push.  I note that the current IMAP submit
>   document pays little heed to SMTP extensions, and apparently expects that
>   by doing so IMAP submit will never have to consider SMTP extensions.  I do
>   not believe that to be a viable position.

The current IMAP SUBMIT document uses annotations to allow use of 
arbitrary SMTP/Submit extensions without tying them to the IMAP 
server.  This does add complexity, however.

----

At 11:29 AM -0800 12/30/03, Ted Hardie wrote:

>   I note, though, that the same problem occurs for PULL; it can 
> force SMTP development
>   to track salient developments in the access protocols for PULL, 
> which would include
>   at least those in IMAP.  No matter which connection we make, we 
> are talking about
>   entanglement here, and we need to go into that with our eyes open.

It seems to me that it will be easier to limit entanglement in the 
Pull mode, since the Submit server doesn't need to know about or use 
most IMAP extensions.  IMAP extensions are targeted at generalized 
IMAP clients, while a Submit server would be a highly specialized 
client.  In contrast, once submission is added to IMAP, the IMAP 
server becomes a generalized Submit server and hence needs a way for 
clients to access arbitrary SMTP/Submit extensions.

----

At 6:48 PM -0800 12/30/03, Mark Crispin wrote:

>    > The strains on SMTP/Submit servers seem to me much higher, since they
>>   are also in the middle of the spam arms race
>
>   Not necessarily!
>
>   UW, like most other large organizations, has separate servers for submit
>   vs. incoming mail.  Not only are they separate boxes, but are on
>   completely different subnets.
>
>    > and a likely need to support
>>   new internationalization mechanisms.
>
>   All the more reason to have one (and only one) submit mechanism.

I agree, plus, I'm not sure there will be collisions between Pull and 
other SMTP work anyway; it seems fairly separate.

----

At 11:28 AM -0800 12/31/03, Ted Hardie wrote:

>   SUBMIT servers are or can be separate from incoming
>   mail.  But the spam arms race includes those servers as well, as one of
>   the key ways folks fight spam is by ensuring that systems within their
>   administrative border aren't injecting it (either because of black hats or
>   "owned" boxes).  That can be reactive (driven by black-list servers) or
>   proactive, but it is still a front in the war, so there is still a 
> strain here
>   on those systems.

But if IMAP servers become Submit servers, then they are now too part 
of the same arms race, for the same reasons.

>   The same entanglement applies to both PULL and PUSH here, but PULL
>   is in a worse spot.  The IMAP server is responsible in PUSH for ensuring
>   that the message is valid before injecting it into the SMTP/Submit
>   system; where the SMTP/Submit server must validate it as it is accessed
>   or assembled in the PULL view.  That's a burden on the SMTP/Submit
>   server, and, as I said before, I think they are already in the more fragile
>   place in the overall system.

But this is a fundamental purpose of Submit servers; in fact, the 
original impetus for Submission as separate from SMTP came from 
clients that were injecting improper messages, and SMTP servers that 
were "correcting" relayed messages.

----

At 1:07 PM -0500 1/2/04, Rob Siemborski wrote:

>   On Fri, 2 Jan 2004, Pete Resnick wrote:
>
>    > Let's also remember that if you want to save a copy of the sent
>>   message, CATENATE gets you that for free. In the BURL/no-CATENATE
>>   model, you either are stuck with a template message or you have to
>>   get the submit server to give you a copy of the real message back.
>
>   CATENATE seems to be the sanest way to solve the "I need an FCC" problem
>   when using BURL to submit the message.

I agree that CATENATE seems the simplest way to solve the FCC 
problem, although there has also been talk of an IMAP extension to 
supply an email address which results in mail being added to a 
particular mailbox (such as Outbox).  The MUA can then include this 
address in the envelope when it submits the message.  It's been 
suggested that such an IMAP extension would be generally useful.

>   Personally, I don't really like the PUSH model in that it adds a new
>   submission mechanism, it assumes "everyone uses IMAP" and implies that if
>   we ever find a new mail retrieval protocol that everyone wants to switch
>   to after IMAP, we'll have to reinvent a submission mechanism for this
>   future protocol as well.

I agree.

>   Up until this thread, the main drawback that I saw with the BURL model was
>   that getting a FCC back onto the IMAP server was hard.  But I think that
>   the use of CATENATE solves that.
>
>   I'm currently favoring a solution that used CATENATE+BURL+URLAUTH.
>   Sadly, I think this solution involves a lot of work on all sides
>   (SUBMIT/SMTP, IMAP, and client).  However, I also feel that it gives us
>   all of the features that LEMONADE needs (forward without download, along
>   with draft messages and fcc capability), and is more future proof than the
>   CATENATE/SUBMIT model.

I concur.

----

At 10:50 AM -0800 1/2/04, Mark Crispin wrote:

>    > Sadly, I think this solution involves a lot of work on all sides
>>   (SUBMIT/SMTP, IMAP, and client).
>
>   Yes, it does.  There are no "easy" ways out.
>
>   However, the benefit of rolling up our sleeves and doing the right thing
>   is that the resulting technology can be leveraged for other purposes.

We also avoid the risk of creating a new, parallel (and possibly 
divergent) submission mechanism for IMAP clients.

----

At 4:24 PM -0800 1/5/04, Ted Hardie wrote:

>   The security model for the restricted CATENATE
>   is pretty easy, since it is providing an IMAP command that uses 
> the same basic
>   permissions for a resource as would a download of that resource.
>   In my personal opinion, the draft needs a very different security 
> analysis if
>   the presumption is going to be that the facility can securely 
> retrieve external
>   non-public resources. It also needs to carefully lay out which 
> schemes can be
>   used (not all uri schemes have the presumption of data retrieval, 
> for example,
>   and the use of those in this context would not make much sense).

I think we should focus on only restricted IMAP URLs for now, and add 
anything else as extensions, just to avoid this extra work at this 
time.

----

At 8:22 PM -0600 1/5/04, Greg M (Greg) Vaudreuil wrote:

>   I mildly favor the push model, in part because it feels more secure.
>   In my corner of the world, there is not a wide diversity of legacy 
> clients.  If I have IMAP push, I can imagine being able to dispense 
> with a client-accessible SMTP-submit server.

It's not just legacy clients that don't use IMAP.  There are a lot of 
"hidden" submit-only clients in most organizations, including small 
programs, scripts, web forms, etc.

You can put your Submit and IMAP servers in the DMZ, or you let 
authorized people use a VPN to get inside and use the interior 
systems.  Many large organizations use the latter approach, in part 
to avoid having boxes in the DMZ with access to user data.

>   This reduces the number of entities that must reside within the 
> DMZ network that need access to fairly sensitive subscriber 
> directory data.  An authenticated submission server generally 
> requires access to the users passwords (or encrypted passwords) 
> stored in a directory by the server in the DMZ.

Or access to an LDAP or AAA or other such server.  Or by using 
client-side certificates and TLS.  The Submit and IMAP servers 
themselves don't need access to the authentication data; they need to 
be able to authenticate users.

>   An IMAP proxy server in the DMZ only needs access to more limited 
> information mapping the recipient identity to a message store... or 
> less.  Gaining control of the IMAP proxy does not gain access to 
> the password data.  Gaining access to a submission server does get 
> me access to enough info to authenticate the user.... and to play 
> off-line guess the password games.

You can use proxy servers for submission as well as for IMAP.  You 
also don't need to give the systems access to the authentication 
credentials.  You also don't need to have these boxes in the DMZ if 
you use VPNs.  There are many approaches.

>   So, if I get push submission via IMAP, I think I get a more secure 
> environment for my end-users.  I also get fewer client-to-server 
> protocols to manage and fewer firewall rules.  Overall, this feels 
> like a pretty good win for push.

If you tunnel everything over SSH then you only have one protocol to 
manage in the DMZ and one firewall rule.  That's even better.  If you 
use Webmail then you run everything over HTTP and again you only have 
one protocol and one firewall rule.

--
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Seduced, shaggy Samson snored.
She scissored short.  Sorely shorn,
Soon shackled slave, Samson sighed,
Silently scheming,
Sightlessly seeking
Some savage, spectacular suicide.
                 --Stanislaw Lem

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade