Subject: [lemonade] Re: Can I offer you some lemonade?

From: Steve Hole <[email-masked]>
Date: Fri, 13 Feb 2004 16:32:10 -0700

On Thu, 12 Feb 2004 17:42:38 -0600 Pete Resnick <[email-masked]> 
wrote:

> Hey Steve,

Hey Pete.   Long time since we last talked.
 
> I don't know if you've been at all following the LEMONADE working 
> group. We're having this big discussion about "forward without 
> download", and it's pretty polarized. I have a lot of respect for 
> your opinion on this sort of thing (and I think others in the IMAP 
> community do to). I'd be interested in hearing your opinion as to 
> whether I'm just nuts and there's really something to the argument 
> the other side is making, or otherwise. Can I rope you in? 

Well thank you.   And yes you can.

> I'll try 
> to objectively summarize for you or point you to particular messages, 
> even if you don't want to read the whole archive.

Prior to this I had not spent any time on Lemonade.   Surprise ... I 
actually have a draft with my name on it inside.

I went and read through the archive of messages starting in January.   I 
believe that it gave me enough to go on.

Nobody is nuts that I can see.   It seems like a good discussion of the 
issues and both sides have valid points.   I was a little surprised that 
it was so polarized, but then I remembered who was participating :-)    
You guys are all getting older, you should have mellowed somewhat by now. 

I'm in agreement with Chris Newman's (very well stated) position, and
with Mark Crispin on this.   That is, I would say that I'm a PULL guy.

Several reasons have been tossed out to support either side and most of 
them carry water even when in total contradiction.   The obvious reason 
for this is that the issue is very complex.

All reasoning about charters, existing infrastructure deployments and 
client vs. server complexity aside, the key decision criteria seems fairly
obvious to me -- what features are offerred by the alternative approaches. 
I don't think that I buy into any arguments for either side that suggest 
one or the other is going to be more or less difficult to implement.    
One of the reasons for this is that I have actually (to my surprise) done 
both of them in some form as private mechanisms.

The PULL mechanism offers a better abstraction of the service that you 
want to employ.   I believe the following to be facts:

1.  The abstraction of where content comes from is both complete and 
source independent.    That is, you can specify anything you want as long 
as it can be expressed in a URI, a mechanism to dereference the URI is 
available in the service implementation and the resource is "visible".  
That's nice.

You'd probably want a mandatory to implement set of supported mechanisms, 
obviously including IMAP (and maybe only including IMAP).

Good abstraction is a fundamental design goal that I rarely select 
against.   Sort of a "SHOULD" on steriods.

2.  The locality within a single service means that it can be used by more
than just IMAP client use.   Other services can then use the same facility
for doing other types of composition work with little or no extension.   I
would think it good engineering practice to provide a general solution 
that exceeds the requirements for which it was conceived. I'm sure that 
there are those who will state that it opens a Pandora's box or some such,
but in practice general solutions are always better than specific ones if 
the development effort is within a magnitude of one another.

In a twist of fate, it turns out that the work that one of my product 
groups is doing maps quite closely (if not exactly) to the PULL 
architecture. The product takes an XML message definition on the way into 
the "submit" server that is then composed into a MIME message and 
submitted.   The XML message definition supports a variety of URI schemes 
for identifying content within the MIME parts.   Specifically: HTTP, FILE,
IMAP, a private JOBDATA scheme which references input queue (MQSeries) data
buffers, as well as "inline" content.

The interesting thing is that the service is used by a variety of 
applications that wish to generate structured mail messages with 
forwarded content (Customer Service Representative web applications), as 
well as a dedicated IMAP mail client. This seems to be anecdotal evidence 
that a generalized composition facility that supports forwarding IMAP 
stored content is both desirable and an eventuality.

There are some serious issues to resolve with the PULL scheme:

1.  Security.   Both PUSH and PULL have this problem though and you just 
have to deal with forwarding credentials.   I wish you great good luck in 
this endeavor.

2.  Representation of a "message that must be filled".   I did it with an 
XML representation of a fully secured (S/MIME) MIME structure.    This 
will have to do it somewhat differently within the constraints of MIME 
itself.   I'm sure that can be done reasonably however.

Cheers.

---
Steve Hole
Chief Technology Officer
ACI Worldwide
<mailto:[email-masked]>
Phone: 780-424-4922

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Pete Resnick <[email-masked]>
Date: Sat, 14 Feb 2004 11:12:43 -0600

On 2/13/04 at 4:32 PM -0700, Steve Hole wrote:

>>Can I rope you in?

Thanks for the excellent analysis. I've got a few followup questions myself.

>Nobody is nuts that I can see.

Well, you mean on this issue, right? :-)

>I don't think that I buy into any arguments for either side that 
>suggest one or the other is going to be more or less difficult to 
>implement. One of the reasons for this is that I have actually (to 
>my surprise) done both of them in some form as private mechanisms.

Well, that's actually good to hear. I don't know that I'm convinced, 
but it's at least comforting to know that we're not in trouble either 
way we go.

>2. The locality within a single service means that it can be used by 
>more than just IMAP client use. Other services can then use the same 
>facility for doing other types of composition work with little or no 
>extension. I would think it good engineering practice to provide a 
>general solution that exceeds the requirements for which it was 
>conceived.
[...]
>This seems to be anecdotal evidence that a generalized composition 
>facility that supports forwarding IMAP stored content is both 
>desirable and an eventuality.

Would your opinion of this change if the mechanism we come up with 
does *not* do the composition work on the PULL server? That is, one 
of the proposals is to do all of the composition work on the IMAP 
server and then PULL only completed messages from there. (Some of the 
reasons for this are that it provides one solution to the "save a 
copy of the sent message" problem and it simplifies the protocol down 
to handing off a single URI.) I personally like this mechanism, for 
either PUSH or PULL. (I did write the draft on it, so it is close to 
my heart. :-) ). But it does limit the flexibility of the mechanism, 
since it would require PULL-ing only completed messages, and that's 
not something you're likely to find today via HTTP, for example.

Would you still be in favor of PULL if it didn't provide the 
composition facility on the PULL server?

>1.  Security. Both PUSH and PULL have this problem though and you 
>just have to deal with forwarding credentials. I wish you great good 
>luck in this endeavor.

One of the suggestions on the list (which so far I believe) is that 
with PUSH, you *could* design your network such that the submit 
server trusts all and only submissions from the IMAP server, thereby 
not having to explicitly forward credentials or even provide an 
authentication/authorization database to the submit server, whereas 
with PULL you couldn't do something similar because having the IMAP 
server trust the submit server would mean that a compromise of the 
the submit server would compromise all of the data on the IMAP 
server. We've not had any security people comment on this to date, so 
maybe this is just nonsense. Are you saying you don't buy the 
argument?

>2.  Representation of a "message that must be filled".

One of the PULL proposals is that the client would actually generate 
this on the fly, basically sending a "insert this data here into the 
message" command when it wants to include something external. Did you 
encounter some sort of problem that required having a pre-composed 
"message that must be filled", or was that just an implementation 
choice?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Randall Gellens <[email-masked]>
Date: Sun, 15 Feb 2004 16:28:47 -0800

At 11:12 AM -0600 2/14/04, Pete Resnick wrote:

> server and then PULL only completed messages from there. (Some of 
> the reasons for this are that it provides one solution to the "save 
> a copy of the sent message" problem and it simplifies the protocol 
> down to handing off a single URI.) I personally like this 
> mechanism, for either PUSH or PULL. (I did write the draft on it, 
> so it is close to my heart. :-) ). But it does limit the 
> flexibility of the mechanism, since it would require PULL-ing only 
> completed messages, and that's not something you're likely to find 
> today via HTTP, for example.

I thought the proposal was that composition could be done on the IMAP 
server and then the composed message be submitted, or composition 
could be done during submission.  That is, if the message has been 
composed on the IMAP server than the simplest case of external 
reference is used on submission (entire message in one reference), 
otherwise muliple references are used.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Majority: That quality that distinguishes a crime from a law.

---

From: Steve Hole <[email-masked]>
Date: Sun, 15 Feb 2004 22:47:38 -0700

> composition facility on the PULL server?

The question is "why would you limit it"?   I would make it any valid URI 
scheme, with a mandatory to implement for the IMAP URL scheme.   That way 
you leave it up to the implementor to decide which they will support.   
I'm not sure how you would advertise what was supported, but it might not 
matter.  Any implementation worth it's salt would make such support 
extensible (factory) oriented so that you can switch on the scheme and try
and find a suitable driver.
 
> maybe this is just nonsense. Are you saying you don't buy the 

Hmm ...

It is clear that you have to solve the issue of passing credentials for 
accessing the IMAP server from the Submit server.   It is not clear that 
you could get around the inverse problem of passing credentials to the 
Submit server from the IMAP server by creating an authorization linkage 
between the IMAP server and the Submit server.   As these things are quite
likely running on separate machines, I think that you have the same 
problem no matter what you do.   Any scheme that would require colocation 
of the Submit server and the IMAP server would be architecturally ill 
conceived.

Either you have to provide for credential forwarding or proxy 
authentication (via SASL) in either case I would think.   In either case, 
I'm not really in love with the security prospects.   I think the idea is 
reasonably cool, but the security problems have always been the blocker 
for doing the work.   I remember talking about this in the core IMAP group
at least 5 years ago without any real good ideas on the security.

We get around this in our implementation by using Kerberos.  Difficult to 
mandate this level of service in a standard though.

Thus the "I wish you good luck" statement.

> "message that must be filled", or was that just an implementation 

We had a choice to provide an external composition specification or extend 
the MIME headers to include the external content references.  In the end, 
it was private, more expressive and just easier to do things in XML. We 
weren't trying to build something that was intended for standardized use. 

I've attached the DTD grammar for interest sake.   It's a stripped version
that deals only with the message structure (822 and MIME structure 
oriented data) version because there are some things in the larger 
composition scope that have competitive significance.

In the end, there really isn't anything magical.   I suspect that a single
new Content-{mumble} header or a variation on an existing one will suffice.

Cheers.

---
Steve Hole
Chief Technology Officer - Billing and Payment Systems
ACI Worldwide
<mailto:[email-masked]>
Phone: 780-424-4922

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Feb 2004 18:12:38 -0800 (PST)

> of the Submit server and the IMAP server would be architecturally ill

I believe that Push assumes that submit servers are either co-located with 
the IMAP server or will promiscuously accept anything from the IMAP 
server.  As such, the claim is made that there is no need for any 
credentials passing in the Push case.

This is one of the major problems that I have with Push.  IMHO, Push is 
designed for an MSA architecture which is rapidly heading towards 
extinction.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Ted Hardie <[email-masked]>
Date: Tue, 17 Feb 2004 10:09:42 -0800

At 10:47 PM -0700 02/15/2004, Steve Hole wrote:
>scheme, with a mandatory to implement for the IMAP URL scheme.

And the answer would be "because different access protocols have
different security characteristics, uses of the network, and presumptions
about change".  "Any valid URI scheme" is clearly the wrong answer
since that would include schemes like urn: and cid: which don't make
any sense for this use.  You'd have to limit it to "any access
protocol commonly used for data retrieval", but there would remain
a lot of slop in that bucket (would epp qualify, for example?).  Even
with that, you'd end up with very different characteristics for retrieval
timing that would effect the deployment.

>   That way
>matter.
>   Any implementation worth it's salt would make such support

And this foster interoperability how?  Without a limited set of URI
schemes, you're going to have to either add an advertisement capability
or run the risk of getting a negative response after the SUBMIT
server has parsed the data, which at best is going to add round
trips we're trying to avoid.

				regards,
					Ted HArdie

---

From: Steve Hole <[email-masked]>
Date: Tue, 17 Feb 2004 11:33:41 -0700

> trips we're trying to avoid.

So you get a negative response?   Do you really think that that is going 
to happen all that often?

If you are just doing good old email, then you would probably only ever 
use one of file, http, or imap -- with imap being by far the most likely.
If you are doing something specific (like a voice mail system) you might 
want some streaming access protocol. In the private use case, the "client"
and the "submit" server are pretty likely to agree.   This is also the 
case where external references really win for you.
 
If you want to promote interoperability over a wider range of 
functionality then add more schemes to the set of mandatory to implement. 
The base specification can still specify a syntax that includes any URI 
scheme.  If you're keen, then add file and http.

In any case, what I was talking about was providing a base from which 
reasonable extension can be done -- most likely by something other than a 
traditional mail client. 

Cheers.

---
Steve Hole

---

From: Ted Hardie <[email-masked]>
Date: Tue, 17 Feb 2004 10:22:37 -0800

At 6:12 PM -0800 02/16/2004, Mark Crispin wrote:
>I believe that Push assumes that submit servers are either 
>co-located with the IMAP server or will promiscuously accept 
>anything from the IMAP server.  As such, the claim is made that 
>there is no need for any credentials passing in the Push case.

I don't agree that Push assumes that submit servers are co-located with
the IMAP server.  I also don't agree with "promiscuously accept anything
from the IMAP server", but I do agree that many deployments would
be able to accept the IMAP server's credentials as sufficient for
injection into the SMTP stream, rather than needing to see the individual's
credentials for itself.
Think of it in terms of normal SMTP operation.  You commonly
have boxes arranged like this:

-----		----			----	----
| 1    |-----------|  2 |-----[internet]-------| 3  |----| 4  |
-----		 ---			----	----

where 1 is inside enterprise A's firewall and 2 is in its DMZ, 3 is in
enterprise B's DMZ and 4 is in its dmz.  2 never sees the end users'
credentials; it is configured to allow 1 to transmit to it based on
the server credentials.  The "upstream" SMTP servers to the IMAP
submit server might be configured to behave in the same way.

As usual, speaking only for myself here,
				regards,
					Ted Hardie

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Feb 2004 10:53:48 -0800 (Pacific Standard Time)

On Tue, 17 Feb 2004, Ted Hardie wrote:
> I also don't agree with "promiscuously accept anything
> Think of it in terms of normal SMTP operation.

As I said in my previous message, you are presuming an architecture that 
is (IMHO) heading towards extinction.  What may be "normal" today is not 
necessarily going to be "normal" for very much longer.

Just as an enterprise inevitably outgrows Linksys NAT boxes and has to 
deploy real networking hardware, the time comes when the "SMTP server in 
the DMZ" architecture is outgrown.

The proliferation of email-based viri makes that painfully clear if 
nothing else does.

-- Mark --

---

From: Randall Gellens <[email-masked]>
Date: Tue, 17 Feb 2004 17:00:03 -0800

At 10:09 AM -0800 2/17/04, Ted Hardie wrote:

>  about change".

I'd support an initial limitation to IMAP URL, since that solves our 
immediate problem, and leaves the protocol in a state that can be 
easily extended.

>  Without a limited set of URI

Since we're talking about Submit here, it seems pretty easy to add an 
advertisement capability.  We can leave that door open in the initial 
version.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The nice thing about standards is that there are so many of
them to choose from.                  --Andrew S. Tanenbaum

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Feb 2004 18:09:35 -0800 (Pacific Standard Time)

On Tue, 17 Feb 2004, Randall Gellens wrote:
> immediate problem, and leaves the protocol in a state that can be easily 
> extended.

I would support it as well, and for the same reasons.

Although I believe that this limitation will soon be overturned (sooner 
than some people think), that is a task for some other WG.  Lemonade will 
have done its duty by (1) solving its tasks and (2) not making the future 
WG's job needlessly harder.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 18 Feb 2004 12:57:59 -0600

On 2/15/04 at 10:47 PM -0700, Steve Hole wrote:

>The question is "why would you limit it"?   I would make it any 
>valid URI  scheme, with a mandatory to implement for the IMAP URL 
>scheme.   That way you leave it up to the implementor to decide 
>which they will support.  I'm not sure how you would advertise what 
>was supported, but it might not matter.  Any implementation worth 
>it's salt would make such support extensible (factory) oriented so 
>that you can switch on the scheme and try

No, that wasn't the question I was asking.

In your implementation, you hand the submit server a template in XML 
that has pointers to the things to collect. That's different than 
what's being proposed here. The pull mechanism envisioned here is 
that the client issues effectively a special DATA command where it 
can say, "Here's some message headers" followed by "here's a URI to 
some data to insert here", followed by more text or more URIs. I'm 
not asking about limiting it to one URI scheme. I'm asking what you 
would think if we limited it to a single URI per command, with no 
surrounding text. That is, what would you think if composition was 
always done elsewhere and the submit server was only ever handed a 
single URI to a completed message?

pr

---

From: Mark Crispin <[email-masked]>
Date: Wed, 18 Feb 2004 11:45:23 -0800 (PST)

On Wed, 18 Feb 2004, Pete Resnick wrote:
> In your implementation, you hand the submit server a template in XML that has 
> pointers to the things to collect. That's different than what's being proposed 
> here. The pull mechanism envisioned here is that the client issues effectively 
> a special DATA command where it can say, "Here's some message headers" followed 
> by "here's a URI to some data to insert here", followed by more text or more 
> URIs.

Not necessarily.  It is possible to dispense with BURL entirely and go to 
a Embedded URL only mechanism such as what Steve did.  Doing so is not 
particularly draft or fcc friendly, which is why I didn't include that 
possibility in the Pull draft; nor did I hear much of a constituency for 
this type of solution prior to Steve's message.

I wouldn't object to it though if that's the way the wind ended up.

> I'm not asking about limiting it to one URI scheme. I'm asking what you 
> would think if we limited it to a single URI per command, with no surrounding 
> text. That is, what would you think if composition was always done elsewhere 
> and the submit server was only ever handed a single URI to a completed message?

That's the CATENATE/BURL variant of Pull, and it is described in the Pull 
draft.  There are benefits to that variant, but also severe deficiencies. 
I don't think that we should talk about restrictions that force that 
variant without first considering the other variants.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 18 Feb 2004 14:04:07 -0600

On 2/18/04 at 11:45 AM -0800, Mark Crispin wrote:

>>In your implementation, you hand the submit server a template in 
>>XML that has pointers to the things to collect. That's different 
>>than what's being proposed here.
>
>Not necessarily.  It is possible to dispense with BURL entirely and 
>go to a Embedded URL only mechanism such as what Steve did.

I didn't intend to (nor do I think I did) say that it wasn't 
possible. Only that it wasn't currently being proposed.

>Doing so is not particularly draft or fcc friendly, which is why I 
>didn't include that possibility in the Pull draft; nor did I hear 
>much of a constituency for this type of solution prior to Steve's 
>message.

I think there was actually some earlier opposition to it due to 
complexity. It is of course compatible with push as well.

>>I'm not asking about limiting it to one URI scheme. I'm asking what 
>>you would think if we limited it to a single URI per command, with 
>>no surrounding text. That is, what would you think if composition 
>>was always done elsewhere and the submit server was only ever 
>>handed a single URI to a completed message?
>
>That's the CATENATE/BURL variant of Pull, and it is described in the 
>Pull draft.

Right. That's what I was asking for an opinion of.

>There are benefits to that variant, but also severe deficiencies. I 
>don't think that we should talk about restrictions that force that 

I'm not asking anyone to disregard the other variants. I am asking 
for an opinion on that particular variant. If Steve (or others) 
responded, "That variant makes me feel even more strongly that pull 
is the way to go for such and so reasons", or alternatively, "Eeek! 
If that were the only variant left on the table, I'd rather go with 
push", I think that would be interesting information to know.

pr

---

From: Steve Hole <[email-masked]>
Date: Wed, 18 Feb 2004 22:11:23 -0700

On Wed, 18 Feb 2004 12:57:59 -0600 Pete Resnick <[email-masked]> 
wrote:

> not asking about limiting it to one URI scheme. 

Ah!

In my implementation the submit server (or moral equivalent) actually 
interprets the MIME structure (as expressed by the XML specification).  It
builds the message from the inline data and the "pulled" external content.

I wasn't able to find the BURL draft so I wasn't sure how you were going 
to do it in the submit server.   I mistakenly assumed that the submit 
server would be MIME smart and would expand URI references within the MIME
body structure (within a Content-{mumble} header) to insert external 
content.

What you have described is simpler in that the submit server doesn't have 
to know anything about MIME and you don't require any extensions to the 
MIME format to represent the insert data.  The submitting MUA has to write
out the data with the appropriate breaks for external inserts -- which 
would be somewhat awkward in the MIME dumpers (formatters) that I've 
worked with, but not impossible.   It's certainly easier than adding MIME 
smarts into the submit agent.

> I'm asking what you 

Well, it just seems like you then have to have composition knowledge in 
three applications (MUA->COMPOSER->SUBMIT) rather than just (MUA->SUBMIT).
I'm not sure why you would want to do that. 

So, once again, I don't see why you would want to limit it.  I suspect 
it's because most MIME formatters dump a complete serialized message into 
a temporary buffer (memory or file) and then just blast it out.   Breaking
that into pieces will mean changing existing implementations.   I don't 
think it's any worse (and likely easier) than adding MIME dumper smarts to 
an IMAP server.

Cheers.

---
Steve Hole