Subject: AW: SMTP filtering use case

From: "Martin Stecher" <[email-masked]>
Date: Mon, 17 Feb 2003 20:52:40 +0100

I have a different view on this.

> Your filtering example is based on delivery to an SMTP server that
> knows that the recipient of the single message is actually an alias.

No, I am not talking about an alias.
It is a very common SMTP behavior that one message is sent to multiple recipients, especially if they belong to the same domain.
This happens in the SMTP dialog.
After the first HELO and MAIL FROM messages the sending MTA sends multiple RCPT TO messages which get all acknowledged one by one.
The receiving SMTP MTA keeps the list of all recipients and delivers to all of them (sometimes it needs then to create copies, sometimes not).

> The OPES service must know the expansion of the alias to the 20
> recipients (any one of which could actually be an alias for another
> list).  This means that the SMTP service and the OPES service are very
> tightly coupled.  As I said, I'd always assumed that the OPES 
> processor
> would have the user preferences, and that's the whole reason for the
> rules, etc., and for having the OPES processor be the 
> enforcement point
> for privacy policy.  But, I guess that assumption is open to debate.

I believe that there will be several implementations of OPES processors that do not want to deal with the complete user preferences and rule engine, but leave this work for the callout service.

> 
> But there's another reason that I'm still not convinced that this SMTP
> example applies to OPES.  If this service and all the user preferences
> are handled by a callout server, there's no reason that the callout
> server cannot just forward the transformed SMTP message to a real SMTP
> server for final delivery.  Because SMTP is a store-and-forward
> service, not end-to-end, this works better than OPES would - there's
> less I/O involved.

This could be done. But we will loose the biggest benefits of the callout strategy at all:
Especially for EMail no message must be lost. The OPES processor is the thing that the user trusts, that has long experience what to do with messages that cannot be delivered immediatly, how to manage queues and so on.
The callout service is "just" for filtering the messages. If it crashes or fails or has a disk crash, it does not affect all waiting messages. That's a very important point I think.

Martin

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 17 Feb 2003 13:52:20 -0700 (MST)

It looks like the big question here is about control over user
preferences. To me, it seems unreasonable to assume that all user
preferences are handled at the OPES processor. It is just too rigid of
a model and often very inefficient since the processor knows very
little about filtering and the filter (OPES server) may need to know a
lot. If we restrict user preference handling to OPES processor, we end
up passing a lot of preference information to the OPES server just so
that it can process a transaction.

Ideally, OPES processor should manage user preferences that affect the
decision of what to filter. OPES server should manage user preferences
that affect the filtering algorithm. But we should not assume a
one-size-fits all scheme.

Thus, it looks to me that Martin's example is valid. I am just not yet
sure whether the performance optimization is worth the complexity.
That is why I asked for more similar uses.

Alex.

---

From: Anil Madhavapeddy <[email-masked]>
Date: Mon, 17 Feb 2003 20:52:12 +0000

On Mon, Feb 17, 2003 at 08:52:40PM +0100, Martin Stecher wrote:
> It is a very common SMTP behavior that one message is sent to multiple
> recipients, especially if they belong to the same domain.
> After the first HELO and MAIL FROM messages the sending MTA sends multiple
> RCPT TO messages which get all acknowledged one by one.
> The receiving SMTP MTA keeps the list of all recipients and delivers 
> to all of them (sometimes it needs then to create copies, sometimes not).
> 

I would note that this is an optimization that not all SMTP 
delivery agents (notably qmail) obey, since it complicates the client
implementation too much.  RFC2821 (Sec 4.5.4.1) says its a 'SHOULD'.

It's also unclear on what to do if there are a huge number of 
recipients at the same mail server (``a limit MAY be imposed'').

It works pretty well in real-life; all servers accept multiple
messages in a single transaction, most clients send them that way,
but you are still allowed to have lightweight clients which choose
to burn some bandwidth for the sake of robustness.

> Especially for EMail no message must be lost. The OPES processor 
> is the thing that the user trusts, that has long experience what to
> do with messages that cannot be delivered immediatly, how to manage 
> queues and so on.

If by 'lost' you mean 'silently dropped', then yes.  I think it's
acceptable for the service to bounce messages back to the sender if
there's some kind of unrecoverable transport error, but silently
dropping them is very, very bad.  And of course, the bounces should
be for rare situations!

-anil

---

From: "Oskar Batuner" <[email-masked]>
Date: Mon, 17 Feb 2003 17:17:18 -0500

> that it can process a transaction.

I'm afraid we are going too deep with analysis of this specific scenario. 
If I understand correctly Martin used preferences just as an example of 
the reason why callout processor may produce multiple responses 
for a single request. 

There may be any number of reasons for the callout processor to 
produce multi-message response, the protocol just SHOULD (MUST?) support 
multiple responses while preserving transactional integrity.

We should not (MUST NOT) impose any limitation on what 
information is available to the callout processor (including 
preferences) and how OPES processor and callout processor assign 
tasks between themselves while performing a collaborative 
processing - this is application specifics and should be 
out of scope for protocol design.

Oskar

> > It is a very common SMTP behavior that one message is sent to 
> multiple recipients, especially if they belong to the same domain.
> > After the first HELO and MAIL FROM messages the sending MTA sends 
> multiple RCPT TO messages which get all acknowledged one by one.
> > The receiving SMTP MTA keeps the list of all recipients and 
> delivers to all of them (sometimes it needs then to create copies, 
> sometimes not).
> >
> > I believe that there will be several implementations of OPES 
> processors that do not want to deal with the complete user 
> preferences and rule engine, but leave this work for the callout service.
> >
> > >
> > The callout service is "just" for filtering the messages. If it 
> crashes or fails or has a disk crash, it does not affect all 
> waiting messages. That's a very important point I think.
> >
> > Martin
> >

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Mon, 17 Feb 2003 15:55:12 -0700

On Mon, 17 Feb 2003 at 17:17:18 -0500 Oskar Batuner avowed, in replying

>  the reason why callout processor may produce multiple responses 

Discussions that reveal schisms in the meaning of the architecture are
very important, whether or not it was the intention of the original
message.  The lack of specifics to date has been difficult for everyone,
and I'm sure that there are more surprises lurking as we discuss a real
protocol for real uses.

>  out of scope for protocol design.

I think it's not as collaborative and loose and all that.  The OPES processor
is the enforcement point for privacy.  It's designed to be a very fast
and fine-grained, policy-based, application-layer switch.  If it is not
the repository for user preferences, then all the policy enforcement
and authentication requirements get pushed to the callout servers.  This
leaves the OPES processor with almost no function - it might as well be
a dumb hardware switch operating at layers 4 through 7.

Hilarie

---

From: Robert Collins <[email-masked]>
Date: 18 Feb 2003 09:49:32 +1100

> multiple responses while preserving transactional integrity.

Placing this requirement on the protocol without a couple of compelling
use cases will add complexity without any gain. I like Alex's cheap-copy
approach more than a 1->N request->response relationship. It's easier to
predict the behaviour of the former.

> out of scope for protocol design.

All protocols exist to support / enable an application. I'm not
suggesting that we should impose limitations - but neither can we design
in a vaccuum.

To me, if user details are in a directory, detection of user preferences
seems to be orthogonal to the OPES transaction. *Identification* of the
user within the OPES transaction may be useful though.

Rob

---

From: "Oskar Batuner" <[email-masked]>
Date: Tue, 18 Feb 2003 10:51:37 -0500

Hilarie,

I did not meant to say that this discussion has no point. I'm
just trying to look at the protocol features that are needed to
support the application requirements that surfaced in the
discussion about Martin's scenario. Essentially this are
transaction integrity and one-to-many mapping between the
protocol messages and the application messages.

Application architecture discussion in fact raises one more
issue - performance requirements for OPES processor and
callout server. One approach is to go from OPES server
performance required to keep up with the request flow.
If callout server has to be in this flow the callout protocol
should be light weight, SOAP and XML do not look good for that.

On another hand we may look at the task performed by the
callout processor. If each request requires significant
processing protocol parsing becomes comparatively less
important - even for heavy weight protocols. If callout
server is powerful enough to keep up with processing requirements
it should be able to do parsing too (this is the situation
when functional request processing is much more demanding than
parsing of heavyweight protocol). Problem of keeping up with
performance requirements that exceed capabilities of single
server may be approached from two directions - either by building
a server farm or by distributing requests from OPES processors to
multiple callout servers. We may look at protocol features that are
helpful to support each approach.

Oskar

> -----Original Message-----
> From: The Purple Streak, Hilarie Orman [mailto:[email-masked]]
> Sent: Monday, February 17, 2003 5:55 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: SMTP filtering use case
>
>