Subject: RE: The power of OCP

From: "Martin Stecher" <[email-masked]>
Date: Mon, 30 Jun 2003 22:55:52 +0200

> 
> > In OCP the callout server does not need to return any data, if the
> > processor preservers all data and sends "Kept" parameters with its
> > DUM messages. During the data trickling phase, the callout server
> > returns DUY messages with small sizes and after scanning is complete
> > it sends one final DUY message that allows the complete rest to be
> > sent to the client. The 5MB is only transferred one way, and only a
> > few bytes for the DUY messages is sent back.
> 
> This works in theory. The hard part is to convince the processor to
> preserve/keep 5MB of data. It is a lot of overhead for the processor
> to keep all the data, and I am sure that a general-purpose
> implementation may not be able to keep the whole thing by default.

I know at least one ICAP/1.0 implementation that can keep copies of very large files (swapping to disk if needed).
Virus filtering is one of the most wanted callout service today. It makes sense to design OPES processors in a way to support those best.
While many OPES processors are caching proxies and because virus filter services often do not change data, it makes sense to store already the original data into the cache, being prepared to replace it later if required and so preserving all data anytime.

> 
> OCP does not (yet?) have any mechanisms to (a) encourage preservation
> or to (b) notify the server that the preservation buffer is limited to
> X bytes and any spills will not be preserved. Do you think it should
> be left to external negotiation channels (configuration files, etc.)?
> Or should we try to support this directly? Which mechanism?

I do not think right now that exchanging this data or even negotiating on this will really change something.
A processor may or may not have limitation for its data preservation capabilities. Preferences of the callout server will probably not change anything here.
Callout servers may like to make intensive use from preserved data but be definition need to be prepared that the processor stops to preserve more data; it has to handle that anyway. No need for a special announcement I guess; it won't change the algorithm I guess.

Am I missing something?
Do you know of examples where exhanging of that information will make a difference?

Martin

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 30 Jun 2003 15:24:59 -0600 (MDT)

On Mon, 30 Jun 2003, Martin Stecher wrote:

> I do not think right now that exchanging this data or even
> negotiating on this will really change something. A processor may or
> may not have limitation for its data preservation capabilities.
> Preferences of the callout server will probably not change anything
> here. Callout servers may like to make intensive use from preserved
> data but be definition need to be prepared that the processor stops
> to preserve more data; it has to handle that anyway. No need for a
> special announcement I guess; it won't change the algorithm I guess.
>
> Am I missing something? Do you know of examples where exhanging of
> that information will make a difference?

A translation service will not use any of the data. An ad banner
stripping service will use all but the prefix data. The virus filter
service will use all the data. Ideally, the processor should not keep
5MB file to be translated but should keep the 5MB file being filtered.
How does the processor distinguish among these cases? Should the
processor assume that most data will be re-requested and preserve
aggressively until it receives a Wont-Use message?

Alex.