Subject: The power of OCP

From: "Martin Stecher" <[email-masked]>
Date: Mon, 30 Jun 2003 21:53:25 +0200

These days I realized that one specific feature of a virus checking callout server can be implemented much better in OCP than in ICAP/1.0.

It is just one use case but it was an "aha-effect" for me, so I like to share it with you.

If a large file is being downloaded through the OPES processor which sends it to the callout server for virus checking, the server probably need to wait for the complete file before it check for viruses. Until then it does not want to send the data to the end user.

But to present at least some information to the end user while downloading there are various strategies in place to show some download progress. No real solution, just workarounds.
We talked about this before (remember LateClearance content encoding).

One popular strategy is to forward a few percent of the original message while the data is being received and processed by the callout server, known as download progress indication or data trickling.

If a callout server wants to implement this, it has to create an ICAP 200 message in ICAP/1.0 and looses the "204 no modification needed" option.
That means for a 5MB file it starts to send a few 100 bytes and after scanning the file it also has to send the other 4MB.
While the 204 message idea works for the smaller files, it does not work with data trickling and larger files. All data has to be sent forth and back. A real waste of bandwidth since no data is changed.

In OCP the callout server does not need to return any data, if the processor preservers all data and sends "Kept" parameters with its DUM messages.
During the data trickling phase, the callout server returns DUY messages with small sizes and after scanning is complete it sends one final DUY message that allows the complete rest to be sent to the client.
The 5MB is only transferred one way, and only a few bytes for the DUY messages is sent back.

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 30 Jun 2003 14:32:26 -0600 (MDT)

On Mon, 30 Jun 2003, Martin Stecher wrote:

> In OCP the callout server does not need to return any data, if the
> processor preservers all data and sends "Kept" parameters with its
> DUM messages. During the data trickling phase, the callout server
> returns DUY messages with small sizes and after scanning is complete
> it sends one final DUY message that allows the complete rest to be
> sent to the client. The 5MB is only transferred one way, and only a
> few bytes for the DUY messages is sent back.

This works in theory. The hard part is to convince the processor to
preserve/keep 5MB of data. It is a lot of overhead for the processor
to keep all the data, and I am sure that a general-purpose
implementation may not be able to keep the whole thing by default.

OCP does not (yet?) have any mechanisms to (a) encourage preservation
or to (b) notify the server that the preservation buffer is limited to
X bytes and any spills will not be preserved. Do you think it should
be left to external negotiation channels (configuration files, etc.)?
Or should we try to support this directly? Which mechanism?

Thanks,

Alex.