Subject: RE: what-parts-to-send-or-skip

From: "Martin Stecher" <[email-masked]>
Date: Fri, 24 Oct 2003 22:58:32 +0200

Hi Alex,

thank you for summarizing this so well.

I agree to all facts but #7:

> 
> 	7) Adapting just HTTP headers is less common than
> 	   adapting bodies. Moreover, adapting just HTTP
> 	   headers may be done better via a dedicated
> 	   profile
> 

There are many services that only deal with headers, especially in the request profile.
Examples are privacy filters (delete Referer and Cookie headers) and URL blocker.

> 
> Given the above, I conclude that:
> 
> 	- We should keep AM-Part and Aux-Part to accommodate (A),
> 	  to have an interface for auxiliary parts in RESPMOD,
> 	  and to support short-circuit operation.

Yes.

> 
> 	- We should rely on OCP Core "getting out of the
> 	  loop" mechanisms to accommodate (B). That would
> 	  let us terminate data transmission early enough,
> 	  though not as early as with Skip-Part in some
> 	  cases. We should delete the Skip-Part interface.

Probably yes.
I have a little concern about the "early enough".
Let's take a service in RESPMOD that deletes cookies from
HTTP headers, nothing else. If that service is not fast
in responding and sending the i-want-out message, the OPES
processor will send much data of the body, wasting bandwidth.

> 
> 	- Optionally, we could add an optional Last-Needed-Part
> 	  parameter to NR to allow specifying the _last_
> 	  original part needed by the service and being semantically
> 	  equivalent to receiving (later) a corresponding Want-Use
> 	  parameter with the actual size of all parts up to the
> 	  last one. Processors MAY ignore Last-Needed-Part.
> 	  With these rules, Last-Needed-Part does not introduce
> 	  any new complicated requirements/dependencies.
> 
> Martin, do you think the above would be sufficient to accommodate most
> of our needs? If yes, do you think adding Last-Needed-Part is worth
> the trouble?

How do you see my concern above? Is it only valid for esoteric services?
Last-Needed-Part would address this. But maybe we can add anoter
parameter "Pause-Offset" to the feature which acts as a data-pause
pre-request. It requests a first data-paused message after n bytes
of the original body.
As with data-pause, OPES processor SHOULD honor this request.
This will be similar to the Preview feature of ICAP/1.0.

I think this fits even better into OCP context and can also be used
for other purposes (always where we have to assume that the callout
service is not as fast as the OPES processor).

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 24 Oct 2003 15:59:26 -0600 (MDT)

On Fri, 24 Oct 2003, Martin Stecher wrote:

> There are many services that only deal with headers, especially in
> the request profile. Examples are privacy filters (delete Referer
> and Cookie headers) and URL blocker.

You are right -- I forgot about URL blocking.

	7) Adapting just HTTP headers of messages with
	   large bodies is less common than adapting
	   large bodies.

> I have a little concern about the "early enough". Let's take a
> service in RESPMOD that deletes cookies from HTTP headers, nothing
> else. If that service is not fast in responding and sending the
> i-want-out message, the OPES processor will send much data of the
> body, wasting bandwidth.

That was my concern as well. Note that the service may be "fast
enough"  but the processor itself may not read service responses "fast
enough" or there could be a network problem. In a case of already
congested network, we are more likely to run into problems, and hence
are more likely to make the situation worse!

> How do you see my concern above? Is it only valid for esoteric services?

URL filtering is not esoteric for sure. When done on requests, it is
probably not a big deal because requests have no or small bodies.
However, adapting based on MIME type or other response header
information may not be that uncommon. It would be very nice to support
that efficiently and Last-Needed-Part does not do that.

> This will be similar to the Preview feature of ICAP/1.0.

Good, except for the Offset part. Since the offset is not known a
priory, services that need just headers will have to overprovision on
average but will not get enough sometimes.

I see two good options:

Option A:

	- add Wont-Need-Part parameter to Profile in HTTP
	  equivalent to Wont-Need, but uses parts, not offsets
	  Meaning: I know for sure that I do not need anything
	  starting with the given part

	- add Data-Pause parameter to Feature in Core
	  equivalent to an a priory Data-Pause message
	  Meaning: Let me tell you later wether I need anything
	  starting with the given offset

	- add Data-Pause-Part parameter to Profile in HTTP
	  equivalent to Data-Pause message, but uses parts
	  Meaning: Let me tell you later wether I need
	  starting with the given part

	- add Wont-Need parameter to Feature in Core?
	  (for symmetry reasons)

Option B:

	- add a new Offset parameter type to Core:

	  Offset: structure {
			  	[Octet: size];
			  	[AM-Part: atom];
	          }

	- add Wont-Need and Data-Pause parameters to Feature
	  in Core; the sender can use Offset and/or Part
	  members.

Option A is more explicit/straight. Option B reduces the number of
virtually identical parameters to deal with and lets other bindings
reuse the Offset type.

Both options do what we need and do not introduce any _new_ conflicts
or concerns. They are equivalent to already existing
messages/parameters.

Which option would you prefer?

Thanks,

Alex.