Subject: RE: transfer- and content-encoding

From: "Martin Stecher" <[email-masked]>
Date: Tue, 14 Oct 2003 12:44:14 +0200

Hi,

>
>...
> 
> There are quote a few sane options available to us, depending on what
> encodings have to be supported and on what to do with custom
> encodings that an OCP agent does not support. Given your feedback and
> the multitude of options we face, I would propose the following:
> 
> 	0) Do not negotiate Transfer-Encodings at all.
> 
> 	1) An OCP agent sending data MUST remove all
> 	   transfer encodings it supports. If any encodings remain, an
> 	   OCP agent sending data MUST specify remaining encodings
> 	   using the Transfer-Encoding parameter of a DUM
> 	   OCP message.
> 
> 	2) If an OCP agent receives Transfer-Encoding parameter
> 	   indicating unsupported encoding, it MAY terminate
> 	   the corresponding OCP transaction.
> 
> Do you think the above rules create any interoperability problems that
> more complex rules can eliminate?
> 
> Can we think of a realistic-enough example where removing supported
> encodings is bad for performance reasons? Note that an agent may be
> _configured_ to leave certain encodings -- that qualifies as lack of
> support for their removal. Perhaps the above "MUST remove" can be
> rephrased to better reflect this caveat?
> 
> Do the above rules still allow a callout service to form a chunked
> HTTP response it order to, for example, indicate service progress on a
> persistent HTTP/1.1 connection? Or should we not expect any processor
> to be able to handle chunked responses from a callout service? Should
> we add the following rule?
> 
> 	1a) OPES processors MUST support chunked transfer coding
> 	    when handling data send by an OCP server.
> 

I think removal of transfer encoding by the OPES processor will work and will probably simplify everything.
If we do this, then adding chunked coding by the callout service seems to be odd to me.
Forcing an OCP client to handle chunked transfer coding is maybe unfair; it it is a simple HTTP/1.0 proxy that is so far unaware of chunked transfer encoding, why should it support it? It will need to remove it again before sending on the HTTP path and it is technically not needed. In ICAP/1.0 we need chunked transfer encoding to track the message body length in OCP we do not need this. So there was a reason to force an HTTP/1.0 ICAP client to support chunked coding on the ICAP path but there is only little reason to do this for OCP.

The motivation you give to do this here is very important though.
Let's for now look at the typical case where the OCP client is a HTTP/1.1 proxy and talks to a HTTP/1.1 client and can therefore use chunked transfer encoding.
Some callout services modify the HTTP body and do not know at the time of header processing how long the content will be.
In terms of low latency and ability to keep persistent connections, it is best if the OPES processor then uses chunked transfer coding when talking to the HTTP client.
I am concerned that many OCP client implementations will not implement this. Callout services that do this kind of content modification are the more reliable place for that transfer coding introduction, unless...

Unless we find another good way to highlight this aspect in the specs.
Reminds me that we had parts of this discussion in early June.
I remember that we agreed that the OPES processor is responsible for correct headers and that we can use the sizep parameter to transmit a known content length.
There we ended with two options I remember:

    i) OPES processor MUST ignore Content-Length header
       and depending on existence of sizep parameter
          - adjust the Content-Length header
          - introduce chunked transfer coding
          - collect all data before sending on
          - close the connection at the end
       We can now add: it MUST also ignore Transfer-Encoding header
       because OCP message do not have transfer encodings
       If we document this aspect and make ignoring the Content-Length
       header a MUST, OPES processors are forced to deal with this problem
       and are likely to find a good or acceptable implementation for it.

   ii) Add a data-you-check message which will then mean that the headers
       sent with a data-use-mine message are correct, including Content-Length
       and Transfer-Encoding.
       I do not like this so much, because it forces the OPES processors will be
       again in the dilemma to trust DUM messages but still having responsibility
       for correct headers. And it will only half way solve the Transfer-Encoding
       problem (callout services may still introduce).

So, here is my today's proposal (sorry for changing my mind quickly ;-)
I tries to combine your proposal with option i) above.

 	0) Do not negotiate Transfer-Encodings at all.
 
 	1) An OCP agent sending data MUST remove all
 	   transfer encodings it supports. If any encodings remain, an
 	   OCP agent sending data MUST specify remaining encodings
 	   using the Transfer-Encoding parameter of a DUM
 	   OCP message.
         A new transfer encoding MUST NOT be applied to a message.
 
 	2) If an OCP agent receives Transfer-Encoding parameter
 	   indicating unsupported encoding, it MAY terminate
 	   the corresponding OCP transaction.

	3) If the Content-Length is known by the callout service, a sizep
         parameter MUST be added to all DUM messages.
         An OPES processor MUST ignore Content-Length and Transfer-Encoding
         HTTP headers.
         XXX: Document the options it has:
          - adjust the Content-Length header
          - introduce chunked transfer coding
          - collect all data before sending on
          - close the connection at the end

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Oct 2003 09:35:49 -0600 (MDT)

Martin,

	I first responded to your message in detail (see below). While
responding I realized that we may be missing an important point that
Rob made. That realization prompted me to drastically simplify
encoding rules (see (***) at the end of the message below).

On Tue, 14 Oct 2003, Martin Stecher wrote:

> Unless we find another good way to highlight this aspect in the
> specs. Reminds me that we had parts of this discussion in early
> June. I remember that we agreed that the OPES processor is
> responsible for correct headers and that we can use the sizep
> parameter to transmit a known content length.

Indeed! I believe decoupling processor actions when forming the final
message to the HTTP agent from OCP agent actions is the right thing to
do here. OCP scope is efficient exchange of application messages being
adapted. If a service wants to adapt transfer encodings it can, in
theory, do that. However, that would be a service-specific operation
built on _top_ of OCP mechanisms; not something we have to support
explicitly.

>           - close the connection at the end

Agreed (some polishing of the above may be required).

>        because OCP message do not have transfer encodings

It can only ignore Transfer-Encoding if services are prohibited from
changing transfer encodings. This is actually related to recomputation
of Content-Length header above because Transfer-Encoding affects
Content-Length presence and value.

> So, here is my today's proposal (sorry for changing my mind quickly
> ;-) I tries to combine your proposal with option i) above.
>

>  	   using the Transfer-Encoding parameter of a DUM

The above should be AMS message (not DUM), I think. Sorry. Encodings
have application message scope, not data chunk scope.

To increase interoperability, I would also add

	An OCP processor SHOULD remove chunked encoding. Most
	callout services are not expected to be able to dechunk.

>          A new transfer encoding MUST NOT be applied to a message.

This requirement is out of scope, I think. An agent can always argue
that the encoding was applied before the application message entered
OCP scope. We should always keep this scope picture in mind:

            +--OCP agent--+                  +--OCP agent--+
            |             |                  |             |
        things that       |                  |      things that
        happen outside   -----  OCP scope  ----  happen outside
        of OCP scope      |                  |     of OCP scope
            |             |                  |             |
            +-------------+                  +-------------+
            (processor side)              (callout service side)

Do you see what I am getting at?

>  	   the corresponding OCP transaction.

Agreed.

>          An OPES processor MUST ignore Content-Length and Transfer-Encoding

I do not like the word "ignore". On a logical level, the processor
should delete Content-Length and Transfer-Encoding headers and either
add correct header(s) or do other things you list below.

>           - close the connection at the end

(***) However, I think Rob has a very good point that we are missing
here. Since HTTP transfer encodings are negotiated hop-by-hop by HTTP
agents, the OPES processor or the HTTP proxy that embeds it _have to_
support encodings they receive (or it becomes an HTTP problem)! For
example, an HTTP/1.0 proxy should not receive chunked encoding (by
default) because it will not be able to find the end of a chunked HTTP
message. Thus, I am thinking it would be acceptable to simplify the
above rules:

	Adaptations that use HTTP transfer-encodings MUST be
	explicitly negotiated. This specification does not document
	such negotiations.

	In the absence of explicit transfer-encoding negotiations, an
	OCP agent MUST NOT send transfer-encoded application messages.
	Informally, this means that the agent or its environment have
	to make sure that all transfer encodings are stripped before
	an application message enters OCP scope.

       If an OCP agent receives transfer-encoded application data
       in violation of the above requirement, the agent MAY terminate
       the corresponding OCP transaction.

The new rules above also leave us an opportunity to document
transfer-encoding negotiations later if we decide they are worth
supporting.

You will need to add Content-Length and Transfer-Encoding
"recalculation" rules you mentioned above. As agreed, the processor is
still the enforcement point for those.

HTH,

Alex.

---

From: Robert Collins <[email-masked]>
Date: Thu, 16 Oct 2003 17:10:05 +1000

On Wed, 2003-10-15 at 01:35, Alex Rousskov wrote:
> default) 

should never actually - it's a MUST NOT in rfc 2616.

> because it will not be able to find the end of a chunked HTTP

Yes.

> 	an application message enters OCP scope.

Yes.

>        the corresponding OCP transaction.

I prefer MUST terminate. It will make things cleaner - there is little
benefit in having an undefined behaviour, when we have explicit
requirement for negotiation already there (three paragraphs up :}).

Cheers,
Rob

-- 
GPG key available at: <http://members.aardvark.net.au/lifeless/keys.txt>.