Subject: Re: HTTP/OCP protocol binding

From: "Martin Stecher" <[email-masked]>
Date: Fri, 30 May 2003 23:51:37 +0200

> 
> > I think it is time to start thinking about application protocol
> > bindings. As we agreed, HTTP first.
> >
> > This will help to validate OCP core and to detect some potential
> > problems. In addition we can then soon start to develop some
> > prototypes that help verifying whether the procotol works.
> 
> I agree and will start putting an HTTP binding draft together unless
> somebody else wants to take the lead. However, I suspect that we may
> need a more general "Common OCP data encodings" draft (see below) as
> well. Or, should common encodings become a part of OCP Core?
> 

I prefer to keep the number of documents someone needs to implement
the protocol at a minimum.

> > Here are some first questions regarding HTTP/OCP:
> >
> > - Transactions for HTTP requests/responses
> > How do OCP transactions look like and differ if they are used at
> > activation points 1-2 and 3-4, i.e. OCP transactions for HTTP
> > requests and responses; or REQMOD vs. RESPMOD for us ICAP guys.
> 
> OCP Core transactions will look identical regardless of the activation
> points because activation points are application-specific things
> beyond OCP Core scope. If activation point is important, it has to be
> passed as metadata or as an [HTTP- or cache-specific] extension OCP
> message/field. HTTP binding draft will have to document how this is
> done, I guess. An extension field to app-message-start message seems
> most appropriate to me:
> 
> 	app-message-start ...
> 	...
> 	Activation-Point: request
> 
> or
> 	app-message-start ...
> 	...
> 	Activation-Point: response
> 
> What do you think?

The activation point itself is not important (in most cases at least)
while it is of course important whether the message encapsulates only
a HTTP request or a HTTP response with original request as meta information.

> 
> > - HTTP meta data
> >
> > Will HTTP headers be simply the payload of a meta-have message? Is
> > the first line special? Will it be coded into named parameters of
> > meta-have messages? What about the empty line between HTTP header
> > and body? Does it belong to the meta data?
> 
> I hope there will be no meta-have messages (see my posting titled "OCP
> metadata == data"). 

I forget about that one. Thanks for the reminder.

> I would use some very simple encoding to pass HTTP
> messages in OCP payload. Something along these lines:
> 
> 	<chunk-type> <chunk-length> <chunk-data>
> 
> where "chunk-type" can be either of
> 	"headers":  HTTP headers including the first line and the last
> 	            CRLF terminator
> 	"trailers": HTTP trailers including the last CRLF terminator
> 	"body":     raw HTTP message payload
> 	"all":      raw HTTP message
> and one OCP payload may contain several chunks in the above format.
> 

For a HTTP response we need the original HTTP request as additional
meta data. That needs to be added somehow.

> [...]
> 
> > - Message length and transfer encoding
> >
> > How to handle HTTP message body in chunked transfer encoding? Remove
> > the encoding before sending via OCP?
> 
> I think this should be left to implementations to decide. The
> recipient MUST handle all valid HTTP encodings, but it is up to the
> sender how to pre-process the message. Recall that, from OCP point of
> view, any kind of preprocessing is out of scope.

While preprocessing is not in OCP scope, it should be o.k. to define
some requirements for application messages that may make some pre-
processing nececssary for some other types of the application message.
What I mean is: I think we can define "only identity transfer encoding
in this version of HTTP protocol binding of OCP" and so force OPES
processors to do some preprocessing for HTTP messages that have a
different transfer encoding.

Not saying that we should make such a rule. I am not 100% sure in
this moment. Could it be an option that is negotiated? Defining
which transfer encodings the callout server supports?

> 
> OCP agents can negotiate more specific requirements, I guess. For
> example, they can negotiate that chunked encoding is always used or
> that identity encoding is used.
> 
> > What is with the Content-Length header? Who is responsible for
> > adding/changing/removing it?
> 
> The sender of the header. For example:
> 
> 	- If a callout server sends "headers" or "all" chunks
> 	  back to the OPES processor, then that callout server
> 	  must ensure that all headers it sends match the body it
> 	  sends. That includes adjusting or removing original
> 	  Content-Length as needed. The OPES processor may
> 	  further adapt the body and Content-Length, of course.
> 
> 	- if an OPES processor sends just the "body" chunk to
> 	  the callout server, then the OPES processor is responsible
> 	  for matching the headers with the adapted body. This
> 	  mode lets us implement services that are not HTTP-dependent.
> 	  Note that data encoding will have to be negotiated in this
> 	  case.
> 
> > Asynchronous OCP data handling and persistent HTTP/1.0 connections
> > is not easy.
> 
> I do not see any complications. Note that we are not adapting
> connections, only messages. Can you give an example where OCP and HTTP
> persistency make things difficult?

What I saw with ICAP implementations:

- Some ICAP servers forget to adjust the Content-Length header although
they change the body length. ICAP clients have a hard time then.

- Often ICAP servers know that they are going to change something but
do not yet know when sending the HTTP response header back how big the
body will be. So, they need to delete the Content-Length header.
If this is a HTTP/1.0 message, the ICAP client needs to close the
connection; or it checks whether it can turn the message into a HTTP/1.1
message and adds chunked transfer encoding.
But most ICAP clients don't do this, so that ICAP servers started to
do the HTTP/1.1 transformation and add chunked encoding thing.
Big problem if the ICAP client does not support this because it is not
really HTTP/1.1 compliant.

I wonder whether we could make things clearer and easier and defining
who is responsible for what.
What about adding a parameter to the app-message-start message that the
OPES callout server returns that has the message size (content-length)
if already known at that time.
The OPES processor should be the only one being responsible to the
real HTTP message exchange with the HTTP client and can select whether
to close the connection, to keep it open, to change Content-Length header
or to add a transfer encoding. The OPES processor does not need to trust
the Content-Length header returned but the value of the new app-message-
start parameter, which if present signals that the callout server
knows what it does.

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 30 May 2003 16:40:49 -0600 (MDT)

> meta data. That needs to be added somehow.

Indeed, I forgot about that requirement. Will having both "request-*"
and "response-*" chunk types be too complicated?

	request-headers, response-headers
	request-trailers, response-trailers
	request-body, response-body
	request-all, response-all

(or the other way around: *-request and *-response).

We will also have to explicitly document what to do with 1xx HTTP/1.1
responses. It would be nice if we can adapt them, but I do not know
yet whether they have to be treated specially from HTTP binding point
of view.

> processing nececssary for some other types of the application

I agree.

> What I mean is: I think we can define "only identity transfer
> encoding in this version of HTTP protocol binding of OCP" and so
> force OPES processors to do some preprocessing for HTTP messages
> that have a different transfer encoding.
>

I think this MUST be negotiated, with a natural caveat that identity
encoding MUST be supported by all parties. We should make sure that
both chained and HTTP-unaware callout services can be supported
efficiently.

> they change the body length. ICAP clients have a hard time then.

Can always happen, I guess. We cannot prevent that at a protocol level
unless we do not want to support persistency. However, a smart OCP
client can detect a OCP broken server, mark it as such, and switch to
non-persistent connections after the first broken response. Also, a
client may be manually configured to ignore server-sent Content-Length
(which would mean no HTTP/1.0 persistency as well, I guess).

> really HTTP/1.1 compliant.

Again, this seems to be a real-world problem without a general
solution because some agents are buggy or too assuming.

We can make the situation better if we explicitly negotiate who is
responsible for what. Defining one-size-fits-all solution (e.g.,
client is always responsible) seems too restrictive for me.

> who is responsible for what.

We should.

> if already known at that time.

Which is what "sizep" optional parameter in the draft is supposed to
do?

> real HTTP message exchange with the HTTP client and can select
> whether to close the connection, to keep it open, to change
> Content-Length header or to add a transfer encoding. The OPES
> processor does not need to trust the Content-Length header returned
> but the value of the new app-message- start parameter, which if
> present signals that the callout server knows what it does.

I agree.

This still does not solve all problems with buggy or assuming servers
that you mentioned above, right? It just helps in some cases.

Alex.