Subject: RE: HTTP/OCP protocol binding

From: "Martin Stecher" <[email-masked]>
Date: Mon, 2 Jun 2003 10:29:17 +0200

> > > I would use some very simple encoding to pass HTTP
> > > messages in OCP payload. Something along these lines:
> > >
> > > 	<chunk-type> <chunk-length> <chunk-data>
> > >
> > > where "chunk-type" can be either of
> > > 	"headers":  HTTP headers including the first line and the last
> > > 	            CRLF terminator
> > > 	"trailers": HTTP trailers including the last CRLF terminator
> > > 	"body":     raw HTTP message payload
> > > 	"all":      raw HTTP message
> > > and one OCP payload may contain several chunks in the 
> above format.
> > >
> >
> > For a HTTP response we need the original HTTP request as additional
> > meta data. That needs to be added somehow.
> 
> Indeed, I forgot about that requirement. Will having both "request-*"
> and "response-*" chunk types be too complicated?
> 
> 	request-headers, response-headers
> 	request-trailers, response-trailers
> 	request-body, response-body
> 	request-all, response-all
> 
> (or the other way around: *-request and *-response).

First, I am not so sure about the "all" chunks. Do we really need them?
While having multiple options and being able to negotiate things is great, especially for OCP core, I think that the application protocol bindings should define a single way how to vector the data to the callout server.
If both OPES processor and callout server agree on HTTP/OCP they have to understand at least the basics of that protocol and so being able to differ between headers and bodies.
If actors in the OPES scenario cannot find a matching protocol binding that both support, there could/should be a File/OCP and Raw-TCP/OCP that may work out sometimes as fallback.

Second, is it necessary that the application procotol chunks encoding is another abstraction layer on top of the data-have messages?
Why do we not just introduce a named parameter (Content-Type or Payload-Type) for data-have and use this core message type to send the chunks? Chunk length is then without additional cost and no extra encoding needs to be parsed.

The protocol binding will then define which payload types exist, whether there is a given order and whether multiple data-have messages per payload type are allowed.

> 
> We will also have to explicitly document what to do with 1xx HTTP/1.1
> responses. It would be nice if we can adapt them, but I do not know
> yet whether they have to be treated specially from HTTP binding point
> of view.

Good point.
It probably makes no sense to handle a 1xx as its own transaction. It could be another response header but we will need to explain this clearly in the draft otherwise we will get interop problems because many implementations will probably oversee this point.

Regarding the transfer encoding:

Seems that we agree that OPES processor and callout server will negotiate which is supported; identity MUST be supported. OPES processor MAY/SHOULD have some pre-processing to remove a transfer encoding that it supports but the callout server doesn't.

Regarding content length and persistency:

We agree that the OPES processor is responsible to handle HTTP persistency and that it is impossible to have a protocol which is so robust that it overcomes all buggy implementations.

But I think that we can reduce the problem and enforce the OPES processor responsibility by defining:
 - the OPES processor MUST ignore the Content-Length header returned in the response header chunk from the callout server
 - a callout server SHOULD indicate the application message length via the sizep parameter in the start-app-message if it is known at that time.
 - the OPES processor SHOULD honor the sizep parameter and use it to adjust the Content-Length header; otherwise it MUST remove the Content-Length header; it MAY then switch to chunked transfer encoding if supported by the client for being able to keep the HTTP connection alive.

With this approach many callout services that only concentrate on the HTTP body can ignore the header and use data-as-is rather than parsing the header, finding and adjusting the ContentLength header, reassembing the header and sending it back.

What do you think about this?

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 2 Jun 2003 10:12:48 -0600 (MDT)

On Mon, 2 Jun 2003, Martin Stecher wrote:

> First, I am not so sure about the "all" chunks. Do we really need
> them? While having multiple options and being able to negotiate
> things is great, especially for OCP core, I think that the
> application protocol bindings should define a single way how to
> vector the data to the callout server. If both OPES processor and
> callout server agree on HTTP/OCP they have to understand at least
> the basics of that protocol and so being able to differ between
> headers and bodies. If actors in the OPES scenario cannot find a
> matching protocol binding that both support, there could/should be a
> File/OCP and Raw-TCP/OCP that may work out sometimes as fallback.

Good point. The reason to keep "all" chunks could be OPES processor
performance. In many cases, it may be more efficient for the processor
to send data as it received it (raw), even though it had parsed it
internally. Nevertheless, I agree that "all" chunks should be removed
from HTTP binding until we have a better HTTP-specific justification for
them.

On the other hand, I think we should define HTTP binding in such a way
that it would be possible to use data (OCP payload) encodings from other
bindings. It should be possible to negotiate data encoding after
negotiating HTTP binding (HTTP binding comes with a default encoding, of
course). This will make it simple to use "raw" and other useful
encodings for HTTP adaptations. We should try to make data encoding (and
other things) reusable across application bindings.

> Second, is it necessary that the application procotol chunks
> encoding is another abstraction layer on top of the data-have
> messages? Why do we not just introduce a named parameter
> (Content-Type or Payload-Type) for data-have and use this core
> message type to send the chunks? Chunk length is then without
> additional cost and no extra encoding needs to be parsed.

It is possible to move the data type field to OCP. Doing so will limit
you to one chunk per OCP message, but that is probably not such a big
deal if we keep data-have messages efficient. It certainly simplifies
the protocol so we should try to do that, I guess.

> The protocol binding will then define which payload types exist,
> whether there is a given order and whether multiple data-have
> messages per payload type are allowed.

I suggest that we keep the App-Message-Part parameter HTTP-specific for
now. Not all application protocols will have different "parts" of
messages to be concerned about or will be able to describe those parts
with a simple parameter value.

> It probably makes no sense to handle a 1xx as its own transaction.
> It could be another response header but we will need to explain this
> clearly in the draft otherwise we will get interop problems because
> many implementations will probably oversee this point.

Treating 1xx as headers is a kludge that may break services, especially
those that mingle with headers. Moreover, you would still need some way
to distinguish 1xx and "normal" headers unless you want the callout
server to rely on response lines as separators. Finally, some callout
servers might _generate_ 1xx responses and, hence, may prefer a more
"direct" approach (but response generation is a much bigger problem, of
course).

Overall, I suspect we should treat a 1xx response as an application
message it is, just like a 200 OK response, but we can postpone this
debate.

> Seems that we agree that OPES processor and callout server will
> negotiate which is supported; identity MUST be supported. OPES
> processor MAY/SHOULD have some pre-processing to remove a transfer
> encoding that it supports but the callout server doesn't.

Yes.

> We agree that the OPES processor is responsible to handle HTTP
> persistency and that it is impossible to have a protocol which is so
> robust that it overcomes all buggy implementations.

Yes.

>  - the OPES processor MUST ignore the Content-Length header returned
>    in the response header chunk from the callout server

This seems too restrictive to me. If, for example, the same company
makes the OPES processor and the callout server, they would want to
avoid extra work and would prefer to honor the Content-Length header
generated by the callout server they trust as much as the processor.

Perhaps you mean that sizep parameter should be used in this case, and
the Content-Length header should still be ignored.

>  - a callout server SHOULD indicate the application message length
>    via the sizep parameter in the start-app-message if it is known at
>    that time.

Why not a MUST? If something so important is known, why not report it?

>  - the OPES processor SHOULD honor the sizep parameter and use it to
>    adjust the Content-Length header;

OK.

> otherwise it MUST remove the Content-Length header; it MAY then
> switch to chunked transfer encoding if supported by the client for
> being able to keep the HTTP connection alive.

Or it can try to buffer the entire response first to calculate the
true Content-Length value.

> With this approach many callout services that only concentrate on
> the HTTP body can ignore the header and use data-as-is rather than
> parsing the header, finding and adjusting the ContentLength header,
> reassembing the header and sending it back.

True, but it seems we are building a very specific/rigid/low-level
solution to the part of the problem where a more general and flexible
solution is possible. What you describe in the above paragraph hints at
the true problem at hand. Some (many?) services have these
characteristics:

    - they need to know message "headers" (processor must send them)
    - they modify "content" or "body"
    - their modifications bring original "header" out of sync
    - they would prefer not to sync message "headers"
    - we think it may be safer for OPES processor to sync the headers

What we may need is a mechanism for the callout server to tell OPES
processor to [re]compute the adapted headers while still providing
original ones:

    * data-i-have
      I tell you what to send;
      I believe the data I have is accurate
      (this is the current data-have message)

    * data-you-have
      I tell you what to send, but I modified nothing, so just
      use your own copy you told me you have;
      I believe the data I have is accurate
      (this is the current data-as-is message)

    * data-you-check
      I tell you what to send;
      I believe the data I have is not accurate enough so please
      check and recompute everything you can as needed

An application binding can use the above mechanism in addition to sizep
and other features to require OPES processor to recompute "headers" when
a callout server sends then via data-you-check OCP message.

The above "content length" thoughts are very unpolished. We need to work
more on this...

Thank you,

Alex.