Subject: HTTP Adaptations issues for f2f discussion

From: Alex Rousskov <[email-masked]>
Date: Thu, 6 Nov 2003 11:25:03 -0700 (MST)

Hi,

        Here are some open HTTP Adaptations issues that may be worth
discussing at the upcoming F2F meeting. I will skip issues already
documented as XXXs in the draft itself.

	1. Security Considerations

	   Are there really no HTTP-specific considerations in OCP?
	   Do things like wrong Content-MD5 checksums belong here?
	   Also see #6,7,9 below; are those related to
	   security considerations?

	2. Scope.

	   Do we need to clearly say (in one place) what it means
	   to adapt HTTP? Is HTTP connection adaptation in scope?
	   Is URL blocking or redirection in scope? Is HTTP
	   authentication (via callout services) in scope?

	   This sounds like a simple question, perhaps, but,
	   as usual, scope is one of the most difficult things
	   to define. See issues #3,5,6,7 below, for example.
	   They are all scope-related in some way!

	3. Partial Content

	   Do we need to say anything special about adapting
	   "206 Partial Content" HTTP responses? There are
	   two cases here: multipart/byteranges media type and a
	   single "part" case.

	   multipart/byteranges are expressed via Content-Type,
	   not Content-Encoding. Thus, we currently have no
	   mechanism to negotiate support for those. On the
	   other hand HTTP does have such a mechanism for its
	   agents (if you do not support it, you do not ask
	   for multiple parts).

	   IMO, we should say something, even if we are not
	   going to provide explicit support for this
	   special content type.

	4. Authentication and other sensitive information.

	   Do we need to specify whether authentication information
	   should be passed "as is" to callout servers by default?

	   IMO, it should be because we pass all other "sensitive"
	   information already. We should probably explicitly say
	   that the processor should treat any sensitive data as
	   regular data in this context (by default). This places
	   stringent requirements on callout server security and
	   trust, but that is how OPES is supposed to work.

	5. FTP -> HTTP

	   HTTP proxies that proxy ftp://host URLs have to
	   do FTP and transform FTP messages to HTTP responses.

	   Do we need to mention this case in some way? Is it
	   special or out of scope (i.e., we assume that all
	   transformations have been already performed).

	6. 100 Continue

	   100 Continue messages are sent as a part of an HTTP
	   transaction but they are not true "responses".
	   What should an OPES processor do about
	   these? Technically, they can be viewed as another
	   original application message to be adapted within
	   the same OCP transaction (multiple original am-ids)!

	   If we want to avoid multiple original am-ids, but
	   still want to adapt 100 Continue messages, we can
	   introduce a notion of "related transactions". Each
	   100 Continue would cause the processor to start
	   a new OCP transaction, related to the original
	   one (so that the request message does not need to
	   be resent). But, on a technical level, multiple
	   original am-ids is exactly what is going in here,
	   I guess.

	   We have to say whether processor must skip them.
	   If processor skips them, does it open a [security]
	   whole for clients or servers to bypass OPES. These
	   messages do not have bodies, but their headers can
	   be used to send information that OPES entities may
	   be interested in.

	7. CONNECT method

	   CONNECT method establishes a tunnel through the
	   proxy. Should OPES processor adapt just the CONNECT
	   request and response, or should it stay in the loop
	   for the tunnel itself? If the processor stays, we
	   do not have a good mechanism in the current draft
	   to adapt a bi-directional tunnel, right?

	   Tunnel adaptation is more like streaming adaptation,
	   isn't it? That's an unchartered territory for OCP,
	   including OCP Core.

	   We have to be explicit what the processor is supposed
	   to do.

	8. SIRs or HTTP WG review

	   Should we solicit SIRs and/or HTTP WG review of
	   this. There is no formal HTTP WG, but there is
	   a semi-alive mailing list that HTTP gurus read.

	9. Concurrent request and response.

	   What happens when the HTTP client is sending a request
	   body _while_ the HTTP server is sending a response, and the
	   callout server requested HTTP request body to
	   adapt the HTTP response. Do we have a deadlock?

	   The above can happen when the HTTP server is
	   sending a constant response regardless of the PUT/POST
	   body or is sending an error message. For example,
	   an error message can say "I got 1MB of your POST
	   request and I will not accept any more. Go away!".
	   OPES has to adapt such a response, but it looks
	   like we cannot until the client stops and the
	   client may not stop for another 100MBs (because
	   it is broken/malicious or because it does not
	   get the error message that is waiting for our
	   adaptation).

	10. Huge request bodies and RESPMOD

	   There is no way for the callout server to
	   skip a huge request body when doing RESPMOD.
	   Is that acceptable? Should OCP Core support
	   "skip N original octets" interface (that
	   does not affect offset/size calculations).
	   Should HTTP draft support "skip this part
	   until its end" interface? Instead?

N.B. Does ICAP address #3,5,6,7? ICAP does not have a problem
     with #9 and #10 because ICAP does not allow sending request body
     in RESPMOD, right?

Note that there are some synchronization issues between OCP Core and
HTTP drafts, but they are probably not worth discussing in public.
Also, some of the above issues might be too technical and
HTTP-specific for the f2f meeting. Again, please pick-and-choose what
you want to discuss.

Thank you,

Alex.

P.S. I think we should have thought of some of the above issues
     earlier. On the other hand, I think we did discuss them briefly
     but ignored when it came to writing the guts of the draft. I hope
     we can "escape" them with little effort, but some look pretty
     tricky to me.

---

From: "Martin Stecher" <[email-masked]>
Date: Thu, 20 Nov 2003 22:13:22 +0100

Hi,

here a few comments.

>    going to provide explicit support for this

I think that this is out of scope because it is a Content-Type thing.
Either an agent should handle these types or should not care, or
it has to try to get a request profile service into the traffic that
removes the support advertisement of partial content from the
HTTP requests.
Is this something we need to describe in the draft?

>    trust, but that is how OPES is supposed to work.

Agreed.

>    transformations have been already performed).

As you said, FTP over HTTP to native FTP transformation
has to be done by the last HTTP proxy server and has
nothing to do with HTTP adaptation of OCP.
The only difference to "normal" HTTP is that the URL starts
with ftp.

Since 3.2.2 of RFC 2616 specifies that a http_URL has to
start with "http:", OCP/HTTP does not apply which is a shame
because it works (if the service supports it). Other protocol
prefixes could also be there; it's not only for FTP.

Is there somewhere something said how HTTP proxies should
handle URLs that have another protocol prefix than "http:",
especially if they have another next-hop-proxy and do not
need to do protocol conversion? This could probably be
copied to our case.

>    be used to send information that OPES entities may

I agree that multipe origin am-ids would solve this issue best
but it seems to be the wrong timing to me to change the concept
now and allow them.
In ICAP/1.0 I have never seen an encapsulated 100 Continue
message. But this does not mean that there are no such messages.
I think that
  a) most ICAP clients do not send 100 Continue messages to ICAP servers
and
  b) most/some ICAP servers will simply return 100 Continue messages
unchanged
or apply some header filters usually without effect since those messages do
not have many header fields in praxis
While ICAP/1.0 has no transaction concept, it is o.k. to send two separate
RESPMOD messages, one for the 100 Continue and one for the final response.

>    We have to be explicit what the processor is supposed

A typical ICAP/1.0 client would generate a REQMOD request for
a CONNECT request but would not try a RESPMOD request
for the response or tunneled data.
HTTPS proxies can generate REQMOD/RESPMOD messages for the
decoded HTTP of course.

>    get the error message that is waiting for our

This is indeed a problem. Do we have use cases where a response profile
OCP service needs to see the original request body? Filters I can think of
that use the request-body part better belong to request profile.

ICAP/1.0 does not support request-body encapsulation in RESPMOD,
so there are no existing services for this feature.

Is it a good moment to remove support for this application message part
from request profile?

>    until its end" interface? Instead?

Removing request-body part would solve this too ;-)

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 20 Nov 2003 14:59:54 -0700 (MST)

On Thu, 20 Nov 2003, Martin Stecher wrote:

> I think that this is out of scope because it is a Content-Type
> thing. Either an agent should handle these types or should not care,
> or it has to try to get a request profile service into the traffic
> that removes the support advertisement of partial content from the
> HTTP requests. Is this something we need to describe in the draft?

I agree that we do not have to provide explicit negotiation at this
time (though I disagree that Content-Type is the excuse here;
Content-Type is [ab]used for _encoding_ here).

I think we must document the issue somewhere to warn implementations
that they may see these responses.

Finally, I think it is a security consideration! I may be able to
bypass OPES filters by constructing a request with two ranges
that cover the entire resource being requested. If the server
does not merge (I do not think it has to), I will get the entire
response while some OPES entities will not be able to handle it.

In fact, requesting a resource part by part can probably fool many
OPES entities even if we add Content-Type negotiation. We must
document this. (Removing Range headers from the request is a solution
but is an ugly one because things like PDF viewers will be screwed big
time by that -- Squid proxy had a long history of fighting with those
issues).

> The only difference to "normal" HTTP is that the URL starts

and that there is not HTTP connection on the server side. We do not
care about HTTP connections for now, but some extensions might. But
let's not document that, I guess.

> Since 3.2.2 of RFC 2616 specifies that a http_URL has to start with
> "http:", OCP/HTTP does not apply which is a shame because it works
> (if the service supports it). Other protocol prefixes could also be
> there; it's not only for FTP.

I hope you misinterpret RFC 2616! Request-URI is defined as:
	URI    = "*" | absoluteURI | abs_path | authority
and absoluteURI comes from generic RFC 2396

http_URL is just one documented case (do document HTTP access
semantics such as default TCP port numbers).

IMO, OCP/HTTP applies to any URI scheme as long as message is
HTTP-formatted.

> Is there somewhere something said how HTTP proxies should handle
> URLs that have another protocol prefix than "http:", especially if
> they have another next-hop-proxy and do not need to do protocol
> conversion? This could probably be copied to our case.

Did the above clarifies why no special handling is needed as far as
URI scheme is concerned?

> but it seems to be the wrong timing to me to change the concept

So, for now, should we say that 100 Continue MUST NOT be forwarded to
callout servers unless their adaptation is negotiated? (and thanks for
the ICAP info).

> for the response or tunneled data.

The latter seems inconsistent. Should we document that both requests
and response are subject to adaptation but the tunnel is not?

> This is indeed a problem. Do we have use cases where a response
> profile OCP service needs to see the original request body? Filters
> I can think of that use the request-body part better belong to
> request profile.

I do not have such use cases.

> so there are no existing services for this feature.

Right.

> Is it a good moment to remove support for this application message
> part from request profile?

That would be another blow to OCP/HTTP "theoretical" coverage of HTTP
protocol, in addition to "100 Continue" non-support.

How about documenting this deadlock and requiring that OCP agents are
able to handle it like any other resource-limitation issue? Either
OPES processor or callout server can terminate the transaction if the
request body gets too long. Document this as a security concern as
well.

> Removing request-body part would solve this too ;-)

True. Still, I do not want to drop coverage of potentially useful
cases just because there are some corner cases that can be abused.
Will the above "treat it like any other resource-limitation issue"
loophole work?

Thank you,

Alex.

---

From: Robert Collins <[email-masked]>
Date: Fri, 21 Nov 2003 20:00:12 +1100

> callout servers unless their adaptation is negotiated? (and thanks for

Huh? This seems orthogonal: the handshaking involved in rfc 2616 uploads
is essentially a buffering mechanism, the adaptation device must handle
the client and server ends as per HTTP semantics. So far, the callout
protocols have no 'send / don't send' concepts in them, so the semantics
there are clear. The corollary is that 100 continue must (not MUST) be
sent http/1.1 clients without delay, and the adaptation device implement
buffering before sending to the http origin / delaying of reads from the
client as appropriate for local policy.

> and response are subject to adaptation but the tunnel is not?

I agree.

Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 21 Nov 2003 08:58:51 -0700 (MST)

On Fri, 21 Nov 2003, Robert Collins wrote:

> Huh? This seems orthogonal: the handshaking involved in rfc 2616
> uploads is essentially a buffering mechanism, the adaptation device
> must handle the client and server ends as per HTTP semantics. So
> far, the callout protocols have no 'send / don't send' concepts in
> them, so the semantics there are clear. The corollary is that 100
> continue must (not MUST) be sent http/1.1 clients without delay, and
> the adaptation device implement buffering before sending to the http
> origin / delaying of reads from the client as appropriate for local
> policy.

Robert,

	I am sorry but I do not understand what you mean. Can you
rephrase please? We are trying to decide what to do when the origin
server responds with 100 Continue before sending 200 OK; there is no
mechanism in current OCP Core to handle multiple original application
messages for the same transaction.

	Are you saying that 100 Continue mechanism is essentially
external to HTTP messages being adapted? Like, for example, HTTP
connections or TCP ACKs are external to the messages that flow on
those connections -- we adapt messages and not the connections or
ACKs. If so, the suggested "MUST NOT forward" rule seems appropriate.

	Or are you saying that 100 Continue messages must be forwarded
to the callout server for some reason?

	Please clarify.

Thank you,

Alex.

---

From: Robert Collins <[email-masked]>
Date: Sat, 22 Nov 2003 07:40:56 +1100

> messages for the same transaction.

Yep.

> ACKs. If so, the suggested "MUST NOT forward" rule seems appropriate.

I guess I'm saying that things emerge from the OCP protocol and the http
protocol. Documenting them would make sense :}.

1) Information status messages MUST be dealt with in the HTTP
guidelines. That means that outside of the OCP environment, protocol
behaviour must be enforced by the adaptation client in whatever degree
of cooperation with OCP we design... That includes 100 continue, and 101
upgrade likewise, and ANY 1XX interim response. (RFC 2616 10.1 - all 1xx
status codes MUST be delivered to the client UNLESS a proxy initiated
the request for that particular 1xx response.)
2) 100 is a special case of these, where the semantics of uploads and
resource management requirements on a http-proxy-like device ensure
there will be some mechanism that can be exploited to implement the 100
semantics (From an external perspective)
3) 101 upgrade has ocp issues if the response circumvents OCP - as
 a) the client will know they have switched
 b) the OCP infrastructure won't [know they have switched].
 c) respmod logic may not even be able to handle the upgraded protocol -
i.e. what if it's a rtsp stream? or a TLS encoded session? (And as more
servers support Upgrade to TLS, port 80 adaptation could be circumvented
quite easily (*))
4) all of the above are no more than special case ramifications of ANY
Http reverse proxy scenario : they are not unique to the adaption
environment, yet they will impact it heavily.

> to the callout server for some reason?

Well I'm not sure.. thats the crux of it.

If you don't give OCP visibilty to 1XX series messages, then OCP cannot
(for instance) handle any messages that goes through a protocol upgrade.
Likewise, we place extra buffering overhead on the adaptation client
when it must treat the OCP subsystem as (for all intents and purposes)
an HTTP/1.0 client.

Conversely, if you expose the 100 and 101 messages (And any extensions
that involve 1XX messages - remember they MAY arrive unexpectedly), then
you may as well use http for OCP as you've just stuffed the remainder of
the protocol in there. 

We've got an issue in squid, where I'm currently designing 100 series
message support, that relates to this, getting all 100 series messages
via the internal pipeline requires significant changes, to have a
multiple-response per transaction concept. 100-Continue is easy by
comparison, as it's literally no more than removing a status flag to
allow the clients req buffer to flow upwards - and issue a notification
to the client to effect the same.

(*) Which comes back to the trust issues discussed already, and the
issue of who is authoritative for the delivered content: if the adapting
device is authoritative, it should implement appropiate policy and
request transformations to ensure that upgrades to tls happen at it's
client facing side, not at the origins - which means it MUST have an
appropriate TLS certificate etc etc etc.

Rob

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 21 Nov 2003 14:15:26 -0700 (MST)

Robert,

	Thanks for the clarification! Your thoughts make it more clear
that the scoping issue (already marked as unresolved!) plays a crucial
role here:

	HTTP/OCP agents are not HTTP agents. They are not adapting
HTTP protocol as a whole. They are adapting isolated HTTP messages
only. They are not adapting HTTP connections, they are not adapting
1xx mechanism, they are not adapting HTTP cache, they are not adapting
TCP flow underneath HTTP connections, etc. HTTP/OCP scope is an atomic
HTTP message. Other OCP Core extensions can be written to adapt other
things if needed.

	Furthermore, we acknowledge that HTTP introduces relationship
between HTTP messages. HTTP/OCP handles one kind of such relationship:
a given HTTP response being adopted is related to a single HTTP
request. This relationship is handled via optional original parts, a
negotiated HTTP profile feature.

	The fact that _multiple_ HTTP responses may be related to a
single HTTP request when 1xx responses are present is acknowledged,
but is out of the draft scope. In other words, HTTP/OCP does not have
a mechanism to relate multiple HTTP responses to a single request as a
part of a single OCP transaction. It is possible to express that
relationship via OCP Core extensions, but the HTTP/OCP specification
does not do that.

	Consequently, callout server (OCP server) MUST be able to
handle all HTTP responses, including 1xx responses. OPES processor
(OCP client) MAY NOT forward 1xx responses to a callout server. If
OPES processor forwards 1xx responses to a callout server, and
negotiated profile requires forwarding of request parts, then OPES
processor MUST forward those request parts. Informally, this means
that callout servers may receive identical request parts multiple
times: each time the processor forwards a 1xx response and one time
the processor forwards the final response.

	Similarly, HTTP upgrade and other HTTP mechanisms are out of
HTTP/OCP scope. The OPES processor MAY forward HTTP message to the
callout server. Informally, this implies that the processor may
forward only a subset of messages. HTTP semantics related to messages
such as Continue or Upgrade is beyond HTTP/OCP specification scope.

Do you see what I am getting at? Will the above scoping rules work
well enough?

Alex.

P.S. The idea may become more clear if you consider adapting protocols
     with a large number of control messages/dialogs. For example, the
     same scoping rules above are meant to apply well to SMTP.