Subject: RE: transfer- and content-encoding

From: "Martin Stecher" <[email-masked]>
Date: Mon, 13 Oct 2003 12:23:23 +0200

Hi Alex,

> > > Do we need to prohibit encoding sequences like
> > > (chunked,identity,chunked,identity)? The above rules do not.
> > >
> >
> > Do you mean multiple usage of the same encoding in the list of the
> > Transfer-Encoding member?
> 
> No. Repetitions in Transfer-Encodings list should not cause any
> problems.
> 
> > Or that a transfer encoding could be applied several times 
> to some data?
> 
> Right, the same transfer encoding(s) applied several times.
> 
> 	Transfer-Encoding: (chunked,chunked,chunked)
> 
> > What kind of problem do you see here?
> 
> If we say nothing about it, them existing rules imply that the
> recipient should handle (chunked,chunked,chunked) encoding if it
> advertised support for chunked. I doubt many implementations would be
> that flexible; I doubt they would be able to dechunk the same content
> three times. It might also be a security vulnerability for the service
> if the list gets rather long or something of that kind. That makes me
> think that we should say something about it, if not outright prohibit
> it.
> 
> RFC 2616 has some text related to this problem, though its motivation
> is probably different:
> 
>    When the "chunked" transfer-
>    coding is used, it MUST be the last transfer-coding applied to the
>    message-body. The "chunked" transfer-coding MUST NOT be applied
>    more than once to a message-body.
> 
> Should we copy that?

I like your general approach below even better.

> 
> It is unlikely that OPES processors would _add_ transfer encoding
> because there is really no need for it with OCP. Content-encoding can
> be used for compression, I guess. So we are most likely talking about
> removing chunking or any extension encoding supported by the
> processor.

I agree.

> 
> I would suggest that we require identity transfer encoding, but I am
> afraid that some processors may not want to dechunk a message (perhaps
> they are doing some kind of zero-copy I/O or TCP slicing?). Another
> problem is that if there is an extension transfer encoding than the
> processor may not be able to dechunk back to identity.
> 
> It's a mess.

So, how about a SHOULD? Agents SHOULD support both identity and chunked
transfer encoding.

> 
> How about this:
> 
> 	If OPES processor applies transfer encoding, it MUST NOT
> 	apply any encoding more than once. Callout services
> 	MAY reject content for which a transfer encoding
> 	was applied more than once (XXX: document this as
> 	a potential exploit area).

Good, I just want to generalize to agent and peer rather than OPES
processor and callout services. Also the callout server MUST NOT
apply any encoding more than once.

> 
> And then perhaps give a typical example with chunked encoding being
> stripped by the processor.
> 

I now wrote this paragraph.

2.2.3 Transfer-Encodings

   The agents need to negotiate their encoding and decoding
   capabilities. Transfer encodings that are not supported by one agents
   MUST NOT be used. HTTP messages with a transfer encoding that is not
   supported by the callout server MUST be decoded by the OPES processor
   and eventually be re-encoded with a supported transfer coding.

   Negotiation is done using the Transfer-Encoding parameter of the
   profile feature. Both agents advertise a list of supported
   transfer-codings for encoding and decoding. Encodings listed earlier
   are preferred. The Transfer-Encodings parameter MAY be omitted if
   only the identity coding is supported.

   An OCP agent receiving application data from its peer, MUST be ready
   to accept multiple transport encodings, in any order, if it
   advertised support for multiple transport encodings. If an agent
   applies transfer encoding, it MUST NOT apply any encoding more than
   once. The peer MAY reject content for which a transfer encoding was
   applied more than once (XXX: document this as a potential exploit
   area).

   OCP agents supporting HTTP profiles SHOULD support and advertise
   identity and chunked transfer coding.

   		Transfer-Encodings  = "Transfer-Encodings:" SP
   		                      "(" [transfer-coding-list] ")"
   		transfer-coding-list = transfer-coding *("," transfer-coding)

   		for example
   			Transfer-Encodings: (identity,chunked)

                                Figure 5

   transfer-coding is defined in section 3.6 of [RFC2616].

What do you think?

Regards
Martin

---

From: Robert Collins <[email-masked]>
Date: Mon, 13 Oct 2003 21:22:54 +1000

>    transfer-coding is defined in section 3.6 of [RFC2616].

Well the HTTP errata removed the 'identity' Transfer coding. I don't see
any reason to reinstate it. Secondly, the prohibition against double
encoding doesn't seem required to me: Chunked has explicit rules (*) -
only one application, and always the last one. Other transfer codings
may have their own rule. As a counter example, one may wish to use a
transfer coding that provides delta-information, and use this twice (say
working of different basis objects).

As TE is hop by hop in HTTP, we need to ensure that any OPES processor's
TE field passed to upstreams, and TE field passed in responses to client
(for them to decide on upload Transfer-Encodings) is the processors
capabilities, not the actual client / origin respectively. Once we do
that, we know that an OPES processor will only receive codings it can
handle, so we can say MUST reject with a 5xx error (sorry to lazy to dig
up the best match) on an unhandlable transfer-coding. With that in
place, the interaction from processor to processor can 'trivially'
follow the HTTP Transfer-coding negotiation rules. That is, from a
protocol viewpoint, all transfer-codings must be removed and applied
anew across hops. By definition - implementations can shortcut this when
a compatible transfer-coding sequence exists across the relevant hop. 

Cheers,
Rob

(*)
RFC 2616 3.6
"   Whenever a transfer-coding is applied to a message-body, the set of
   transfer-codings MUST include "chunked", unless the message is
   terminated by closing the connection. When the "chunked" transfer-
   coding is used, it MUST be the last transfer-coding applied to the
   message-body. The "chunked" transfer-coding MUST NOT be applied more
   than once to a message-body. These rules allow the recipient to
   determine the transfer-length of the message (section 4.4)."
-- 
GPG key available at: <http://members.aardvark.net.au/lifeless/keys.txt>.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 13 Oct 2003 22:59:10 -0600 (MDT)

On Mon, 13 Oct 2003, Robert Collins wrote:

> Well the HTTP errata removed the 'identity' Transfer coding. I don't
> see any reason to reinstate it.

The reason for documenting "identity" encoding tag is to express the
lack of support for identity encoding. For example, a particular
service may want to receive all data using some custom transfer
encoding. I doubt such design is worth supporting though. That is,
this reason is probably not good enough. A service that requires
custom encoding can probably use content-encoding instead or rely on
manual configuration rather than run-time negotiation.

> Secondly, the prohibition against double encoding doesn't seem
> required to me: Chunked has explicit rules (*) - only one
> application, and always the last one. Other transfer codings may
> have their own rule. As a counter example, one may wish to use a
> transfer coding that provides delta-information, and use this twice
> (say working of different basis objects).

This is debatable. I would say that two delta encodings with different
base objects are two different encodings. On the other hand, there is
no pressing need to prohibit encoding repetitions where they are
allowed by HTTP.

> As TE is hop by hop in HTTP, we need to ensure that any OPES
> processor's TE field passed to upstreams, and TE field passed in
> responses to client (for them to decide on upload
> Transfer-Encodings) is the processors capabilities, not the actual
> client / origin respectively.

Proposed TEs headers are exchanged among OPES agents only and are not
passed to HTTP agents. It is an OCP header, not an HTTP header. We
cannot assume that OPES agents will control HTTP headers when adapting
HTTP payloads. This caveat is at the core of our problems here.
Consider an virus scanning service -- it should not affect TE headers
on the "outside" wire, but it should be able to handle any common
content that the corresponding HTTP intermediary proxies. That is, OCP
agents should be able to handle a variety of common transfer encodings
without being able to affect "outside" encoding negotiations.

> Once we do that, we know that an OPES processor will only receive
> codings it can handle, so we can say MUST reject with a 5xx error
> (sorry to lazy to dig up the best match) on an unhandlable
> transfer-coding.

HTTP proxy capabilities may be different from an attached OPES
processor capabilities (which, in turn, may be different from an
attached callout service capabilities). This is true for
Transfer-Codings and for some other features. We cannot simply assume
that OPES processor is an HTTP proxy, even if it adapts HTTP messages.

> With that in place, the interaction from processor to processor can
> 'trivially' follow the HTTP Transfer-coding negotiation rules. That
> is, from a protocol viewpoint, all transfer-codings must be removed
> and applied anew across hops. By definition - implementations can
> shortcut this when a compatible transfer-coding sequence exists
> across the relevant hop.

I agree with the above. However, we still need to provide a
negotiation mechanism for OPES agents to agree on the actual transfer
encoding to be used. We cannot rely exclusively on HTTP specs because
our agents, especially callout service, may not be HTTP agents. For
example, many callout services will work with message payload and
disregard any HTTP headers; those services will be very sensitive to
encoding issues; they may not, for example, support chunked encoding.

XXX: we need to document HTTP Adaptation scope. Does the draft apply
only to agents that are HTTP compliant? Or to any adaptation of an
HTTP messages, including those where the service is not HTTP-aware? Or
something else?

There are quote a few sane options available to us, depending on what
encodings have to be supported and on what to do with custom
encodings that an OCP agent does not support. Given your feedback and
the multitude of options we face, I would propose the following:

	0) Do not negotiate Transfer-Encodings at all.

	1) An OCP agent sending data MUST remove all
	   transfer encodings it supports. If any encodings remain, an
	   OCP agent sending data MUST specify remaining encodings
	   using the Transfer-Encoding parameter of a DUM
	   OCP message.

	2) If an OCP agent receives Transfer-Encoding parameter
	   indicating unsupported encoding, it MAY terminate
	   the corresponding OCP transaction.

Do you think the above rules create any interoperability problems that
more complex rules can eliminate?

Can we think of a realistic-enough example where removing supported
encodings is bad for performance reasons? Note that an agent may be
_configured_ to leave certain encodings -- that qualifies as lack of
support for their removal. Perhaps the above "MUST remove" can be
rephrased to better reflect this caveat?

Do the above rules still allow a callout service to form a chunked
HTTP response it order to, for example, indicate service progress on a
persistent HTTP/1.1 connection? Or should we not expect any processor
to be able to handle chunked responses from a callout service? Should
we add the following rule?

	1a) OPES processors MUST support chunked transfer coding
	    when handling data send by an OCP server.

Thanks,

Alex.

---

From: Robert Collins <[email-masked]>
Date: Thu, 16 Oct 2003 17:30:53 +1000

> manual configuration rather than run-time negotiation.

I think that the solution implied by your later message (I've already
replied to that) is much better. That is - everything is identity
between agents unless explicitly negotiated into a different transfer
coding. Even so sending via chunked may be a better lowest common
denominator due to the built in end of data indicator. I'm not fully up
to speed on the current draft - I've been skim reading due to general
busyness unfortunately.

> without being able to affect "outside" encoding negotiations.

Which directly conflicts with HTTP's hop by hop requirements - anything
that is hop by hop will break, and break badly, unless we explicitly
handle the hop by hop semantics. (And any other message based protocol
other than HTTP that splits hops and messages will have similar issues).
I hope I'm not missing an existing solution when I point out that the
OCP device performing the interception and upstream forwarding has the
responsibility of negotiating in the intercepted protocols boundaries,
and gatewaying into the OCP protocol. So yes: the internal OCP encodings
should stay in the OCP area, but that wasn't what I referred to. What I
was saying is that:

Origin 
   |
AV interceptor(OPES processor)  ---- AV engine
   |
Client

in this diagram, the AV interceptor is responsible for Connection:
semantics, for TE and transfer-encoding semantics, and Trailer:
semantics, to pick a few common headers. Thus: the client will see the
AV Interceptors TE headers, not the origins TE headers.

> that OPES processor is an HTTP proxy, even if it adapts HTTP messages.

But we can assume that the OPES processor will only negotiate what it
can handle, in the protocol(s) it exposes to the clients/origins. So for
the OCP protocol to have unhandlable transfer-codings arrive is in fact
a failure of the OPES processors responsbilities to the protocol it's
exposing, and IMO a OPES error of some sort is appropriate.

> encoding issues; they may not, for example, support chunked encoding.

Ah. Very good point. Any reason not to leverage the connection
termination, Connection: TE: and Transfer-Encoding: headers wholesale?
(Yes, showing my skim reading again, I know).

> 	   the corresponding OCP transaction.

for 2) I think MUST terminate is a requirement. 0) seems short sighted
to me - I'd rather see an analogous environment to HTTP:
1) messages pushing data to the agent require foreknowledge of supported
encodings - i.e. during initial handshaking. 
2) messages from the agent back to the processor use metadata supplied
by the processor to determine acceptable codings. 
3) all agents and processors must support chunked, which is trivial to
implement efficiently.

> more complex rules can eliminate?

Other than what I mention above, no.

> rephrased to better reflect this caveat?

Yes - rproxy. That is rsync over http. Mind you, to do *anything* nearly
all agents will need the data in identity format eventually. So perhaps
the strip everything to identity is the sanest option.

> 	    when handling data send by an OCP server.

IMO Yes. chunked is a good thing.

Rob

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 16 Oct 2003 09:39:41 -0600 (MDT)

On Thu, 16 Oct 2003, Robert Collins wrote:

> replied to that) is much better.

I will not argue with the rest of this e-mail then, except that I want
to clarify one important point.

> handle the hop by hop semantics
>
> Origin
>    |
> semantics, to pick a few common headers. Thus: the client will see
> the AV Interceptors TE headers, not the origins TE headers.

I agree, but let's not forget that our scope is OPES agents and not
the entire intermediary. A more accurate/detailed picture would look
like this:

  Origin
    |
  HTTP proxy -?- OPES processor -*- OPES callout service
    |
  Client

OCP scope is the "-*-" link. All "|" links are HTTP links. OPES
processor can be viewed as an HTTP hop from HTTP proxy view _or_ it
can be integrated with the HTTP proxy. Processor traffic and actions
on the "proxy side" (the "-?-" link) would depend on the degree of
that integration. Thus, it is not accurate to say that OPES processor
has to be responsible for Connection semantics; that would depend on
the link. However, if that OPES processor is using an HTTP link (and
claims HTTP compliance), then it becomes responsible.

Finally, the callout service may be interested in hop-by-hop headers.
For example, a logging service may want to log them. Thus, it is not a
good idea to require OPES processor to strip them. We should require
correctness on "|" links. What happens on "-*-" link is not important
from HTTP point of view because "-*-" link is not an HTTP link.

Hope this clarifies,

Alex.