Subject: what-parts-to-send-or-skip

From: "Martin Stecher" <[email-masked]>
Date: Thu, 23 Oct 2003 17:04:39 +0200

Hi,

Alex is right, the current message part handling is a mess.
I do not understand why I could forget about request satisfaction mode (short circuits).

I tried several ways to list the possibilities of message part sending, skipping, responding and found no easy way :-(

First here are some thoughts about short-circuits in the request-profile. I think handling this one is not so complicated and can be added to the existing draft:

  Before the callout server sends back any application message part,
  it MUST decide whether it wants to adapt the request or to short-curcuit.
  It will either send back request-header, request-body, request-trailer parts
  or response-header, response-body, response-trailer parts.
  Some parts could be optional due to conditions defined below but it MUST NOT
  mix parts of these two lists; on violation client MUST terminate transaction.

Then here some thoughts on parts in HTTP messages:
  A request header is always in HTTP messages.
  There are HTTP servers that do not send response headers.
  OPES processor MUST either create a dummy HTTP header part,
  or not use OCP/HTTP.

  Trailers are often not in messages. There is no difference between
  an empty trailer and a non-existing trailer.

  Bodies can or cannot exist. Empty but existing bodies are technically
  possible. Can an empty body be deleted without changing the meaning
  of the message?

Then the best way to describe what needs to be done I found for me is a pseudo programming language style.
I do not take the trailer parts in account, it's long enough without ;-)

OCP_Client_Send_Routine {
    if ( NOT skip(request-header))
      send (request-header)
    if (inMessage (request-body) AND NOT skip(request-body))
      send (request-body)
}

OCP_Client_Receive_Routine {
    lastMessagePart = 0
    while (read (newMessagePart)) {
        if (newMessagePart == request-header) {
            if (lastMessagePart != 0)
                goto terminate_transaction;
            buildHTTPMessage (newMessagePart);
        } else if (newMessagePart == request-body) {
            if (skip(request-header)) {     // client has skipped part in send
                if (lastMessagePart != 0)
                    goto terminate_transaction;
                buildHTTPMessage (original_request-header);
            } else {
                if (lastMessagePart != request-header)
                    goto terminate_transaction;
            }
            buildHTTPMessage (newMessagePart);
        } else if (newMessagePart == response-header) {
            if (lastMessagePart != 0)
                goto terminate_transaction;
            buildHTTPMessage (newMessagePart);
        } else if (newMessagePart == response-body) {
            if (lastMessagePart != response-header)
                goto terminate_transaction;
            buildHTTPMessage (newMessagePart);
        }
        lastMessagePart = newMessagePart;
    }  
}

OCP_Server_Receive_Routine {
    lastMessagePart = 0
    while (read (newMessagePart)) {
        if (newMessagePart == request-header) {
            if (lastMessagePart != 0)
                goto terminate_transaction;
            filterHTTPMessage (newMessagePart);
        } else if (newMessagePart == request-body) {
            if (lastMessagePart != request-header) {
                if (lastMessagePart != 0)
                    goto terminate_transaction;
                if (skip_requested(request-header))
                    goto terminate_transaction;
            }
            filterHTTPMessage (newMessagePart);
        }
        lastMessagePart = newMessagePart;
    }  
}

OCP_Server_Send_Routine {
    if (short_circuit) {    // do request satisfaction
        send (response-header)
        if (message_has_body())
            send (response-body)
    } else {
        if (have_received (request-header)) {
            send (request-header)
        } else {
            if (have_own (request-header)
                send (request-header)
        }
        if (have_received (request-body)) {
            if ( NOT want_to_delete (request-body))
                send (request-body)
        } else {
            if (have_own (request-body)
                send (request-body)
        }
    }
}

This is not trivial and describing this in a good way is not easy.
Especially the problem that an OCP transaction that does not contain request-body parts in neither client nor server messages can mean multiple things:
  a) There was no body part in the HTTP message and there should not be one
  b) The body part has been skipped and should be used again in the adapted message
  c) The body part has been skipped and should be removed from the message

b) and c) can only be solved if we require that the callout service MUST send an empty body part, if it wants to make sure that a skipped original part is not included in the adapted message.

Is this the solution?

If not, we should consider to either skip the Skip-Parts feature or to redefine it in another way.

Please check this idea:
-----------------------

Payload becomes optional in DUM messages. A new named parameter "Skipped-Payload" has a value which is the size of the payload but the payload itself is not sent.

The Skip-Parts negotiation remains the same but the OPES processor must still send DUM messages including Kept parameter (becoming mandantory in this case).
The callout server replies with normal DUM or DUY messages (DUM if it wants to replace the content it never saw, DUY if it wants the processor to re-use the original data as usual). If the part is not sent, it shall be deleted from the message.

The overall message part handling becomes much easier. There are DUM/DUY messages for all message parts that are in the HTTP message and parts that are not in the HTTP message (or that shall be deleted) will not be sent.

Does this solve the problems and does it simplify the draft and implementations?

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 23 Oct 2003 11:14:37 -0600 (MDT)

On Thu, 23 Oct 2003, Martin Stecher wrote:

>   it MUST decide whether it wants to adapt the request or to
>   short-curcuit. It will either send back request-header,
>   request-body, request-trailer parts or response-header,
>   response-body, response-trailer parts. Some parts could be
>   optional due to conditions defined below but it MUST NOT mix parts
>   of these two lists; on violation client MUST terminate
>   transaction.

I agree with the above in principle. If we make any changes to the
what-parts-to-send-or-skip mechanism, the above may need to reflect
those changes, of course. As of now, the above together with adding
adapted response-* parts to request profile should suffice.

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 24 Oct 2003 10:43:04 -0600 (MDT)

I think we should step back a little and try to recall our primary
goals here:

	A) parsing optimization:
	   do not spend time isolating message parts twice

	B) bandwidth optimization:
	   do not send data that is not needed by recipient

	C) accommodating HTTP-unaware services:
	   just let them see/adapt the content part

To start with, I think that a non-HTTP profile must be used for
HTTP-unaware services. This leaves us with (A) and (B) above only.

The following are facts to take into account (Martin mentioned
most of them already):

	1) HTTP messages have many parts, most of which may be
	   absent, and some may have zero length

	2) all HTTP messages we should care about have headers

	3) most HTTP messages do not have trailers

	4) HTTP message bodies are the only parts that can
	   save "enough" bandwidth if skipped

	5) Skipping bodies if there are no trailers is
	   already supported by OCP Core

	6) Most processors will not be able to skip HTTP
	   message bodies if both headers and trailers need to
	   be adopted (because they may have to cache the
	   entire body while waiting for adaptations)

	7) Adapting just HTTP headers is less common than
	   adapting bodies. Moreover, adapting just HTTP
	   headers may be done better via a dedicated
	   profile

	8) The interface to accommodate all possible combinations
	   is rather complex and it is not very likely that
	   implementations will get all corner cases right

	9) While important, both (A) and (B) are "just"
	   optimizations

Given the above, I conclude that:

	- We should keep AM-Part and Aux-Part to accommodate (A),
	  to have an interface for auxiliary parts in RESPMOD,
	  and to support short-circuit operation.

	- We should rely on OCP Core "getting out of the
	  loop" mechanisms to accommodate (B). That would
	  let us terminate data transmission early enough,
	  though not as early as with Skip-Part in some
	  cases. We should delete the Skip-Part interface.

	- Optionally, we could add an optional Last-Needed-Part
	  parameter to NR to allow specifying the _last_
	  original part needed by the service and being semantically
	  equivalent to receiving (later) a corresponding Want-Use
	  parameter with the actual size of all parts up to the
	  last one. Processors MAY ignore Last-Needed-Part.
	  With these rules, Last-Needed-Part does not introduce
	  any new complicated requirements/dependencies.

Martin, do you think the above would be sufficient to accommodate most
of our needs? If yes, do you think adding Last-Needed-Part is worth
the trouble?

Any other comments or preferences?

Alex.

On Thu, 23 Oct 2003, Martin Stecher wrote:

>   OPES processor MUST either create a dummy HTTP header part, or not
>   use OCP/HTTP.
>
>   Bodies can or cannot exist. Empty but existing bodies are
>   technically possible. Can an empty body be deleted without
>   changing the meaning of the message?
>
>
> Then the best way to describe what needs to be done I found for me
> is a pseudo programming language style. I do not take the trailer
> parts in account, it's long enough without ;-)
>

> Especially the problem that an OCP transaction that does not contain
> request-body parts in neither client nor server messages can mean
> multiple things:

> b) and c) can only be solved if we require that the callout service
> MUST send an empty body part, if it wants to make sure that a
> skipped original part is not included in the adapted message.
>
> Is this the solution?
>
> Payload becomes optional in DUM messages. A new named parameter
> "Skipped-Payload" has a value which is the size of the payload but
> the payload itself is not sent.
>
> The Skip-Parts negotiation remains the same but the OPES processor
> must still send DUM messages including Kept parameter (becoming
> mandantory in this case).
>
> The callout server replies with normal DUM or DUY messages (DUM if
> it wants to replace the content it never saw, DUY if it wants the
> processor to re-use the original data as usual). If the part is not
> sent, it shall be deleted from the message.
>
> The overall message part handling becomes much easier. There are
> DUM/DUY messages for all message parts that are in the HTTP message
> and parts that are not in the HTTP message (or that shall be
> deleted) will not be sent.
>