Subject: RE: copying commitment and deadlock

From: "Abbie Barbir" <[email-masked]>
Date: Tue, 25 Mar 2003 10:04:04 -0500

Alex,

I see the points,
Can u please provide a scenario

abbie

> -----Original Message-----
> From: Alex Rousskov [mailto:[email-masked]] 
> Sent: Tuesday, March 25, 2003 9:27 AM
> To: [email-masked]
> Subject: Re: copying commitment and deadlock
> 
> 
> 
> 
> On Tue, 25 Mar 2003, The Purple Streak, Hilarie Orman wrote:
> 
> > What's the compelling reason for message by message copy directives?
> 
> The reason for per-message [copied] flag is to optimize for 
> the common case where callout server does not modify the 
> data. OPES processor will often keep a copy anyway, for many 
> reasons. There is no need to send the same data back and 
> forth; one direction is enough in this case.
> 
> The reasons for copying commitment or a similar mechanism is 
> to optimize for the common case where the callout server 
> wants to get out of the loop because it does not expect 
> any/further modifications.
> 
> Alex.
> 
>

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 25 Mar 2003 10:00:16 -0700 (MST)

> Can u please provide a scenario

Scenario where [copied] flag is useful is handling of any application
message that is not modified by the callout server. For example, virus
scanning (most messages are probably virus-free and, hence, require no
modifications). Another, more complex example is application header
manipulation for, say, anonymization purposes: only a small portion of
the application message is affected; the rest can be sent as is. A
third example is appending "company signatures" to outgoing SMTP
messages: only the tail of the message is affected.

The first example is a scenario where "getting out of the loop"  is
very beneficial provided the virus scanner can make a "no virus"
decision without seeing the entire application message. The second
example is also a scenario where "getting out of the loop"  is very
beneficial. Getting out of the loop in the third example is not
possible.

HTH,

Alex.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Tue, 25 Mar 2003 11:10:18 -0700

In my previous note I left out the rule that makes the stream
restartable ...  if stream A is "copy" and the callout server signals
"no further copy", the easiest way to proceed with "no copy" mode is
for the OPES processor to open a new stream, B, with the directive
"this is the no-copy continuation of stream A".  Anything un-acked on
stream A is now sent on stream B.

This does mean that "copy" streams cannot be multiplexed on a single
reliable connection unless the callout server promises not to switch
to "no-copy" mode.

I'm still wondering how useful this will be; I'd always assumed that
the intelligence about which parts to forward would reside with the
OPES processor, or that the directives would be at a higher semantic
level.

Hilarie

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 25 Mar 2003 11:51:43 -0700 (MST)

On Tue, 25 Mar 2003, The Purple Streak, Hilarie Orman wrote:

> restartable ...  if stream A is "copy" and the callout server
> signals "no further copy", the easiest way to proceed with "no copy"
> mode is for the OPES processor to open a new stream, B, with the
> directive "this is the no-copy continuation of stream A".  Anything
> un-acked on stream A is now sent on stream B.
>

You seem to be solving a very different problem. Could you please give
an example or a use case? It seems to me that you are trying to
introduce some negotiation here (callout server tells the processor to
copy/move, and the processor may copy/move in response). That could be
a useful feature _on top_ of the basic mechanisms we are trying to
finalize in this thread.

> OPES processor, or that the directives would be at a higher semantic

We may be talking about different use cases here. See my earlier
response to Abbie Barbir with specific examples I have in mind.

The callout server may modify the message. Thus, the resulting data
stream from the callout server to the OPES processor consists, in
general, of new and old data chunks. The copy mode is meant to reduce
the number of old data chunks, nothing else.

Alex.

---

From: jfcm <[email-masked]>
Date: Wed, 26 Mar 2003 12:32:49 +0100

Sorry, to say it again, but you are talking here of a protocol between X 
and Y without having modelized X and Y. The callout protocol, X and Y 
modelization must be an harmonious interative process.

At 19:51 25/03/03, Alex Rousskov wrote:
>You seem to be solving a very different problem.

modelize what you are talking about.

>Could you please give an example or a use case?

examples are too specific and involving to many aspects.
The must be used to build and check a model. Not to
replace it.

>It seems to me that you are trying to

Sure he is. But what are you both really calling a "processor"?

>The callout server may modify the message.

what do you call the message? If it received only a part of the
stream, do you call message the chunk it receives or a set of
copied and non copied data chuncks. Dispatchers rules may
say that (A+B ) calls for B only to be modified into B', and that
(A+B' ) calls for A to be modified too, or in conjunction with B'.

>Thus, the resulting data

Copying seems hazardous anyway because if several services are called in 
conditional succession, how will a service know the nature of what is in 
the buffer. Are they original or already modified data.

jfc

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 26 Mar 2003 08:55:02 -0700 (MST)

On Wed, 26 Mar 2003, jfcm wrote:

> Sorry, to say it again, but you are talking here of a protocol
> between X and Y without having modelized X and Y. The callout
> protocol, X and Y modelization must be an harmonious interative
> process.

jfc,

	Please understand that such comments are not constructive
enough to cause any action. There are working group documents that
describe architecture and protocol requirements. Obviously, people who
wrote them and who are now working on the protocol think that they
have "modelized X and Y"! Sure, some models can be improved and some
scope questions need to be answered, but those improvements and
answers are unlikely to affect the subject of this thread IMO.

	It is perfectly fine to disagree with the published documents,
but please be constructive! Tell the group what exactly is wrong with
the current models. Suggest specific improvements if possible. It is
the only way we all can understand and address your concerns. "You
have not modelized" is not something others can address given the fact
that some models do exist in the working group documents.

> modelize what you are talking about.

Please see architecture and protocol requirements documents. They are
my models.

> But what are you both really calling a "processor"?

Whatever receives raw application messages and talks to the callout
server for the purpose of adapting those application messages. That's
enough detail/modelization for the subject of this thread, I think.
This thread does not really care what the processor or dispatcher is
beyond what is currently defined in the architecture and requirements
documents. Why do you ask?

> what do you call the message?

Whatever that application protocol calls a message. I should have said
"may modify the application message", to be precise.

> If it received only a part of the stream, do you call message the
> chunk it receives or a set of copied and non copied data chuncks.

[Copied] flag applies to an application data chunk. Copying commitment,
if any, applies to several chunks (possibly the entire application
message).

> Dispatchers rules may say that (A+B ) calls for B only to be
> modified into B', and that (A+B' ) calls for A to be modified too,
> or in conjunction with B'.

Sorry, I do not follow. I do not understand what "(A+B) calls for B"
means.

> Copying seems hazardous anyway because if several services are
> called in conditional succession, how will a service know the nature
> of what is in the buffer. Are they original or already modified
> data.

Original data may have a [copied] flag. Modified data must not have a
[copied] flag unless the dispatcher made a copy of the modified data
before calling the next service. Apart from that distinction,
coordination between individual OPES services is outside of this
thread scope, I guess.

Alex.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 26 Mar 2003 11:24:04 -0700

I think that the comment below refers to applying dispatch rules to
the result of a dispatch.  I.e., assume the content is two parts,
{A,B}. That content triggers a modification to part B, resulting in
{A,B'}.  The dispatch rules applied to {A,B'} trigger a modifiction to
A, resulting in {A',B'}.  What might happen in an implementation is
that the first condition would trigger a callout, causing the OPES
processor to start sending B to the callout server; the callout server
would have logic asking the OPES server if A existed, and if so, to
send it also, so that it can generate A' as a function of B'.

I can see this happening, but I'm not sure how it affects the current
discussion about copying commitment.  I think we need to be selecting
the absolutely minimal set of mechanisms to support transport of
content to and from callout servers.  We should lean heavily towards
mechanism reuse over mechanism extension.

Hilarie

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 26 Mar 2003 11:51:54 -0700 (MST)

On Wed, 26 Mar 2003, The Purple Streak, Hilarie Orman wrote:

> A, resulting in {A',B'}.

OK. This seems to have nothing to do with OCP. This is about rule
processing, right?

> What might happen in an implementation is that the first condition
> would trigger a callout, causing the OPES processor to start sending
> B to the callout server; the callout server would have logic asking
> the OPES server if A existed, and if so, to send it also, so that it
> can generate A' as a function of B'.

We need to decide whether OCP should support partial message exchange.
So far, I have assume a simple model when entire messages are
exchanged and there is no "give me part X" dialog. After the IETF
meeting, I tend to stay away from application specific parts (would
prefer to limit the language to byte ranges). However, without
application-specific parts, it becomes difficult to describe "part X"
unless the entire message is known (the callout server would not know
where X starts and where it ends).

Added this topic to the todo list.

> discussion about copying commitment.

Neither am I.

> I think we need to be selecting the absolutely minimal set of
> mechanisms to support transport of content to and from callout
> servers.  We should lean heavily towards mechanism reuse over
> mechanism extension.

Yes, of course. The difficulty is in defining the "absolute minimum",
as usual.

Thanks,

Alex.

---

From: Markus Hofmann <[email-masked]>
Date: Wed, 26 Mar 2003 15:02:40 -0500

Alex Rousskov wrote:

> prefer to limit the language to byte ranges).

I can only second this view!

-Markus

---

From: jfcm <[email-masked]>
Date: Wed, 26 Mar 2003 22:54:43 +0100

>OK. This seems to have nothing to do with OCP. This is about rule

No. the question was about a message definition and copying.
I say you have two chuncks A and B. What may make or not
one single message.

(In between you responded that the message definition was
application dependent - what I have to digest as not my own
thinking).

The first dispatch rule leaves A untouched, so A can be sent.
Now the second rule says that A is to be modified.

I did not say where the second dispatch rule is applied.
It can be on a cascading dispatcher or a loop back into the
first dispatcher before sending. Depends on the modelization.

Where am I to find A depending on the cases?
- it may have been sent - if there are considered as 2 messages
- it may have been copied to a server and not returned since unchanged
- it may have been copied back
- it may not have been copied to the precedeing service

I do not think it is complex but it needs to be made clear.
Because we may certainly find cases where it is complex.

jfc

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 26 Mar 2003 16:09:14 -0700 (MST)

> No. the question was about a message definition and copying.

.. and copying is pretty much unrelated to an application message
definition! OCP dispatcher copies bytes (opaque message chunks), not
application messages.

> I say you have two chuncks A and B. What may make or not one single
> message.

OK. A single application message may be received and handled by OCP in
chunks.  Each chunk belongs to one and only one application message.

> application dependent - what I have to digest as not my own

Since [copied] flag applies to data chunks (not entire application
messages), the definition of a message seems to be irrelevant.

Application message definition is application dependent, by its very
nature (by definition if you will). I am not sure how you can have it
any other way. Care to explain? For example, RFC 2616 defines what an
HTTP/1.1 message is. No other spec defines that. However, OPES and OCP
are application agnostic. Whatever the message is, OCP treats it as a
bunch of opaque bytes. We just need to know the message boundaries,
nothing else.

> Now the second rule says that A is to be modified.

You said above that "A" is a chunk. I think that rules should not be
applied to chunks, only messages, but it probably does not matter for
this discussion.

> I did not say where the second dispatch rule is applied. It can be
> on a cascading dispatcher or a loop back into the first dispatcher
> before sending. Depends on the modelization.

IMO, OCP takes place when a single rule is applied to a single
application message. It does not really matter (from OCP point of
view) what happens before or after that. There might be 10 more rules,
but OCP does not know that and does not care. Why are you talking
about multiple rules in an OCP copying context? What is the relation
of rules to copying?

> Because we may certainly find cases where it is complex.

I do not know what you mean by "finding A". If "A" is a message, then
it may not have a single "location" because it is comprised of opaque
chunks, and each chunk may be at a different processing
stage/location. Recall that we are trying to use a pipeline model
here; each chunk will be at some processing point of the pipeline.

Moreover, until the pipe ends, it may be impossible to reassemble A
from "current" chunks because each chunk may be at a different
processing stage. Only when all processing stages are complete, you
can have a complete application message again. Furthermore, since the
application message chunks may be sent out as they "arrive" from the
pipe, there can be no single point in time where an OPES system sees a
complete application message. Finding message "A" is not possible, in
general! This is true for HTTP non-caching proxies, for example.

Alex.

---

From: jfcm <[email-masked]>
Date: Thu, 27 Mar 2003 14:01:08 +0100

hmmm... you will excuse me, but this response seems contradictory to me 
(you seem to both question and  document my questions). I will then drop 
the debate here. I will continue to listen and to look for what I may use 
in pratical, real life development specifications and tests.
jfc

At 00:09 27/03/03, Alex Rousskov wrote:

>---
>Incoming mail is certified Virus Free.
>Checked by AVG anti-virus system (http://www.grisoft.com).
>Version: 6.0.463 / Virus Database: 262 - Release Date: 17/03/03