Subject: More comments on ocp-00

From: "Martin Stecher" <[email-masked]>
Date: Fri, 4 Apr 2003 12:31:10 +0200

First of all:
Good work Alex!
I think this draft covers pretty well what has been discussed on
the list so far. I vote for turning a later version of this
document into a WG draft.

- The sentence "A SERVER MUST NOT send this message" should also
  be added to the connection-service message and maybe also to
  connection-priority

- I am not sure whether the ability to change the service with
  connection-service and within xaction-start is really needed.
  It increases the implementation requirements for a callout
  server but is it really needed?
  Setting a service will usually also mean to negotiate further
  parameters.
  Couldn't we just put this into connection-start?
  Using different services will then mean to start different
  connections. Isn't this just ok?

- I forget about the motivation of the "ack" flag and data-ack
  messsage. I thought we wanted to have this done by the
  transport layer binding. Can you point me to the thread in
  which this has been discussed?

- Isn't there a problem with data-pause?
  If the SERVER sends data-pause to the CLIENT, the CLIENT
  MUST stop sending but before the CLIENT receives and
  processes this message it may already have sent additional
  data-have messages. The SERVER will then receive unexpected
  data any MAY abort.
  Second problem: A CLIENT may not be able to pause as long
  as the server likes it to pause. If it is not able to keep
  a copy of the complete message and if it is not be able to
  pause the origin server for the same time, it may easily run
  into an internal buffer overflow problem.

  How about defining that data-pause MUST NOT be send by the
  SERVER but adding a new message data-pause-request which is
  only to be sent by the SERVER. Upon receiving this message
  the CLIENT MAY pause by sending data-pause to the SERVER.

Regards
Martin

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 4 Apr 2003 09:19:09 -0700 (MST)

On Fri, 4 Apr 2003, Martin Stecher wrote:

> > 	a) Thus, somebody needs to tell OPES processor
> > 	  and callout server what is an "application message"
> > 	  definition that they should both use during OCP
> > 	  communications. Should OCP support auto-negotiation or
> > 	  rely on out-of-band (e.g., manual) configuration?
>
> How about this:
> We discussed that we probably end with multiple application
> protocol bindings of the OCP. One application protocol binding
> could be a generic binding which is not really related to any
> Internet protocol but handles data as if it was a file on disk.
> Application messages could have a very simple and generic format
> here. Many callout services may want to support this protocol
> binding and an OPES processor may have pre-processing code to
> translate Internet protocol data to that simple format.
> Other protocol bindings will handle native HTTP and SMTP messages.
> Every application protocol binding defines a fixed application
> message format.

I think both Reinaldo Penno and myself want a scheme very similar to
what you described above. There will be a set of supported application
protocols and message encodings. Known message "formats" will be
documented. OPES agents will support some subset of them. It is
probably a good idea to document a scheme where OCP sees only message
payloads and is unaware of the actual application protocol (still, one
would need to agree on things like payload encoding).

> Negotiations between OPES processors and callout services is then
> limited to exchange supported application protocol bindings.

This is the only uncertain area. You are, essentially, voting for
supporting auto-negotiations (over a finite set of known
protocols/encodings, of course). If nobody objects to this automation,
OCP will implement auto-negotiation. My understanding is that Reinaldo
is already working on specific messages to handle that.

> > 	e) Do we need to group processing of a single application
> > 	   message together using OCP transaction concept? Should
> > 	   OCP transaction mean something else? Do we need OCP
> > 	   transactions at all?
>
> Yes, I think so.
> In most cases not more than one application message will be
> transferred from CLIENT to SERVER and back within a transaction.
> But we have two examples where multiple messages make sense:
> 1. Allowing the SERVER to return different SMTP messages for
> one email to multiple recipients it received from the CLIENT.
> 2. Pre-processing of a message at CLIENT which results in
> multiple application messages, e.g. MIME sections of an email
> to be transferred as multiple application messages
>
> The first case requires to have the messages being wrapped into
> a single transaction.
> The second case could benefit from a transaction wrapping
> because the SERVER may want to know which MIME sections belong
> together.

The current draft does not support the second case in a situation where
both cases happen at the same time (e.g., multiple responses to a
pre-processed message part). This combination can be supported within one
OCP transaction only if server responses specify which pre-processed
application message [part] they are adapting. That is, an
app-message-start message from a callout server will have to include a
second am-id parameter, carrying the am-id of the corresponding
application message part from the processor. While possible, this
defeats the purpose of having an OCP transaction -- since all server
responses have to carry two am-ids, the transaction wrapping is useless
for anything other than aborting the whole thing with one message.

I think the second case must be supported by generating multiple
(possibly interleaved) OCP transactions, one for each pre-processed part.
The overhead, if any, will be minimal, and the only missing
functionality would be an ability to abort all parts with one message. We
can add 'xaction-ends' to support the latter if needed.

> I agree with Alex that the transaction overhead could (and should)
> be limited to a minimum so that no additional transport layer packet
> is needed. For the normal case the xaction-start app-message-start
> couple could almost look like a single message I think.
> But let's keep the xaction-start message two allow support of above
> cases.

Yes, there is not much difference between parsing two simple messages
with 1 attribute and one complex message with 2 attributes.

>   be added to the connection-service message and maybe also to

Ack.

>   connections. Isn't this just ok?

Yeah, that will work. The rationale was that you can use the
connection to specify common/default services and then overwrite that
on a transaction basis. You make a good point regarding associated
negotiations though. Will add this question to the to-do list. The
answer will probably depend on what kind of negotiations we would end
up supporting.

>   which this has been discussed?

Ack/Data-ack sequence is useful if one side wants to track (or debug)
progress in receiving/parsing OCP messages on the other side. Perhaps
OPES processor can optimize its buffer management by knowing how fast
the callout server is reading the data. Do you think that enough
motivation to support the ack/data-ack pair?

>   data-have messages. The SERVER will then receive unexpected

Indeed. I've added the "MAY abort" part at the last minute and
missed the bug. Will fix.

>   into an internal buffer overflow problem.

I do not think that can happen because the CLIENT can always stop
reading incoming data if it has no place to put it. Right?

There is a deadlock problem, however (see the "copying commitment and
deadlock" thread on this list). That problem is on the to-do list.

>   the CLIENT MAY pause by sending data-pause to the SERVER.

The above logic is what data-pause from the SERVER was supposed to mean.
We can fix the current definition or use data-pause-request instead.
Will fix.

Thanks a lot for the comments,

Alex.

---

From: "Reinaldo" <[email-masked]>
Date: Fri, 4 Apr 2003 14:44:36 -0500

----- Original Message -----
From: "Alex Rousskov" <[email-masked]>
To: "OPES WG (E-mail)" <[email-masked]>
Sent: Friday, April 04, 2003 11:19 AM
Subject: RE: Comments on ocp-00

> is already working on specific messages to handle that.

Folks,

Are we talking about the same thing? This is already a MUST on the
requirements draft. We have several parameters that need to be negotiated
that have (a priori) nothing to do with the application protocol/bindings
OCP will be carrying.  They are go from basic things like version number to
performance, security, etc.

"  Capabilities and parameters that could be negotiated between an OPES
   processor and a callout server include (but are not limited to):
   callout protocol version, fail-over behavior, heartbeat rate for
   keep-alive messages, security-related parameters etc.

"
and yeah, I'm already working on that...

thanks,

Reinaldo

My statements

---

From: jfcm <[email-masked]>
Date: Sat, 05 Apr 2003 14:59:48 +0200

I went through a very interesting real life "something" which very much 
looks like an OPES but has none of its attributes and makes big real money 
(a ONES?). The box is on the edge of the network since it is actually a 
gateway to X.25 Minitel protocol. So there is protocol conversion at lowest 
layer  as well.

That box reads a web page and transforms it in a Minitel page on the fly. 
Since in France there are still more Minitel users than Internet users, I 
think the example is relevant.

The interest is that other OPES could be added to transform the page not 
only from HTML to Minitel format, from TCP/IP to stable cheap X.25, from 
free access services to pay network access with good return for the content 
provider, but also from French into English,  etc...

I note that these services have an interesting specificity: no callout 
protocol the way it is proposed in here. But a much more simple "callout 
path". Depending on the load of the next Service Processors and on the next 
rule to apply the precedeing dispatcher send the whole traffic to a 
processor or to another, dasy chaining with all the possible loop, 
redirect, etc. that an X.25 native net can provide including multitarget, 
closed user groups, etc..

I discussed with the people there and the way they are able to support 128 
users with a simple cheap X25 board and asked them about a simple 
architecture where I could use an X.25 edge network, daisy chaining with as 
much PC I would like (standard plug, stable and secure flow, flow control, 
simply formated extremely fast exchanges, easy priority support), looping 
back into the old internet in good old TCP/IP once the whole bunch of 
sophisticated OPES have been performed.

I am waiting for a price from them (we developped that interface in 1983 so 
one probably improved since then). We discussed an interesting feature BTW: 
the Gatekeeper. To be able to store at the entrance of the X25 system 
(first dispatcher) everything confidential which may not interest the OPES 
operators and restore them at the output. Obviously the X.25 system can be 
used as a star network or a mshed network (the processed documents 
proceeding along a dasychaining or coming back to the same dispatcher until 
cleared).

Comments welcome. jfc

On 21:44 04/04/03, Reinaldo said:

>---
>Incoming mail is certified Virus Free.
>Checked by AVG anti-virus system (http://www.grisoft.com).
>Version: 6.0.463 / Virus Database: 262 - Release Date: 17/03/03

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Sat, 5 Apr 2003 14:45:33 -0700

The Minitel/Internet example is certainly interesting.  I think that it is
useful to keep the architectural separation between transport and
application in mind here.

Good example of OPES service: Minitel page <-> HTML page
   question: is there a cache of translated pages at the gateway?

Good example of OPES service: English -> French
   question: are French network service providers allowed to translate
	     French -> English  :-)

Good example why OPES has privacy requirements: removing privacy-related
   information from data going to callout servers.  I don't know what kind
   information X.25 carries or whether or how hard it is to remove/restore
   it, but the overall principle is an important one

Example of service on an generalized OPES architecture: X.25 <-> TCP/IP
   question: this raises a lot of questions about addressing, flow control,
             naming, character set conversions, etc.  Seems like a
             nightmare, ..., does it really work?  It would seem especially
             awkward to support the barely disguised OPES model of forwarding
             IP packets on the callout path - it's not a requirement, but
             it certainly works well in an all TCP/IP environment.

Hilarie

---

From: Markus Hofmann <[email-masked]>
Date: Sat, 05 Apr 2003 20:52:37 -0500

> performance, security, etc.

The intention behind this requirement was to allow for very basic, but 
fundamental and absolutely required parameter exchange. It should 
*not* be seen as justification for very complex capability 
auto-negotiation schemes. Key point - keep it simple, any added 
complexity will have to be justified with a very specifc "must-have" 
example.

-Markus

---

From: "Reinaldo Penno" <[email-masked]>
Date: Sat, 5 Apr 2003 21:28:11 -0500

Markus,

although I agree with the simple approach, the idea is to have a capability
negotiation scheme that can be extended to negotiate whatever people want.
The WG itself should standardize only a few parameters such as version
number, keep-alive, etc.

would you agree?

---

From: Markus Hofmann <[email-masked]>
Date: Sat, 05 Apr 2003 22:30:38 -0500

> The WG itself should standardize only a few parameters such as version

If we can keep the extensible negotiation scheme reasonably simple and 
don't end up specifying a complex negotiation protocol - sure.

-Markus

---

From: jfcm <[email-masked]>
Date: Sun, 06 Apr 2003 19:32:06 +0200

>              it certainly works well in an all TCP/IP environment.

Well, there are certainly a lot of issues here and it could be much 
application dependant (TCP/IP). Only a test can show what is the fastest. 
Also depends on the Window size, etc. But the real point is that it shows a 
totally different logical model and a different philiosophy of protocol. We 
have a fast pipe with bypasses and loops.

And no callout protocol.
jfc

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 9 Apr 2003 10:43:27 -0600 (MDT)

OCP Core and Tracing drafts need to talk about application-independent
mechanisms. "Something(s)" need to talk about application-specific
aspects of OCP and tracing.

For example, OCP Core documents OCP messages that can be used to
communicate HTTP messages and HTTP metadata (as opaque sequences of
octets) between OPES processor and callout servers. Some other
document has to explain what HTTP metadata really means to OPES
processor and callout server when it is sent over OCP (and document
metadata format if we decide that the format itself is
application-specific).

Similarly, Tracing draft documents general rules (e.g., OPES
processors MUST/SHOULD/MAY add their IDs to the trace). Some other
document has to explain how trace entries can be added to SMTP
messages (MIME header field names and such).

It would help authors to know the set of final OPES documents so
that appropriate cross-references can be made before all documents
are written. The set can change at any time, of course.

I would like to suggest the following set. It is based on a principle
that it is better to have one application-specific document per
application. An alternative is to have one application-specific
document per application, per application-independent draft. The
former places all, say, HTTP-related things in one draft, yielding
	(#Facilities + #Applications) documents
(convenient for those who implement/deploy/debug OPES _systems_ for
HTTP).  The latter allows for more fine-grained, facility-specific
application drafts, yielding
	(#Facilities * #Applications) documents
(convenient for those who implement/debug specific OPES facilities for
HTTP).

The set is incomplete, I only mention some documents we care about at
this time:

	Application-independent documents:

		OPES Architecture
		draft-ietf-opes-architecture

		OPES Callout Protocol Core
		draft-ietf-opes-ocp

		OPES Tracing
		draft-ietf-opes-trace

		OPES Bypass
		draft-ietf-opes-bypass

		...

	Application-specific documents:

		OPES adaptation of HTTP
		draft-ietf-opes-http

		OPES adaptation of SMTP
		draft-ietf-opes-smtp

		OPES adaptation of FooBarP
		draft-ietf-opes-foobarp

		...

Does the above make sense? Would you prefer the alternative (more of
more specific drafts)? Other options like combining all
application-independent drafts into one document?

Also, given a relatively large number of related drafts, would it make
any sense to have one short "OPES Requirements" draft that lists all
available drafts and describes the relationship between them? Or do we
want to repeat pretty much the same explanations in every draft
instead? Are there similar cases in other WG?

Thanks,

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Wed, 9 Apr 2003 15:52:36 -0400

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Alex Rousskov
> Sent: Wednesday, April 09, 2003 12:43 PM
> To: [email-masked]
> Subject: set of OPES documents
>
>
[...]

> 		OPES Callout Protocol Core

If I understand correctly the last one is about - very limited - set
of directives. I'd suggest having a single document "OPES tracing,
notification and directives"

		draft-ietf-opes-dirs-trace

The rationale: subjects are tightly connected. There will be requests that are
tracing directives (e.g. Set-tracing-level), there will be responses to
directives
that are very close to tracing/notification info (<Via (OP:abcd> -
<Whois:abcd> - ...).
"Bypass" looks like low-granularity instruction about the desired level of
transformation.

-------------
And again - no, OPES is not evil! It is MAY be very useful for both content
producers and content consumers. Let's not focus just on ways how to
reduce possible harm, let's also look at the ways to maximize benefits. Why
the first thing that comes in mind is "bypass"? Why not "activate",
"translate",
"enable-transformation"?
-------------

If there will be OCP-related tracing features we need another document:

		draft-ietf-opes-ocp-trace

Rationale: OCP is just one recommended callout protocol. We do not exclude
usage of
other callout protocols conforming to the OPES requirements (e.g. icap-based).
Inserting OCP-related things into the document describing OPES flow will make
OCP an integral part of the OPES spec. IMHO OCP should be positioned as a
replaceable
building block.

> 		OPES adaptation of FooBarP

For the reasons described above I'd suggest a separate set of docs:

		draft-ietf-opes-opcp-http
 		draft-ietf-opes-opcp-smtp
		draft-ietf-opes-opcp-...

> instead? Are there similar cases in other WG?

I agree - this is an important problem. There already is certain interrelated
information scattered through different drafts. For example at some moment
Hilarie referred to the figure 2 in the policy draft to explain some
architectural
elements. I am not sure that a person studying OPES architecture will
easily find this connection. Adding a big group of documents without
proper systematization may create even more confusion.

Oskar

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 9 Apr 2003 14:18:34 -0600 (MDT)

On Wed, 9 Apr 2003, Oskar Batuner wrote:

> The rationale: subjects are tightly connected. There will be
> requests that are tracing directives (e.g. Set-tracing-level), there
> will be responses to directives that are very close to
> tracing/notification info (<Via (OP:abcd> - <Whois:abcd> - ...).
> "Bypass" looks like low-granularity instruction about the desired
> level of transformation.

Good point! I agree that it may make sense to have one draft that
covers all communications between end points and OPES processors
(tracing, bypass, etc.) If we go down that path, we need a more
appropriate name though. How about:

	"Processor-to-end communications in OPES"
	draft-ietf-opes-end
or
	"OPES processor and end points communications"
	draft-ietf-opes-end

or something like that? Does that cover all uses we know of right now?

> Why the first thing that comes in mind is "bypass"? Why not
> "activate", "translate", "enable-transformation"?

"Bypass" is shorter and easier to grasp with little context. :-)

> Rationale: OCP is just one recommended callout protocol. We do not
> exclude usage of other callout protocols conforming to the OPES
> requirements (e.g. icap-based). Inserting OCP-related things into
> the document describing OPES flow will make OCP an integral part of
> the OPES spec.

I agree with you rationale, but not with your conclusion/suggestion.
IMO, If there will be OCP-related tracing features, they can go into
OCP document (draft-ietf-opes-ocp) or (draft-ietf-opes-ocp-core).

> IMHO OCP should be positioned as a replaceable building block.

I agree. However, a "replaceable building block" assumes some common
interface. The presence of such (undocumented) interface has an effect
on what you propose below.

>  		draft-ietf-opes-opcp-smtp

I assume you meant "-ocp-". I see your reasons, but have to say that
if we go down that path, we would _also_ need application bindings for
draft-ietf-opes-end:

		draft-ietf-opes-end-http
		draft-ietf-opes-end-smtp
		draft-ietf-opes-end-...

Instead, we could try to keep OCP as a replaceable block, but assume
(keep in mind) that whatever interface is needed for *-app-protocol
drafts would be common among all "OCPs". For example, metadata meaning
and service IDs would be common among all OCPs. This approach limits
our theoretical abilities/flexibility, but, since chances of two OCPs
long-term survival are pretty slim, we may be OK.

Comments?

Alex.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 9 Apr 2003 18:21:15 -0600

Some WG's produce a "roadmap" document that does this.  However, experience
shows that a large number of related drafts is a situation to be avoided
wherever possible.  It seems like a good idea at first, but if they
need to cross-reference each other it becomes a nightmare to keep them
all in sync.  Years later backlash sets in.

Hilarie

---

From: Markus Hofmann <[email-masked]>
Date: Thu, 10 Apr 2003 13:38:47 -0400

> are written. The set can change at any time, of course.

Yup. In general, however, let me second Hilarie''s advise (in a later 
email) that we should try to avoid a *large* number of related drafts. 
This does *not* preclude having separate, focused drafts as you 
propose, but if there are multiple choices on how to split them, a 
general guideline would be to keep the number of separate documents 
low (as feasible).

> document per application, per application-independent draft. 

Following above guideline, I would be tempted to lean towards the 
former approach, namely having one application-specific document per 
application.

> 		OPES Callout Protocol Core

As suggested later, I like the idea of combining the later two, since 
both are concerned about conveying information towards the end-user(s).

> 		OPES adaptation of FooBarP

Yes, but only the first one (on HTTP) would be within the current 
charter of the WG.

-Markus

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 11 Apr 2003 14:58:21 -0600 (MDT)

On Sun, 6 Apr 2003, jfcm wrote:

> But the real point is that it shows a totally different logical
> model and a different philiosophy of protocol. We have a fast pipe
> with bypasses and loops. And no callout protocol.

An OPES system does not require the use of OCP. OPES processor can use
other means to adapt content, including adapting content using
processor's internal resources/code. OCP is just something generally
(but not universally) useful that we want to standardize, besides
other things that are OCP-independent (like tracing and bypass).

Thus, I do not see any "totally different" logical model or
philosophy in your example. What is the difference between a
chain/mesh of OPES processors and the model you describe?

Thanks,

Alex.

---

From: jfcm <[email-masked]>
Date: Sat, 12 Apr 2003 12:33:05 +0200

>chain/mesh of OPES processors and the model you describe?

Totally different is may be too big a word. However in my present
OPES model understanding it is, but I may be wrong. This is due
to the lack of proper layer modelization. Why would two chained
OPES have necessarily to fall down to http level and not to chose
to stay at callout protocol level?

jfc

---

From: Alex Rousskov <[email-masked]>
Date: Sat, 12 Apr 2003 08:15:31 -0600 (MDT)

> to stay at callout protocol level?

The answer is very simple: an OPES processor will proxy application
protocol (e.g., HTTP) instead of "branching out" using the callout
protocol (e.g., OCP) when the processor is not interested in the
result of the adaptation that the next hop (the chained OPES
processor) will perform.

These kinds of decisions are usually done when interpreting the rules
at the first processor. Thus, the decision is really done by whoever
writes the processing rules.

Alex.