Subject: Re: Need to look at tracing and debuggig

From: Markus Hofmann <[email-masked]>
Date: Tue, 01 Apr 2003 13:30:55 -0500

Oskar,

great input, thanks. Just some quick, minor comments:

> 1. Tracing information has to be provided in-band, I see no
> other way to satisfy current architecture requirements. The
> OPES architecture states that:

I agree with that. The question is, though, whether the callout 
protocol itself will also carry some tracing information, or whether 
the callout server will embedd possible tracing information directly 
into the application message.

> 2. We have to decide on the in-band - out-of-band balance for tracing
> facilities. Two extreme approaches are:
> 
> - in-band data provides only a reference to the point to the facility where
> the tracing information may be obtained;
> 
> - include all information in-band.
> 
> Advantage of the first approach: a) Tracing information may be provided
> in application protocol independent manner. b) Level of details is
> determined by direct request, lengthy descriptions may be provided
> without an impact on the application protocol efficiency.
> Disadvantages: a complex identification mechanism is needed to retrieve
> application message specific information (like "virus checking applied"), and
> getting such simple notification will involve overhead of creating or
> keeping an additional connection.

Nice comments. I was tempted to lean towards the first approach with a 
refrence to a point for retrieval, but.... Where would such point of 
tracing information retrieval be? Would you assume that, for example, 
I can retrieve detailed tracing information about a provided OPES 
service directly from the callout server the service has been 
executed? Such a callout server might very well sit behind a NAT and 
might have a non-routable IP adress... If it's not directly on the 
callout server, how would the callout server provide the required 
tracing information to the other entity?

> The second approach provides a kind of opposite advantages and disadvantages:
> simple per-message information binding but application protocol may be
> cluttered
> with excess information that is not relevant to the current session (may be
> due to the lack of interest of the participants).
> 
> An additional advantage of in-band approach is end-to-end coverage: tracing
> information and related directives are available to all OPES flow
> participants,
> no topology knowledge is required.

Yup, good point, have to agree.

> A reasonable combination may include message-related information in band and
> a reference to the session-related information. To adjust in-band information
> level to the needs of session participants some trace control directives
> should
> be defined (as an application protocol extensions).

Could you provide a specific example for session-related information 
(as opposed to message-related)?

> The purpose of identification (exposure) should be defined by the
> intended use.  Only the information points (where some participant
> may call for additional information) and reference points (those that should
> be identified in related request, e.g. to the center of authority) should be
> exposed. If there are points that may accept directives (e.g. privacy
> directives) - they should be exposed.
 >
> Also session participants should be notified about the center of
> authority for the OPES server.
 > [...]

Can we develop a few example scenarios that illustrate the various 
concepts of "information points", "reference points", "identifier", 
etc., and how they play together?

> Any such explicitly identified point may be on the path or out of the path,
> this should not be a factor. Points on the path are exposed by IP,
> as according to IAB requirements connections are terminated at these points.

The IAB considerations state that "the OPES intermediary must be 
explicitly addressed at the IP layer by the end user", which 
translates  that (only) the first "hop", i.e. the  first OPES 
intermediary on the path, has to be addreessable explicitely. Others 
might very well sit behind a NAT or so.

> Information about services should be provided in-band and should uniquely
> identify the service provider and service type, but not the service point
> (OPES
> server). This makes service information location independent and facilitates
> system reconfiguration, including failover and recovery: agreements and
> parameters
> may be transferred to another OPES server (within the same trust domain)
> without
> renegotiation with end-point.

If we don't identify the exact server, how would a service provider 
trace a problem I report to him? How would he know whichserver to 
check, if I tell him that something went wrong and check him to ask 
this? With email, for example, I know exactly which mail servers have 
been o nthe path, thus being able to trace down to the exact server.

> Direct point exposure (let's say by inclusion of URL into the tracing
> information)
> raises a question about out-of-band protocol used to access this point.
> SOAP looks like a good candidate for that.

Do we have to decide on a specific protocol to be used for this 
purpose, or can we leave this open and just indicated the protocol to 
be used (e.g. withing the embedded URI).

-Markus

---

From: "Oskar Batuner" <[email-masked]>
Date: Wed, 2 Apr 2003 02:12:19 -0500

Let me summaraze relevant issues for the reference purposes:

1. Choice of the OPES application model.

The OPES architecture provides two possibilities for placing 
application modules - OPES service application on the same computer 
as the OPES dispatcher and callout server. First case (initially 
called proxylet) comes as a natural extension of caching proxies. 
Some commercially available caches have proprietary API for adding 
application logics, like filtering capabilities. The proprietary 
nature of such extensions prevented extensive deployment, and I 
believe the whole OPES idea started as an attempt to standardize 
triggers (rules) and proxylet API, environment and deployment. 

Callout server comes either as a natural extension of the first 
model to offload application processing and create a scalable 
application structure or as a way to use a different class of 
devices - fast L7 switches - as an application building platform. 

What is common to both models - the central point, dispatcher, that 
is assigned the role of policy enforcement point. 

OPES facilities should not prefer one model over another, and this 
may be achieved by keeping OPES processor as a main representation 
center. It should be responsible for complying with tracing and 
other OPES requirements. It does not mean that it always has to 
keep persistent information, but in this case callout protocol 
should support directives for tracing control. Callout protocol 
may also support negotiation about insertion tracing information 
into the message. OPES processor should be able either to request 
necessary information from callout server or to issue directives 
for information insertion and verify that directive is accepted.

Why I'm going into this lengthy discussion is that I got 
an impression that there is s shift to the second model that is 
causing some misunderstanding. Maybe I'm wrong.

2. Tracing information granularity and persistence levels. 
The information may be:

- message-related, e.g. "virus checking done - passed", "content 
filtering applied", "translated from quibbish to danqush". Such 
information should be supplied with each message and indicate 
that specific action was taken. All data that describes specific 
actions performed for the message should be provided with that 
message, as there is no other way to find message level details 
later. OPES application (including OPES processor and all 
application modules and callout servers involved) is not 
supposed to keep volatile information  after request 
processing is done. 

- session related. The session knowledge may be not directly 
supported by the protocol, as the case is for HTTP. In this 
situation OPES processor is responsible for keeping the  
session context. Session related information may be provided 
once per session, some details may be replaced by id or a 
reference for subsequient information retrieval.

Session level data must be preserved for the duration of 
the session. OPES processor is responsible for inserting 
notifications if session-level information changes. 

Examples of session-related information is "virus checker 
abcd build 123 enabled", "OPES server id=xyz present". 

- log information id. This may be used e.g. for accounting 
and non-repudiation purposes. Detailed information referenced 
by this id may be not available online but can be retrieved 
later by some off-line procedure.

- server related persistent information, e.g. "OPES center of 
authority <URI>", "privacy policy <URI>". It may be also 
presented once per session and it does not change between 
sessions.

- end-point related data: what profile is activated (profile ID), 
where to get profile details, where to set preferences. I'm not 
sure how far we should go in this direction. 

I see other work going on in this area 
(e.g. [draft-barbir-opes-spcs-03.txt]). I thing 
OPES should provide a framework for such development 
by defining flexible and extensible 
tracing and informational facilities.

3. Some terminology.

> concepts of "information points", "reference points", "identifier", 

- REFERENCE POINT - a reference that may be used out-of-band to 
  perform a specific function. 

  An example may be URI for the privacy policy, center of authority 
  URI, server address, etc. Usually no protocol is provided to access 
  the reference point.
  
- INFORMATION POINT - implies presence of the protocol to access 
  detailed information at this point. Example may be URI to get 
  a certificate for virus checker or content filter, examine 
  and set profile setting and active preferences.

- IDENTIFIER - provides a unique binding to detailed persistent 
  information. For example "transformation-applied : fe123" gives 
  a participant ability to enquire (and maybe cache) details of 
  the transformation fe123. Use of such (opaque) identifiers 
  does not require prior knowledge and does not create a burden 
  of storing additional information - this is just a tag for 
  persistent information (not message-specific).

4. Using discretion of what points should be exposed.
 
> trace a problem I report to him? How would he know which server to 
> been o the path, thus being able to trace down to the exact server.

It is the choice of the service provider - what servers should be exposed. 
For example currently if pictures coming from some site are distorted 
or data is corrupted it is extremely difficult and often even impossible 
to tell what front-end or back-end servers are malfunctioning, especially 
in the presence of dynamically addressed CDN and multi-tier backend 
application. Usually notification containing the main URL and request 
parameters should be sufficient. 

Mail server is also a good example: you may see only representative 
of a server farm, some processing, like virus checking or spam 
filtering may be performed by invisible back-end servers. Still servers 
that are directly identified in the headers give resonable information 
for problem analysis. 

I'd recommend to minimise number of points exposed - in order to hide 
application complexity and dynamic reconfiguration but provide a separate 
logical places for information requiests and references. In most cases 
OPES processor should hide underlying application structure and care the 
burden of relayng some requests (both in-line and out-of-band) to callout 
processors. This does not require storage of additional 
data - at each moment OPES procesor knows all underlying configuratiuon 
details and can determine what callout processor should answer the 
request.

5. Additional protocol and schema definitions.

> be used (e.g. withing the embedded URI).

As we are building the OPES framework from top to bottom we understandably 
delay details introduction until we are at the appropriate level of 
specification. But at some point this specifics has to be defined. If we 
define all HTTP extensions necessary to implement HTTP-based OPES system 
but for the information point only URI is defined, then interoperation of 
different implementations may become a problem.

Oskar

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Markus Hofmann
> Sent: Tuesday, April 01, 2003 1:31 PM
> To: OPES Group
> Subject: Re: Need to look at tracing and debuggig
> 
> 
> 
> Oskar,
>

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 2 Apr 2003 11:49:41 -0700 (MST)

> is assigned the role of policy enforcement point.

Actually, the two models are equivalent except for communication costs
between OCP agents. I cannot think of any other difference between
"same computer" and "other computer" placements, not to mention that
"same computer" is very difficult to define (same CPU? same processor
block? same server farm? same owner? etc.)

Thus, I would suggest that we keep varying communication costs in mind
but make no other assumptions or distinctions.

> causing some misunderstanding. Maybe I'm wrong.

I do not see a clear connection between "OPES facilities should not
prefer one model over another" and the rest of the paragraphs. I do
agree with the opening statement. However, I think it is premature to
make conclusions regarding OCP involvement.

I would suggest that we agree on what tracing information must be
supported first, and only after that talk about how to support it (in
OCP or elsewhere).

> information should be supplied with each message and indicate that
> specific action was taken. All data that describes specific actions
> performed for the message should be provided with that message, as
> there is no other way to find message level details later. OPES
> application (including OPES processor and all application modules
> and callout servers involved) is not supposed to keep volatile
> information after request processing is done.

Agreed.

> abcd build 123 enabled", "OPES server id=xyz present".

I am not convinced we have to support these kind of tracing. The end
does not usually care whether "virus checker abcd build 123 is
enabled"; it cares only wether that virus checker has seen or modified
the application message, which is already covered by "message-related"
bullet above. Same for "OPES server id=xyz present".

What is a session? What are session boundaries? How do those
boundaries correspond to message/connection boundaries? And, finally,
why should we care about anything that does not affect our application
message?

> later by some off-line procedure.

This belongs to the "message-related" category, IMO. For example,

	OPES-Actions: "virus checker FooBar applied
		(result=clear, logid=34341, version=123)"

> presented once per session and it does not change between

This has to be per-message unless you somehow can define sessions so
that the end-user can distinguish them. For example, two pipelined
HTTP request on the same TCP connection (from end-user point of view)
may pass through very different OPES intermediaries and reach
different content providers. How are you going to maintain sessions if
not on a per-message basis (which makes session concept unnecessary)?
Please give an example of a session in this context.

> where to get profile details, where to set preferences. I'm not sure
> how far we should go in this direction.

OK.

> I see other work going on in this area (e.g.
> [draft-barbir-opes-spcs-03.txt]). I thing OPES should provide a
> framework for such development by defining flexible and extensible

I agree, but I hope we can limit ourselves to per-message facilities
only (no "sessions").

>   URI, server address, etc. Usually no protocol is provided to access

If reference point is a URI (and it probably should be), then the
schema part of the URI (e.g., "http") usually determines ways to
access the information.

>   and set profile setting and active preferences.

I see no difference with the "REFERENCE POINT". The protocol
distinction is too vague. Can you give a reference point example that
lacks protocol? Do we need to distinguish the two points?

>   the transformation fe123.

Identifier can be an information point as well, right? I guess I am
missing the motivation behind this terminology and these distinctions.

>   Use of such (opaque) identifiers

The same is true for REFERENCE and INFORMATION POINTS, right?

Why cannot an IDENTIFIER be message specific? For example,

  transformation-applied: http://service.org/explain?msgkind=foobar

> It is the choice of the service provider - what servers should be exposed.

I agree. Same for HTTP's Via headers. I would also add "what servers
should be exposed, if any".

> that are directly identified in the headers give resonable information

Exactly. OPES should not be responsible for solving general
finger-pointing problems on the Internet.

> I'd recommend to minimise number of points exposed - in order to
> hide application complexity and dynamic reconfiguration but provide
> a separate logical places for information requiests and references.

I would suggest that we leave this choice to the administrator. We
should provide means to expose every OPES point. The administrator
should be able to configure their intermediaries to expose just what
they want/need.

> In most cases OPES processor should hide underlying application
> structure and care the burden of relayng some requests (both in-line
> and out-of-band) to callout processors. This does not require
> storage of additional data - at each moment OPES procesor knows all
> underlying configuratiuon details and can determine what callout
> processor should answer the request.

If we only trace OPES processors and service IDs, then OPES processor
should be responsible for adding tracing (it can use a trace-adding
callout service but that should not be required). A good example is a
text translation callout service. Such a service can be implemented in
an application protocol-agnostic manner and, hence, will not be able
to trace itself. OPES processor will always be able to add tracing
information though.

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Thu, 3 Apr 2003 10:28:51 -0500

> [mailto:[email-masked]]On Behalf Of Alex Rousskov
> Sent: Wednesday, April 02, 2003 1:50 PM
> To: OPES Group
> Subject: RE: Need to look at tracing and debuggig
>
>
>
>
> > Let me summarize relevant issues for the reference purposes:
> >

This is a kind of extreme approach. It implies that no matter where the
application module resides all interaction with dispatcher goes through
OCP. This is possible, but initially we had some attempts to define a
proxylet environment and API without regard to protocol. In this approach
"same computer" means something that supports a fast and reliable (usually
proprietary) method to get information from the OPES dispatcher to the
application module.

Currently I do not see any developments in the proxylet area, so your
approach becomes an appointed winner :).

> > reference for subsequent information retrieval.
> >

Taking virus checker as an example I can see two situations when
detailed information is important. The end user might wish to verify
virus checker signature and certificate. Very reasonable thing to do if
one is going to rely on the "check passed" notification. Another
possibility is a second OPES processor with a virus checker present
in the OPES flow. When signed as "passed" message reaches this second
processor it may be interested not only in the first virus checker
credentials but also in type and version information - in order to
adopt it's actions.

> why should we care about anything that does not affect our application

I do not have a "standard" session definition at hand, so for our
purposes I'd define session as an application dependent persistent
context that propagates certain parameters to all messages belonging
to that session. In this sense session notion does affect our application
messages: if there was a specific OPES application participating
in the session its credentials and functional abilities extend to all
messages within that session. Reintroducing and re-verifying complete
list of all participating OPES servers' credentials, settings and
functional capabilities with each message may have a serious
performance impact.

> 		(result=clear, logid=34341, version=123)"

Yes, this belongs to message-related. I was thinking about
designated a separate (additional) category because of two
specific information of logging information:

- it can not be used  as a parameter in subsequent real-time requests
for detailed information

- it is persistent and should be added to the other logs to facilitate
future log collation.

I'm not sure we really need this as a separate category - this is
just a subject for discussion.

> Please give an example of a session in this context.

Let's say a series of requests from the same user to the same site.
It is application dependent, so I'd live determination of the session
boundaries to the application. This creates a possibility that different
participants may define session differently, but a) they may add a session
id or use the existing one (application defined), b) they may do certain
thing that are safe even if session context is lost, e.g. put unique
identifiers instead of descriptions. If end user lost related description
he/she may just send a kind of whois request for that identifier.

I do not understand your statement about two requests in the same
connection passing through different OPES server to different content
providers. My understanding was that the requirement of explicit
IP exposure and per-hop connection termination prevents such situations.
Please explain.

> I agree, but I hope we can limit ourselves to per-message facilities

Knowledge of information persistence permits replacement of detailed data (
which is required for volatile information) by references. Sure server URI
is an object better defined than a session. We may further clarify what
kind of persistency should be taken into consideration, but considering
all information as volatile (per-message) looks like too strong
simplification.

> lacks protocol? Do we need to distinguish the two points?

[email-masked] looks for me as a reference point. The main
difference is presence of a programmatic facility for information
retrieval. Such facility permits to remove from the protocol detailed
information even if this information may be needed by the
participating applications.

> missing the motivation behind this terminology and these distinctions.

Information point is an address (URI) and protocol combination,
identifier is just a tag.

If you want to tell that this terminology is far from perfect - yes,
I agree. But still I hope we can clarify

> The same is true for REFERENCE and INFORMATION POINTS, right?

REFRERENCE and INFORMATION POINTS are not opaque. Application and/or
a human being should be able to interpret them and use for information
sending and/or retrieval.

>   transformation-applied: http://service.org/explain?msgkind=foobar

Sure IDENTIFIER can be message specific, e.g. log-id. But one you are
introducing here is a combination of identifier and an information or
reference point (the difference is determined by presence of predefined
response format and ability of application to interpret this response
programmatically).

I am a strong proponent of schema normalization, and I prefer to keep
separately all things that can be separated.
This example should look like:

transformation-applied: server=1.2.3.4 definitions=http://service.org
	transformation-type=foobar profile=ffe123

> > that are directly identified in the headers give reasonable information
> > for problem analysis.
>
> > I'd recommend to minimize number of points exposed - in order to
> > a separate logical places for information requests and references.
>
> > structure and care the burden of relaying some requests (both in-line
> > storage of additional data - at each moment OPES processor knows all
> > underlying configuration details and can determine what callout

Agreed.

Oskar

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 3 Apr 2003 11:34:54 -0700 (MST)

> proprietary) method to get information from the OPES dispatcher to the

Oh, that's perfectly fine. I was only talking about the tracing impact
on OCP. Other protocols or interfaces may have different models, of
course.

> virus checker signature and certificate. Very reasonable thing to do
> if one is going to rely on the "check passed" notification. Another
> in the OPES flow. When signed as "passed" message reaches this
> second processor it may be interested not only in the first virus
> checker credentials but also in type and version information - in
> order to adopt it's actions.

I agree! However, I think this information should be transmitted on a
per-message basis because the application protocol we care about most
(HTTP) does not have sessions. Not sure about SMTP.

> to that session. In this sense session notion does affect our
> application messages: if there was a specific OPES application
> participating in the session its credentials and functional
> abilities extend to all messages within that session. Reintroducing
> and re-verifying complete list of all participating OPES servers'
> credentials, settings and functional capabilities with each message
> may have a serious performance impact.

I agree. However, HTTP does not have sessions. Nothing in HTTP
"propagates certain parameters to all messages belonging to X" because
HTTP messages do not belong to any end-to-end X. The closest you can
come is an HTTP persistent connection, but since it is hop-by-hop, it
cannot be used as a session.

> IP exposure and per-hop connection termination prevents such situations.

Yes, that statement is in the core of our disagreement. Here is an
example:

	- Client issues two requests A and B. Both requests
	  are pipelined on the same HTTP/TCP connection to
	  client-side proxy P0. The requests may be to the
	  same origin server, but do not have to be.

	- Proxy P0 reads both requests and sends them off
	  using two HTTP/TCP connections: request A
	  is forwarded to proxy PA, and request B
	  is forwarded to proxy PB. There are many reasons
	  to "split" two requests: caching, processing
	  rules, peering relationships, etc.

	- Both proxy PA and PB have OPES processors.
	  Processors belong to different OPES systems,
	  with different privacy policies, etc. Each
	  forwards the request its origin server.

It is not possible to establish a real "session" between a client and
proxy PA because the client would not be able to distinguish that
session from the "session" established by proxy PB. Moreover, the same
request A sent 5 seconds later may go to PB and not PA (perhaps P0
load balances or whatever).

The only option to maintain session information is to send some kind
of session ID with every response so that the client can distinguish
sessions on a per-message basis and maintain its own session state.
This becomes not a session but rather a "temporary alias"
optimization:

first message:
    OPES-Agent-Details: a very long header ... (id=1231)

next messages:
    OPES-Agent-Details: see info with id 1231, if you still have it

Is that something you are after?

> Knowledge of information persistence permits replacement of detailed
> data ( which is required for volatile information) by references.
> Sure server URI is an object better defined than a session. We may
> further clarify what kind of persistency should be taken into
> consideration, but considering all information as volatile
> (per-message) looks like too strong simplification.

"replacement of detailed data by references" smells like a
OPES-Agent-Details example above. I would not call it a session-based
approach. This is more like caching or temporary aliasing. If that is
something you are after, perhaps we can rename it to avoid "session"
confusion?

> participating applications.

To me, [email-masked] implies SMTP (e-mail) protocol to access
the information so it becomes an INFORMATION POINT. I can send an
e-mail and expect an automated or personal response with more info.

I guess we can ignore this confusion until the difference between
REFERENCE and INFORMATION POINTs becomes clear or important.

Alex.