Subject: RE: Need to look at tracing and debuggig

From: "Abbie Barbir" <[email-masked]>
Date: Tue, 1 Apr 2003 14:27:13 -0500

see inline,
abbie

> -----Original Message-----
> From: Markus Hofmann [mailto:[email-masked]] 
> Sent: Tuesday, April 01, 2003 1:31 PM
> To: OPES Group
> Subject: Re: Need to look at tracing and debuggig
> 
> 
> 
> Oskar,
> 
> great input, thanks. Just some quick, minor comments:
> 
> > 1. Tracing information has to be provided in-band, I see no 
> other way 
> > to satisfy current architecture requirements. The OPES architecture 
> > states that:
> 
> I agree with that. The question is, though, whether the callout 
> protocol itself will also carry some tracing information, or whether 
> the callout server will embedd possible tracing information directly 
> into the application message.
> 

---- abbie

I agree tracing should be done inline. whether the ocp will carry the
tracing info or not has to be answered with the next question below. I think
that tracing has to go throu the opes processor, the callout server is just
a slave in this case. I know that this will add more burden on the processor
in particular, if info need to be saved.

> > 2. We have to decide on the in-band - out-of-band balance 
> for tracing 
> > facilities. Two extreme approaches are:

SNIP

> Nice comments. I was tempted to lean towards the first 
> approach with a 
> refrence to a point for retrieval, but.... Where would such point of 
> tracing information retrieval be? Would you assume that, for example, 
> I can retrieve detailed tracing information about a provided OPES 
> service directly from the callout server the service has been 
> executed? Such a callout server might very well sit behind a NAT and 
> might have a non-routable IP adress... If it's not directly on the 
> callout server, how would the callout server provide the required 
> tracing information to the other entity?
> 

-----------abbie
see above

> > The second approach provides a kind of opposite advantages and 
> > disadvantages: simple per-message information binding but 
> application 
> > protocol may be cluttered with excess information that is 
> not relevant 
> > to the current session (may be due to the lack of interest of the 
> > participants).
> > 
> > An additional advantage of in-band approach is end-to-end coverage: 
> > tracing information and related directives are available to 
> all OPES 
> > flow participants, no topology knowledge is required.
> 
> Yup, good point, have to agree.
> 

-----------abbie

agree too

> > A reasonable combination may include message-related information in 
snip

> 
> > Direct point exposure (let's say by inclusion of URL into 
> the tracing
> > information)
> > raises a question about out-of-band protocol used to access this 
> > point. SOAP looks like a good candidate for that.
> 

-----abbie

agree, but how about duration and the extra burden of storing the info.

> Do we have to decide on a specific protocol to be used for this 
> purpose, or can we leave this open and just indicated the protocol to 
> be used (e.g. withing the embedded URI).
> 
> -Markus
> 
-------abbie

I have discussed embedded URI before, it could been argued that this is out
of band.
even with that we need to answere questions such as who did what when and
may require signatures etc..
This sounds like the trace (debugg token) that we spoke about before.

abbie

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 1 Apr 2003 23:53:24 -0700 (MST)

More thoughts on tracing and debugging.

1. I could not find any reference to "debugging" in architecture
   and OCP requirements drafts. Shouldn't we talk just about
   tracing/reporting? I am not sure what exactly others mean by
   debugging, but since it is not required, we do not have to
   worry about it. Tracing helps with debugging, of course,
   but let's concentrate on immediate goals. Please correct me
   if I missed debugging requirements.

2. It is very important to keep in mind that tracing is limited
   to the trust domain. Anybody outside of that domain may or may not
   see any traces, depending on domain policies/configuration. In
   other words, we are not talking about mandatory end-to-end tracing
   facility here. For example, if an OPES system is on the content
   provider "side", end-users are not guaranteed any traces. If an
   OPES system is working inside end-user domain, the origin server is
   not guaranteed any traces related to user requests.

3. There are two distinct purposes/uses of traces. First, is to
   enable the "end (content producer or consumer) to detect OPES
   processor presence within end's trust domain. Such "end" should be
   able to see a trace entry, but does not need to be able to
   interpret it beyond identification of the trust domain(s).

   Second, is the domain administrator. The administrator should be
   able to take a trace entry (possibly supplied by an "end"  as an
   opaque string) and interpret it. The administrator must be able to
   identify OPES processor(s) involved and may be able to identify
   applied adaptation services along with other message-specific
   information. That information should help to explain what OPES
   agent(s) were involved and what they did, but it is impractical to
   provide all the required information in all cases. A trace record
   is a hint, not an exhaustive audit.

   Moreover, since trust domains and their administration vary a lot,
   I would argue that we must give implementors a lot of freedom in
   what to put in trace records and how to format them. Trace records
   should be easy to extend beyond basic OPES requirements. Trace
   management algorithms should treat trace records as opaque data to
   the extent possible.

   Markus asked for tracing use cases. If we start collecting those, I
   would suggest to clearly document "end" and "admin" role in each
   use case.

4. (This is a consequence of #2 and #3 above) We should not expect
   entities in one trust domain to be able to get any OPES-related
   feedback from entities in other trust domains. For example, if I am
   an end-user, and I think that the page I am getting is corrupted by
   a callout service, I should not expect to be able to identify
   that service, contact its owner, or debug it _unless_ the
   service is within my trust domain. This is no different from
   the current situation where it is impossible, in general, to know
   the contact person for an application on an origin server that
   generates broken HTML; and even if the person is known, one should
   not expect that person to respond to end-user queries (in general).

4. We know that traces must be in-band. This [very reasonable]
   requirement limits both the number of application protocols that
   OPES can adapt and the amount of details a trace record can carry.

   The former limitation must be clearly documented somewhere so that
   folks do not try to apply OPES to unsupported applications only to
   find out months later that they cannot trace them.

   Some of us may want to supply additional information out-of-band to
   address the second limitation. Since architecture and protocol
   requirements drafts do not require support for out-of-band tracing
   details, I suggest that the WG should not spend much time on them
   and treat them as possible extensions to the tracing facility.
   Let's concentrate on in-band tracing for now.

5. There is a question on whether OPES processor (OCP client) or
   callout server (OCP server) must be responsible for adding trace
   records to application messages. I am not 100% sure, but I would
   suggest that we try to assign this task OPES processor exclusively.
   Here are my reasons:

	a) Exclusive assignment simplifies the protocol.

	b) There are use cases where callout services adapt payload
	regardless of the application protocol in use and leave header
	adjustment to OPES processor or other services. For example,
	think of a generic text translation or image modification
	service; such services require payload encoding knowledge but
	can be application-independent if OPES processor can supply
	them with just the payload.

	c) OPES processor is always _able_ to trace its own invocation
	and service(s) execution because OPES processor must understand
	the application protocol. Assigning these tracing tasks to
	callout servers is just an optimization in cases where callout
	servers manipulate application message headers.

	d) We are not required to trace services, just processors,
	AFAIK.

	e) It makes OPES compliance checks easier when remote 3rd
	party callout servers are used.

	f) Servers or services MAY add their own OPES trace records,
	of course.

6. #5 suggestion implies that tracing is out of OCP scope! :-)

7. How tracing is added is application protocol-specific and may be
   documented in separate RFCs/drafts. We can only document what tracing
   information is required and, perhaps, some common tracing elements.

HTH,

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Wed, 2 Apr 2003 10:10:44 -0500

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Alex Rousskov
> Sent: Wednesday, April 02, 2003 1:53 AM
> To: OPES Group
> Subject: RE: Need to look at tracing and debuggig
>
>
>
>

I'm afraid the situation is more complicated. While ability to control
tracing facility and level of details available inside and outside of
the trust domain may be different there are some requirements (e.g.
privacy) that are not limited to the trust domain. Both IAB requirements
and the architecture mandate some end-to-end disclosures.

>    would suggest to clearly document "end" and "admin" role in each

There are more participants interested in tracing information. Independent
OPES processors in the same OPES flow might want to know about each other
existence and might need to cooperate. For example virus checker may be
satisfied by the signature of another virus checker, ad insertion/localization
servers may avoid stepping on each other toes, etc.

In fact you are describing two roles. We should use this approach - roles,
not topology based description - end vs. intermediate.

>    not expect that person to respond to end-user queries (in general).

Yes, but... For example you expect to be able to access content provider's
privacy policy and security credentials. Same is true for OPES processors.
Again, the amount of information and level of control may be different
but still some information should be provided.

>    Let's concentrate on in-band tracing for now.

There are certain things that should be provided as references, e.g. privacy
policy. Extending combined approach with some persistent information replaced
by references give much greater flexibility and makes application protocol
more efficient.

See detailed discussion in my previous message - posted exactly at the
same time as yours :). Well, yours is fifteen minutes earlier.

> 	f) Servers or services MAY add their own OPES trace records,

Yes, can not agree more! See my previous post.

> 6. #5 suggestion implies that tracing is out of OCP scope! :-)

I'm afraid - not. For example OPES processor should be able to get
information about callout server privacy policy and might want to
control insertion of tracing information into application messages.
This has to be supported by the OCP :-(

>    information is required and, perhaps, some common tracing elements.

We can delay specification details while we are describing high level
architecture. But the ultimate goal is an implementable specification,
and to support application interoperation we have to specify all essential
formats and protocols. This may span multiple RFCs/draft but the final
result should be the same - implementable specification with guaranteed
interoperability.

Oskar

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 2 Apr 2003 11:10:57 -0700 (MST)

On Wed, 2 Apr 2003, Oskar Batuner wrote:

> I'm afraid the situation is more complicated. While ability to
> control tracing facility and level of details available inside and
> outside of the trust domain may be different there are some
> requirements (e.g. privacy) that are not limited to the trust
> domain. Both IAB requirements and the architecture mandate some
> end-to-end disclosures.

Let's be more specific. I did not find any IAB considerations that
mandate automated end-to-end disclosures. There is some discussion
that content providers may benefit from knowing client-side OPES
actions, but the only consideration is that "The overall OPES
framework needs to assist content providers in detecting and
responding to client-centric actions by OPES intermediaries". This
consideration is too vague; it can be satisfied by, for example,
asking the user to manually supply OPES-specific headers to the
content provider if the provider suspects inappropriate OPES behavior
by observing HTTP Via headers or equivalent.

As IAB notes, implementing this assistance goes against implementors
objectives (client-side OPES system does not care about server-side
desires). Thus, it would be a waste of time in implementing any
sophisticated/detailed (hence expensive) notification facility that
spans trust domains. Moreover, requiring such facility on a MUST level
clearly violates client-side privacy considerations.

I agree that we should require (at a SHOULD level) some minimal form
of cross-trust-domain tracing similar to current HTTP Via headers, but
I would not go far beyond that.

> There are more participants interested in tracing information.
> Independent OPES processors in the same OPES flow might want to know
> about each other existence and might need to cooperate. For example
> virus checker may be satisfied by the signature of another virus
> checker, ad insertion/localization servers may avoid stepping on
> each other toes, etc.

Sure.

> In fact you are describing two roles. We should use this approach -
> roles, not topology based description - end vs. intermediate.

I have no problems with that. Both approaches may co-exist actually.

> Yes, but... For example you expect to be able to access content
> provider's privacy policy and security credentials. Same is true for
> OPES processors. Again, the amount of information and level of
> control may be different but still some information should be
> provided.

Provider != OPES intermediary.

The mechanism for end users to determine the privacy policy of a
server-centric intermediary is to visit the corresponding web page on
provider's web site. For end users, there can be no clear distinction
between server-side OPES intermediary and the "server-side" as a
whole. For starters, a particular server-side [OPES] intermediary may
not be directly addressable by the end-user. IAB RFC clearly states
that only the first intermediary in the chain must be addressable.
That first intermediary may be a L7 load balancer, for example. L7
load balancers do not have a privacy policy.

> There are certain things that should be provided as references, e.g.
> privacy policy. Extending combined approach with some persistent
> information replaced by references give much greater flexibility and
> makes application protocol more efficient.

Sure. I have no problems with allowing additional information to be
provided via references. We can discuss whether it would make sense to
require any specific references.

> information about callout server privacy policy

Why should it?

The architecture draft says "Protocol(s) for policy/rule distribution
are out of scope for this document, but the OPES architecture assumes
the existence of such a mechanism". You do not really expect OPES
processors to care about every callout server privacy policy, do you?
Content providers have privacy policies. Server-side callout servers
do not have to (because they are operating under the content provider
policy).

> and might want to control insertion of tracing information into
> application messages. This has to be supported by the OCP :-(

Depends on what tracing information we want to support. For example,
if we want OPES processor to insert service identifier into an
application message, the processor can do it without OCP-level
support.

> result should be the same - implementable specification with guaranteed

Yes, of course.

Thanks,

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Wed, 2 Apr 2003 22:11:44 -0500

See inline.

> Sent: Wednesday, April 02, 2003 1:11 PM
> To: OPES Group

Yes,  for cross-domain I was looking at Via-type notification. I think
final protocol binding should define vendor-independent format suitable
for automated analysis.

As for MUST/SHOULD - I was always trying to prevent mandatory disclosure
of information outside of trust domain boundaries. Sometimes disclosure
of filter information may create a security threats facilitating attacks
on knows vulnerabilities and attempts to bypass protection.

But for some optional services well defined notification and control
mechanism may help to adjust services to the requirements of participants
even beyond trust domain, so I would not consider such mechanism a complete
waist of time.

> load balancers do not have a privacy policy.

As it was stated in IAB RFC additional entities introduce additional risks.
Your approach is absolutely valid if server-centric OPES servers are
deployed as a part of data provider application and are completely under
his control. Situation becomes more complicated if OPES service is provided
by a third party - with the content provider consent. In this situation
OPES server may have a privacy policy different from the content
provider's one and it should be accessible separately. OPES framework should
provide appropriate mechanisms and rules of deployment. Sure there is no need
to expose a separate privacy policy for each server in a server farm, but
there should be a clear rule that describes how privacy policy is propagated
and that ensures that any OPES entity is covered by one of declared policies.

As for a L7 load balancer: if this balancer represents an OPES PEP (controls a
group of callout servers) it should obey all OPES rules, including privacy
notification.

> do not have to (because they are operating under the content provider

1. Yes, I do care about callout server privacy policy. The fact that a
specific server obeys a policy declared by the OPES processor completely
satisfies my concerns, but this fact should be stated and verified by
the OPES processor.

2. Privacy policy is just an example. Tracing information may cover
other functional aspects, for example content filtering abilities and
policies, types of ads injected, personalization effects, etc. OCP
should provide a generic mechanism that can be used to satisfy a wide
variety of application specific needs.

3. Absence of tracing and adaptation support from OCP makes it
incomplete - tracing directives go out-of-band and results are not
verifiable. This may defeat one of the main reasons for creating a
standard for such protocol - to permit creation of plug-and-play
(the word "appliance" is more in fashion today) application
servers that can interoperate with any caller understanding the
protocol.

4. Server-side (content provider side) may have a different meaning.
It may mean an integral part of content provider application - in
which case your model is correct - but it may also mean a third-party
service operating with consent of service provider, the case that
was causing most concerns in discussions about privacy and
information distortion.

5. If we adopt a model were OPES server represents all applications that
it controls - and looks like we are looking favorably at such model - this
OPES server should be able to control all relevant aspects of callout servers'
behavior, including tracing information insertion. OCP is very natural place
for this extended control mechanism. It already controls most aspects of
OPES server - callout server interaction, deploying additional/different
mechanism may result in multiple complications (out-of-band control of
per-message information).

Oskar

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 2 Apr 2003 22:32:49 -0700 (MST)

Oskar,

	It looks like our disagreements are not as deep as I
originally thought. I agree with most of your latest comments.

	Indeed, we should be careful about distinguishing a content
provider system and a 3rd party system operating under content
provider permission. I was pushing for treating any system as a single
entity once the message leaves that system. You are saying that there
are at least three kinds of systems: content providers, end users with
their proxies, and 3rd party services like CDNs. I agree, as long as
we can treat a 3rd party service as a single entity once the message
leaves that service.

	How about starting with the following simple set of
unpolished rules:

	1. An OPES processor SHOULD add its identification
	   to the trace.

	2. An OPES processor SHOULD add to the trace
	   identification of every callout service that received
	   the application message.

	3. An OPES processor MUST add to the trace identification
	   of the "system/entity" it belongs to. "System"
	   ID MUST make it possible to access "system" privacy
	   policy.

	4. An OPES processor MAY group the above information
	   (items 1-3) for sequential trace entries having
	   the same "system/entity" ID. In other words, trace
	   entries produced within the same "system/entity"
	   MAY be merged/aggregated into a single less detailed
	   trace entry.

	5. An OPES processor MAY delegate trace management to
	   a callout service within the same "system/entity".

We need to agree on what an "identifier" in items 1-3 represents and
how trace entries are encoded (the latter is application-specific).

Overall, do the above 5 items sound reasonable at all? Are there any
huge gaps they do not cover? (I think I know of one hole, but would
prefer to hear others' comments before diving any deeper)

Thank you,

Alex.

---

From: Weisong Shi <[email-masked]>
Date: Thu, 03 Apr 2003 08:10:55 -0500

Given theses traces, is it possible for clients or content providers to 
verify the correctness of them? For example, if an OPES processor 
reduces the resolution of an image, but he record something else. How 
can you verify this?

-Weisong

---

From: Weisong Shi <[email-masked]>
Date: Thu, 03 Apr 2003 08:16:04 -0500

Hello,
	Does anyone know the number of squid web caches deployed on the 
Internet? and how about its percentage among all Web caches? Thanks.

Weisong Shi

Department of Computer Science
Wayne State University

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 3 Apr 2003 06:46:26 -0700 (MST)

On Thu, 3 Apr 2003, Weisong Shi wrote:

> Given theses traces, is it possible for clients or content providers
> to verify the correctness of them? For example, if an OPES processor
> reduces the resolution of an image, but he record something else.
> How can you verify this?

You cannot automatically detect all inconsistencies, of course. It is
impossible to guarantee that a program does not lie or to auto-detect
every possible lie. What we can and should do is:

	- assist the user or content provider in contacting
	  administrators of broken OPES systems once the
	  user or content provider suspect a problem
	  (i.e., OPES trace can help in locating the
	  broken intermediary)

	- support bypass of OPES intermediaries when
	  possible (i.e., you may be able to get a
	  raw, unprocessed image if you suspect that
	  processing is corrupted _and_ if the system
	  does not require processing)

	- optionally, we can ask for digital signatures
	  for trace entries so that you have better
	  protection from the trace itself lying to you
	  (but most intermediaries will not pay $$$ for
	   key certificates so this will work just for
	   a few big companies handling most(?) of the
	   traffic)

Please note that OPES does not introduce new problem categories (what
you describe can happen today if, say, a CDN gets hacked), but it may
help solving some of the existing ones, at least partially.

Alex.

---

From: jfcm <[email-masked]>
Date: Thu, 03 Apr 2003 17:43:07 +0200

>            ID MUST make it possible to access "system" privacy

I disagree with the MUST.

- first there are system which does not want to be known.
- most of all this would block innovation. Information services could be 
actually built as a core system and a daisy chain of OPES. Information on 
the system structure is propreitary. There is no "system" privacy policy by 
application components.

jfc

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 3 Apr 2003 10:59:09 -0700 (MST)

> - first there are system which does not want to be known.

Yes, but it looks like IAB prohibits us from "supporting" an OPES
system which does not want to be known:

(5.1) Privacy: The overall OPES framework must provide for mechanisms
   for end users to determine the privacy policies of OPES
   intermediaries.

Thus, an OPES intermediary that does not want to be known cannot be
OPES compliant, but will, of course, interoperate just fine. Also, we
still need to decide what that ID is. We may be able to support IDs
that do not reveal much of private info.

Furthermore, since IAB does not seem to care about content provider
privacy, we may want to polish the above rule 3 so that it applies to
providers only (and is only a MAY for client-side intermediaries).

> - most of all this would block innovation. Information services
> could be actually built as a core system and a daisy chain of OPES.
> Information on the system structure is propreitary. There is no
> "system" privacy policy by application components.

This is not a problem due to the rule 4 that allows aggregation of
tracing information to hide system structure. We just need one trace
entry from each system, we do not care about individual OPES
intermediaries inside that system:

This is very similar to Via headers in HTTP that face similar
challenges.

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Thu, 3 Apr 2003 16:16:42 -0500

> Sent: Thursday, April 03, 2003 12:33 AM
> To: OPES Group

The OPES processor is the only entity that is always present, 
so if there is a MUST - it is here. On another hand if OPES 
processor belongs to the content producer trust domain (and in fact 
is an integral part of the origin) - why should it be exposed? 

I'd say MUST unless it belongs to the content producer trust domain 
and is covered by the appropriate privacy policies and other 
regulations (TBD). 

> 	   identification of every callout service that received

Unless those callout processors belong to the same trust domain 
and covered by the same policies. 

> 	   ID MUST make it possible to access "system" privacy

"system/entity" is not well defined, it is absent in the 
architecture. I'd put more stress on OPES processor as a 
traceable entity. We may introduce "system" (TBD), but it is 
a MAY. The second sentence belongs to paragraph 1.

> 	   MAY be merged/aggregated into a single less detailed

Agreed.

> 	   a callout service within the same "system/entity".

I'm not sure what do you mean by delegate here. If this means 
that tracing information is inserted by callout processors 
according to the OPES processor instructions and under its 
supervision - OK, but in this case the exact point of trace 
info insertion is not essential. 

If you mean delegation of authority - this does not look right. 

I'd set two types of requirements. First one is requirement to 
identify the OPES flow participant. It will be covered by rules 
1-3 (or whatever they will be transformed to). Second set of 
requirements should state what kind of tracing information 
MUST/SHOULD/MAY be inserted.

> prefer to hear others' comments before diving any deeper)

In general your practical approach to building a tracing framework 
is very reasonable. Items may look a little different.

Oskar

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 3 Apr 2003 15:14:28 -0700 (MST)

On Thu, 3 Apr 2003, Oskar Batuner wrote:

> The OPES processor is the only entity that is always present, so if
> there is a MUST - it is here.

I used a SHOULD because rule 3 is a MUST. The idea is that OPES
processor must/should/may include several pieces of tracing
information. The processor ID is a SHOULD because that information is
not required by IAB and is not very useful for agents outside of the
"system".

Rule 3 is a MUST because IAB requires that privacy policy is known.

In other words, knowing individual OPES processors benefits only the
"system" admins and is not really important to others. It is a SHOULD.
Knowing privacy policy and "system" in charge is important to the
other "end" and is a MUST, the only MUST.

> On another hand if OPES processor belongs to the content producer
> trust domain (and in fact is an integral part of the origin) - why
> should it be exposed?
>

I agree with the intent -- outside agents only care about the whole
"system" identification and policies, not its individual components.
On the other hand, "system" admins may need more detailed trace. My
solution to the problem was rule 4 which is designed to give "system"
admins full control over how the system looks to the "outside" world.

In other words, what is added by individual OPES processors can be
aggregated (i.e., partially removed) by the "system" as a whole.

> and covered by the same policies.

See above. I wanted to avoid repeating "unless, ..." for every rule.
That is why I added rule 4: What is added can be removed by the same
or other agents within the same "system".

> > 	   of the "system/entity" it belongs to.
>
> architecture. I'd put more stress on OPES processor as a traceable
> entity. We may introduce "system" (TBD), but it is a MAY.

Yes, the "system/entity" needs to be defined. If we manage to define
it, I think it will keep the rules much simpler. Here is my reasoning:
The architecture draft talks about OPES processors and trust domains.
OPES processor is too low-level entity in this context because you
have to add "... unless the processor belongs to the same ..."
exceptions to every rule.

The "trust domain" is too high-level entity in this context because a
single trust domain may include several systems with different privacy
policies and we MUST identify every applicable policy in the trace. As
you have pointed out previously, there are more than two players here;
there can be only two trust domains. Trust domains are based on
delegation of authority. We need something based on the authority
itself and its privacy policy.

Actually, the architecture draft does use "entity" (i.e., "system")
term without defining it! See section 3.1, "Trust Domains":

   The delegation of authority starts at either a data consumer or data
   provider and moves to more distant entities in a "stepwise" fashion.
   Stepwise means A delegates to B and B delegates to C and so forth.
   The entities thus "colored" by the delegation are said to form a
   trust domain

The entity in the above wording is probably my "system".

How about defining an "OPES system" in this context as a "logical
group of OPES entities operating under the same authority and using
the same privacy policy"?

Will such a definition work for our purposes? Any better ideas?

> >        "System"
> The second sentence belongs to paragraph 1.

Or, better, a separate rule.

> info insertion is not essential.

Yes, that is what I meant. Yes, it is not essential in this context. I
was just trying to address concerns that sometimes it is better for a
callout service (and not the OPES processor) to add tracing
information.

> If you mean delegation of authority - this does not look right.

No, I do not.

> MUST/SHOULD/MAY be inserted.

Sounds good to me.
	- Identify traceable entities (definitions, not requirements)
	- Document trace entries (definitions, not requirements)
	- Document who MUST/SHOULD/MAY insert what (requirements)

Most of already proposed rules fall into the last category. I think we
almost agree regarding what is traceable (OPES processor, applied
services, and system authority and/or policy). We need to define
"system" (or rewrite the rules without it) and agree what the trace
entries should look like.

Alex.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Sun, 6 Apr 2003 18:19:15 -0600

Stepping back for a moment, it seems to me that you need to identify
four essential things relevant to a "traceable entity" (Oskar called
these "reference points", I believe):

	1. The privacy policy at the time it dealt with the message.
	2. Identification of the party responsible for setting and
	   enforcing that policy.
	3. Information pointing to a technical contact
	4. Information that identifies, to the technical contact, the
	   OPES processors involved in processing the message

Item 4 can be opaque, i.e., only has meaning to the technical contact.

I don't know if we want to standardize the representation of any of this,
other than the tracing header names.

From a protocol standpoint, it is certainly easiest to just say that
every OPES processor is a traceable entity and that callout servers
MAY be traceable entities.

There has not been any discussion of what triggers tracing.  I suppose
it must originate from an endpoint (consumer or provider).  Is it "per
request" or "per connection" or "per something-else" (application-specific)?
Or required all the time?

There's also the question of "who's asking?"  Items 1-3 might be
considered sensitive information, relevant to competitors, or even
security and privacy relevant.  So a consumer might intend a trace
request to mean "tell me what is applied by my network service
provider and entities he contracts with, and anything else you can
find out from the provider, but don't tell the content provider
anything."  And a caching OPES processor might send requests to
content providers with a tracing request that meant "one or more of my
consumers has requested tracing" and would cache the response with
tracing headers.

So, maybe tracing responses should be optional, and there should be a
header that means "tracing request received and denied."  Of course, that's
weird - who received it, who denied it?  Some unidentified OPES processor
somewhere?

I seem to have run into a conundrum here --- should we define tracing
for all systems that do not trace themselves? :-)

Hilarie

---

From: Alex Rousskov <[email-masked]>
Date: Sun, 6 Apr 2003 22:08:39 -0600 (MDT)

> 	1. The privacy policy at the time it dealt with the message.

Maybe, though #2 is probably sufficient. If we include #1, we would
have to define what a "privacy policy" is, which is not going to be
fun. Moreover, we would have to define what an _acceptable_ (OPES
compliant?!)  privacy policy is.  See my earlier e-mail about that.
For starters, are "we have no privacy policy" and "our privacy policy
is only available through a court order or for $1,000" acceptable
privacy policies?

As long as the "responsible party" is identified, OPES responsibility
ends. If a user does not like (or cannot find!) that "party" policies,
the user can bypass.

> 	2. Identification of the party responsible for setting and

Yes.

> 	3. Information pointing to a technical contact

Should be optional. Info in #2 is the only "point of contact" that
outsiders should be able to see.

> Item 4 can be opaque, i.e., only has meaning to the technical contact.

Yes, but also optional, of course.

> I don't know if we want to standardize the representation of any of
> this, other than the tracing header names.

The only reason to standardize representation is to expect some
automated processing of this information. I would not spend too much
time on this. I would suggest something very basic like:
	<URI|""> [comment]

for every identifier (required URI or empty string, followed by an
optional free-flow comment, encoded appropriately depending on the
application protocol). This should be good enough for log analyses and
similar automated tools but flexible enough to accommodate virtually all
human needs.

> There has not been any discussion of what triggers tracing.  I
> suppose it must originate from an endpoint (consumer or provider).
> Is it "per request" or "per connection" or "per something-else"
> (application-specific)? Or required all the time?

I would start by requiring it at all times because:

	- you cannot, in general, re-trace an old message;
	  network paths and intermediary decisions are
	  not constants; if you do not get a trace
	  with the message, you cannot be sure the bug
	  you saw was caused by the same intermediaries
	  you are now tracing one second later

	- some messages are impossible, dangerous, and/or
	  costly to repeat

The only objection to always-on tracing is the performance and
bandwidth overhead. Our design limits the amount of information to the
necessary minimum, so we should be safe given we are adapting
application protocols and not transport-level things that have to
count every bit.

> content providers with a tracing request that meant "one or more of
> my consumers has requested tracing" and would cache the response
> with tracing headers.

I do not think traces reveal anything sensitive other than identities
of "systems" that touched our message. Moreover, the latter can be
obscured if really needed! I would start with a simple always-on
scheme and make it more complex only if there are very specific
reasons for that added complexity.

Thanks,

Alex.