Subject: Re: An opes usage question.

From: "John G. Waclawsky" <[email-masked]>
Date: Wed, 03 Mar 2004 13:08:07 -0500

Alex, thanks for the discussion. I'll take your suggestion to provide a 
more detailed example that we can discuss. I am considering  the case of 
a large number of users accessing content and also needing adaptations 
for the proper delivery of the content. Also the adaptations for the 
services may be done in stages. The environment (summarized from our 
previous e-mails, ...but please let me know if I have a 
misunderstanding) that I envision is:

1)  The opes framework allows services to be distributed (or pipelined), 
with incremental services being added to each traffic flow at each 
stage. This is an opes proxy to proxy communication model.
2) A pool of multiple opes proxies can be provisioned at each stage to 
support a large number of flows
3) Installing load balancers between stages to distribute the flows is 
ok in an opes framework (and this is a typical business scenario). If 
all the flow processing can be achieved in-line then there is no need to 
identify any specific proxy in any pool. In this case we probably don't 
care which previous stage opes proxy did the prior adaptation step. 
4) The crux of the problem is how to share information between two 
stages of the flow. This sending of metadata from one stage to a 
previous stage will require knowledge of specific server addresses (a 
more general case might be be to send the metadata in  either direction).

Let's assume, for a specific example, a network allows access to premium 
content that is billable by the number of bytes being delivered. The 
content flows to the users from a premium content server through a pool 
of billing gateway proxies and then proceeds to the next stage where the 
content may be modified by other proxies for rendering on a user's 
device (color to black/white, compressed for efficiency, adding 
advertisements or banners etc.). If any particular flow adaptation 
reduces the number of bytes then billing needs to be adjusted.  Now two 
specific proxies in each pool need to exchange metadata to coordinate 
information about a specific flow to get the bill correct for a 
particular flow. The location of a load balancer between the two sets of 
proxy pools hides the association for individual flows. I "think" this 
may be a different problem than the general peer discovery problem you 
brought up. In this case peer discovery refers to "flow participant 
discovery" in the previous stage and does not refer to locating useful 
servers. The need for discovering peer participants for a particular 
flow arises if any specific metadata exchange has to be made between 
participants, like the billing record consolidation example.

I think "flow participant discovery" problem I described is an opes 
specific problem. So I am back to my initial hypothesis that the opes 
framework needs a "flow participant discovery" protocol between proxies 
that isn't disturbed/effected by load balancing or uses load balancing 
to some advantage (assumes it is there and leverages it somehow). I have 
some thoughts on that, but first I like to make sure the usage situation 
is clear. Your thoughts are greatly appreciated. Thanks again.
Regards  John

Alex Rousskov wrote:

>On Fri, 27 Feb 2004, John G. Waclawsky wrote:
>
>  
>
>>Alex, you had mentioned in a previous e-mail that the "framework is
>>not complete. It does not have ready-to-use tools for all possible
>>intermediary adaptations". Would this include a load balancing
>>environment?
>>    
>>
>
>Possibly, but it is too early to say. I do not know if any
>opes-specific tools will be needed in a load balancing environment.
>For example, as far as I can see, to load balance callout server
>adaptations, no new tools are needed and OCP can be used as is.
>
>  
>
>>Specifically, load balancing is not the problem that I currently
>>see. I believe I can make the load balancers work with what I
>>anticipate doing within an opes framework.
>>    
>>
>
>Looks like we are in agreement here.
>
>  
>
>>But, I am looking for a general opes solution that allows peer
>>discovery so I can perform adaptations (pipelined over load
>>balancers, if you will) on a number of flows, web pages, file
>>downloads,...etc.
>>    
>>
>
>By "peer discovery", do you mean discovering callout servers that
>provide OPES services that the application proxy needs? If yes, then
>this discovery is out of OCP and possibly even OPES scope. It can be
>implemented using existing or new protocols on top of OPES protocols.
>
>The closest thing to peer discovery supported in OCP is querying an
>OCP peer for a list of supported services or negotiating service
>support. Both can only be done if we have established a connection to
>the peer, which is what peer discovery should help us with.
>
>  
>
>>I consider load balancing as an essential part of a "typical" high
>>capacity services situation. My thoughts are that peer discovery is
>>necessary as an option for a load balanced opes environment.
>>    
>>
>
>I see no direct relationship between load balancing and peer
>discovery. I define load balancing as spreading the load among a known
>set of servers. I define peer discovery as locating potentially useful
>servers. The two can be combined (discover what servers we can use for
>load balancing), but are independent.
>
>  
>
>>Are you saying in your reply that OCP can also be used for peer
>>discovery?
>>    
>>
>
>No.
>
>  
>
>>If not, shouldn't the opes framework be extended to satisfy a peer
>>discovery use case over a load balancer?
>>    
>>
>
>Only if there is something really OPES-specific in this peer discovery
>problem. As you know, peer discovery in general is an old problem with
>a few semi-working solutions and no good ones. What OPES-specific peer
>discovery features would you like the framework extension to support?
>And, again, I am not sure how you connect peer discovery and load
>balancing.
>
>  
>
>>I was thinking another protocol between peers might be needed (or we
>>can use an existing protocol).
>>    
>>
>
>Hopefully, that another protocol already exists and can be used as-is.
>Can you give a very specific example: who needs to discover what and
>how is that related to load balancing?
>
>Thanks,
>
>Alex.
>
>
>
>  
>
>>Alex Rousskov wrote:
>>
>>    
>>
>>>John,
>>>
>>>	OPES protocols do not have features specific to a
>>>load-balancing environment, and I do not think they have any features
>>>that do not work in such an environment. If your load balancer can
>>>balance opaque TCP sessions, then OPES callout protocol should work
>>>just fine with multiple callout servers. If you are proxying native
>>>HTTP without outsourcing services via OCP, then load balancers can
>>>balance HTTP "sessions" as well.
>>>
>>>	OCP is easier to load balance than HTTP because it does not
>>>have valuable state outside of the OCP connection. While HTTP claims
>>>to be stateless, things built on top of HTTP often require
>>>client/server affinity. Hopefully, OCP will not be abused in such a
>>>way because it has mechanisms to maintain necessary state within the
>>>connection.
>>>
>>>HTH,
>>>
>>>Alex.
>>>
>>>On Fri, 27 Feb 2004, John G. Waclawsky wrote:
>>>
>>>
>>>
>>>      
>>>
>>>>I have another question. Does an opes environment handle the
>>>>requirement of a peer interacting with another peer with a load
>>>>balancer between them. The situation is that I require two proxies
>>>>to provide a service.  An "A" proxy and a "B" proxy that are peers
>>>>in providing the service to a data flow. Because of the scale, the
>>>>likely deployment will involve multiple boxes of types A and B. This
>>>>means I will have two groups of boxes with a load balancer between
>>>>them. I wish to dynamically associate an specific box of type A with
>>>>a specific Box of type B (across the load balancer) and a particular
>>>>flow. Is it possible to perform peer discovery and the dynamic
>>>>association between the two peers and a flow within the opes
>>>>framework? Thanks for any help and guidance.
>>>>
>>>>Regards  John
>>>>
>>>>John G. Waclawsky wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Alex, I very much appreciate your help and the additional
>>>>>information. I need a little time to digest this and think some more
>>>>>about my problem and opes "needs"....
>>>>>Thanks.   Regards  John
>>>>>
>>>>>Alex Rousskov wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On Thu, 8 Jan 2004, John G. Waclawsky wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Alex, thank you for your reply and the pointer. I agree with you
>>>>>>>that the use case I am interested in is not explicitly documented in
>>>>>>>the opes material. This gives the wrong impression, since all the
>>>>>>>examples show the data flows returning to the first hop data
>>>>>>>dispatcher. To restrict data flow in this way would be inefficient
>>>>>>>for high capacity applications and would tend to make the first hop
>>>>>>>a bottleneck.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>John,
>>>>>>
>>>>>>	You are misreading OPES architecture diagrams because you are
>>>>>>thinking about a much "simpler" case than most diagrams attempt to
>>>>>>document and, hence, you assign wrong roles to "boxes" that may not
>>>>>>even exist in your environment. This is the architecture draft fault,
>>>>>>not yours. Let me try to explain.
>>>>>>
>>>>>>Here is a "classic" OPES proxying case (horizontal lines are
>>>>>>application data/protocols being proxied, proxies may be adapting data
>>>>>>internally):
>>>>>>
>>>>>>  Figure A.
>>>>>>
>>>>>>      end -- proxy -- proxy -- ... -- end
>>>>>>
>>>>>>Here is a "classic" OPES callout case (vertical lines are callout
>>>>>>data/protocols, proxy is not adapting data, callout servers may be
>>>>>>adapting data):
>>>>>>
>>>>>>  Figure B.
>>>>>>
>>>>>>      end --- proxy --- end
>>>>>>       _________|__________
>>>>>>       |        |         |
>>>>>>       |        |         |
>>>>>>    callout   callout   callout
>>>>>>    server    server    server
>>>>>>
>>>>>>A combination of the above is possible and common, of course. In a
>>>>>>combined case, some proxies are adapting data internally and some use
>>>>>>callout servers to adapt.
>>>>>>
>>>>>>In your particular case (the adapted data flows to the next hop), you
>>>>>>do not have callout servers. You have proxies that adapt data
>>>>>>internally. However, your case may not be a classic proxying case
>>>>>>because you may want proxies to use a protocol that differs from the
>>>>>>original application protocol (so that you can ship metadata and
>>>>>>perhaps pipeline more efficiently). I will use curly (~) lines to show
>>>>>>that new protocol below. You may also want to do some load balancing:
>>>>>>
>>>>>>  Figure C.
>>>>>>
>>>>>>                  ~~~~~ proxy1 ---
>>>>>>                 /
>>>>>>    end -- proxy ~~~~~~ proxy2 ---  ??  end
>>>>>>                 \
>>>>>>                  ~~~~~ proxyN ---
>>>>>>
>>>>>>
>>>>>>I do not know how the proxies will get application data to the right
>>>>>>"end", but it is not important for this discussion.
>>>>>>
>>>>>>Note that the curly path may use a completely new protocol (e.g., OCP)
>>>>>>or can use a combination of the original application protocol to
>>>>>>deliver data and some other protocol for metadata (billing, etc.)
>>>>>>records.
>>>>>>
>>>>>>The above is OPES, but not a callout case. There are no callout
>>>>>>servers, just proxies. Whether you want to use OCP for the curly path
>>>>>>depends on what kind of data/metadata you want to exchange and what
>>>>>>other protocols you have available.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>To be a little more specific on my use case and to make sure I
>>>>>>>understand your response let me add more detail. Fundamentally, I wish
>>>>>>>to have an "open" IETF network environment. The second requirement is
>>>>>>>speed in adaptation processing, so I am thinking that callout will be
>>>>>>>faster than using a proxy (maybe this is an incorrect assumption).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>It's an incorrect question :-). Adaptations at the callout server can
>>>>>>be as "fast" or as "slow" as adaptations performed internally at the
>>>>>>proxy. The primary reason to use a callout server is to "outsource"
>>>>>>(from business, development, support, logistics, legal, etc. points of
>>>>>>view) adaptations. The primary reasons not to use a callout server are
>>>>>>security and overheads of the data having to return back to the proxy.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>I may be a little confused about the distinction between classic
>>>>>>>proxy and classic call out in this situation (this word proxy isn't
>>>>>>>even mentioned in the opes architecture document as a
>>>>>>>consideration).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I hope the above diagrams clarify the distinction. OPES processor is
>>>>>>the term closest to the "proxy", I think.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Also, I have a requirement to adapt a large volume of content for
>>>>>>>numerous devices.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>Both callout and builtin adaptations can handle large volumes of
>>>>>>content. There are many factors at play here, from legal concerns to
>>>>>>the kind of adaptation being performed.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>I expect that the volume of data will require a number of adaptation
>>>>>>>(call out) processors. I estimate 10 of them and therefore I would
>>>>>>>need load balancing as part of the solution for directing data to
>>>>>>>the call out servers (doing the adaptation).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>Load balancing can be done with both callout and proxying schemes.
>>>>>>OPES mechanisms are usually per-message so any load balancing method
>>>>>>that does not split individual application messages should work just
>>>>>>fine.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>I am looking at Figure 3 in the opes architecture draft and
>>>>>>>considering a load balanced collection of ten call out servers. As I
>>>>>>>had mentioned previously, I'd like to have parallel pipeline
>>>>>>>approach where the adapted data from each of the opes call out
>>>>>>>server is simply forwarded on, directly to the producer or data
>>>>>>>consumer (what ever the case would be).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>The latter makes the adapter a proxy, not a callout server (by
>>>>>>definition). See Figure C above.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>In addition each of the opes call out servers would then send
>>>>>>>billing and trace data back to the load balancer (or another network
>>>>>>>location). Is it realistic to expect that this could this be
>>>>>>>accomplished within the opes framework?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>It is. The only question is what protocol(s) you will use between your
>>>>>>proxies (the curly lines in Figure C).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>In fact a more fundamental question is the opes framework the best
>>>>>>>way to solve this problem and maintain an open system. I think this
>>>>>>>is what opes is all about.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>IMO, OPES framework accommodates, in principle, any intermediary
>>>>>>adaptation. Thus, your problem is within the framework. However, the
>>>>>>framework is not complete. It does not have ready-to-use tools for all
>>>>>>possible intermediary adaptations. I hope that you will find OCP Core
>>>>>>and OPES communications "tools" useful for your problems, but I lack
>>>>>>information to tell your whether they are the best.
>>>>>>
>>>>>>The system is "open" if it uses "open standards". If current OPES
>>>>>>tools are not right for you, and there are no better open tools, then
>>>>>>we can work together, within the OPES Framework, on defining open
>>>>>>standards (protocols and interfaces) that will solve your specific
>>>>>>problem.
>>>>>>
>>>>>>Alex.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>      
>>>
>
>
>  
>

---

From: Markus Hofmann <[email-masked]>
Date: Fri, 05 Mar 2004 23:23:09 -0500

John G. Waclawsky wrote:

> reduces the number of bytes then billing needs to be adjusted.

My first question about this specific example would be why the billing 
is done before the adaptation. If the user is billed for the adapted 
content, do the billing after the adaptation and there's no need for 
the kind of "callout server interaction" you're talking about, right?

Anyway, I believe the problem you're interested in boils down to 
tracing callout servers that did some earlier operations on the 
message. In your hypothetic example above, the adaptation server would 
have to be able to identify (the IP address?) of the billing server, 
so that it can contact the billing server, right? Could the OPES 
tracing functionality be of any help?

-Markus

---

From: "John G. Waclawsky" <[email-masked]>
Date: Sat, 06 Mar 2004 13:15:26 -0500

Markus, thanks for your comments. Depending on what is being done 
billing could be done after adaptation but this doesn't solve the 
general case and always makes billing the last stage in the opes proxy 
pipeline. I think this would overly restrict opes configurations and 
probably cause issues elsewhere? Also, this restriction might not be 
practical especially if billing is done in the network where the content 
is located and adaptation is done in another network. In fact, the 
content may not even be delivered  because of policy in the access 
network, e.g. parental controls is one situation that comes to mind.  In 
my particular situation the billing server address for each particular 
flow must be in the opes traces to be used by any downstream proxy and 
if there are multiple proxy addresses present which one would be the 
billing proxy?  I am not sure the tracing functionality will help but I 
will investigate.
Thanks for the discussion
Regards  John

Markus Hofmann wrote:

>
> John G. Waclawsky wrote:
>
>> Let's assume, for a specific example, a network allows access to 
>> premium content that is billable by the number of bytes being 
>> delivered. The content flows to the users from a premium content 
>> server through a pool of billing gateway proxies and then proceeds to 
>> the next stage where the content may be modified by other proxies for 
>> rendering on a user's device (color to black/white, compressed for 
>> efficiency, adding advertisements or banners etc.). If any particular 
>> flow adaptation reduces the number of bytes then billing needs to be 
>> adjusted.
>
>

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Sat, 6 Mar 2004 12:13:30 -0700

If the billing is based on actual bytes delivered to the consumer,
there must be a business relationship between the billing stage
servers and adaptation servers.  The relationship will usually allow
the adaptation servers to cache the content and report to the billing
servers about the number of bytes delivered.  However, the reporting
is a reverse flow in the client-server model, and that is why it seems
problematical.  By convention, the adaptation server might make a
request back to the provider with the number of bytes encoded in the
request, or it might, on occasion, upload a report.  The rules
governing this behavior would most naturally be encoded on the
adaptation server, not on the OPES callout server.  So, this seems to
me to be an issue for the rules language.

An alternative model would have this byte reporting functionality
handled on the callout server, and it would directly contact
the provider's billing service with reports.

Hilarie

---

From: "John G. Waclawsky" <[email-masked]>
Date: Sat, 06 Mar 2004 17:11:45 -0500

Thanks for your comments Hilarie. My comments are in-line below.

>servers and adaptation servers.  The relationship will usually allow
I think we are talking about the same thing which I have been calling a 
"flow participant discovery" problem (not a services discovery problem). 
The question is how, within the opes framework, do I establish this 
business relationship between two proxies in different stages of a 
services pipeline separated by a load balancer. Packets can travel 
through the billing server and later be changed by any adaptation proxy 
where (in the billing by bytes case) bytes could be reduced 
(compression) or increased (adding banners or advertisements or other 
correlated content). Because of previous e-mails I have been considering 
both the billing server and any adaptation servers as opes proxies. ...I 
believe this is the simplest opes environment.

>problematical.  By convention, the adaptation server might make a
I agree the reverse flow is a problem.

>me to be an issue for the rules language.
I am not following your logic here. I am working with a simple proxy to 
proxy opes framework and not doing any callouts. Am I missing something? 
Please clarify.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Sun, 7 Mar 2004 12:35:10 -0700

I think I see your problem, but I don't think you've established a
need for communication between a specific billing server and a
specific adaptation server.  If this sort of thing is the model:

  ContentServer     ContentServer     ContentServer    
        |               |                |
        \               |               /
         \              |              /
          ----------------------------
          |      Load Balancer       |
          ----------------------------
            |           |         |
            |           |         |
            |           |         |
  BillingServer  BillingServer   BillingServer
        |               |                |
        \               |               /
         \              |              /
          ----------------------------
          |      Load Balancer       |
          ----------------------------
            |           |         |
            |           |         |
            |           |         |
       AdaptServer  AdaptServer  AdaptServer
        |               |                |
        \               |               /
         \              |              /
          ----------------------------
          |      Load Balancer       |
          ----------------------------
                      .....

Then I'd think that a Billing Server, upon seeing a request and
response, might open a record saying "customer A got 10K bytes at 10
am", but the Adaptation Server would send back a "request" encoding
the information "the customer really got 8K bytes at 10am".  Your
question is "how would that information get back to the same billing
server?"  and my first reaction is to say it just naturally would
if they keep their connection open.  

If they decide to close the connection, then they need to agree on
some kind of identifier for the transaction.  If the Billing Server
wants to include an OPES header with a unique ID, it can do that, and
it can write a billing record into its database saying "8K bytes for
unique ID xyzzy".  The Adaptation Server then sends back the
information, using the same ID.  This might not get delivered to the
same Billing Server, but it doesn't matter, it all goes into one
database and can be correlated for billing purposes.

In any case, the Adaptation Server needs to do extra communication,
above and beyond proxying.  That extra work should be encodable in
the Rules Language, I believe.  It is what I'd call a "local callout
service".

I agree that in general the problem of identifying entities in a
pipeline that is interesting.  Load-balancing, hot backup, failover,
etc. all introduce some interesting problems.  If there are provably
correct solutions that can be encoded with OPES (without complicating
it), that would be very interesting.  Perhaps out of charter for OPES,
but possibly worth a new WG (unless there already is one!).

Inimitably, on Sat, 06 Mar 2004 at 17:11:45 -0500 John G. Waclawsky
murmured:
>  correlated content). 

Hilarie

---

From: Geetha Manjunath <[email-masked]>
Date: Mon, 08 Mar 2004 10:17:32 +0530

On the contrary, I feel the accounting model should be slightly
different in the OPES case:
(a) We should charge the consumer for bytes output from the content
server (we can debate on whether the #bytes is pre-adaptation or not)
PLUS
(b)A pay-per-use model for specific adaptation services used - this
information can be provided  to the billing server by the adaptation
server.

Comments?

regards
Geetha

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 08 Mar 2004 10:46:24 -0500

Thanks for your input Geetha. I have found that billing can be a 
difficult problem because people expect solutions to fit into their 
existing billing model/system, no matter how it might work (it might be 
good for some things and a problem for others). So a solution that 
accommodates a number of  billing approaches would be the best because 
it is flexible and will be able to handle in-line billing, pre-paid, 
pay-per-use...etc.

Also, I'd like to build an open infrastructure to solve my problem. The 
feedback that I have gotten from past opes e-mails leads me to believe 
that using the opes framework would be the best way accomplish this 
important  goal. But is seems the opes framework needs a solution to the 
flow participant discovery problem. I emphasis the word discovery. My 
thoughts are that once you know the other flow participant you can 
always establish out of band communications to do whatever you need. 
This seems to be the most flexible approach. Your thoughts?
Regards  John

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 08 Mar 2004 10:43:54 -0500

My comments are in-line below:

>specific adaptation server.  If this sort of thing is the model:
But I believe a solution is needed that allows communication between a 
specific billing server and a specific adaptation server. I think or 
thought that was clear from the past e-mails.

>  ContentServer     ContentServer     ContentServer    
Yes, that is the model.

>if they keep their connection open.  
I am not sure I understand why this would be true if there is a load 
balancer between stages of the services pipeline?  Also connections can 
be very short lived.

>database and can be correlated for billing purposes.
You are proposing billing system changes. This introduces an additional 
dependency and complicates deployment and can lead to other problems. I 
would prefer not to have to modify the billing system.

>the Rules Language, I believe.  It is what I'd call a "local callout
I believe this extra communication can be done out of band  Is the 
reconciliation of billing information what you are referring to as a 
local call out service?

>but possibly worth a new WG (unless there already is one!).
As I mentioned before, I believe that any extra communication (for 
billing reconciliation) can be done out of band and therefore may not 
have to be part of opes, but I believe finding a  flow participant is an 
opes issue that needs resolution.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 8 Mar 2004 23:08:01 -0700 (MST)

I am afraid the latest discussion/diagrams unnecessary complicated
things. We suddenly started talking about IP addresses of individual
proxies, persistent connections between load balanced proxies, and
other complicated low-level details that should be kept outside of
most OPES protocols. Let's step back a little:

> care which previous stage opes proxy did the prior adaptation step.

Agreed.

> more general case might be be to send the metadata in  either direction).

I hope the last statement is false. IMO, identifying or communicating
directly with individual proxies behind a load balancer makes sane
load balancing impossible. Sane load balancing, by definition,
includes load balancing of identical proxies (identical from external
protocol agents point of view). If all proxies are truly identical
from external agents ping of view, there should be not reason to
identify them individually.

If proxies are not identical for any reason, then we are not load
balancing them; we are managing a pool of different proxies, with some
complex per-protocol selection criteria. The latter model is what
origin server load balancing evolved into and is exactly why HTTP load
balancing requires AI techniques and ugly hacks to work, despite the
fact that pure HTTP is stateless. We should avoid this model (on a
protocol level) if at all possible.

In your specific case, this implies that external proxies must not try
to identify individual proxies behind the load balancer. While we can
build such identification mechanisms, the long-term effect would be
the same as for HTTP: expensive and relatively rigid load balancing
schemes causing headaches for all the parties involved.

The task of such identification should be assigned to load balancers.
If the protocol is designed correctly, a load balancer should be able
to reliably identify the proxy/server it should talk to when the
external proxy sends a follow-up message to the load balancer. We
tried hard to make this possible with the OPES tracing approach. It
should be possible in reverse direction as well.

Instead of using this diagram:

let's use the following diagram when designing your billing adjustment
algorithm/protocol:

                  ContentServer
                        |
                 BillingServer
                        |
                   AdaptServer

and require that if any load balancing is introduced, it does not
change the algorithm/protocol in any way. This implies that if IP
addresses are used to identify proxies, then load balancers should put
their own IP addresses instead of the addresses of the proxies being
balanced (and embed known-to-balancer-only meta information to map
flow ID to individual proxy address). OPES tracing allows for such
substitutions and meta information, for example. The notification
algorithms working in the opposite direction should allow them as well
and can reuse the techniques discussed when OPES tracing was
developed.

In other words, instead of allowing a load balancer as a separate protocol
entity that everybody has to worry about, you require that protocols are
designed so that a group of load balanced agents is visible as a single agent,
and nobody has to worry about the presence or specifics of load balancing
(except for the load balancer itself).

Can you think of any real world problem that cannot be solved using the above
simplified framework? Can you think of a reason why load balancers will not be
able to hide the presence of multiple proxies from the outside agents? In
other words, is there a point in drawing load balancers and multiple proxies
when discussing the protocols you need?

Thanks,

Alex.

---

From: "John G. Waclawsky" <[email-masked]>
Date: Tue, 09 Mar 2004 23:42:06 -0500

I agree things have gotten a little too complicated and I would also 
like to get the discussion back on track. It seems we are in agreement 
with items 1, 2, and 3 below (previous e-mail). So to proceed, I would 
like to just focus on item 4 and the reason for sharing  information 
between two proxies (in different stages of a services pipeline) 
separated by a load balancer. I believe that item 4 is true and there 
are no load balancing issues. Let me explain.

The flow participant discovery problem that I am describing does not 
effect load balancing. We can safely assume all proxies do the same 
thing. The proxies are identical.  Let's assume a pool of proxies at a 
stage (of the pipeline) are all measuring content bytes and amassing 
billing information.  The traffic flows from the content servers are 
being distributed over the pool of billing proxies in a particular stage 
by a load balancer. The reason to identify a billing proxy individually 
is simply because that is where a particular flow's billing information 
is located. Even though all the proxies are doing the same thing, 
monitoring the content passing by and billing by the byte, they will be 
working with different content on a flow by flow basis. So what they do 
in the case of each flow will be different and each flow will have a 
different bill. If we change the number of bytes at an adaptation stage 
down stream from the billing proxies how do we get the information about 
the adaptation change back to the correct billing proxy. This is the 
crux of the problem and "sane" load balancing is preserved.  Can we 
agree on this point?

Regards  John

---

From: Red K <[email-masked]>
Date: Wed, 10 Mar 2004 13:20:34 +0800 (CST)

Dear all,

Could anybody please forward me the original question?
Thank you very much!

Best Regards
Red K

__________________________________________________
Do You Yahoo!?
Stand a chance to win a dream date, join the Dream Guy Contest!
http://sg.yahoo.com/dreamguy

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 9 Mar 2004 23:31:30 -0700 (MST)

On Tue, 9 Mar 2004, John G. Waclawsky wrote:

> Let's assume a pool of proxies at a stage (of the pipeline) are all
> measuring content bytes and amassing billing information.  The
> traffic flows from the content servers are being distributed over
> the pool of billing proxies in a particular stage by a load
> balancer. The reason to identify a billing proxy individually is
> simply because that is where a particular flow's billing information
> monitoring the content passing by and billing by the byte, they will
> be working with different content on a flow by flow basis. So what
> they do in the case of each flow will be different and each flow
> will have a different bill.

I agree that each proxy behind a load balancer works on different
flows and, hence, keeps different information. However, I argue that a
good protocol (and a compliant load balancer) will hide that fact from
outside observers. The load balancer will provide mapping between its
trace entry/ID and particular proxy in the pool. The outside observers
will only see an opaque ID and will not care that it maps to one of
the pooled proxies. Please see below for an example.

> If we change the number of bytes at an adaptation stage down stream
> from the billing proxies how do we get the information about the
> adaptation change back to the correct billing proxy.

Let me sketch a solution:

	1) billing-proxy-i inserts its ID "B-i" into the trace

	2) load balancer replaces "B-i" with "LB-1/i" in the trace
	   the tracing protocol requires that the optional
	   information in the ID (stuff "after the slash") is
	   forwarded but does not affect ID comparison and
	   protocol-level addressing

	3) if an adaptation proxy wants to send an update about
	   the current flow it is adapting, it talks directly to
	   the load balancer, using the "LB-1/i" ID supplied by
	   that load balancer

	4) upon receiving a notification from an adaptation
	   proxy, the load balancer converts "LB-1/i" ID to
	   "B-i" and forwards the notification to billing-proxy-i

Note that neither the billing proxy nor the adaptation proxy know that
they are talking to a load balancer. Each of them, perceives the load
balancer as a "standard" OPES proxy, just like they are themselves.
Only the load balancer knows about the address translation (to use a
forbidden expression; but it is fine to do address translation on this
level, we are not talking about IP NATs here, of course, more like
URNs).

Does this clarify my take on the problem?

Thanks,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 9 Mar 2004 23:35:06 -0700 (MST)

> Could anybody please forward me the original question?

Mailing list message headers point to the OPES WG archive[1] that
contains the entire thread, starting with [2].

HTH,

Alex.

[1] http://www.imc.org/ietf-openproxy/mail-archive/maillist.html
[2] http://www.imc.org/ietf-openproxy/mail-archive/msg02834.html