Subject: Re: An opes services usage question

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 12 Apr 2004 16:06:43 -0400

Alex, sorry for the delay in my latest response. I have been doing some 
investigation and also some travel and other things ...with my day job 
   :-)  

To get back to the discussion. It seems that many of the assumptions in 
the background of our discussions about load balancers are based on web 
load balancers that provide virtual addresses and hide the server 
cluster from the user. But, I have been thinking a little differently in 
considering usage of an opes framework in the mobile wireless market 
segment. This segment may be unique in that load balancers do not 
rebalance for every client connection. Instead they typically rebalance 
on a "per client" basis. This is done because the devices that are being 
balanced have client information associated with their traffic such as 
$, access rights, browser form factors, XML style sheets...etc. Affinity 
to a particular server is strongly maintained for each client. Once 
affinity is in place the load balancer could operate, for example, by 
just replacing the destination MAC address as the traffic arrives based 
on the source IP address (you could view the load balancer as just a 
"bump on the wire").  I believe this additional information should help 
clarify what I am thinking, a bit. I guess I am looking for the simplest 
solution for this environment and I keep thinking the most elegant 
solution would be lower layer one.... just some of my thoughts...

It seems that of the two limitations you suggest, the best way to 
proceed (IMO) would be to assume proxies with exposed/routable adresses. 
I am thinking this assumption is the best one because it probably 
provides a faster solution, would be more flexible, and satisfies small 
deployment needs that do not have load balancers but still need the 
metadata returned to a specific server. I think we are also in agreement 
that with either limitation we still need something to support the 
metadata information exchange. I was also wondering if anyone else has 
any suggestions or additional information to consider?  

Regards  John

Alex Rousskov wrote:

>On Thu, 18 Mar 2004, John G. Waclawsky wrote:
>
>  
>
>>I believe the client server affinity problem for metadata is really
>>the simpler case of just needing client affinity for the metadata to
>>be sent back to a previous proxy stage.
>>    
>>
>
>I agree that we need affinity for the metadata to be sent from a
>processing node down the application protocol flow (next proxy stage)
>back to a processing node up the application protocol flow (previous
>proxy stage). I am not sure why you say "simpler case".  Simpler than
>what?
>
>  
>
>>I believe this affinity problem should and can be solved at layer 3
>>if the source (client) IP address is available in whatever protocol
>>is decided to be used in the transfer of the metadata. I did some
>>checking and it seems that most load balancers maintain this IP
>>address affinity for you at layer 3.
>>    
>>
>
>I am not sure what you mean. The load balancer (L4-7) usually
>maintains some kind of connection/address map to match clients and
>servers. The load balancer can use that map to support affinity
>features for existing connections and for returning clients (in some
>cases). However, the L3 mapping is usually not exposed to clients and
>may change. That is, a client, in general, does not know the IP
>address of a proxy that served its request if that proxy is behind a
>load balancer. Note that availability of a proxy IP address in
>application protocol headers does not mean that the load balancer will
>use that proxy for the next request from the same client and does not
>mean that the IP address is reachable. In most cases, visible IP
>addresses of balanced proxies are simply an architectural bug (a load
>balancer should have stripped them as unreliable/incorrect/private
>info, but it did not).
>
>I am sure there are installations where IP address information in the
>application protocol headers (e.g. Via header in HTTP) leaked from
>behind a load balancer is more or less reliable, at least short term.
>Do you want to limit your solution to these environments?
>
>  
>
>>So a layer 3 solution can be application protocol
>>agnostic if we send the metadata in-band back through the load
>>balancer.
>>    
>>
>
>Yes, _provided_ that load balancer balancing algorithm matches your
>affinity needs. For example, if a load balancer uses a simple round
>robin on request basis, then you cannot get to the same proxy. Or, of
>an HTTP load balancer uses URL-based switching, you may not be able to
>get to the same proxy unless you request the same URL.
>
>  
>
>>My current thinking is that the same layer 3 load
>>balancing / affinity information could be leveraged to transfer
>>proxy IP address information and resolve the flow participant
>>discovery solution for sending metadata out-of-band.
>>    
>>
>
>My understanding is that layer 3 load balancing / affinity information
>is not generally available. Only load balancer knows it. Can you give
>a couple of specific examples that satisfy your needs?
>
>  
>
>>Driving the solution down to layer 3 allows metadata to be sent
>>either in-band (through the load balancer) or out-of-band in those
>>situations where proxies have IP reachable addresses. I did some
>>more checking and also found that IP reachable addresses are often
>>the case because many of the these load balanced environments have
>>management LANs associated with them.
>>    
>>
>
>I am not sure how existence of a [properly secured and isolated]
>management LAN indicated IP-routable addresses. I am sure there are
>cases where proxies are globally addressable, of course (which either
>indicated poor network setup OR some external requirements that
>exposed proxy IPs).
>
>  
>
>>I agree we need a new protocol to support the metadata exchange (and
>>possibly for acknowledgments of metadata). But, if we want the most
>>general solution that also allows the sending of metadata
>>out-of-band then maybe the new protocol (or some existing protocol
>>used or adapted for this situation) could solve the flow participant
>>discovery problem too.
>>    
>>
>
>IMO, the three requirements (that we agree on) imply that no general
>solution is possible without requiring proxies with exposed/routable
>IPs (something you seem to require in the above discussion about L3
>information) or requiring load balancers that are aware of the new
>protocol. Do you agree? Which of the two limitations do you want to
>assume?
>
>Alex.
>
>  
>
>>>Are the following three requirements correct?
>>>
>>> 1) Agents need to communicate metainformation to each other
>>> 2) metainformation is related to some application protocol, but
>>>    agent communication protocol should be application agnostic
>>>    (i.e., should work with many application protocols)
>>> 3) some agents are being load balanced with respect to the
>>>    application protocol in question
>>>
>>>If yes, I conclude:
>>>
>>> - some agents will not have IP addresses reachable from other
>>>   agents
>>>
>>>And, hence,
>>>
>>> - agent communication without assistance of load balancers
>>>   would be impossible in general
>>>
>>>To resolve the conflict, you have two options, I think:
>>>
>>> (a) require load balancers to handle part of the communication
>>>     (this is how HTTP, SSL, and ICAP are load balanced today)
>>>
>>> (b) limit the environment to agents that are IP-reachable
>>>     despite being load-balanced
>>>     (this would exclude some (many?) load balancing environments)
>>>
>>>Do you see what I am getting at? In short, if something is behind a
>>>load balancer, that something may not have a routable IP address
>>>and, hence, cannot communicate directly with the world.
>>>
>>>Whether you chose (a) or (b), you still need a new protocol or new
>>>protocol profile to support the metainformation exchange.
>>>      
>>>
>
>
>  
>

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Apr 2004 14:34:37 -0600 (MDT)

John,

	I agree that if your problem space consists of routable
proxies with strong client affinity, then that is what you should
address first (while keeping more general load balancing demands in
mind, iff possible).

	My understanding is that we will start discussing OPES
rechartering very soon. Adding "OPES metadata exchange" protocol into
the discussion mix would be nice. Ideally, for the discussion to be
effective, there has to be a draft with a good outline of what the
problem space and possible solution directions are. Active
participation of interested parties in the WG would be key as well.

Thank you,

Alex.

On Mon, 12 Apr 2004, John G. Waclawsky wrote:

> To get back to the discussion. It seems that many of the assumptions
> in the background of our discussions about load balancers are based
> on web load balancers that provide virtual addresses and hide the
> server cluster from the user. But, I have been thinking a little
> differently in considering usage of an opes framework in the mobile
> wireless market segment. This segment may be unique in that load
> balancers do not rebalance for every client connection. Instead they
> typically rebalance on a "per client" basis. This is done because
> the devices that are being balanced have client information
> associated with their traffic such as $, access rights, browser form
> factors, XML style sheets...etc. Affinity to a particular server is
> strongly maintained for each client. Once affinity is in place the
> load balancer could operate, for example, by just replacing the
> destination MAC address as the traffic arrives based on the source
> IP address (you could view the load balancer as just a "bump on the
> wire").  I believe this additional information should help clarify
> what I am thinking, a bit. I guess I am looking for the simplest
> proceed (IMO) would be to assume proxies with exposed/routable
> adresses.  I am thinking this assumption is the best one because it
> probably provides a faster solution, would be more flexible, and
> satisfies small deployment needs that do not have load balancers but
> still need the metadata returned to a specific server. I think we
> are also in agreement that with either limitation we still need
> something to support the metadata information exchange. I was also
> wondering if anyone else has any suggestions or additional
> information to consider?
>
> Regards  John

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 12 Apr 2004 17:17:57 -0400

Alex, I am assuming, but just to make sure, that the "opes metadata 
exchange" protocol will include solving what I have been calling the 
flow participant discovery problem (finding out who to exchange data 
with is a big part of the problem). Is this correct?
Regards  John

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Apr 2004 15:38:01 -0600 (MDT)

> with is a big part of the problem). Is this correct?

It will include whatever _you_ want it to include since you are the
driving force behind this :-). I just gave it a, perhaps
inappropriate, label based on your last response. To avoid serious
misunderstandings, we would need some kind of a brief but well-thought
statement-of-work to put this work on the rechartering agenda (IMHO).

Alex.

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 12 Apr 2004 17:45:23 -0400

Thanks for the clarification.  I understand... the ball (so to speak) is 
in my court...   :-)     Thanks
Regards  John

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Mon, 12 Apr 2004 16:52:19 -0600

Flow participant discovery should be compared against the midcom
charter.

Hilarie

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 12 Apr 2004 19:10:13 -0400

I have looked at midcom but I am still not sure I understand this 
statement?   ...can you expand a little please?
Regards  John

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Mon, 12 Apr 2004 17:31:17 -0600

If you want to have opes rechartered to include flow participant
discovery you may need to explain why it should be part of opes
and not part of midcom.  The answer "because I want it work with
opes and don't care about anything else" might not be enough to
convince the authorities.

Hilarie

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Apr 2004 18:34:04 -0600 (MDT)

I agree with Hilarie that we should be careful about potential
overlaps with MIDCOM WG charter. I would suggest that we still start
with a concise but clear-enough proposal that, among other things,
defines "flow participant discovery" problem and lists possible
solution directions. Having such a document at hand, we can do a more
meaningful evaluation of MIDCOM conflicts.

Alex.

---

From: "John G. Waclawsky" <[email-masked]>
Date: Mon, 12 Apr 2004 23:02:57 -0400

Hilarie thanks for your clarification. I'll review midcom more closely 
to check into overlap. I do agree with you and Alex that we should avoid 
any overlaps, or misunderstandings with any WG.  But, I am now a bit 
confused because I thought we were past the point of  the this problem 
being considered an opes problem from the earlier (what I thought to be) 
thorough problem discussions on this e-mail thread.  Am I missing 
something?   ....Also, I don't think that I'd represent my posture as 
"don't care about anything else".  I am investigating how to get this 
problem solved with an open standards solution. I just want to get it 
right and any help you can give me is much appreciated.    :-) 
Regards  John

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Mon, 12 Apr 2004 21:28:01 -0600

> I'll review midcom more closely to check into overlap

OK, that will be good.

>  ... I thought we were past the point of  the this problem 
>  something? 

Am I?  I haven't been convinced that this is an opes problem.  It seems
to be generic problem affecting many protocols.

Hilarie

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Apr 2004 22:23:04 -0600 (MDT)

I think it would be useful to see the problem statement (which will
naturally reflect past discussions) before saying whether the proposed
topic is in OPES scope.

FWIW, it is clear to me that at least parts of the problem domain are
OPES-specific. Yes, the problem affects or works with many protocols
but so does OPES framework. It is too early to tell whether the
problem has a simple general solution that goes well beyond OPES scope
(that would be a good thing!) and that is better developed outside of
OPES scope.

Alex.