Subject: An opes services usage question

From: "John G. Waclawsky" <[email-masked]>
Date: Wed, 10 Mar 2004 22:23:47 -0500

Alex, from prior e-mails we agreed the use case I described fits within the opes framework (basically we agreed on items 1, 2 and 3 but not item 4 in my last e-mail). I am now trying to resolve item 4, I have been looking over the last couple of e-mails and the current discussions around item 4 can be summarized around four potential suggestions to the "flow participant discovery" problem for the use case.

A) Tracing -  My conclusion of this discussion is that tracing is not a general purpose solution that will work with all application protocols (I am assuming that application protocol refers to any protocol enveloped by TCP or UDP). The comment that "each application protocol needs its own tracing profile and some 
application protocols will not be able to support embedded tracing at all." is very significant. I find it unrealistic to expect all application protocols (or even a significant number in common use today) will support embedded tracing any time soon. Also, it seems using tracing would be "overkill" for this problem.

B) HTTP Metadata -  While HTTP flows support the in-band transfer of small amounts of metadata this may not practical. The size and volume of the metadata could be too much for HTTP. Also, this does not provide a complete answer because HTTP is not the only application protocol we might have to consider.  This suggestion does not seem to be the best general purpose approach.

C) OCP -  OCP can't handle any proxy to proxy exchanges because there are no OCP profiles for proxy to proxy flows.

D) Load Balancer changes - Load balancers are usually high performance boxes and anyone building a load balancer would most likely object to adding additional processing to deal with "address translations". It seems impractical and risky to require the addition of new code into load balancers to support opes (even if it is only to append and replace IP addresses). Also load balancers will not be balancing metadata if the metadata is targeted to return to a particular proxy. Sending metadata through a load balancer will likely bring about scaling issues. All that is required to transfer the metadata, in this case, is simple out of band communication (once the correct address is  known).

All this discussion bring me back to where I started with my initial opes e-mail. It is apparent that a practical opes solution, useful today, is not available for this use case. I recognize that the opes framework is not complete. But I would really like to see the framework extended to handle this services use case, which I believe will become one of the first opes deployments.  I continue to believe that the opes framework is the best way to solve this services usage problem and maintain an open system. I think this is what opes is all about.

Shouldn't we now look into how to extend the opes framework to satisfy the need for a "flow participant discovery" solution?  Solving the flow participant discovery  problem in the most general way without dependencies on load balancers opens up lots of ways of sending any metadata out-of-band between proxies. Your thoughts are greatly appreciated. Thanks again.
Regards  John

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 11 Mar 2004 12:20:45 -0700 (MST)

On Wed, 10 Mar 2004, John G. Waclawsky wrote:

> current discussions around item 4 can be summarized around four
> potential suggestions to the "flow participant discovery" problem
> for the use case.
>
> A) Tracing - My conclusion of this discussion is that tracing is not
> a general purpose solution that will work with all application
> protocols

Agreed. Tracing is one-way and what you want seems to be two-way
(tracing + notification).

> B) HTTP Metadata ...  This suggestion does not seem to be the best
> general purpose approach.

Agreed. Piggybacking metainfo to HTTP is only good if your application
traffic is HTTP and you do not need notifications (only tracing).

> C) OCP - OCP can't handle any proxy to proxy exchanges because there
> are no OCP profiles for proxy to proxy flows.

OCP Core can handle proxy to proxy exchanges. One would need to write
a profile to document rules/messages specific to your environment.

> D) Load Balancer changes ...

(D) is not a solution like A, B, and C. Whether you change load
balancers or not, you still need a [new] protocol to communicate your
metainformation. (D) is, IMO, an unavoidable requirement (see below).

Note that L7 load balancers have been doing content manipulation for
years. This is not a new requirement. The nature of a L4/7 load
balancer requires them to understand the protocol being load balanced.

> Also load balancers will not be balancing metadata if the metadata
> is targeted to return to a particular proxy.

Right. L7 load balancers (aka content switches) are used to such tasks
as keeping client-server affinity. Pure load balancing is only a part
of what they have to do.

> Sending metadata through a load balancer will likely bring about
> scaling issues. All that is required to transfer the metadata, in
> this case, is simple out of band communication (once the correct
> address is known).

A L7 switch or a proxy can do that and can do that fast. Of course, a
L2 switch can do that even faster, but I am not sure I would recommend
trading speed for exposing proxies behind a load balancer.
Architectural concerns usually trump performance optimizations.

Moreover, in most environments, the traffic would have to go through
the load balancer to the proxy because that is the only way to reach
the proxy. Thus, it would be impossible to bypass the load balancer
box in a typical network setup where proxies are load balanced. (Your
environment may not be typical though, not sure).

> All this discussion bring me back to where I started with my initial
> opes e-mail. It is apparent that a practical opes solution, useful
> today, is not available for this use case.

I agree. No existing protocol seems to be able to support what you
want: bidirectional exchange of application-agnostic metainformation
among agents that are possibly being load balanced.

> I recognize that the opes framework is not complete. But I would
> really like to see the framework extended to handle this services
> use case, which I believe will become one of the first opes
> deployments.  I continue to believe that the opes framework is the
> best way to solve this services usage problem and maintain an open
> system. I think this is what opes is all about.

> Shouldn't we now look into how to extend the opes framework to
> satisfy the need for a "flow participant discovery" solution?
> Solving the flow participant discovery problem in the most general
> way without dependencies on load balancers opens up lots of ways of
> sending any metadata out-of-band between proxies.

Are the following three requirements correct?

  1) Agents need to communicate metainformation to each other
  2) metainformation is related to some application protocol, but
     agent communication protocol should be application agnostic
     (i.e., should work with many application protocols)
  3) some agents are being load balanced with respect to the
     application protocol in question

If yes, I conclude:

  - some agents will not have IP addresses reachable from other
    agents

And, hence,

  - agent communication without assistance of load balancers
    would be impossible in general

To resolve the conflict, you have two options, I think:

  (a) require load balancers to handle part of the communication
      (this is how HTTP, SSL, and ICAP are load balanced today)

  (b) limit the environment to agents that are IP-reachable
      despite being load-balanced
      (this would exclude some (many?) load balancing environments)

Do you see what I am getting at? In short, if something is behind a
load balancer, that something may not have a routable IP address and,
hence, cannot communicate directly with the world.

Whether you chose (a) or (b), you still need a new protocol or new
protocol profile to support the metainformation exchange.

Alex.

---

From: "John G. Waclawsky" <[email-masked]>
Date: Thu, 18 Mar 2004 13:31:09 -0500

Alex, It looks like we are ruling out solutions A, B, C. I have been thinking about your response to solution D (below). It seems your perspective consists of two aspects. One aspect is that a layer 7 solution is required for affinity and the second aspect is that metadata is only sent in-band (through the load balancer). I believe that is one specific solution. But I don't see it as the most general way to extend the opes framework. Let me explain. 

I believe the client server affinity problem for metadata is really the simpler case of just needing client affinity for the metadata to be sent back to a previous proxy stage. I believe this affinity problem should and can be solved at layer 3 if the source (client) IP address is available in whatever protocol is decided to be used in the transfer of the metadata. I did some checking and it seems that most load balancers maintain this IP address affinity for you at layer 3. So a layer 3 solution can be application protocol agnostic if we send the metadata in-band back through the load balancer. My current thinking is that the same layer 3 load balancing / affinity information could be leveraged to transfer proxy IP address information and resolve the flow participant discovery solution for sending metadata out-of-band. 

Driving the solution down to layer 3 allows metadata to be sent either in-band (through the load balancer) or out-of-band in those situations where proxies have IP reachable addresses. I did some more checking and also found that IP reachable addresses are often the case because many of the these load balanced environments have management LANs associated with them. I also think this satisfies the three requirements you mention in your previous e-mail response below, which I also believe are correct. 

I agree we need a new protocol to support the metadata exchange (and possibly for acknowledgments of metadata). But, if we  want the most general solution that also allows the sending of metadata out-of-band then maybe the new protocol (or some existing protocol used or adapted for this situation) could solve the flow participant discovery problem too.

Your thoughts?
Regards   John

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 18 Mar 2004 12:03:59 -0700 (MST)

On Thu, 18 Mar 2004, John G. Waclawsky wrote:

> I believe the client server affinity problem for metadata is really
> the simpler case of just needing client affinity for the metadata to
> be sent back to a previous proxy stage.

I agree that we need affinity for the metadata to be sent from a
processing node down the application protocol flow (next proxy stage)
back to a processing node up the application protocol flow (previous
proxy stage). I am not sure why you say "simpler case".  Simpler than
what?

> I believe this affinity problem should and can be solved at layer 3
> if the source (client) IP address is available in whatever protocol
> is decided to be used in the transfer of the metadata. I did some
> checking and it seems that most load balancers maintain this IP
> address affinity for you at layer 3.

I am not sure what you mean. The load balancer (L4-7) usually
maintains some kind of connection/address map to match clients and
servers. The load balancer can use that map to support affinity
features for existing connections and for returning clients (in some
cases). However, the L3 mapping is usually not exposed to clients and
may change. That is, a client, in general, does not know the IP
address of a proxy that served its request if that proxy is behind a
load balancer. Note that availability of a proxy IP address in
application protocol headers does not mean that the load balancer will
use that proxy for the next request from the same client and does not
mean that the IP address is reachable. In most cases, visible IP
addresses of balanced proxies are simply an architectural bug (a load
balancer should have stripped them as unreliable/incorrect/private
info, but it did not).

I am sure there are installations where IP address information in the
application protocol headers (e.g. Via header in HTTP) leaked from
behind a load balancer is more or less reliable, at least short term.
Do you want to limit your solution to these environments?

> So a layer 3 solution can be application protocol
> agnostic if we send the metadata in-band back through the load
> balancer.

Yes, _provided_ that load balancer balancing algorithm matches your
affinity needs. For example, if a load balancer uses a simple round
robin on request basis, then you cannot get to the same proxy. Or, of
an HTTP load balancer uses URL-based switching, you may not be able to
get to the same proxy unless you request the same URL.

> My current thinking is that the same layer 3 load
> balancing / affinity information could be leveraged to transfer
> proxy IP address information and resolve the flow participant
> discovery solution for sending metadata out-of-band.

My understanding is that layer 3 load balancing / affinity information
is not generally available. Only load balancer knows it. Can you give
a couple of specific examples that satisfy your needs?

> Driving the solution down to layer 3 allows metadata to be sent
> either in-band (through the load balancer) or out-of-band in those
> situations where proxies have IP reachable addresses. I did some
> more checking and also found that IP reachable addresses are often
> the case because many of the these load balanced environments have
> management LANs associated with them.

I am not sure how existence of a [properly secured and isolated]
management LAN indicated IP-routable addresses. I am sure there are
cases where proxies are globally addressable, of course (which either
indicated poor network setup OR some external requirements that
exposed proxy IPs).

> I agree we need a new protocol to support the metadata exchange (and
> possibly for acknowledgments of metadata). But, if we want the most
> general solution that also allows the sending of metadata
> out-of-band then maybe the new protocol (or some existing protocol
> used or adapted for this situation) could solve the flow participant
> discovery problem too.

IMO, the three requirements (that we agree on) imply that no general
solution is possible without requiring proxies with exposed/routable
IPs (something you seem to require in the above discussion about L3
information) or requiring load balancers that are aware of the new
protocol. Do you agree? Which of the two limitations do you want to
assume?

Alex.

> >load balancer, that something may not have a routable IP address
> >and, hence, cannot communicate directly with the world.
> >