Subject: [v6ops] Why v6ops should think about load balancers [ draft-carpenter-v6ops-label-balance-01.txt]

From: Brian E Carpenter <[email-masked]>
Date: Thu, 09 Feb 2012 09:05:50 +1300

Every serious content provider runs load balancers.

Missing IPv6 support in load balancers has been a delaying factor
for IPv6 deployment.

This draft discusses the only aspect of IPv6 that seems to be
significantly different from IPv4: the flow label may enhance load
balancer efficiency.

The authors suggest that the topic belongs in v6ops because it
potentially affects all large server operators, but calls for no
protocol changes. We'd like to know who agrees or disagrees.

Regards
   Brian Carpenter

On 2012-01-30 11:33, Brian E Carpenter wrote:
> Hi,
> 
> I'm not seeing more comments on this. The authors would like advice:
> 
> Should we ask for v6ops to adopt this?
> Should we ask another WG to adopt it?
> Should we propose it as an individual submission?
> Should we forget about it?
> 
> Regards
>    Brian Carpenter
> 
> On 2012-01-18 10:14, Brian E Carpenter wrote:
>> This is a significant update since the version discussed in Taipei,
>> with an additional author who is a practitioner in the field,
>> and incorporating comments from implementors.
>>
>> Of course we would like more feedback. For the moment I suggest
>> discussion on the v6ops list, although that may not be the
>> final home for this draft.
>>
>> Please keep Willy on CC since I'm not sure he's on this list.
>>
>>     Brian Carpenter
>>
>> -------- Original Message --------
>> Subject: I-D Action: draft-carpenter-v6ops-label-balance-01.txt
>> Date: Tue, 17 Jan 2012 12:40:05 -0800
>> From: [email-masked]
>> Reply-To: [email-masked]
>> To: [email-masked]
>>
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>
>> 	Title           : Using the IPv6 Flow Label for Server Load Balancing
>> 	Author(s)       : Brian Carpenter
>>                           Sheng Jiang
>>                           Willy Tarreau
>> 	Filename        : draft-carpenter-v6ops-label-balance-01.txt
>> 	Pages           : 12
>> 	Date            : 2012-01-17
>>
>>    This document describes how the IPv6 flow label can be used in
>>    support of layer 3/4 load distribution and balancing for large server
>>    farms.
>>
>>
>> A URL for this Internet-Draft is:
>> http://www.ietf.org/internet-drafts/draft-carpenter-v6ops-label-balance-01.txt
>>
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>>
>> This Internet-Draft can be retrieved at:
>> ftp://ftp.ietf.org/internet-drafts/draft-carpenter-v6ops-label-balance-01.txt
>>
>> _______________________________________________
>> I-D-Announce mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/i-d-announce
>> Internet-Draft directories: http://www.ietf.org/shadow.html
>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>
>>
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 25 Feb 2012 10:17:34 +1300

It seems from the lack of discussion that nobody in v6ops cares
about server load balancers. I find this strange, since when
I talk to enterprise operations folk, they always flag load
balancers as one of the main holdups with supporting IPv6.

The author's won't drop this draft, but unless the WG shows
some sign of interest, we will take it elsewhere.

Regards
   Brian

---

From: james woodyatt <[email-masked]>
Date: Fri, 24 Feb 2012 13:44:52 -0800

On Feb 24, 2012, at 13:17 , Brian E Carpenter <[email-masked]> wrote:
> 
> The author's won't drop this draft, but unless the WG shows some sign of interest, we will take it elsewhere.

I'm boggled too.

Anyway, I read the draft between compiles.  Some comments...

-- Page 3, in the list of techniques for redirecting traffic to target servers, this would be a good place to mention NPTv6 as an alternative to NAPT66, if not *instead* of NAPT66.

-- Three-way FTP is worth mentioning in section 3.

I have no opinion about where this draft belongs, and will defer to those with more experience in relevant operational environments.

--
james woodyatt <[email-masked]>
member of technical staff, core os networking

---

From: Joel jaeggli <[email-masked]>
Date: Sat, 25 Feb 2012 01:34:40 -0800

> balancers as one of the main holdups with supporting IPv6.

I doubt that very much...

Every load balancer platform that I have access too today has some level
ipv6 support. f5 a10 netscalar, along with ha-proxy and nginx are all in
the field providing production level ipv6 traffic delivery.

I would think given the topic that the involvement of the flow label is
implicated the in lack of excitement.

> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 26 Feb 2012 07:29:22 +1300

"some level [of] ipv6 support"

I can assure you that this is only just filtering out to
the more remote ends of the Internet, but I agree that it's
happening. However, this draft looks a little beyond current
practice.

Regards
   Brian

---

From: Joel jaeggli <[email-masked]>
Date: Sat, 25 Feb 2012 11:15:07 -0800

> "some level [of] ipv6 support"

I'm going to be provocative since I've had my third cup of coffee this
morning.

An f5 which is a complex appliance has a rather different feature set
that nginx which is a http/pop/imap alg that runs on an off-the-shelf
linux box.

I'm missing features on the f5 like layer3 ecmp routes with ipv6  bgp
sessions, that's kinda far down in the weeds feature parity-wise
compared to, "can I use it for l4 proxy or l7 session termination, can I
hash connections across LBs" which it does fine.

The question with rehabilitating the flow label for use with load
balancers is imho orthagional to the question of feature parity with
ipv4 load balancers.

3697 behavior I imagine still dominates, which implies that if I myself
am rewriting the flow label for my own purposes that I'm doing so on the
basis of other criterion (for example the 5 tuple, class of service,
origin interface, evil bit etc) which means I could just as well make
the lb decision on that basis as well.

My reading of 6346 is that I'm now more or less free to do what I want
with the flow label, and I have access to hardware forwarding platforms
that can rewrite anything in the first 256 bytes of a packet at line
rate, so hey, like with DSCP bits in ipv4 if I'm going to the use them
for my own purpose the first thing I'll do is rewrite every externally
set value on ingress because I can't trust them. What could I do with 20
bits that the routers can ignore and that doesn't require a new header?
The flow label is big enough bit field to do this in spades:

http://www.nanog.org/meetings/nanog51/presentations/Monday/NANOG51.Talk45.nanog51-Schaumann.pdf

in fact it more bits than I've ever had to hack about with in an ip
header, so hey go crazy.

joel

---

From: Randy Bush <[email-masked]>
Date: Mon, 27 Feb 2012 15:15:11 +0530

> Every load balancer platform that I have access too today has some
> level ipv6 support. f5 a10 netscalar, along with ha-proxy and nginx
> are all in the field providing production level ipv6 traffic delivery.
> 

what is a 'flow label'?

randy

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 27 Feb 2012 10:09:50 +0000

> what is a 'flow label'?

ok, i'll humour you:  it's bits 13 to 32 of the standard ipv6 header.

But you already know this and we all know you know this, so what point are
you making?

Nick

---

From: Randy Bush <[email-masked]>
Date: Mon, 27 Feb 2012 16:13:11 +0530

> But you already know this and we all know you know this, so what point are

that we have spend over a decade wasting time trying to find a noble use
for those bits.  perhaps we should find operationally useful work.

randy

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 27 Feb 2012 11:22:04 +0000

> for those bits.  perhaps we should find operationally useful work.

flow labels are used as a hashing token for lags and ecmp load balanced
links - this has been the case for a long time.  And no, I'm not going to
get into protracted arguments about the necessity of load balancing N*10G
ipv6 links - the point is that the field is used by various kit out there
today.  And who knows, maybe on June 6th this year, this might start
becoming important.

Nick

---

From: Gert Doering <[email-masked]>
Date: Mon, 27 Feb 2012 16:11:05 +0100

Hi,

On Mon, Feb 27, 2012 at 11:22:04AM +0000, Nick Hilliard wrote:
> links 

Is anybody setting these to anything but "0"?

(This is a serious question, and I don't know the answer - but if it's
used as a balancing hash, there needs to be some diversity in the values
to make it useful)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Mon, 27 Feb 2012 15:54:36 +0000

> used as a balancing hash, there needs to be some diversity in the values

Yes, you may remember the old CSCsd13298 problem?  I'd also like to point you
at the old surveys from Orla McGann and David Malone.

I think part of the load balancer issues is that the latest documents do not
make it fully suitable for "hosts" anymore where an e-to-end (edge-to-host or
better end-to-end) flow label would be essential (controllable by the host
ideally) or uniquely by an entire management domain, which already breaks with
a customer host  (I'll ignore Fernando's immediate concerns there;-)

I hope that some RSS people could speak up here, given the toeplitz etc problems.
Talking to NIC vendors (with clue) might also be a good idea.

/bz

-- 
Bjoern A. Zeeb                                 You have to have visions!
   It does not matter how good you are. It matters what good you do!

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 27 Feb 2012 16:17:08 +0000

> used as a balancing hash, there needs to be some diversity in the values

in theory it ought to be enabled on freebsd and consequently mac os/x,
depending on whether net.inet6.ip6.auto_flowlabel is set to 1 (the
default).  However, wireshark shows inconsistent results compatible with
the Orla/David's paper of 2005.  Linux still has no code to set it.
Windows doesn't appear to set it either.  Disappointing.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 27 Feb 2012 16:21:12 +0000

> Windows doesn't appear to set it either.  Disappointing.

should add: this is on the basis of 5-10 minutes of poking around.  One
wouldn't want to call it useful data or meaningful investigation.

Nick

---

From: Ray Hunter <[email-masked]>
Date: Mon, 27 Feb 2012 18:29:34 +0100

Off Topic:
I work with enterprises and care about load balancers and high 
availability systems.

To be honest though, my cares are much much more low-level than a new 
IETF draft. e.g. lack of any IPv6 support at all in some brand new 
devices that are still shipping, IPv6 support in existing IPv4-based 
mechanisms (e.g. WCCP & WCCPv2 although I believe this is now fixed 
since 2H2011), any impact on security when introducing IPv6 in load 
balancers, how to deploy IPv6 across 18 different services in 18 months 
whilst keeping all of the services running 7x24 with minimal downtime.

On Topic:
I have read this draft. As usual with work from the authors, it is well 
written and easy to understand for operational type people.

A form of load balancing that I have not yet seen addressed is for on 
the fly processing more akin to multipath routing, such as WAN 
compression or protocol optimization e.g. for CIFS, where the cost of 
processing an individual packet may be much greater than switching it.

In this case, an in-line device (the interceptor/ director) may 
intercept packets and switch based on L3/L4 info to either
i) forward the original traffic flow to other in-line devices for 
further processing (e.g. caching/ protocol acceleration/ WAN 
compression) before the traffic is forwarded over a tunnel to a remote 
partner device for decompression,
or
ii) farm out service requests to an array of specialist 
compression/cache/protocol acceleration engines which then return data 
to the director for forwarding.

Ensuring that content is appropriately balanced between a number of 
compression/cache/protocol acceleration devices can potentially balance 
the load on each device, increase the overall system availability, and 
also increase the efficiency of individual cache hits for requests.

Another form of load balancing that hasn't been mentioned is Layer 7 
load balancing.

A layer 7 switch may examine traffic on the fly and switch the requests 
to an appropriate server based on the actual content of the request 
(perhaps utilizing NAT44 to ensure proper routing of the return 
traffic). This may allow separate provisioning of servers for servicing 
HTTP requests for fixed content such as static images, compared to 
servers used for handling transactional requests, whilst all content is 
located from a browser perspective via a single base URL and IP address. 
Of course there's no official equivalent of NAT44 in IPv6 to perform 
this return routing.

As far as I can see, the IPv6 flow label may not be an effective method 
of caching layer 7 load balancing decisions, because a single persistent 
HTTP session defined in RFC 2616 (aka HTTP keep-alive) may transport 
requests for many different forms of content via pipelining.

Also the cost of Layer 7 processing for IPv6 for load balancing could be 
very different than in IPv4. See e.g. draft-gont-6man-oversized-header-chain

regards,
RayH

-- 
Ray Hunter
[email-masked]
Globis Consulting BV, Fazantlaan 23, 5613CB Eindhoven NL,
Registered at the KvK, Eindhoven, under number BV 17098279
mobile: +31 620 363864

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 28 Feb 2012 09:32:52 +1300

>> implicated the in lack of excitement.

Today, it's usually 00000000000000000000

That's what we want to change, by documenting use cases (as RFC6438 does).

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 28 Feb 2012 09:40:07 +1300

> Another form of load balancing that hasn't been mentioned is Layer 7 load balancing. 

Actually Ray, it *is* mentioned in the draft, but it's a little beside the point,
because the flow label is a layer 3 construct. However, if layer 7 deep packet
inspection has been used to assign a particular flow to a particular server, a
layer 3 flow label has value for identifying future packets of the same flow.

(I'm not going to respond to the other messages that are a rehash of the 2 year
debate in 6man that led to RFC6437. Been there, reached rough consensus on that.)

Regards
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Feb 2012 12:18:29 +1300

Joel,

> I'm going to be provocative since I've had my third cup of coffee this

You could try camomile.

> balancers is imho orthagional to the question of feature parity with

True. The point is to use the flow label to make LBs work better. What
I've learnt from Willy Tarreau is that biggest problem for load balancers
is when the client's address/port for a given session changes, but the
LB still has to identify each packet of the flow at line speed to guarantee
session persistence.

> the lb decision on that basis as well.

The behaviour that dominates is default (flow label =0) and that won't change
unless host stack maintainers see a good reason to change it.

> with the flow label, 

Not quite, there are clear recommendations for default actions, but we have
dropped the idea that it's genuinely immutable, and that does allow for
innovative use. But the real benefit will come when sources set it.

and I have access to hardware forwarding platforms
> set value on ingress because I can't trust them. 

You don't particularly need to trust them, since they aren't used
for e2e signalling. But yes, if you are worried about attacks on
the flow label itself, the border device could rewrite them. However,
that *will* defeat the kind of advanced usage Willy is thinking of
to support persistence.

What could I do with 20
> http://www.nanog.org/meetings/nanog51/presentations/Monday/NANOG51.Talk45.nanog51-Schaumann.pdf

Well, the way they use the DSCP is unconventional, but fundamentally
harmless as far as I can see. And yes, in the domain between a LB
and the target server, I can't see any real harm in twiddling the
flow label bits too. It's actually orthogonal to what we talk about
in the subject draft, which happens before the packet leaves the
load balancer.

   Brian

---

From: Joel jaeggli <[email-masked]>
Date: Tue, 28 Feb 2012 15:37:16 -0800

> LB still has to identify each packet of the flow at line speed to guarantee

To be clear though if the source ip/port changes that's a new flow...
e.g. in the expected case of unicast tcp we need another handshake to
restablish a session with a  new source port.

It may be not be a new session from the vantage point of the application
layer which may well maintain persistence across connections (which
cookies, http, magic, or otherwise are particularly suitable for).

So I'm not sure I  grok that's a case where you'd expect the flow label
to remain constant when I have to establish a new connection.

---

From: Hesham Soliman <[email-masked]>
Date: Wed, 29 Feb 2012 10:53:47 +1100

>to remain constant when I have to establish a new connection.

=> Agreed. Can't see how a flow label can help here. It's not globally
unique! 

Hesham

>>>>>level

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Feb 2012 13:20:22 +1300

>> to remain constant when I have to establish a new connection.

If you want to use the flow label instead of much less palatable
techniques to identify the transaction, using the same flow label
is indicated. But that's getting beyond v6ops scope, I fear.

> => Agreed. Can't see how a flow label can help here. It's not globally

Strangely, it doesn't need to be. It doesn't matter if the load balancer
happens to send two different flows to the same target server, because
they happen to have the same flow label. (Which is only a 1 per million
probability, since we have 20 bits.) What matters is that all the
packets of the same transaction go to the same server.

    Brian

---

From: Hesham Soliman <[email-masked]>
Date: Wed, 29 Feb 2012 18:15:29 +1100

>packets of the same transaction go to the same server.

=> Agreed again. But the point is that the flow label doesn't really buy
you anything for that specific scenario because it's a new connection
anyway and it can be load-balanced to any server. So, there doesn't seem
to be an advantage for that scenario.

Cheers,
Hesham

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 29 Feb 2012 08:30:45 +0100

Hi,

> to be an advantage for that scenario.

The advantage of having the flow label is to tell the load balancer what
user-level session the connection belongs to so that it can forward the
connection to the proper server. That's exactly what is currently done
in HTTP with cookies and in SSL with SSL-ID, except that with the flow
label we wouldn't have to process layer-7 information.

Basically a browser would pick a random 20-bit number when opening a
page and would assign this 20-bit random number to all outgoing connections
to the same site, whether they're HTTP or HTTPS btw. The LB would then
either hash this flow label or load-balance + stick on it, at the admin's
option. The general idea is to insert a session identifier in the packets
so that all packets related to a same user session can be processed equally
and directed to the same place, which is the hardest thing to do right now
in load-balancing.

Regards,
Willy

---

From: Hesham Soliman <[email-masked]>
Date: Wed, 29 Feb 2012 21:43:05 +1100

-----Original Message-----
From: Willy Tarreau <[email-masked]>
Date: Wed, 29 Feb 2012 08:30:45 +0100
To: Hesham Soliman <[email-masked]>
Cc: Brian E Carpenter <[email-masked]>, IPv6 Operations
<[email-masked]>
Subject: Re: [v6ops] So nobody cares about load balancers? [
draft-carpenter-v6ops-label-balance-01.txt]

>label we wouldn't have to process layer-7 information.

=> But aren't you now comparing a unique id known locally in the load
balancer to the flow label?

>to the same site, whether they're HTTP or HTTPS btw.

=> Is it safe enough to rely on the browser's 20 bit RNG and ignore IP
addresses? 
The discussion was related to the change in IP address and whether the
flow label helps in that case.
Aside from the fact that this case is rare and probably not worth the
optimisation, my contention is that the flow label doesn't help because
it's not unique. 

>The LB would then

=> I don't know if this is a sound idea to be honest. I can understand the
use of the flow label instead of port numbers when combined with IP
addresses for load balancing. I think that's a fine thing to do. I just
don't think using the flow label alone as a session id is such a good idea
given the reliance on its uniqueness. It seems to optimise for a rare case
and for a small benefit.

Hesham

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 29 Feb 2012 14:38:48 +0100

> balancer to the flow label?

The goal would be to use the flow label as a session ID. Not a *unique* ID
since it won't be unique due to its small size. Just a random enough ID to
ensure a smooth enough distribution.

> => Is it safe enough to rely on the browser's 20 bit RNG and ignore IP

I'd say it differently. It's safe to rely on anything but the IP address.
One user != one IP address. It's only an approximation which is valid
enough where it's only used for optimization, but not where user session
stickiness is required. For instance, hashing a source IP address is fine
to load-balance an SSL farm, because most of the users will remain on the
same SSL endpoint, and the small part of those who will constantly change
will just experience a minor degradation which is not a show stopper. But
it's totally different when you want to ensure that once you log into a
server you need to remain on this one until you click the "logout" button,
because this time you cannot accept to drop connections for even 1% of your
visitors. That's why LBs mostly rely on cookies and have to ignore IP
addresses for this task.

With a flow label, we have a lower layer alternative to stickiness cookies.

> flow label helps in that case.

Yes precisely it does help. If you make it possible for the client to set
the flow label on purpose on outgoing connections, and you make the LBs
only rely on the flow label, then you're not sensitive to IP address changes
anymore. These ones happen at enterprise proxies and in the mobile world
where it's very common to lose access after some idle time.

> optimisation, my contention is that the flow label doesn't help because

We don't need unicity (and we don't even want it). We need a large enough
value so that not everyone goes to the same server. Some L3 load balancers
are still hashing on the lower 8 bits of IPv4 addresses right now. It's
far from being unique and still it works for most purposes, provided that
the number of servers is much less than 256. With 20 bits of randomness,
you can have a fair distribution up to 1 million servers, or a bit less
if you want to apply smooth weighting.

> given the reliance on its uniqueness. It seems to optimise for a rare case

It's not the rare case, it's the main case. Look at how load balancing is
performed everywhere. IP address is never used when you need to guarantee
user stickiness (eg: HTTP, RDP, ...). That's even why LB vendors claim to
support a wide variety of protocols. In fact what they have to do is to
pick the relevant part in the protocol to find the server associated with
the session. For instance, in HTTP you have to parse L7 to find cookies,
in HTTPS you first have to decipher SSL then to extract HTTP cookies. In
RDP you have to either find the user ID or the server's address which are
hashed in RDP cookies, etc...

Regards,
Willy

---

From: "Marc Lampo" <[email-masked]>
Date: Wed, 29 Feb 2012 17:17:03 +0100 (CET)

Hello,

Some considerations, trying to bring constructive feedback for this document 
:

1) seems to me the document, in its present state, is not using
   the term "session" in a consistent way.

   Having dome some programming on specific layers of the OSI model,
   I interpret "session" as something from layer 5 : the session layer.

   By situating session on a different layer (than transport),
   the following should be clear :
   - one transport "connection" can transport multiple sessions
     (compare to : during one phone call - transport connection -
      partners have a "greeting" / "say what they have to say" /
       and "goodbye" part in the conversation;
       each of which could be considered a "session" by itself)
     --> seems to me, flow label can cope with this.

   - one session can span multiple transport connections
     The present text uses ftp as an example :
      one control connection + (each ?) data connection
       are all one "session"
       (I'm hesitant about this)
     Another example, also referred to in the text, are http cookies.
      The final server uses cookie to decide whether or not
      "this" data is actually a continuation of a previous session.
      Potentially client IP and port could change during the session.
     Nothing forbids that different transport layers are
      used for one session : though not affected by flow label,
      but think of DNS where the initial Q over UDP yields TC-bit,
      and is followed by the same Q over TCP : one session ?
     --> I'm unsure if flow label can cope with this.

For the text :
Please review the use of "connection" and "session".
I see, in 2, second paragraph : "transport session" !?
Still in 2, top of page 7 : "first packet of a new session"
 --> shouldn't this be : "new connection"

Looking at the draft and keeping the 7 layer definition in mind,
it seems that what is actually attempted is to put layer 5 info
(session "id") into a field of layer 3.
I'm unsure if this is possible at all (especially : how to span
 multiple transport "connections").
Of course, the challenge (speed/number of connections per second)
for load balancers is not made any easier if they must take layer 5
Information into account.  I acknowledge that this draft tries to
help them with that.

2) Security considerations
(actually a remark on the same section in RFC 6437,
 but this draft refers to it.  Sorry for being late ...)
RFC 6437 states, in 6.1 "Covert channel risk" :
 "... isolated UDP packets ...".
But what are "isolated" UDP packets ?
Isn't it overlooked that, since a layer 5 session can span
 multiple layer 4 packets, one session can easily span
 multiple UDP packets (think of tftp to name one).

I hope this contribution helps fixing ideas and avoid confusion.

Kind regards,
Marc Lampo

---

From: Warren Kumari <[email-masked]>
Date: Wed, 29 Feb 2012 13:26:58 -0500

> and directed to the same place, which is the hardest thing to do right now

I think that some of the reluctance to this document / idea is that it gives too much control to the client app and seems like it could be a DoS vector (unless I have completely misunderstood the mechanism).

By using the flow label to determine which backend to ship traffic to, the clients can basically choose a server to hit -- this means that a bunch of zombies could all initiate connections to 192.0.2.1:80 with flow id 42 and then, every N seconds, increment the flow label. This would basically allow targeting of backends, dumping the full volume of attack traffic at a single backend (and then the next backend, and then the next and so on). 
While this is currently possible with the current load balancing algorithms, many of them use H(dest_ip, dest_port, src_ip, src_port, secret)%N -- the inclusion of src_ip, src_port spreads the hash (and the secret prevents the attackers from calculating src_ports to generate matching hashes). By doing H(dest_ip, dest_port, flow_label, secret)%N, the attacker has control over all hash inputs....

W

---

From: Tina TSOU <[email-masked]>
Date: Wed, 29 Feb 2012 18:31:56 +0000

Tina

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Warren Kumari
> Sent: Wednesday, February 29, 2012 10:27 AM
> To: Willy Tarreau
> Cc: IPv6 Operations
> Subject: Re: [v6ops] So nobody cares about load balancers? [ draft-
> carpenter-v6ops-label-balance-01.txt]
> 
> 
> I think that some of the reluctance to this document / idea is that it
> gives too much control to the client app and seems like it could be a DoS
> vector (unless I have completely misunderstood the mechanism).
> 
> By using the flow label to determine which backend to ship traffic to, the
> clients can basically choose a server to hit -- this means that a bunch of
> zombies could all initiate connections to 192.0.2.1:80 with flow id 42 and
> then, every N seconds, increment the flow label. This would basically
> allow targeting of backends, dumping the full volume of attack traffic at
> a single backend (and then the next backend, and then the next and so on).
> While this is currently possible with the current load balancing
> algorithms, many of them use H(dest_ip, dest_port, src_ip, src_port,
> secret)%N -- the inclusion of src_ip, src_port spreads the hash (and the
> secret prevents the attackers from calculating src_ports to generate
> matching hashes). By doing H(dest_ip, dest_port, flow_label, secret)%N,
> the attacker has control over all hash inputs....
But can people always have a better hash algorithm? For example, make the hash tables dynamic; that is, they may have to vary in size substantially as the process goes on.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 29 Feb 2012 13:42:53 -0500

> But can people always have a better hash algorithm? For example, make the hash tables dynamic; that is, they may have to vary in size substantially as the process goes on.

But if you are only hashing over H(dest_ip, dest_port, flow_label, secret):

dest_ip is a constant.
dest_port is a constant.
flow_label - provided by the attacker.
secret is a constant.

This means that the attacker is in control of all of the inputs (other than secret, which has to stay the same to maintain consistency), so:
H(192.0.2.1, 80, 42, "hunter2") will always generate the same value (and has to, otherwise your LB will not be providing a consistent mapping)

You need *something* that the attacker is not in control of...

W

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 01 Mar 2012 09:04:00 +1300

>    the term "session" in a consistent way.

Marc is correct that the terminology needs to be defined and used
consistently. As Willy has pointed out, a user session is not
the same thing as a transport session, and that is a fundamental
issue in load balancing.

> I think that some of the reluctance to this document / idea is that it gives too much control to the client app and seems like it could be a DoS vector (unless I have completely misunderstood the m…

Absolutely, the attack scenarios need to be considered. Actually
this is discussed in RFC 6437 and in the subject draft, but it needs
more thought if we pop up from transport-layer labels to session labels.

Regards
   Brian Carpenter

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 29 Feb 2012 22:27:00 +0100

Hi,

> You need *something* that the attacker is not in control of...

This is totally pointless because the goal of "persistence" or "stickiness"
in load balancing precisely is to ensure that some form of user controlled
data is used to find the proper server. If you want to bring a random server
down, that's very easy. Just connect to it first, and feed the cookie to a
botnet which will always connect to the same server. This is the expected
behaviour. There are counter measures against abuse, some of them involve
counting the per-session request rate for instance and blocking. Quite
frankly right now this has not been an issue and will probably not be one
for a long time to come since the biggest issue still is to find this
user-server association.

Regards,
Willy

---

From: Warren Kumari <[email-masked]>
Date: Wed, 29 Feb 2012 17:13:21 -0500

On Feb 29, 2012, at 4:27 PM, Willy Tarreau wrote:

> botnet which will always connect to the same server.

Um, not really -- this only applies if you use cookies for stickiness (and are load-balancing something like HTTP that uses cookies!)

There are a large number of other load balancing algorithms and a large number of protocols that don't have anything approaching a cookie...

W

> This is the expected

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 01 Mar 2012 11:53:13 +1100

>I'd say it differently. It's safe to rely on anything but the IP address.

=> No one suggested that you rely on the address alone.

>stickiness is required.

=> And the flow label can ensure that stickiness by itself? That's a
contradiction of what you're trying to do.
If you want proper stickiness then you need unique application
identifiers. If you think the user's IP address is not stable enough and
you want stronger stickiness then I'd say the flow label is much worse.
More below

>For instance, hashing a source IP address is fine

=> Ok, but my point is that you can't compare flow labels and cookies
because you'll end up with either incorrect load distribution (because the
flow label doesn't tell you enough about the connection, or with users
being distributed onto different servers, depending on which side your
load distribution algorithm favours. What am I missing?
If you LB flow label 27 on server 1 blindly then you can very well end up
with server 1 being busier than others. If you split flow label 27 between
servers 1 and 2 without any other considerations then you'll end up where
you don't want to be now.

Relying on applications to do the right thing (use a good RNG) for your
infrastructure to work seems like a proposition that could lead to
unreliable results.

>where it's very common to lose access after some idle time.

=> Yes but they don't use flow labels for that. They use a unique
application-specific identifier.
flow label != cookie. The cookie is not created by the client.

>value so that not everyone goes to the same server.

=> Large enough numbers of clients producing a uniform distribution of
flow label values. What happens when the clients don't behave that way?

>Some L3 load balancers
>user stickiness (eg: HTTP, RDP, ...).

=> Changing IP address is a rare case, I wasn't saying that identifying
users is a rare case, lets not mix things. And yes of course user
identification happens but this is done with a *unique* identifier. You
keep using *user* stickiness as a reason while ignoring that to achieve
*user* stickiness you need a unique identifier. The combination of
unreliable number generators and load balancing based on those numbers
alone can end up with an undesirable effect. Do you disagree with this
statement? 

>That's even why LB vendors claim to

=> Right, all about finding user-specific credentials that are unique on
the server side. You're replacing that with arbitrary numbers generated by
the client and claiming that it will have the same effect. In an ideal
world where clients behave, I think you have a good point, but if clients
don't behave, what will you do?

Hesham

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 01 Mar 2012 16:07:57 +1300

Hesham,

> load distribution algorithm favours. What am I missing?

As the draft explains, you don't use the flow label to take the
initial decision about which server gets which new session. That
must be done by looking at much more of the first packet, exactly
as today. You use the flow label to accelerate treatment of the
subsequent packets. Please tell us where in the draft we need
to make this more clear.

> If you LB flow label 27 on server 1 blindly 

You don't do that.

> then you can very well end up

Indeed, so you don't do that either.

You do accept the 1 in 2^20 chance that two unrelated sessions have the
same flow label; in that case they will need to be redirected to the same
server. In any case the LB is going to need a hash table to look up current
flow labels, but it certainly contains a monstrous hash table today for
other reasons. Server LBs are stateful beasts, unfortunately.

(In contrast to ECMP/LAG, which should be stateless.)

> infrastructure to work seems like a proposition that could lead to

It can only be better than today, where there is zero entropy
in the flow label. The draft also discusses the case where the
source doesn't set the label.

As for choice of RNG, yes, it needs to be reasonable. I have some
results on this coming very soon.

> flow label != cookie. The cookie is not created by the client.

Correct, and it's a fairly horrible solution, isn't it? It forces
you into DPI for every packet in order to recognise the session.
That's *exactly* what needs to change.

> flow label values. What happens when the clients don't behave that way?

The worst case is that the LB has to drop back to today's solutions.

> => Changing IP address is a rare case, 

I'll let Willy comment on that, but he tells me that it's a very common
case, viewed from a server site.

I wasn't saying that identifying
> alone can end up with an undesirable effect. Do you disagree with this

Again: that is not the model we describe in the draft.

> don't behave, what will you do?

The worst case is that the LB has to drop back to today's solutions.
Also, there's an incentive for client stacks to do the right thing:
improved average application response time.

Regards
   Brian

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 1 Mar 2012 07:40:20 +0100

On Wed, Feb 29, 2012 at 05:13:21PM -0500, Warren Kumari wrote:
> Um, not really -- this only applies if you use cookies for stickiness (and
> are load-balancing something like HTTP that uses cookies!)

Let me clarify. Stickiness is the opposite of load balancing. Load balancing
applies an algorithm and stickiness is the principle which consists in
finding the information which lets you pick the correct server without
applying the load balancing algorithm. In HTTP, cookie-based stickiness
precisely does that. Same for the RDP cookies used in TSE or with the
SSL-ID in SSL which is assigned by the server and works exactly like a
cookie.

Some hash-based load balancing algorithms are an alternative to the
combination of LB + stickiness. The principle is that if the information
is always present including the first time, then it is worth hashing to
find the appropriate server. This includes MAC/IP/ports where stickiness
is not critical (eg: etherchannel in switches, any stateless TCP-based
offloader, etc...), URLs for HTTP when caches are involved, username for
POP/IMAP servers, etc...

> There are a large number of other load balancing algorithms and a large
> number of protocols that don't have anything approaching a cookie...

I perfectly agree. What I meant is that there are a huge number of
situations where load balancing is applied on purpose to some data
fed by the client and this is not a problem at all. Even when hashing
is involved on source/dest addresses (think about SYN floods), ports
(TCP connections are free to select the source port), URL, username,
etc...

It would be exactly the same with the flow label : the client is able
to choose is as well as it can choose its source ports. What is needed
is to define clearly how this should be processed in order to avoid
caveats such as a flow label changing in the middle of a TCP connection
and packets being sent to a wrong server and being resetted (this would
leave the old connection open until the timeout, just like when a client
suddenly disconnects from the net).

Another point to consider is that it would probably make sense to have
the server use the same flow label for the return packets belonging to
the same connection. This would allow the label to be used for firewall
load balancing which needs to be symmetric.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 1 Mar 2012 08:05:44 +0100

Hi Brian,

> subsequent packets. Please tell us where in the draft we need

I have talked about that in this thread. In fact, I'm fairly certain that
once everyone will observe that flow labels follow an almost uniform
distribution, it will be safe to hash them in low level components such
as switches. Many switches are already able to hash MAC or IP addresses
right now. In a datacenter full of two-port servers, MAC addresses are
very poorly distributed, and nobody complains about the risk to cause an
unbalanced distribution on etherchannels. Same for IP addresses where .0
and .255 are almost never seen. But I agree that right now we should focus
only on load balancers learning the flow label and not on hashing.

> other reasons. Server LBs are stateful beasts, unfortunately.

Let's talk about "lookup table" instead of "hash table" to avoid confusion
here.

> results on this coming very soon.

In fact there are many other alternative to RNGs to provide a valid
flow label inside a browser (sorry for insisting on browsers but I
think that HTTP right now is the most load-balanced protocol). It
could be anything such as a hash of the precise time the user clicked
the link to the page, it could be whatever you like. It could even be
a hash of the MAC address of the device itself which would always tie
it to the same server for a given site. That's not really an issue, as
we don't need it to be unpredictable nor cryptographically strong, we
just need it to be uniformly distributed across all clients.

> >> Look at how load balancing is performed everywhere.
> >> IP address is never used when you need to guarantee

Yes I can confirm that and everyone dealing with load balancers is well
aware of this fact too which forces you to use something else to stick
on if stickiness is critical (eg: your shopping basket).

Depending on the sites, you can see between 1 and 5% of visitors using
more than one source address in their whole session. Even if it was only
0.01% it would be huge because it would mean that for a site seeing 1
million visitors a day (not a big site), you would have 100 people
complaining every day that your site is broken. And here I'm talking
about much many ones. I have identified three reasons for changing
IP addresses :
  - use of load-balanced proxy farms (the "AOL effect" as it used to be
    called among web developers) : the client connects to a farm of
    caching proxies which perform URL-based stickiness to improve
    caching. They reach the net with a different source address, so
    many URLs are fetched from different IP addresses ; This practise
    now tends to be masked by NAT in enterprises but not much at ISPs.
    With V6, it should reappear better in enterprises with NAT disabled.

  - mobility : if you look at the address of a mobile user along a session,
    you'll see that *many* of them regularly change. They change when the
    device is moving (I suspect that sites such as Facebook where people
    connect to in the train observe this effect a lot), and after long idle
    periods ("long" meaning something like 5-10 seconds with some operators).

  - load-balanced ADSL links. Since ADSL is of extremely poor stability in
    some places, we've seen the rise of ADSL link loadbalancing gateways.
    Customers want to use all of their links since they're paying for them,
    and this results in traffic being randomly spread over the multiple
    links depending on their instant load or queue size. Each link has a
    different address and very often is connect to a different ISP in order
    to maximize reliability. So a client session again uses multiple addresses.

Regards,
Willy