Subject: [v6ops] Happy-eyeballs - P-value & behavior

From: Javier Ubillos <[email-masked]>
Date: Fri, 25 Feb 2011 15:16:32 +0100

Hi.

As I mentioned previously, we're implementing happy-eyeballs in
name-based sockets (all in kernel-space).

Here are some thoughts about things we've run into:

* Increasing/decreasing the P-value

So, the current action table is
       |    Winner   |
|   P  | IPv4 | IPv6 | Action | Suggestion
+------+------+------+--------+-----------
| P<=0 |   X  |      | P -= 1 |
+------+------+------+--------+-----------
| P>=0 |   X  |      | P /= 2 | P /= -2
+------+------+------+--------+-----------
| P<=0 |      |   X  | P /= 2 | P /= -2
+------+------+------+--------+-----------
| P>=0 |      |   X  | P += 1 |
+------+------+------+--------+-----------
|      |  Both fail  |   ?    |  P = 0
+------+------+------+--------+-----------
And a case which wouldn't fit in the table, what to do if we cannot
determine a winner i.e. both sockets get connected very quickly. I'd
like to give IPv6 the edge in that scenario.
Also, if both connections fail, I suggest we reset P to 0.

Multiplying with -1, consider the scenario:
P is high, favouring IPv6 and delaying IPv4 connections.
If IPv4 still wins, it means that suddenly is significantly better than
IPv6 (it won dispite IPv6's headstart).
Shouldn't we multiply P by -1?
Does it matter if we start "flapping" between IPv4/6?
If it does matter, e.g. we want SSL, wouldn't a slow-flapping
("normal"-P management) be just as bad?

If we, instead of dividing by two, divide by -2. We both switch
protocol, yet move towards zero, expressing that we are not as confident
about the preference.

* Why should we keep a P-value at all?

After a mail-exchange with Dan Wing, my conclusion is basically, to
avoid unnecessary SYN-spamming. Which makes sense to me, SYNs don't have
a back-off mechanism as an "ordinary" TCP flow has. However, using a
P-value, where P is a delay to be introduced before trying the "other"
AF seems sub-optimal to me.
Perhaps some other congestion-control for SYNs is more appropriate?
A trivial algorithm _could_ be:

  | Always send a SYN for both IPv4 & IPv6 (be aggressive)
  | If some SYN does _not_ receive a SYN-ACK begin
SYN-congestion-control
  --> Select the best AF (v4 or v6) according to some criteria
    | During time T, only use the best AF
    |* When time T expires, test sending a SYN on the other AF, and see
if it gets through (multiple tests?)
    | If the congestion is gone go to start (i.e resume an aggressive
approach)
    | Else - reset T and resume the timid approach.

The reason I'm suggesting a "default" aggressive approach, that I don't
think that SYN packets make out a noticeable large fraction of the total
amount of traffic. I've started some captures to compare our offices
ratio of tcp/tcp-syn packets (overall and for www traffic).

* P-scope
Currently we're choosing the easy-way out, and keeping a system-wide P.
However, I'm assuming that a per-destination P is more relevant. Perhaps
even destination+service. I think destination+service is overkill. We
_want_ to re-use P and not have to re-init/test too many times.
We'll continue with a system wide P for now, but we'll likely go the
per-destination P, hopefully with some nice aggregation feature too.

My 3 * 2cents.
// Javier Ubillos

---

From: Simon Perreault <[email-masked]>
Date: Fri, 25 Feb 2011 09:40:12 -0500

> |   P  | IPv4 | IPv6 | Action | Suggestion

Agreed.

> Also, if both connections fail, I suggest we reset P to 0.

What would justify discarding all accumulated evidence in one shot?

> protocol, yet move towards zero, expressing that we are not as confident

I understand what you're trying to accomplish, but I'm wondering why
you're giving so much more importance to the latest result than to the
accumulated evidence. Why is it important that we switch right away?
Over time, if IPvX maintains its advantage over IPvY, the switch will
happen anyway.

> ratio of tcp/tcp-syn packets (overall and for www traffic).

I agree: SYN spamming does not sound like a big issue to me (need data
to back that up). But your algorithm still tries to avoid SYN spamming.

I'd go further:

- Always send SYN for both IPv4 & IPv6.
- If the IPv6 connects first, use it and cancel IPv4.
- If the IPv4 connects first, wait 100ms to give IPv6 a chance.
  - If IPv6 connects within 100ms, use it and cancel IPv4.
  - Otherwise, cancel IPv6 and use IPv4.

The cool thing about this is that it is super simple and stateless.
There is one tweakable parameter (sounds like a sysctl to me!).

But it assumes that SYN spamming is not an issue. Any idea on how we can
test that assumption?

> per-destination P, hopefully with some nice aggregation feature too.

I have evidence that a global P is broken. I'll post it soon.

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 25 Feb 2011 18:35:09 +0100 (CET)

Hi Javier,

> |   P  | IPv4 | IPv6 | Action | Suggestion

I agree. (we want the IPv6 to be the preferred protocol, all other things 
equal).

> Also, if both connections fail, I suggest we reset P to 0.

Seems like it won't be a big deal - but do you have more details on the 
motivation behind ? If both connections failed, that would say either about the 
very sketchy connectivity of the target, or possibly that the source has lost 
their connectivity on that interface.

If the latter, then I think resetting would make sense under the assumption that 
we have a different attachment later. If that happens more often than either the 
target total unreachability or the 'spurious connectivity losses' (like, bad 
connectivity on wireless).

> ("normal"-P management) be just as bad?

I think this would optimize the case where we switched a link - but deoptimize 
the case where we had a one-time blip in the connectivity ?

> protocol, yet move towards zero, expressing that we are not as confident

I think in a much smaller part it might have the same property as my previous 
paragraph - but I think this might be worth experimenting with.

> a back-off mechanism as an "ordinary" TCP flow has. However, using a

I think not only SYN-spamming - but also accept()-spamming on the server side I think.

Arguably from the capacity planning a host should be able to support 2x 
accept()-load - but if this is normal scenario, that seems like a bit of a 
waste.

(This is a theoretical view 'in case everyone implemented this', obviously do 
not apply in the beginning)

> ratio of tcp/tcp-syn packets (overall and for www traffic).

While we're thinking radically different ideas: how about a "zebra retransmit".

Start with your "favourite" AF, and if you need to retransmit - then send the 
SYN for a different AF. This way the amount of SYNs will always be exactly the 
same. If more than 50% of the time over a certain period you fallback - change 
your favourite protocol.

This idea would get nuked by the purists, I suspect :-)

> per-destination P, hopefully with some nice aggregation feature too.

How is the PMTUD info stored ? I think that the connectivity 
characteristics may be related to the PMTUD. P might then sit nicely 
in one of the address families alongside with MSS ?

cheers,
andrew

---

From: Javier Ubillos <[email-masked]>
Date: Mon, 28 Feb 2011 13:57:37 +0100

> On 2011-02-25 09:16, Javier Ubillos wrote:
*cut*
> Over time, if IPvX maintains its advantage over IPvY, the switch will

I'm unsure about which approach would give the best user-value.
I'd be happier with a compromise, where we _do_ react quickly, but still
make use of the historic data.

Thinking about it a bit:
I'm seeing two extreme scenarios, given IPvX and IPvY connectivity:
a) We're using IPvX, one connection attempt fails (e.g. dropped syn).
    * Inverting (and dividing) P -> 
        If there is a large difference between IPvX/Y we run the risk
        having one connect() to give us a connection of worse service
        (IPvY).
        Or, if IPvY is so bad that IPvX still wins, we'll get IPvX +
        connection delay.
    * Decrementing P until it 'flips' ->
        If IPvX is better, we'll still get IPvX.

b) We're using IPvX, and IPvX connectivity dies 
    * Inverting (and dividing) P ->
        We switch over to IPvY w/o delay.
    * Decrementing P until it 'flips' ->
      We'll get IPvY connection after connection-delay + max(P)*10ms
(4seconds)

Conclusion.
In the "just missed out a packet" scenario, inverting P is just nuts, i
retract my invert-P suggestion ;)
In the "IPvX just died" scenario, I guess inverting P is 'better'. But I
imagine that scenario to be rarer, and that delays are more acceptable
in it.
Combined with some test for whether or not we have changed networks, it
ought to be negligible.

In other words, let's ignore my invert-P suggestion ;)

What I wanted to do was to minimize delays for the user.

> > * Why should we keep a P-value at all?
*cut*
> > Perhaps some other congestion-control for SYNs is more appropriate?
*cut* 

> to back that up). But your algorithm still tries to avoid SYN spamming.
Yes, the idea is to use some congestion-control on SYNs.

> But it assumes that SYN spamming is not an issue. Any idea on how we can

So, I've run tcpdump at home during the weekend.
TCP packets [tcpdump tcp and port 80]: 865890
TCP-syn packets (www, syn-flag only): 20833

That is, of _all_ tcp-packets, ~2.4% where SYN-packets.
W/o a P value, that would climb to ~4.81%

As a user, I don't care, but perhaps my ISP disagrees :)

> I have evidence that a global P is broken. I'll post it soon.

My intuitive guess is that a global P does 80% of the work. An IPv6
bottleneck is most likely to be located at the host (assuming a user
connects mostly to data-center located and well connected servers).

// Javier

---

From: Javier Ubillos <[email-masked]>
Date: Mon, 28 Feb 2011 14:53:04 +0100

On Fri, 2011-02-25 at 18:35 +0100, Andrew Yourtchenko wrote:
*cut*
> connectivity on wireless).

Perhaps that's a more sensible approach.
If the hosts IPs change, reset P (we *might* have changed networks).

> the case where we had a one-time blip in the connectivity ?

After some closer consideration.. Yes, you'r right.
I commented on that in the mail to/from Simon Perreault.
So, as mentioned above, perhaps a better method to achieve what I'm
trying to get at is to detect local address changes.

> not apply in the beginning)

Correct me if I'm mistaken, but...
If the responder dosen't allocate the TCB until it recieves the ACK
(third handshake packet), then the implementation shouldn't consume much
resources until that point (protecting against SYN-floods).
And, the initiator, shouldn't be sending two ACKs, but rather one ACK
and one RST (as it has chosen a winning socket).

> This idea would get nuked by the purists, I suspect :-)

Isn't the core problem that we don't want to start with AF_INET6 (our
favourite protocol) and have to wait for it to time-out before we switch
over to AF_INET?
Isn't it that which is what's causing service-providers to start use
whitelisting for their AAAA records?

> in one of the address families alongside with MSS ?

I'm not sure I understand what you're getting at.
The name-based socket will keep a readable P value. We could allow an
interface which could access AF_INET6 data (such as PMTUD).
In the happy-eyeballs case, we could also make an interface to get the
MSS and have it point to the active socket (IPv4/v6).

// Javier