Subject: [v6ops] Happy eyeballs implementation

From: Rashmi <[email-masked]>
Date: Wed, 19 Jan 2011 23:24:01 +0100

Hello,

I am Rashmi, a Master thesis student at KTH, Stockholm. I studied "Happy 
Eyeballs: Trending Towards Success with Dual-Stack Hosts 
draft-wing-v6ops-happy-eyeballs-ipv6-00" and I am very interested to 
know if there is any current implementation of "Happy Eyeballs".

-- 
Regards,
Rashmi Purushothama

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 17:57:14 -0500

> know if there is any current implementation of "Happy Eyeballs".

I've done a somewhat aggressive Happy Eyeballs implementation for an iPad ssh app that I use.   Its behavior is quite satisfactory.   It does not delay startup on either protocol--it just launches connections on both protocols at once, and uses the connection that comes back first.   This is usually IPv6 within the house, and often IPv4 when I'm out and abroad.

My implementation is not a generic library, though--it's part of the application.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 19 Jan 2011 21:01:49 -0200

I would be willing to collaborate in writing a generic implementation.

regards

Carlos

On Wed, Jan 19, 2011 at 8:57 PM, Ted Lemon <[email-masked]> wrote:
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

-- 
--
=========================
Carlos M. Martinez-Cagnazzo
http://www.labs.lacnic.net
=========================

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 18:06:33 -0500

> I would be willing to collaborate in writing a generic implementation.

It's a bit challenging; I used the Apple event-driven socket API, which made it easy, but to deliver functionality equivalent to the BSD socket API that did happy eyeballs would be quite challenging, I think.   You would have to make a lot of assumptions about how the application interacts with the network layer that would restrict the set of applications that could use your library.

What I'd really like to see would be happy eyeballs in Google Chrome and Firefox, neither of which currently appears to do it.   These are the applications that seem to me to suffer the most when they guess wrong on which stack to try first.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 19 Jan 2011 21:11:13 -0200

I was thinking more along the lines of a socket wrapper, similar to
what some "socksifiers" (software that makes your network client
"socks aware") do

I agree that modifying the BSD socket API would be quite a large job,
and probably it's too early to think about that and having a user-mode
implementation would help in gaining experience that could be applied
to changing kernel-level APIs later on

warm regards

Carlos

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 19:17:43 -0500

> to changing kernel-level APIs later on

Actually, the problem is that it's simply not as easy as you suggest, because the BSD sockets API is essentially synchronous at connection time.   Most applications are not prepared to deal with asynchronous behavior on connect.

This is not to say that doing an in-kernel implementation is easier--there are serious problems with burying the functionality in the kernel, and I actually think it's a bad idea.   What I _am_ saying is that you might get the most bang for the buck from fixing a couple of really high-profile applications, rather than trying to come up with a generic solution.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 19 Jan 2011 22:35:14 -0200

I see your point, and I mostly agree. We can further discuss the idea.

(BTW I never suggested it was easy :-) )

warm regards

Carlos

---

From: Fred Baker <[email-masked]>
Date: Wed, 19 Jan 2011 16:38:36 -0800

> This is not to say that doing an in-kernel implementation is easier--there are serious problems with burying the functionality in the kernel, and I actually think it's a bad idea.   What I _am_ say…

very possibly. That said, a generic solution in application space might well have the effect of testing out a kernel solution that was deployed later. In other words, put in a library function in that has an API similar to

	unsigned int connect_by_name (char *name, socket *socket, ...)

and returns either an open connection to the named system with an error code of zero or returns some other error code.

Viola - not a kernel function, but it could be, if appropriate, something that spawns several threads and then closes all except the one that gave it an open socket first, and finally (apparently synchronously) returns.

If the right thing happens - if random unnamed vendors all fix their OS's and applications - except in the case of OpenBSD, in time that problem will correct itself...

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 19 Jan 2011 22:20:10 -0200

(I never suggested it was easy :-) )

warm regards

Carlos

-- 
Carlos M. Martinez
LACNIC I+D
PGP KeyID 0xD51507A2
Phone: +598-2604-2222 ext. 4419

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 19:56:51 -0500

> Viola - not a kernel function, but it could be, if appropriate, something that spawns several threads and then closes all except the one that gave it an open socket first, and finally (apparently s…

I think this is a reasonable idea, but it actually brings up the reason why I think pushing this down into the kernel is a bad idea.   In the case of my ssh application, what I really want is to get all the way to mutual authentication before I give up on the other connection.   If all I get back from the kernel is the first connection to complete a mutual handshake, I can't do that.

Strictly speaking, there's no reason to use threads for this, by the way--you can just use select/poll/whatever.   I don't know precisely what Mac OS X is doing under the covers to give me my event-driven connects, but I don't get the impression that it's forking threads.   It's probably better not to use threads unless you're sure that the application is going to be threaded anyway.   But you could certainly do precisely what you suggested either with or without threads, and I think it would be a useful exercise.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 01:54:39 +0100 (CET)

Hi Rashmi,

On Wed, 19 Jan 2011, Rashmi wrote:

> draft-wing-v6ops-happy-eyeballs-ipv6-00" and I am very interested to know if 
> there is any current implementation of "Happy Eyeballs".

I've done the initial prototyping for the idea as part of the links browser.

To minimize the changes, I had tried the following approach:

1) using the posix threads to start two threads of execution ("v4" and "v6" one 
- both doing lookups and connect attempts)

2) in order to fool the rest of the code that the "connection has not happened 
yet", I initially returned the non-writable file descriptor from the pair that 
pipe(2) returns, and then when the connection succeeded, then I used dup2(2) to 
impose the successfully connected socket onto the old file descriptor.

(1) is a big no-no in a well-behaved library, and the (2), while initially it 
seemed to be a good parlor trick, turned out to be nonportable and incompatible 
with the behavior of the "true" not-connected-yet socket file descriptor - so I 
never bothered to let it see the light.

I think a LD_PRELOAD-style variation of the happy eyeballs can be made, with 
some ugly jumps over the hoops, here are my napkin notes:

a) as part of the initialization, fork a separate process that would do the 
"monitoring" part using threads, so that the threading structure of the client 
process were not altered.

b) intercept all the socket calls, and return the socket of the type (INET or 
INET6) that you think will most likely succeed (based on the current value of 
P).

c) Here be dragons: intercept all name resolutions. Thanks to blocking nature of 
the resolver functions, there are quite a few async resolver libs - so merely 
intercepting the syscalls would not be enough, you'd need to intercept and parse 
the udp/53 packets. Not too easy and very ugly hack.

d) the forked "miscellaneous" process would check whether the returned socket 
was indeed the correct one and if not - then would force the correct one upon 
the client by means of an injected signal handler + sendmsg. This is also a very 
ugly hack.

So in summary - I think it is technically possible, but the resulting code at 
least in my estimates may cause severe indigestion.

I'm not sure how well that is doable on Windows, but I know that antivirus 
programs are generally very sensitive to most of the actions I've described 
above, so it'd be a tough game to make it abstract enough.

A challenge in doing it in user space is gracefully gluing it into the different 
process and I/O handling models of the existing apps that assume the two 
separate stages "lookup the address(es)" and "connect to address" - they can be 
quite far apart in the applications sometimes.

cheers,
andrew

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 20:17:29 -0500

> quite far apart in the applications sometimes.

I don't see the point of this.   No matter how hard you try, your solution is going to be rickety.   Why not just fix the application?

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 02:16:28 +0100 (CET)

On Wed, 19 Jan 2011, Ted Lemon wrote:

> I think this is a reasonable idea, but it actually brings up the reason why I 
>think pushing this down into the kernel is a bad idea.   In the case of my ssh 
>application, what I really want is to get all the way to mutual authentication 
>before I give up on the other connection.   If all I get back from the kernel 
>is the first connection to complete a mutual handshake, I can't do that.
>
> Strictly speaking, there's no reason to use threads for this, by the way--you 
>can just use select/poll/whatever.   I don't know precisely what Mac OS X is 
>doing under the covers to give me my event-driven connects, but I don't get the 
>impression that it's forking threads.   It's probably better not to use threads 
>unless you're sure that the application is going to be threaded anyway.   But 
>you could certainly do precisely what you suggested either with or without 
>threads, and I think it would be a useful exercise.

The reason I used threads in my proof-of-concept code was that I wanted to stick 
to getaddrinfo() which does not lend itself to more than one in-progress lookup 
at a time. The motivation for sticking to getaddrinfo() was that 
non-RFC1035 things like EDNS0, DNSSEC, etc. are Operating System Problem.

But assuming the host who just needs to know the IPv4 or IPv6 address of a 
hostname can survive with simpler resolver, it may be beneficial to roll one's 
own. During the new year's holidays I looked at existing resolvers - and the 
resulting frustration resulted in the rubber duckie at 
https://github.com/ayourtch/ydns - which does not do anything terribly useful 
now apart from asking questions and parsing answers, but with some more work I 
think can be used as part of such a "connect_by_name" api that would be 
eventloop-compatible.

cheers,
andrew

---

From: james woodyatt <[email-masked]>
Date: Wed, 19 Jan 2011 17:28:03 -0800

On Jan 19, 2011, at 16:56, Ted Lemon wrote:
> 
> I don't know precisely what Mac OS X is doing under the covers to give me my event-driven connects, but I don't get the impression that it's forking threads.

As the source code is open, this shouldn't be a mystery.  The answer for the impatient is that it's a stack of abstractions built around a distinguished set of operations on file descriptors with the O_NONBLOCK flag set and judicious use of the kevent(2) system call.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 02:28:57 +0100 (CET)

On Wed, 19 Jan 2011, Ted Lemon wrote:

> I don't see the point of this.   No matter how hard you try, your solution is 
>going to be rickety.   Why not just fix the application?
>

Because sometimes the fix can require a massive refactoring.

For starters - a nonblocking DNS lookup.

Like I wrote, I could not find an async resolver library that does not try to 
surreptitiously launch interstellar shuttles from behind my back.

cheers,
andrew

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 02:09:12 +0100 (CET)

On Wed, 19 Jan 2011, Ted Lemon wrote:

> Actually, the problem is that it's simply not as easy as you suggest, because 
>the BSD sockets API is essentially synchronous at connection time.   Most 
>applications are not prepared to deal with asynchronous behavior on connect.

well, actually if you set O_NONBLOCK then you get EINPROGRESS on connect and 
have to keep poll()-ing/select()-ing - which most of the apps that deal with 
more than one socket do just fine.

In my experience the problem was less with the asynchronous behaviour per se, 
but with:

1) giving back a "neigher v4-nor-v6" fd back to socket() - since the connect() 
was far far far away yet.

2) convincing the resolver-du-jour that I can do it instead.

3) fabricating the "connection in progress" condition on that fake fd from (1) 
while I do the lookups and connection attempts.

As I wrote to Rashmi - I think (1) and (3) might be mitigated by overriding the 
domain of the socket requested by the app with the one that you *think* may 
likely succeed based on the past behaviour, and if it does not, then shoehorning 
the one that does using dup2() call, but I haven't tried this yet.

>
> This is not to say that doing an in-kernel implementation is easier--there 
> are serious problems with burying the functionality in the kernel, and I 
> actually think it's a bad idea.

yes and no. Doing anything than the blocking 'hello world' style enumeration of 
addresses (after the slow-ish getaddrinfo succeeds, oh by the way, it also does 
not seem to cache the addresses) is a royal pain.

And the async DNS resolvers out there all imagine that they are the most 
important part of the program. Maybe there's just no other solution.

> What I _am_ saying is that you might get the 
> most bang for the buck from fixing a couple of really high-profile 
> applications, rather than trying to come up with a generic solution.

Absolutely agree.

I think putting this into FF + Chrom(e|ium) would bring a lot of value.

cheers,
andrew

---

From: Fred Baker <[email-masked]>
Date: Wed, 19 Jan 2011 18:18:57 -0800

> I don't see the point of this.   No matter how hard you try, your solution is going to be rickety.   Why not just fix the application?

Because if we fix the application, we have to fix every application individually. If we can get it into the OS or into a library that looks like the OS, we still have to convince existing applications to to use it, but we can debug it once and future applications don't need to ask the question.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Jan 2011 21:35:13 -0500

> Because if we fix the application, we have to fix every application individually. If we can get it into the OS or into a library that looks like the OS, we still have to convince existing applicati…

I think you and I are talking about different things.   I think you are talking here about your connect_by_name() api, right?   I don't entirely agree that that's the right API, but I agree in principle that an API is a good idea, for just the reason you've stated.

What I'm arguing probably isn't going to work is a shim that works with an application that calls gethostbyname() and then connect().   I can imagine it working in some cases, but it would almost certainly also cause instability.

---

From: Simon Perreault <[email-masked]>
Date: Thu, 20 Jan 2011 08:46:45 -0500

On 2011-01-19 20:16, Andrew Yourtchenko wrote:
> The reason I used threads in my proof-of-concept code was that I wanted
> to stick to getaddrinfo() which does not lend itself to more than one
> in-progress lookup at a time.

Strictly speaking, it is impossible to properly implement happy eyeballs
in terms of getaddrinfo() (with or without threads). That's because in
case getaddrinfo() performs multiple queries (e.g. a DNS A query and a
DNS AAAA query), it will only return when all queries have completed.
This prevents you from initiating connections as soon as you have any
answer.

With or without threads, a good happy eyeballs implementation needs
something less synchronous than getaddrinfo().

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 18:26:31 +0100 (CET)

> This prevents you from initiating connections as soon as you have any

What I was doing was firing up two threads:

thread 1:
   af =  AF_INET
thread 2:
   af = AF_INET6;

both threads, after:
   hints.ai_family = af;
   hints.ai_socktype = SOCK_STREAM;
   hints.ai_flags = 0;
   hints.ai_protocol = 0;
   s = getaddrinfo(node, service, &hints, &result);

   // try to connect here.

Could be that I was wrong in my tests that showed the two lookups in two threads 
were running ok. But setting the hints does appear to limit the lookups to only 
one AF. (here's the above snippet wrapped into the proper main(): https://gist.github.com/788223)

> something less synchronous than getaddrinfo().

Definitely cleaner that way, yes. Not that one can't abstract away the 
getaddrinfo() by having a pool of fork()ed blocking workers, but that design 
would not win the beauty contests.

cheers,
andrew

---

From: Simon Perreault <[email-masked]>
Date: Thu, 20 Jan 2011 13:02:13 -0500

>   s = getaddrinfo(node, service, &hints, &result);

It's a good approximation.

But, again strictly speaking, it doesn't take into account the fact that
getaddrinfo() will potentially look into places other than DNS. For
example, the well known /etc/hosts file will return results much faster
than DNS. Or other sources, as configured in e.g. /etc/nsswitch.conf
(e.g. SMB names). Even only considering DNS there is the possibility
that the answer section could span multiple packets.

So it's all these corner cases that speak to the need for a truly
asynchronous getaddrinfo(), one that could be used with or without threads.

Simon

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 20:03:20 +0100 (CET)

> that the answer section could span multiple packets.

Well, in this case it'd be the getaddrinfo()'s problem, not mine ? (that's 
precisely why I hoped to offload all these details to getaddrinfo which is 
already supposed to take care of them.)

> asynchronous getaddrinfo(), one that could be used with or without threads.

Ideally, asynchronous connect_by_name(), yes. Async getaddrinfo() could be a 
good start to that - though it would solve half of the problem.

cheers,
andrew

---

From: Simon Perreault <[email-masked]>
Date: Thu, 20 Jan 2011 14:15:15 -0500

On 2011-01-20 14:03, Andrew Yourtchenko wrote:
> Well, in this case it'd be the getaddrinfo()'s problem, not mine ?
> (that's precisely why I hoped to offload all these details to
> getaddrinfo which is already supposed to take care of them.)

My point is that getaddrinfo() will only return once it has *all* the
results it could get, from *all* possible sources. What we'd like
instead is to be notified of results as they come in so that we can
initiate connections as soon as possible.

> Ideally, asynchronous connect_by_name(), yes. Async getaddrinfo() could
> be a good start to that - though it would solve half of the problem.

Agreed.

(But I think the async getaddrinfo() is the hard part. The rest is
already possible (and fairly easy) using regular BSD sockets.)

Simon

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 20 Jan 2011 20:34:00 +0100 (CET)

> initiate connections as soon as possible.

Not for the Happy Eyeballs case - there we'd want two two threads. Blue pill 
thread where the getaddrinfo() does only the IPv4-related lookups and then 
connects, and red pill, where the getaddrinfo() does only IPv6-related lookups 
and then connects.

Within each of these parallel worlds the reply should come in a single 
packet - so we can start connecting.

The fact that we want two lookups is dictated by trying to avoid an extra 
lookup and a connection attempt if we have a reasonable assurance it won't work 
anyway.

> already possible (and fairly easy) using regular BSD sockets.)

Yes. The only detail (which I feel we also agree on) is that it is boring 
to have to write the code to start the connection attempt after the async DNS 
lookup result comes back.

I have a hostname in the config or entered by the user, and I want to get a 
working connection. I'd be happy to get an fd that is "EINPROGRESS" until what 
needs to happen happens somewhere in the background.

cheers,
andrew

---

From: Mohacsi Janos <[email-masked]>
Date: Fri, 21 Jan 2011 09:05:15 +0100 (CET)

On Thu, 20 Jan 2011, Andrew Yourtchenko wrote:

> Could be that I was wrong in my tests that showed the two lookups in two 
> threads were running ok. But setting the hints does appear to limit the 
> lookups to only one AF. (here's the above snippet wrapped into the proper 
> main(): https://gist.github.com/788223)
>
>

I don't see the point in your implementation. The speed of the DNS lookup 
for specific records does not have any relation to speed of connectivity 
or responsiveness for any specific protocol. I think what can help:
1. do DNS lookup via getaddrinfo()
2. do parallel connection for all the IPv4 and IPv6 address returned.
3. first succesful - keep it. terminate all connection attempts for the 
others.

Best Regards,

Janos Mohacsi
Head of HBONE+ project
Network Engineer, Deputy Director of Network Planning and Projects
NIIF/HUNGARNET, HUNGARY
Key 70EF9882: DEC2 C685 1ED4 C95A 145F  4300 6F64 7B00 70EF 9882

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 21 Jan 2011 10:47:39 +0100 (CET)

On Fri, 21 Jan 2011, Mohacsi Janos wrote:
>
> I don't see the point in your implementation. The speed of the DNS lookup for 
> specific records does not have any relation to speed of connectivity or 
> responsiveness for any specific protocol. I think what can help:

Take a look at

https://bugzilla.redhat.com/show_bug.cgi?id=505105

for an example why.

cheers,
andrew

---

From: Mark Smith <[email-masked]>
Date: Fri, 21 Jan 2011 21:20:06 +1030

<snip>

SCTP being attempted / used in any of these implementations? I'm dealing
with a few IPv6 residential CPE implementations who's firewalls still
believe the only transport protocols are TCP and UDP. It'd be nice to
be able to show that other transport protocols are available and
useful once NAT isn't in the picture.

<snip>

Thanks,
Mark.

---

From: Mohacsi Janos <[email-masked]>
Date: Fri, 21 Jan 2011 12:28:52 +0100 (CET)

Dear  Andrew,

On Fri, 21 Jan 2011, Andrew Yourtchenko wrote:

>> for specific records does not have any relation to speed of connectivity or 

For example:

- Broken Fedora Core 11 gai implementation using same source port- Fedora 
needs to be fixed.

- Broken stateful inspection of the DNS UDP traffic in some 
firewalls/load-balancers - broken firewalls/load-balancers needs to be 
fixed.

But beware brokenness of AAAA DNS resolution nothing to do with brokenness 
of IPv6 connectivity - which is is the original intention of happy eyeball 
draft.
 	Best Regards,
 			Janos

---

From: Pekka Savola <[email-masked]>
Date: Fri, 21 Jan 2011 13:54:28 +0200 (EET)

On Fri, 21 Jan 2011, Mohacsi Janos wrote:
> But beware brokenness of AAAA DNS resolution nothing to do with brokenness of 
> IPv6 connectivity - which is is the original intention of happy eyeball 
> draft.

The eyeball won't be happy when s/he tries to connect to an IPv4-only 
destination, it fails.

But I agree that multiple and different mitigation methods for various 
error conditions is another and possibly useful approach to tackle the 
overall problem.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Fri, 21 Jan 2011 11:53:12 +0000 (UTC)

Dear Andrew,

> https://bugzilla.redhat.com/show_bug.cgi?id=505105

That is a rather bad example to show people for your case.  Do you
really want to

1) have people having a good experience,
2) work around obvious brokeness in device for standards compliant
    queries,
3) show that despite how much people (grudgingly) try to work around
    brokeness, brokeness will stay, and
4) at the same time add more complexity to the procotol/standards
    just asking for more problems?

The result of that will be obvious.

Ignoring the above sample, I see the point that people think that the
sum of DNS lookup time + connect time is a good metric for better user
experience.  I am not saying I confirm that.

/bz

-- 
Bjoern A. Zeeb                                 You have to have visions!
         <ks> Going to jail sucks -- <bz> All my daemons like it!
   http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 21 Jan 2011 13:05:23 +0100 (CET)

Dear Mohacsi,

On Fri, 21 Jan 2011, Mohacsi Janos wrote:

> firewalls/load-balancers - broken firewalls/load-balancers needs to be fixed.
>

No, the original intent of the happy eyeballs is improving the end-user 
experience.

As a user I do not care if it is the AAAA lookup that fails or the connect() 
that fails. They are done in sequence, and the end result is exactly the same, 
right ?

cheers,
andrew

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 21 Jan 2011 13:13:10 +0100 (CET)

> useful once NAT isn't in the picture.

SCTP was not in the proof of concept code that I was doing - but there was a 
separate effort on it that was demoed right after the happy eyeballs 
presentation at IETF75 - if my memory serves me well, it was in the TSVAREA 
meeting.

cheers,
andrew

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Jan 2011 14:19:22 +0000

On Jan 21, 2011, at 7:05 AM, "Andrew Yourtchenko" <[email-masked]> wrote:

> As a user I do not care if it is the AAAA lookup that fails or the connect() that fails. They are done in sequence, and the end result is exactly the same, right ?

Right. My reaction to this bug report (which is based on skimming it, so I may have missed some detail) is to use a different socket for every query to avoid triggering broken middleboxes. Of course, this creates new brokenness.  But the bottom line is that all we can ever do is try to make things work, and observe the ways they fail to work, and try again.

---

From: Mohacsi Janos <[email-masked]>
Date: Fri, 21 Jan 2011 16:14:06 +0100 (CET)

> No, the original intent of the happy eyeballs is improving the end-user 

Does you intention also to improve IPv4 copnnectivity brokenness?

> that fails. They are done in sequence, and the end result is exactly the 
> same, right ?

Then I think the algorithm is slighlty more complex:

1. define list_of_sockaddr
2. do_parallel lookup for_each address_family with timeout < tolerable_for 
end_users/2
----     if (found_result) list_of_sockaddr.append(lookup_result)
3. do_parallel connect() for_each list_of_sockaddr with timout < 
tolerable_for_end_users/2
----     if (connected) terminate all other_connect_threads
4. communication as usual

Also causes problem of not possible to honor RFC 3484 policy tables....

Best Regards,
 		Janos Mohacsi

---

From: Jack Bates <[email-masked]>
Date: Fri, 21 Jan 2011 09:39:29 -0600

On 1/21/2011 9:14 AM, Mohacsi Janos wrote:
> 2. do_parallel lookup for_each address_family with timeout <
> tolerable_for end_users/2

Actually, 2/3 should be combined. What should be parallel is the 
combined lookup/connect. First protocol to make connect wins.

> Also causes problem of not possible to honor RFC 3484 policy tables....

While it will circumvent the v4/v6 policy, it should still follow the v6 
policy itself. Not sure if this can be done with the mechanism you're 
wanting to do, but the v6 address selection should be utilized.

Jack

---

From: Mohacsi Janos <[email-masked]>
Date: Sat, 22 Jan 2011 09:47:18 +0100 (CET)

On Fri, 21 Jan 2011, Jack Bates wrote:

> Actually, 2/3 should be combined. What should be parallel is the combined 
> lookup/connect. First protocol to make connect wins.

You can't, since more than one IP address per address family possible. E.g 
if a particular name has 5 A record and 2 AAAA, then the point 3 should be 
done in 7 thread, while point 2 only in 2 threads.

> policy itself. Not sure if this can be done with the mechanism you're wanting 
> to do, but the v6 address selection should be utilized.

RC 3484 is protocol independent. You can setup RFC 3484 policy table for 
prefer ipv4 for dual-stack servers ....

Best Regards,
 		Janos Mohacsi

---

From: Jack Bates <[email-masked]>
Date: Sat, 22 Jan 2011 12:05:24 -0600

On 1/22/2011 2:47 AM, Mohacsi Janos wrote:
> RC 3484 is protocol independent. You can setup RFC 3484 policy table 
> for prefer ipv4 for dual-stack servers ....
>

It can, but it should still be taken into account. Address selection is 
critical in IPv6 within the protocol itself, not counting address 
selection cross protocol. Ignoring it completely will break some IPv6 
connections that otherwise might have succeeded faster than IPv4.

Jack

---

From: Mark Smith <[email-masked]>
Date: Sun, 23 Jan 2011 08:23:28 +1030

On Wed, 19 Jan 2011 21:35:13 -0500

> What I'm arguing probably isn't going to work is a shim that works with an application that calls gethostbyname() and then connect().   I can imagine it working in some cases, but it would almost c…

It may not be ideal, however if hiding it behind getaddrinfo() et al.
and connect() provides a better experience than the vanilla versions
would today, then it is worth while I think.

Regards,
Mark.

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Sat, 22 Jan 2011 22:09:18 +0000 (UTC)

> What I'm arguing probably isn't going to work is a shim that works with an application that calls gethostbyname() and then connect().   I can imagine it working in some cases, but it would almost c…

Yes, that certainly will not work as gethostbyname() [usually] is IPv4 only
and should probably not be used in any new code.  Thanks.  [1]

/bz

References:

[1] http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap03.html#tag_22_03_01_08

---

From: Andrew Yourtchenko <[email-masked]>
Date: Sun, 23 Jan 2011 10:57:31 +0100 (CET)

Dear Bjoern,

> The result of that will be obvious.

This case only shows that by doing the parallel threads of DNS+connect within 
the address family we avoid that sort of nastiness for free.

The reason it came to my mind because I was helping to fix the box in the 
middle that was dropping the AAAAs.

Re. the complexity and brokenness - I think Ted summed it up great in his other 
mail.

> experience.  I am not saying I confirm that.

What metric would you have in mind ?

cheers,
andrew

---

From: Andrew Yourtchenko <[email-masked]>
Date: Sun, 23 Jan 2011 11:34:45 +0100 (CET)

On Sat, 22 Jan 2011, Mohacsi Janos wrote:
> You can't, since more than one IP address per address family possible. E.g if 
> a particular name has 5 A record and 2 AAAA, then the point 3 should be done 
> in 7 thread, while point 2 only in 2 threads.

We are not making ourself the point to connect with *the* specific address 
family. We are making the point to have a working TCP connection, be it IPv4 or 
IPv6. But above is an interesting idea - I am though curious how, if 16 
addresses are returned, you avoid opening 16 connections.

A suggestion.

Since this thread started with a question about running code, 
yesterday I had a little fun and made eyewear for happy eyeballs:

https://github.com/ayourtch/gncci : "general network connections contortion 
interface".

It gives the ability to do anything in the middle between the application and 
the Linux socket layer. In a high-level language that Lua is. So prototyping 
something Happy Eyeballs-like in this middle layer is not too difficult.

The other reasons that I made it this way one can induce the "intentional 
breakage" via this middle layer and see how the application behaves, or do 
parental control proxy, or whatever else shiny. It's just an eyewear.

Feel free to play with it and send me the pointer to the code you come up with.

There may still be bugs of course since I've done the entire thing in a single 
sitting - the fixes are appreciated.

> prefer ipv4 for dual-stack servers ....

I tried to do it on my linux box, but failed... Do you have any advise ?

(short of disabling the IPv6 of course:)

cheers,
andrew

---

From: Mohacsi Janos <[email-masked]>
Date: Sun, 23 Jan 2011 14:33:39 +0100 (CET)

> I tried to do it on my linux box, but failed... Do you have any advise ?

Try to setup  ::ffff:0:0/96    with higher precedence than any ipv6 
precedence. Typically 50 is the largest precedence if they use the 
default RFC 3484 policy table, therefore 60 or 100 will suffice. 
Best Regards,
 		Janos Mohacsi

---

From: Jack Bates <[email-masked]>
Date: Mon, 24 Jan 2011 08:50:07 -0600

On 1/23/2011 3:57 AM, Andrew Yourtchenko wrote:
> This case only shows that by doing the parallel threads of DNS+connect
> within the address family we avoid that sort of nastiness for free.
>
> The reason it came to my mind because I was helping to fix the box in
> the middle that was dropping the AAAAs.
>

I've contacted three companies who used broken load balancers that 
returned nxdomain on AAAA but a response for A (ie, when AAAA nxdomained 
the resolver didn't ask for A). All companies fixed the problem in a 
reasonable time.

I agree and understand connect() reachability and performance. I 
disagree with circumventing buggy DNS that needs to be fixed.

Jack

---

From: Jeroen Massar <[email-masked]>
Date: Mon, 24 Jan 2011 16:03:29 +0100

> disagree with circumventing buggy DNS that needs to be fixed.

That is because you are talking about three companies (the bbc was once
also one of them btw ;) People on this list are talking about thousands
upon thousands of CPE devices which have the same annoying behavior of
just dropping AAAA records to the floor and then you get a time out and
then the user thinks it is all broken.

One good thing about that problem is that it occurs the moment one
IPv6-enables a host, it does not happen because one introduces AAAA
records for a host into DNS.

As such, except for broken connectivity this whole "World IPv6 day"
experiment will not going to break too much unless a lot of people
suddenly start enabling IPv6 stacks on their hosts because then the
broken connectivity will show up and of course the broken DNS recursors.

Greets,
 Jeroen

---

From: Jack Bates <[email-masked]>
Date: Mon, 24 Jan 2011 09:52:33 -0600

> then the user thinks it is all broken.
I see. I haven't experienced that issue yet, and the helpdesk has a 
policy of never disabling IPv6 (except for dialups, as v6 screws up 
dialing into a v4 only modem bank).

> records for a host into DNS.
I don't see this as a good thing. IPv6 is enabled on Windows Vista and 
above automatically. At some point, I expect Windows updates to activate 
all XP machines.

> broken connectivity will show up and of course the broken DNS recursors.

A decent percentage of my customer base is now running Windows Vista/7 
with default enabled IPv6. In addition, many of them have walmart 
special routers. A quick sampling of my recursive resolvers shows 
roughly a 1:9 ratio AAAA to A.

Jack Bates

---

From: Jeroen Massar <[email-masked]>
Date: Mon, 24 Jan 2011 17:04:28 +0100

On 2011-01-24 16:52, Jack Bates wrote:
[..]
> above automatically.

In the case where one enables an IPv6 stack on Windows + receive IPv6
connectivity AAAA resolving starts to happen. As such, this will then
directly expose people to the AAAA-resolving bug in their CPE devices.
That is a good thing. As it does not depend on a remote site enabling
IPv6 in DNS, it all depends on local problems which also can be resolved
locally.

> At some point, I expect Windows updates to activate all XP machines.

As XP is effectively EOL don't expect that to happen. MS wants people to
upgrade and I also think that is the right thing to do if you look at
the differences in the stacks.

> roughly a 1:9 ratio AAAA to A.

That might mean that 1:9 of your customers actually have IPv6
connectivity ;)

Greets,
 Jeroen
   (who is still faithfully using Windows XP ;)

---

From: Andrew Yourtchenko <[email-masked]>
Date: Mon, 24 Jan 2011 16:49:21 +0100 (CET)

On Mon, 24 Jan 2011, Jack Bates wrote:

> I've contacted three companies who used broken load balancers that returned 
> nxdomain on AAAA but a response for A (ie, when AAAA nxdomained the resolver 
> didn't ask for A). All companies fixed the problem in a reasonable time.
>
> I agree and understand connect() reachability and performance. I disagree 
> with circumventing buggy DNS that needs to be fixed.
>

Jeroen put it in his mail very well, I'll just add my own 2 cents for what it's 
worth.

The subscribers of this list should have all the possible sharp edges sharpened 
so they could see what breaks.

But the "casual user" folks who type "facebook login" into search engine to get 
onto facebook 
(http://www.readwriteweb.com/archives/facebook_wants_to_be_your_one_true_login.php)
are not going to fix the DNS.

They will either call their SP (who, if they are after a quick cost-savings, 
will tell then "disable IPv6"), or find themselves on the web "internet slow ? 
disable IPv6!", and disable it.

This is not a large problem support cost-wise, but it increases the end-user 
resistance towards moving to IPv6 - they have read it on the interwebs that it 
is harmful and have a proof of it - when they disabled it, it started to work 
faster. And maybe the ISP got a few calls as well.

This is hard to measure from the "far end" by anything other than 
http://www.google.com/trends?q=disable+ipv6 and handwaving, so treat this for 
what it is.

But it seemed that even if the above were an empty worry, splitting the query 
into A and AAAA would not be catastrophic (especially since we already have two 
parallel paths - one for A and one for AAAA).

If the above were to be correct, then we would not help too much those users who 
happen to have the broken CPEs - making this only half as useful.

Again, the first passage is guesses all the way down - so any extra clue from 
ISP folks who have the end-user experience in this area is appreciated.

I am not extra strong about this point - just wanted to clarify what went into 
the decision, FWIW.

cheers,
andrew

---

From: Jack Bates <[email-masked]>
Date: Mon, 24 Jan 2011 12:21:46 -0600

> That might mean that 1:9 of your customers actually have IPv6

I'm not completely sure on this. For example, doesn't the stack query 
for AAAA to support 6to4, even if 6to4 is broken by a NAT device?

This is the only scenario I can think of that would cause a broken CPE 
to be an issue (ie, why query for AAAA if we can't get a global 
assignment from the CPE?).

It would also break connectivity (which happy eyeballs would fix 
connection wise) if the remote site had a 6to4 AAAA record (which most 
policies will choose by default over IPv4).

I'm rusty on the RFCs, but a timeout mechanism should have been built 
into AAAA queries similar to how we stale out bad resolvers. Given 
current state, I can see how issuing dual queries and taking the first 
response could be useful, though I recommend that the response be given 
a window with IPv6 preference (say 50ms?). This would allow for the AAAA 
response to return slightly slower but still be applicable) and a low 
enough timeout to not register human perception. Address selection 
policies, when receiving both responses, or even within a protocol 
should be taken into account (ie, am I using the 6to4 address, the 
globally assigned address, the toredo address). It also applies when 
dealing with IPv4. I disagree with connecting to all addresses within a 
protocol with all available addressing. It arbitrarily increases load on 
remote servers. Pick 1 IPv4 address and pick 1 IPv6 address to connect to.

Jack

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 25 Jan 2011 16:32:31 +0100

> for AAAA to support 6to4, even if 6to4 is broken by a NAT device?

Depends on the stack, afaik Win7 at least does some reachability test to
determine if 6to4 can be properly used.

> assignment from the CPE?).

6to4 can be considered a global assignment ;)

> policies will choose by default over IPv4).

Anyone publishing 6to4 AAAA records is doing something they should not
be doing.

6to4 is great for temporary connectivity but if anybody wants to do
something more long term they should arrange proper connectivity.
If one is going to publish AAAA records then one is thinking long term
and one should not be using 6to4. (It is just too difficult to debug and
there are too many breakage scenarios IMHO)

> into AAAA queries similar to how we stale out bad resolvers.

There is a timeout, it is about 30 seconds and is general for any DNS
queries. This is why 'IPv6 makes the internet slow', because of that
timeout. Disabling the IPv6 stack disables the AAAA query and presto
'the internet is fast'...

> Given
> a window with IPv6 preference (say 50ms?).

50ms is too much for a site like Google who want instant results.. of
course when the DNS record is cached locally the repeats will be coming
in at a near zero delay. Thus might be an option but still.

The happy eyeballs stuff is all fine, the big problem is getting that
code distributed everywhere. Unless all the major vendors push this as a
security fix it won't change anything. Oh and don't forget to fix all
the code. Of course, patching getaddrinfo() to do this will go quite a
long way, except for the connect() time outs in the loop following it.

Maybe a longer-term goal is a better API, which is just
makeconnect(hostname, service) instead of doing separate resolves?

Still, that requires all software to be upgraded, and just check the
history of how long it takes to get IPv6 support in a lot of
applications, we don't have another 10 years to get this out there.

Greets,
 Jeroen

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Jan 2011 17:41:00 +0100 (CET)

> applications, we don't have another 10 years to get this out there.

Since a lot of them still don't have it, perhaps rapidly developing a new 
API would mean the ones who already don't have it will go for the new one 
instead?

Even if it takes longer and it's not an short term help, I still feel that 
a new API that does happy eyeballs is of value.

-- 
Mikael Abrahamsson    email: [email-masked]