Subject: Failover for a multihomed site with unreachable ISP

From: Christian Schild <[email-masked]>
Date: Wed, 26 Mar 2003 13:39:32 +0100

Hi ho,

inspired by the 'dual homing experiment' document of Christian Huitema, 
we (JOINers) thought about a solution to offer robustness for a site that 
is multihomed and multiaddressed and that we like to discuss here. 
Exteral connectivity of such a site is affected by failures of

- the sites border router
- the sites uplink
- the ISPs infrastructure (spec. the router with the link to the customer)
- the ISPs global border router
- the ISPs global uplink

To recover from such a failure in the fist three cases, the site could
communicate with the ISP a set of possible prefixes and connectivity could
get reestablished via a tunnel technology. We already thought about this,
but the solution is quite complex to explain it in short. It is not
discussed here.

The approach I try to explain here is a solution for the last two cases,
where there the ISP is no longer reachable and a tunneled solution is not
possible.

First, we consider a failure a seldom and abnormal event. Only if a direct
connect fails, the network (or the ISP) has to take some failover action.
This means that - if you think of the size of the global routing table - 
in default behaviour the table is small (only /32 prefixes) and only in 
case of a failure a more specific prefix (/48 or shorter) is neccessary.

                     +----------------------------+
                     |  'Global Internet'/'DFZ'   |
                     +--+-----------------------+-+
                        |                       |
                        |                       |
            +-----------+--+                 +--+-----------+
            |    ISP A     |                 |     ISP B    |
            | Prefix PA/32 |                 | Prefix PB/32 |
            +-----------+--+                 +--+-----------+
                        |                       |
                        |                       |
                       ++-----------------------++
                       |       Customer C        |
                       |      Prefix PAC/48      |
                       |      Prefix PBC/48      |
                       +-------------------------+

In this scenario customer C gets a /48 from every ISP (PAC/48 from ISP A
and PBC/48 from ISP B) and communicates the existance of these prefixes to
every provider (as mentioned earlier, how this is done is not explained
here). Thus, e.g. ISP A knows that it has a multihomed route to
customer C with prefix PBC/48.

Usually, all traffic from the outside to PBC/48 will go through ISP B.
If ISP B detaches from the DFZ now, ISP A has to announce to the DFZ
somehow, that it has a valid route to PBC/48. This can be done in two
different ways within (e)BGP:

Approach 1:
When ISP B detaches, it's BGP announcement of PB/32 will vanish from the
global routing table. ISP A's border router could use this as a trigger to
announce PBC/48 to the DFZ. It will remove this announcement when PA/32
reappears.

The advantage here is that the /48 will only be present in the DFZ, when
the usual routes fail. The disadvantages are bad convergence and possible
multiple routes. If ISP A and ISP B are very distant, it may take some
time until the /48 is known everywhere. If ISP B detaches from the DFZ
only for a split second - or even flaps - both prefixes will be visible
in the DFZ for some time.

Approach 2:
The second approach is more severe and requires a change in BGPs default
routing behaviour.

Usually BGP choses the route for a packet based on the longest prefix
match calculation. The suggestion here is, that (e)BGP - despite this
rule - chooses the _shortest_ prefix in the DFZ. This behaviour has to
be restricted to prefixed between /32 and /48. This restriction is
neccessary, because we only want this to happen in the routing area.
Within a site, longest prefix match should still be possible. And,
shortest match needs to be prevented for prefixes shorter than /32, else
anyone could create a _real_ black hole by announcing an /<32 to the
DFZ.

Assuming this behaviour of (e)BGP, in the above scenario ISP A could simply
announce PBC/48 to the DFZ. It will get announced to everyone, but it
should not get used in the forwarding table, cause the (now) better prefix
PB/32 exists. If now ISP B detaches and the prefix PB/32 is retrieved from
the routing table, every node in the DFZ will add the PBC/48 to the
forwarding table and the new route to the customer establishes.

Advantages here are the faster convergence. A disadvantage is the major
change in BGPs behaviour. This 'shortest path' criteria may be critical,
because calculation of the best route might get complex and expensive.
Also, again the routing table may grow large, but in return the
forwarding table will stay small. It is not know what other impact a
'shortest prefix' selection will have.

So long,
     Christian (Schild :-)

-- 
JOIN - IP Version 6 in the WiN  Christian Schild
A DFN project                   Westfaelische Wilhelms-Universitaet Muenster
http://www.join.uni-muenster.de Zentrum fuer Informationsverarbeitung
Team: [email-masked]      Roentgenstrasse 9-13
Priv: [email-masked]    D-48149 Muenster / Germany
GPG-/PGP-Key-ID: 6EBFA081       Fon: +49 251 83 31638, fax: +49 251 83 31653

---

From: marcelo bagnulo <[email-masked]>
Date: 26 Mar 2003 16:20:21 +0100

Hi Cristian

Wouldn't this be somehow similar to RFC 2260?

Regards, marcelo

-- 
marcelo bagnulo <[email-masked]>
uc3m

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 26 Mar 2003 23:09:20 +0100

Christian,

On woensdag, maa 26, 2003, at 13:39 Europe/Amsterdam, Christian Schild 

> case of a failure a more specific prefix (/48 or shorter) is 

I see two problems with this:

1. An ISP's aggregate route (the /32) disappearing from the global 
routing table
    is an extremely rare event. Last mile problems are infinitely more 
common,
    and after that there is the class of partial ISP failures (for 
instance,
    a single POP is partitioned from the network) where the aggregate 
remains
    visible.

2. Conditional announcement makes the global routing system less 
predictable,
    which is dangerous. The sudden appearance of thousands of more 
specific
    /48s could easily break systems that could just about cope with the 
regular
    aggregated situation.

---

From: Christian Schild <[email-masked]>
Date: Thu, 27 Mar 2003 07:57:31 +0100

Hi Marcelo,

> Wouldn't this be somehow similar to RFC 2260?

Hm, I don't think so. RFC 2260 seems to be similar but isn't the same. In 
fact, it could be used to help in one of the first three cases of failures I 
talked about.

But RFC2260 talks about connecting a site to an ISP (e.g. by extending BGP to 
the customer). In my mail I talk about how an ISP A could inject a customers 
prefix (that is part of another ISP B's /32) to the DFZ, without endangering 
the size of the routing table. This is not solved by RFC2260.

Christian

---

From: Christian Schild <[email-masked]>
Date: Thu, 27 Mar 2003 08:14:39 +0100

Hi Iljitsch,

>     is an extremely rare event. Last mile problems are infinitely more 

Well, rare is not zero :). We just want to cover that case. 

>     a single POP is partitioned from the network) where the aggregate 

Hm, if just one POP disappears, the ISP (and the customer) is still rachable,
we don't have a loss of connectivity and no action has to be taken.

>     /48s could easily break systems that could just about cope with the 

I see. But in our approach 2, the longer prefix is already present in the 
routing table. Only if the shorter prefix disappears, it will be added to the 
forwarding table. It is some kind of 'automatic aggregation'. Do you think it 
will be that extremely costly? Moreover I think, systems 'that could just 
about cope' should be replaced, and they shouldn't be a concern here.

So long,
     Christian

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 27 Mar 2003 08:44:50 +0100

On donderdag, maa 27, 2003, at 08:14 Europe/Amsterdam, Christian Schild 
wrote:

>> routing table is an extremely rare event. Last mile problems are 
>> infinitely more common,

> Well, rare is not zero :). We just want to cover that case.

Maybe a better solution would be a backup organization that announces 
the /32 if it disappears and tunnels the packets to the customers over 
the secondary ISPs.

>> remains visible.

> we don't have a loss of connectivity and no action has to be taken.

??? If the POP the customer connects to goes down, the customer isn't 
reachable over the addresses from that ISP anymore.

>> predictable, which is dangerous. The sudden appearance of thousands 
>> of more
>> specific /48s could easily break systems that could just about cope 
>> with the
>> regular aggregated situation.

> will be that extremely costly?

Not costly, dangerous.

> Moreover I think, systems 'that could just

If the routers must be able to handle the full set of unaggregated 
prefixes anyway, why bother aggregating? Maybe this can work in theory 
but not in practice. Many people will just filter the /48s to avoid 
problems.

---

From: Christian Schild <[email-masked]>
Date: Thu, 27 Mar 2003 09:49:57 +0100

> the /32 if it disappears and tunnels the packets to the customers over 

I'm not sure if I got your point here. Are you suggesting that another ISP 
has to take over all the traffic for the disconnected ISP? I hardly believe
this will ever happen in the real world :) Things like this could only be 
done administratively and we want a protocol-based solution. 

> reachable over the addresses from that ISP anymore.

Oh, this is our failure case 3 and shouldn't be considered here. A tunneled 
solution could handle this. (Please note that we are preparing a draft for a 
protocol based solution that could handle cases 1-3. Later :-). 
 
> prefixes anyway, why bother aggregating? 

Well, a large routing table is only a memory/msg-size problem. A large 
forwarding table also slows down the forwarding process. With automatic 
aggregation at least the latter is solved. 

> Maybe this can work in theory 

Well, if BGP's behaviour would change to aggregation, filtering of 
/48 is done automatically, and configured filtering of /48 is no longer 
neccessary. 

Christian

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 27 Mar 2003 10:36:15 +0100

On donderdag, maa 27, 2003, at 09:49 Europe/Amsterdam, Christian Schild 

> this will ever happen in the real world :)

It could. An ISP could set up a related, but independent backup 
network. Since this network would only have to peer or buy transit in a 
couple of locations and maintain a bunch of tunnels, this wouldn't be 
too costly.

> Well, a large routing table is only a memory/msg-size problem.

Memory scales linear so that's not the biggest problem. Processing 
scales worse than linear so in the end that will be the downfall of 
huge routing tables.

> A large

This isn't a protocol issue. Vendors can implement this today, but they 
choose not to.

---

From: marcelo bagnulo <[email-masked]>
Date: 27 Mar 2003 12:11:03 +0100

> fact, it could be used to help in one of the first three cases of failures I 

Agree, specially the tunnel part of the solution.

> prefix (that is part of another ISP B's /32) to the DFZ,

I understand that the difference would be that RFC 2260 proposes that
the end-site injects the alternative route information and in the other
case is the ISP that injects the address space assigned by the other ISP
to the end-site.
However, i am not sure that this is a feature of the alternative
proposal. I mean RFC 2260 does not requires coordination between ISPs
while the other proposal does.

>  without endangering 

I do not see any difference between the two proposals in this point
(probably i am failing to understand something)
I both cases a more specific prefix in injected when an outage occurs,
so both proposal contribute with an additional entry in the  DFZ routing
table during the outage (and no longer than that)
Am i missing something?

Regards, marcelo

---

From: Christian Schild <[email-masked]>
Date: Thu, 27 Mar 2003 13:08:34 +0100

> while the other proposal does.

No, neither does our proposal. The ISP A reacts on signals from the DFZ 
(vanishing of PB/32), the other ISP B is not directly involved. 
A feature may be, that it is not nesseccary to establish a BGP peering to the 
customer and therefore the customer doesn't need to have a non-private ASN. 

> table during the outage (and no longer than that)

You are right, I missed the conditional clause, that the customer injects a 
prefix _only_ when the other connection fails. But this may be a problem, as 
the customer may not always be able to detect a failure in the other ISP's 
global uplink (e.g. cause the BGP session customer<->ISP B is still 
established), while our proposal could handle that. 

So long,
     Christian

---

From: marcelo bagnulo <[email-masked]>
Date: 27 Mar 2003 13:44:38 +0100

[...]

> (vanishing of PB/32), the other ISP B is not directly involved.

Ok.
But the ISPB has to know which prefix to inject, but this may be
acceptable i guess...

> customer and therefore the customer doesn't need to have a non-private ASN. 

Yes, i think so.
However, the end-site needs some mechanisms to known how to route
packets. I mean, if there is an outage above ISPa, the end-site needs to
know that it must send packets through ISPb, how do you achieve that if
you do not use BGP?

Regards, marcelo

---

From: Christian Schild <[email-masked]>
Date: Thu, 27 Mar 2003 14:49:32 +0100

> know that it must send packets through ISPb, how do you achieve that if

Ok, well. This can be done with some kind of multihoming information exchange 
protocol, that signals failures and simultanously securely exchanges prefix 
information between the customer and the ISP, opens and closes secure tunnels 
between them and sets proper ingress filters. Don't worry, should we write it 
down in detail? :-)

Christian

---

From: marcelo bagnulo <[email-masked]>
Date: 27 Mar 2003 15:14:54 +0100

On Thu, 2003-03-27 at 14:49, Christian Schild wrote:
> information between the customer and the ISP,

I would say that this can be done using BGP, what advantages do find in
providing a new tool?  

>  opens and closes secure tunnels 
> between them and sets proper ingress filters.

I think this may be interesting...
Christian's Host centric proposal  deals with ingress filtering
problems, but i do not recall that this possibility is considered,
perhaps it would be interesting to study it (IMHO).

Regards, marcelo

>  Don't worry, should we write it 
> down in detail? :-)
> 
> Christian
>