Subject: Re: A tunneling proposal

From: Daniel Senie <[email-masked]>
Date: Mon, 16 Jul 2001 18:01:01 -0400

At 05:40 PM 7/16/01, Ramakrishna Gummadi wrote:
>I hypothesize that the only scenario where tunneling for TCP
>and UDP fails is when the entire ISP is affected in a major way.

Given economic conditions and the relative health or lack thereof of some 
providers, this alone is a significant concern. If a network is powered 
down by their bankers, customers who were multihomed via tunneling would be 
out of luck. This isn't the type of redundancy that is going to make people 
sleep well at night.

People want to multihome so that any outage, from a local loop to a 
backbone carrier can occur without obliterating their connectivity. I think 
it important to keep that in mind.

Tunneling is quite useful for fixing temporary problems, but I'm not 
convinced it's a worthwhile solution to the multihoming problem.

-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.                    http://www.amaranth.com

---

From: Daniel Senie <[email-masked]>
Date: Mon, 16 Jul 2001 18:35:24 -0400

At 06:20 PM 7/16/01, Ramakrishna Gummadi wrote:
>One can think of a simple extension to tunneling under such ISP-wide
>outages---if tunnel creation fails, the second ISP initiates
>non-aggregatable route announcements for the prefixes from the address
>space of the first ISP who has become unreachable. This two step approach
>may prove effective for fixing both small-scale temporary and large-scale
>persistent problems.

This could work, provided the alternate upstream ISPs are willing, and 
other networks are willing to take the announcements. The concern is that 
RADB and similar are used by providers to restrict where announcements will 
come from. Keep in mind also that what we're talking about here is human 
intervention in the case of an outage. The question to be answered by 
prospective users of such a multhoming solution is how many hours of outage 
can be tolerated? I suspect it unlikely that automated tunnel creation and 
announcement setup would be well received by those running backbones, so 
there's likely to be a human element at play.

>It may be a good idea to examine a breakdown of the kind of failures
>experiences within an ISP, because this will help judge the effectiveness
>of the tunneling approach. But the primary motivation in tunneling is not
>to introduce unnecessray routes into the DFZ if there is an alternative.
>For example, I know that, in the context of server farms, about 20% of
>outages and errors are because of local power failures, but this may not
>hold in the ISP case. Does anyone have any info about the main problems
>and their frequencies?

Where's that 20% figure come from? With all the backups the colo vendor 
salesfolk talk about, that's a surprising figure.

While I understand the desire to find ways to avoid introducing routes into 
the DFZ, I firmly believe any solution should provide an automated 
resilience in the face of outages. I'm not yet convinced there's a good way 
to do this with tunnels.

>On Mon, 16 Jul 2001, Daniel Senie wrote:
>

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Mon, 16 Jul 2001 15:20:07 -0700 (PDT)

thanks,
ramki

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Mon, 16 Jul 2001 15:43:41 -0700 (PDT)

> there's likely to be a human element at play.

Is it not possible to create a *static* entry in the RADB or other policy
database permitting each of a site's ISPs to advertise the site's prefixes
drawn from other the address space of other ISPs? I understand it is not
straightforward, but because it is static, perhaps it can be automated...

> salesfolk talk about, that's a surprising figure.

I heard this number from Jim Gray's seminar talk at UC Berkeley when he
was analyzing systems (databases, in particular) failures. But this was
from a couple of years ago....

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 17 Jul 2001 01:35:10 +0200 (West-Europa (zomertijd))

On Mon, 16 Jul 2001, Daniel Senie wrote:

> >space of the first ISP who has become unreachable.

Speaking from operational experience: this is not going to work. As a
multihomed site, I have to regularly check if I can still reach the entire
internet when one line goes down, because it wouldn't be the first time
someone placed the wrong filters at the wrong place. If you don't test
something like this, it's not going to work when you need it.

Also, when huge amounts of more specific prefixes appear, this could be just
the kind of thing to drive a lot of routers over the edge, especially when
we're getting close to a common limit, such as 128 MB in Cisco routers.

And when you can announce more specifics some of the time, why not make
"some" "a lot" or even "all"? And then people will start to filter and we're
back to square one.

> salesfolk talk about, that's a surprising figure.

Power is getting less and less stable these days. Those anti-globalisation
people do have a point, you know.

I got this from http://www.apcc.com/power/problems.cfm

"Power problems are the largest cause of data loss"

Power Failure/Surge: 45.3% 
Storm Damage: 9.4% 
Fire or Explosion: 8.2% 
Hardware/Software Error: 8.2% 
Flood & Water Damage: 6.7% 
Earthquake: 5.5% 
Network Outage: 4.5% 
Human Error/Sabotage: 3.2% 
HVAC Failure: 2.3% 
Other 6.7% 

Source: Contingency Planning 

So power problems are about 8 times as common as earthquakes. That's not so
bad.

Iljitsch van Beijnum

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Mon, 16 Jul 2001 17:20:32 -0700 (PDT)

> So power problems are about 8 times as common as earthquakes. That's not so

The following excerpt is from the results of the web-polygraph cache
benchmarking bake-off:
(http://www.measurement-factory.com/results/public/cacheoff/N03/report.by-alph.html)

----------------------
 The IBM-230 entry was unable to complete the downtime test. The system
requires manual intervention to boot up. IBM informed us that the product
is designed to never lose power. It has triple-redundant power supplies.
----------------

It costs $25,000, BTW. I wonder why they did not anticipate that
someone can pull a plug by mistake...

-ramki

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Mon, 16 Jul 2001 17:07:32 -0700 (PDT)

> something like this, it's not going to work when you need it.

In this case, at least, I don't see a big scalability problem as long as
everyone tests randomly, and the routers forward using longest prefix
match.

> we're getting close to a common limit, such as 128 MB in Cisco routers.

It may be useful to know what % of current router memory each ISP
currently uses, and how many networks he aggregates. That should give a
back-of-the-envolope figure about how much a single ISP can overload a
network. While, clearly, this figure will be much better than what is
today (assuming every multihomed site today is advertising
non-aggregatable routes into the DFZ, and there is no one major ISP
serving all multihomed sites), it may give a better idea of the rough
number of possible excess routes that can appear due to a single ISP
breakdown.

> "some" "a lot" or even "all"? And then people will start to filter and we're

I think we can safely assume that the probability of the event of an
entire ISP going down is randomly distributed across time, which means
that even if one ISP goes down completely, the overload will be less than
that in the case where every ISP advertises non-aggregatable routes (as is
the current case today).

thanks,
ramki

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Tue, 17 Jul 2001 20:14:31 -0500 (CDT)

Forgive me...I'm running behind...see notes below.

> there's likely to be a human element at play.

I'm sorry, but you're incorrect here.  Multihoming is meant to be
"automatic," no intervention required.  You're talking about manually
rerouting traffic, not multihoming.  What's the point of this if you have
to do it manually?

> resilience in the face of outages. I'm not yet convinced there's a good way 

-Taz

-- 
        "Be liberal in what you accept,
      and conservative in what you send."
--Jon Postel (1943-1998) RFC 1122, October 1989