Subject: Re: your mail

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 18 Jul 2001 11:34:21 -0500 (CDT)

On Wed, 18 Jul 2001, Ramakrishna Gummadi wrote:

> On Wed, 18 Jul 2001, Jon (Taz) Mischo wrote:
> 
> > Yes you are.  You are asking a provider to take all traffic destined for a
> > specific egress point and re-route it through their network to a peering
> > point.  You are *DOUBLING* the traffic they are carrying for that
> > subscriber.
> 
> I am sorry, but I may be missing something here---true, the traffic is
> doubled, but how to deal with the doubled traffic should be a decision
> between the provider and the subscriber according to the SLA. Increased
> load could cause higher packet loss, greater latency, etc. Indeed, if the
> provider were using fair queuing, this would what happen, without
> affecting any other customers of the provider. 

The last sentence is what's wrong.  A single router does not a backbone
make.  You seem to forget that a tunnel is not a wormhole.  Packets don't
enter a tunnel in one router and magically appear at the other end.  The
still traverse the network, following a different path than the one they
would normally, messing up a LOT.

> Like I said before, how is this case different than the one where the
> subscriber cancels the service from the other provider, due to which the
> load on the first provider would be increased anyway? Remember that we
> are not even requiring the first provider to participate in the tunnel
> setup...

Because you're completely munging up the ultilization.  There are suddenly
many megabits of traffic where there was none before.  I'd say that's
pretty different.

> > This isn't based on political reasons, but feasibility reasons, and we
> > don't outlaw, we just decide what we predict will be best.
> 
> Well, there is an easy, although socially questionable, answer to
> that---use IPSec on the tunnel so that the provider can not find out
> anything about the payload...

I fail to see how IPsec is going to get you around the problem.  The
problem isn't, "Oh, I see that's tunnelled data, I'll drop it!"  It is, "I
see that packet is coming from someone I don't peer with.  I'll drop it."

The main problem here is that some providers won't peer with others.  I
can guarantee that those same two providers won't configure their routers
to automatically build tunnels to each other.

> Once again, I don't see how doing this, while suboptimal and probably
> unfair, is going to violate the contract between the provider and the
> subscriber. All the provider would see is a surge in the traffic for the
> subscriber, and he will probably use the usual tools (fair queuing,
> increased latency and drop rate, etc. ) to deal with them...

Hello!  The provider will see a surge along his entire network!  This
isn't the magic schoolbus full of 0 length packets.  It's someone's OC-3
at full capacity being re-routed through other peoples' networks.  There's
a LOT more impact than you think.

-Taz

-- 
        "Be liberal in what you accept,
      and conservative in what you send."
--Jon Postel (1943-1998) RFC 1122, October 1989

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Wed, 18 Jul 2001 10:02:24 -0700 (PDT)

> make.  You seem to forget that a tunnel is not a wormhole.

But that is the beauty of fair-queuing and variants---the single router
implementing FQ can perform virtual admission control, guaranteeing that
no more than the specified percentage of traffic from a subscriber can be
injected into the network. Also, it only needs to be implmented in the
border routers.

thanks,
ramki

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Wed, 18 Jul 2001 09:51:39 -0700 (PDT)

> would normally, messing up a LOT.

Which is why tools such as fair queuing and fair dropping exist--they give
you the perfect isolation and bandwidth management guarantees. And I think
they are present on most cisco's already on most high speed interfaces..

> to automatically build tunnels to each other.

We are not requiring anybody to accept traffic from those they don't peer
with or have subscriber-provider relationships (and we don't require any
extra physical links):

D served by ISP2 and ISP3 that don't peer:

Scenario A: (normal operation):
S-ISP1-Internet-ISP2-D.

Scenario B: (failure-mode operation):
S-ISP1-Internet-ISP2-Internet-ISP3-D.

How is B different than A, and how does ISP3 know the tunneled traffic
originated in ISP2 (without looking at the internals of the packet, which
can be forbidden by IPSec anyway)?

If ISP2 and ISP3 cooperatate, the system as a whole can achieve a more
optimal operation point than if they don't. In both cases, however, it
continues to work.

As an aside, should not one be first concerned that the Internet expects
end-to-end congestion control out of hosts who can directly collapse an
ISP if they are greedy? Because people got concerned about such problems,
they invented fair queuing, etc., that allow the ISP full control over how
to fairly manage traffic. These tools can be used in this scenario as
well, if required.

Finally, I don't see how any other multihoming scenario (end-to-end
solutions included) that has to
provide failure protection can behave significantly better---after all,
the goal is to redistribute the load onto working links, with a possible
degradation in performance.

thanks,
ramki

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 18 Jul 2001 12:13:19 -0500 (CDT)

> they are present on most cisco's already on most high speed interfaces..

I'm sorry, but as a customer, I wouldn't want my traffic dropped because
someone else's OC-3 went down.

> with or have subscriber-provider relationships (and we don't require any

Yes, you ARE.  For it to work right, that's EXACTLY what you have to
require.

> S-ISP1-Internet-ISP2-Internet-ISP3-D.

ISP2 and 3 would HAVE to peer to do this.  I'm sorry, but as a provider I
wouldn't want just anybody to be capable of building a tunnel.  Any
potential tunnels of this sort I'd want to be pre-configured so there
couldn't be abuse.  But who's to say it still won't be abused
anyway?  What if ISP2 brings up the tunnel because of a low bandwidth
situation?  As ISP3, I didn't sign up for that.

> can be forbidden by IPSec anyway)?

You're missing the point.  ISP3 can see the tunnel originated at
ISP2.  The tunnel would be from ISP2's router to ISP3's router.  Hard to
miss.  And if I don't trust ISP2 as ISP3, I'm sure as hell not going to
let them create a pipe and pour packets into my network.

> optimal operation point than if they don't. In both cases, however, it

Again, they have to be peers or have a mutual trust relationship.  We're
right back to square one.  This can't work unless they cooperate.

> to fairly manage traffic. These tools can be used in this scenario as

Again, NO.  If you can't afford the bandwidth to do this, don't do it,
cause as a customer, I won't tolerate high packet loss.  Why bother at
that point?

> degradation in performance.

Lots of scenarios can behave FAR better than this.  You're not looking at
the big picture here.  You're assuming someone's T1 fails over.  Most
people with T1's aren't going to be buying this these days.

-Taz

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 18 Jul 2001 12:15:12 -0500 (CDT)

> injected into the network. Also, it only needs to be implmented in the

Think about what you just said.  Why the hell would I pay someone to
redirect a percentage of my packets?  That's not multihoming, that's
ridiculous.

-Taz

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Wed, 18 Jul 2001 10:39:28 -0700 (PDT)

> someone else's OC-3 went down.

Fair queuing says: If there are n customers, there is a scheduling
algorithm that guarantees that each of these customer's traffic has a fair
share of the link bandwidht. This means, an ISP can choose to set the
weights for its n customers such that:
a) each customer can inject no more than
1/n of the traffic into the outgoing link, no matter how much they try
to pump via their respective access links, and
b) the packet losses and delays suffered due to an increased load
on one customer are restricted to that customer *alone*, and are
*completely* invisible to the other customers---the other customers'
traffic does not experience *any* increased delay or packet loss.

Think of it FQ's role as being similar to a  virtual memory manager in
an Operating system. The customers would correspond to processes, and
the memory manager guarantees that no process can affect any other
processes in terms of access to memory pages, crashing via sigment
faulting, etc...

> situation?  As ISP3, I didn't sign up for that.

I don't think ISP2 and ISP3 have to *directly* peer---it is enough for
each to be reachable from the other. The ISPs could be regional providers
with no direct peering, while their parents can peer at some internation
point, for example.

> > optimal operation point than if they don't. In both cases, however, it

Agreed, I made a mistake---ISP3 can figure out that the traffic originated
in ISP2's network. But, then again, in the context of fair queuing above,
I don't see why it is illegal for ISP2 to originate traffic for the site's
border router (in fact, some of the traffic may have been originating from
ISP2's customers).

> right back to square one.  This can't work unless they cooperate.

One end of the tunnel terminates at the site's border router, the other
end at one of ISP2's routers. ISP3 can choose not to participate in the
tunnel creation, and can enforce bandwidth restrictions using FQ on this
traffic as well.

> cause as a customer, I won't tolerate high packet loss.  Why bother at

If one of a site's two links breakdown, I don't see how the site can not
expect to see performance degradation at all. Of course, we require, and
enforce, that no other site's are affected by this degradation using FQ.

Finally, there is no need for the tunnels to be dynamically configured,
although that facility will allow them to possibly withstand additional
failures. Also, the tunnel is between ISP2 and the site, and does not
need to have anything to do with ISP3 at all.

thanks,
ramki

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 18 Jul 2001 13:35:04 -0500 (CDT)

> traffic does not experience *any* increased delay or packet loss.

I know how fair queueing works.  What I'm saying is that if I'm
multihoming via tunnels like this, fair queuing is not what I'm paying
for...I'm paying for QoS beyond that.  Besides, FQ does NOT come into play
here.  If the router where the access link was is where you're tunneling
from, you may or may not be subject to fair queuing, just as your link
was.  However, through the rest of the network, fair queuing does not come
into play, as it would really mess things up.  Or, if it does, you're
still changing the characteristics of the network traffic.

The point here is traffic patterns.  You're arguing when I'm trying to
make it clear that you're missing the point.

> with no direct peering, while their parents can peer at some internation

I can tell you this: if ISP2 and ISP3 aren't directly peered, they're
going to be paying for someone else's traffic.  So you're saying that to
multihome like this, if ISP2 and ISP3 aren't peers, the customer is going
to have to pay for 4 links instead of 2, because those two ISPs are going
to be buying extra transit to carry this.  You're not winning this
argument.

> border router (in fact, some of the traffic may have been originating from

Ummm...I'm sorry, but do you have operational engineering experience?  I
don't know a whole lot of people who do that would feel comfortable with
someone else causing a change in their provisioning and routing, even if
it was temporary, especially if they didn't trust those people.  I just
can't see it happening.

> tunnel creation, and can enforce bandwidth restrictions using FQ on this

In this case, though, you're still doing bad things.  See, now you're
saying that the tunnel can cause legitimate packets for that ingress point
to be dropped.  Furthermore, you've just taken control of any of that from
the router the tunnel is terminating on, and moved it upstream.  Do you
see what I'm getting at?  Now you're dropping the packets in the tunnel
AND the legitimate packets bound for that ingress point that DON'T have
problems.

You're making this assumption that everyone has dormant circuits laying
around with no traffic on them.  This is NOT the case.

> If one of a site's two links breakdown, I don't see how the site can
> not expect to see performance degradation at all. Of course, we
> require, and enforce, that no other site's are affected by this
> degradation using FQ. Finally, there is no need for the tunnels to be

Ummmm...why are you so obsessed with fair queuing?  It is not God, I
assure you.  It won't be upset if you see that it has flaws.  If you
multihome a site properly and smartly, and have adequate bandwidth, you
don't have to take any significant performance hit.  However, if you do it
this way, you're breaking provisioning for everyone, cost everyone a lot
of money, and making it more and more unattractive.

When I first read this I liked the idea...you've made me see how ugly this
idea is, though.

> dynamically configured, although that facility will allow them to
> possibly withstand additional failures. Also, the tunnel is between
> ISP2 and the site, and does not need to have anything to do with ISP3
> at all.

Read my comments above.  I don't think you understand the issues involved
in doing this.  You're looking at certain pieces of the puzzle, but not
the big picture.  There has GOT to be a better way.

-Taz

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Wed, 18 Jul 2001 12:52:46 -0700 (PDT)

Hi,

> for...I'm paying for QoS beyond that.

First of all, the site uses normal links if there are no problems. It only
switches to tunnels when there is are failures. I don't think it is fair
to say we are "multihoming via tunnels" in all contexts.

I may be missing something, but, as far as I know, FQ can provide the
strongest possible QoS guarantees as far as flow isolation, bandwidth
allocation, delay, and loss are concerned. It is not widely deployed so
far because of implementation complexities (lots of approximations exist),
and is currently being deployed at least in the sprint backbone.
I am interested in knowing what additional QoS you have on your mind.

> Besides, FQ does NOT come into play

If an ISP guards its exits/entrances, it does not have to worry about
whether it changes the internal traffic patterns because FQ guarantees it
does not.

> to be buying extra transit to carry this.  You're not winning this

Let us pick the following concrete example, count the extra packets, and
figure out who is paying the extra cost for whose failures:

                             D
                             |
                             |
                          /--+---\
                        //        \\
                       |   Internet |
                        \\         /
                   L7    / \------/ \   L6
                      //            \\
                     /                \
                   //                  \
               +--/---+      L1      +--\---+
               |  ISP4+--------------+ ISP3 |
               |      |              |      |
               +--+---+              +--+---+
                  |                     |
               L2 |                     | L3
                  |                     |
                  |                 +---+--+
               +--+---+             |  ISP2|
               | ISP1 |             |      |
               |      |             +---/--+
               +------*\              //
                        \\          //
                     L4   \\      //   L5
                            \\  //
                           +--\*--+
                           |      |
                           | Site |
                           +---+--+
                               |
                               |
                               S

Suppose the multihomed site gets connectivity from regional ISPs, ISP1 and
ISP2 who
do not trust and peer with each other, while their parents, ISP4 and ISP3,
peer using link L1. The site has a host S that is talking to a destination
D on the Internet using the path: D-(Internet)-L6-ISP3-L3-ISP2-L5-S.

Suppose there is a major problem within ISP2, and link L5 goes down.
However, assume ISP2 is still partially functional, so assume L3 continues
to work. Packets would now follow the route:

D-(Internet)-L6-ISP3-L3-ISP2-L3-ISP3-L1-ISP4-L2-ISP1-L4-S.

The number of extra packets introduced on the inter-provider links is 4
(L3, L1, L2, L4).

However, consider the perspective of ISP1 and ISP4. Theoretically, the
destination D could have used the path through them (and this would be
true in any end-to-end failover rerouting scheme as well), meaning the
flow of packets would be:

D-(Internet)-L7-ISP4-L2-ISP1-L4-S.

This means that ISP1 and ISP4 are *not* taking any additional load in the
first case versus the second case---they are not resposible for the
problem, and they don't pay *any* extra price in terms of extra
packets than they would otherwise have had to.

The extra penalty is paid by ISP2 and ISP3, but because it is fault of
ISP2, it should be willing to bear this cost both for itself and its
transit provider.

The multihomed site does not have to pay any penalty because it is not its
problem.

If ISP3 had a problem, on the other hand, it alone would have had to pay
the extra bandwidth price, and would not burden ISP2 and the site. We
should not consider sending packets via L1-ISP4-L2-ISP1 a burden because
ISP3 is, by definition, allowed to send packets via L1 to ISP4.

It would be great if you can use this scenario to make your objections.

> it was temporary, especially if they didn't trust those people.  I just

I don't have operational experience, but I still think this is not a case
of changing provisioning as the same could have happened in a non-failure
mode scenario where the site simply stopped using one link and diverted
the entire traffic over the other link. And, I definitely don't see this
requiring any routing  changes. I am interested in seeing what
other peoples' reactions are to this.

> of money, and making it more and more unattractive.

I am just highlighting a technique that can be used to provide
worst-case guarantees. I think it is useful in general because
it is theoretically sound, backbone carriers (like Sprint) are
beginning to deploy it in their networks, and router implementations are
appearing for highspeed interfaces (an OC-48 channel can be split up
fairly between n customers, for example). It helps in strictly enforcing
SLAs.

thanks,
ramki

---

From: Daniel Senie <[email-masked]>
Date: Wed, 18 Jul 2001 16:21:36 -0400

>to say we are "multihoming via tunnels" in all contexts.

So let's make sure everyone is clear on this point. You're saying the only 
reason to multi-home is for failover purposes. As long as the "primary" 
link is up, the "secondary" interface (which is used solely to carry 
tunneled traffic in the event of an outage with the primary) is never used.

Sites multi-home for reasons other than line outage recovery, but you don't 
take those into account. It is common for sites to select a second upstream 
provider for multihoming based on a business need for reliable 
communications to a specific destination (business partner, for example). 
I'm not convinced a multi-homing solution based solely on the notion of 
disaster recovery is sufficient.

-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.                    http://www.amaranth.com

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 18 Jul 2001 23:01:16 +0200 (West-Europa (zomertijd))

> disaster recovery is sufficient.

Here in The Netherlands many fairly small ISPs only have one transit
connection, but they connect to the Amsterdam Internet Exchange because
that way they save a bundle on transit costs.

The "real" multihomed networks I've worked with multihome for disaster
recovery reasons, but since multihoming is expensive, they also want to
balance traffic over both connections.

Just an idea I just thought of: a large transit network could have a
backup-network that announces the same routes as the primary network and uses
tunnels to send traffic over the second ISP of multihomed networks. Under
normal circumstances the primary network collects all incoming traffic
(better routes) and the backup network does nothing. When the primary network
goes down, other ISPs send the traffic to the backup network and the backup
network tunnels it to the customers. As long as the primary and backup
networks don't suffer from (much) shared fate, this could be a very resilient
solution with no routing impact.

As long as we're talking about tunnels: we should fix the path MTU discovery
problems they expose.

Iljitsch

---

From: Ramakrishna Gummadi <[email-masked]>
Date: Wed, 18 Jul 2001 13:56:33 -0700 (PDT)

> tunneled traffic in the event of an outage with the primary) is never used.

No, all I am saying is a normal working link can carry tunneled traffic
in the face of an outage in addition to the normal traffic; thus, if a
site has 11 links (possibly to 11 different providers) that each operate
under 90% utilization in normal conditions, when a provider or link goes
down, the remaining 10 links carry both the spillover (9% each) and the
normal
traffic, so that they now operate at near-full (99%) utilization.

thanks,
ramki