Subject: plug: thesis on site multihoming

From: Pekka Savola <[email-masked]>
Date: Fri, 28 Mar 2003 17:29:33 +0200 (EET)

Hi,

Sorry for a plug, but it seems relevant and may be interesting for lots of
people.

I'm about to submit a MSc thesis on the subject of "Examining Site
Multihoming in Finnish Networks", first looking at IPv4 and then IPv6.

If you're interested, it's available at:

http://staff.csc.fi/psavola/di.ps (.pdf is also there but quality is 
worse)

There's some minor work to be done still (like writing the abstract,
making a few pictures clearer and actually reading it properly myself ;-),
but it should pretty close to done (submitting for review in the beginning
of next week, making small modifications up until about 2 weeks from now).

Even if you don't have the time, I intend to split off a modified section
(likely based on sections 6.1 and 6.2) and submit it as an I-D in a week
or two.  By breaking down the sites by a few different types and looking
at which multihoming requirements they have, we seem to be able to solve
almost all problems at least to some extent quite nicely.  Of course, with
long term, things can be improved a lot.

Comments etc. are of course really appreciated; off-list is probably best
at least for the most of them.

In another w.g. I made a joke about most thesis's being non-relevant and
useless (for the IETF).  Hopefully this doesn't fall into that category
:-).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Pekka Savola <[email-masked]>
Date: Sun, 30 Mar 2003 17:45:56 +0300 (EEST)

An update,

1) yes, the thesis _is_ in English ;-)

2) the files have been updated slightly, mainly editorial updates but I've
also added the abstract and some other relatively minor changes.

For the curious, the abstract is below.

Thanks for interest.

======
Site multihoming means end-sites connecting to multiple separate network
service providers; currently, no IPv6 site multihoming
mechanism has been widely accepted yet.

This thesis studies both IPv4 and IPv6 site multihoming mechanisms using
literature, other similar studies, analysis of route advertisements at the
Finnish exchange point FICIX, and queries on multihoming practices to
major ISPs in Finland.

Currently in IPv4, there seem to be three-four main mechanisms which are
used to achieve at least some of the multihoming benefits: obtaining own
address space and AS number and advertising those, advertising more
specific routes with different path, using multi-connecting and leveraging
NAT.

In IPv6, the first two of IPv4 mechanisms which are considered
architecturally unscalable have been operationally prevented for now and
the fourth does not exist.

Based on a tentative roadmap introduced in this thesis, organizations are
split to four categories: minimal, small, large and international; each
have different multihoming requirements which can be met in different
ways.

Focusing on immediate and short term solutions, minimal organizations do
not seem to require a solution, small ones could use multi-connecting,
host-centric multihoming or multihoming at site exit routers approaches,
large ones the same or possibly separate provider independent address
allocations, and international ones either provider independent
allocations or be broken down to multiple large organizations and using
those techniques.

It is apparent that only a limited amount of work is needed to enable
sufficiently good multihoming mechanisms which should provide the required
features.  However, as the mechanisms are unarguably more difficult for
the end-site, while taking the global Internet routing architecture better
into the account, it is unclear whether they might be adopted.
======

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 30 Mar 2003 18:16:54 +0200

> also added the abstract and some other relatively minor changes.

3) It's 97 pages. Do they pay you by the word?  :-)

I'm afraid that my geo proposal has again been mangled beyond  
recognition. Let me say it once more: NOBODY announces the aggregates  
(except maybe to customers). They are local to each ISP network, hence  
the "provider-internal aggregation" part.

(The draft is going to expire soon, but it will remain available at  
http://www.muada.com/drafts/draft-van-beijnum-multi6-isp-int-aggr- 
00.txt )

Iljitsch

BTW, what makes the PDF look so ugly??

---

From: Pekka Savola <[email-masked]>
Date: Sun, 30 Mar 2003 19:23:44 +0300 (EEST)

> 3) It's 97 pages. Do they pay you by the word?  :-)

Nope, and even then I had to leave stuff out .. when you can't expect the
readers to be aware of anything related to multihoming, things will get 
noisy.
 
> the "provider-internal aggregation" part.

And as I've said before, I fail to see how it actually solves the relevant
multihoming problem then :-).  Clearly this seems to be somethng that
needs to be said more clearly.. Having geographical addresses that are
specific to an ISP is a complete non-solution, we aren't worried about 
the scalability of provider-internal geo more specifics...
 
> BTW, what makes the PDF look so ugly??

The use of fonts which are not included by default in PDF.  I'd have to 
put that through Adobe Distiller or do some tricks, which I didn't bother 
yet.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 30 Mar 2003 18:37:32 +0200

> specific to an ISP is a complete non-solution,

You don't say.

"To make multihoming (as we know it today) possible, individual routes
must be present in the global routing table. But in order to fit the
routing table into a router, there must be aggregation. These
requirements seem at odds with each other. This is because there is an
unspoken assumption: the full global routing table must be present in
all routers that are part of the default-free zone. Dropping this
requirement makes everything much more complex, but it is possible. The
global routing table can then be split into several parts, where
individual routers all handle one (or a few) of those parts.

This works as long as traffic for a certain subset of the destination
networks present in the global routing table is always sent to a router
containing that part of the global routing table. The obvious way to
accomplish this is for each router to announce an aggregate covering the
part of the global routing table it serves. For instance, if a network
has four routers and wants to divide routing information for the IPv6
global unicast address space over those routers, it could have router A
handle 2000::/5, router B 2800::/5, router C 3000::/5 and router D
3800::/5. So if this network peers with another network that announces
2200:abc::/35 and 3ffe:def::/35, all routers except router A filter out
the first route, and all routers except router D filter out the second
route. When router C then has a packet for 2200:abc:1:2::1, it sends the
packet to router A (because router A announces the 2000::/5 aggregate)
and router A delivers the packet to the right peer. Note that this
behavior is completely hidden from the peer: the aggregates are only
used within the local network, they are not announced to peers. To avoid
confusion with regular provider aggregatable routes, the term "pilot
routes" will be used for this type of private aggregates."

---

From: Pekka Savola <[email-masked]>
Date: Sun, 30 Mar 2003 19:51:17 +0300 (EEST)

> individual routers all handle one (or a few) of those parts.

All routers inside an ISP certainly don't need to have full routing table, 
but if *one* or those with external connections to DFZ don't, you might 
end up in problems.

If you want to remove full routing table from an AS that's in routing 
table -- well, that may be doable to an extent, but I'm not sure how it 
helps, and that's not explicitly stated.

On the other hand, if you want to reduce the number of routes in the full 
routing table inside your AS, that's a non-goal.

> routes" will be used for this type of private aggregates."

A,B,C,D are part of one AS.  If you try to expand that so that they 
aren't, you get into trouble.  In the same way, if you don't expand it, 
you are just solving the provider-internal problem -- and are requiring 
the propagation of more specifics in DFZ.

Could you work out an example of AS's rather than routers?  It seems to me 
that there's some huge confusion somewhere.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 30 Mar 2003 19:15:43 +0200

On zondag, maa 30, 2003, at 18:51 Europe/Amsterdam, Pekka Savola wrote:

>> if a network
>> and router A delivers the packet to the right peer.

> aren't, you get into trouble.

And that's exactly the point: we aggregate inside provider networks.

> In the same way, if you don't expand it,

Yes. But since no single router has to keep a copy of the entire DFZ 
routing table, this is no longer a problem.

It all boils down to the question: why would someone in Europe need 
more specifics for people in the US, or vice versa?

---

From: Pekka Savola <[email-masked]>
Date: Sun, 30 Mar 2003 20:37:07 +0300 (EEST)

> routing table, this is no longer a problem.

Ok -- now I think I understand the concept.

If the max number of routes a router implementation can handle is N and
more-or-less-complete routing table (non-existant but as a concept) would
be Y (>N), you require that each ISP would have at least Y/N (rounded up)
routers each connected separately to the Internet -- that, between
operators you'd have at least Y/N (rounded up) interconnections if you
wish to have optimal paths, etc.

Forgive me saying this, but this kind of model which requires a lack of
aggregates in the core, and sets requirements for ISPs' border routers
just doesn't seem to fly.

My argument as an operator is that unless you can divide & conquer the the
full routing table to such proportions that it can't be uniform in one AS,
there's something very broken in the design of the internet routing
architecture, or the definition of the AS.

Certainly, the basic concepts of how these different routers with partial
knowledge interact with peers' and upstreams' routers of partial knowledge
seems quite fuzzy .. and this is one thing that seems critical to have
clarified (some pictures might be illustrative).

But as this seems a different proposal than Tony's, I'll remove the
reference.
 
> more specifics for people in the US, or vice versa?

Depends on the level of interconnection and how the routes are set up, 
there.

Currently, there is no abstraction, so it's difficult to estimate..  But
if there was, people would probably very much like *their* pretty little
route going through optimal paths and being in the full routing table of
all the routers.

I haven't really been able to form a picture in my head how the proposal
would have to be practically enabled for example through a path from an
European ISP to U.S. ISP, in between having or maybe not having transits
and ISP's which may or may not honor this model.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 30 Mar 2003 21:12:56 +0200

On zondag, maa 30, 2003, at 19:37 Europe/Amsterdam, Pekka Savola wrote:

> routers each connected separately

Yes.

> to the Internet

Each separately connected to peers. If they are "connected to the 
internet" that would be a default route which you can have in just one 
location if you want.

> -- that, between

Yes. So it doesn't scale forever but one order of magnitude is no 
problem at all and a second one (20 or so interconnects per continent) 
should be doable.

> just doesn't seem to fly.

Why not?

> architecture

I'm not entirely sure what you're saying. Is it "if you can't hold the 
entire DFZ in a single router your architecture is broken"? That's one 
view. I'd say it's the other way around: if your architecture requires 
you to keep information about the entire world in all routers, you've 
painted yourself into a corner.

Forget current practices for a moment. Doesn't it make sense to keep 
detailed information about local stuff and not so detailed information 
about what happens farther away?

> clarified (some pictures might be illustrative).

I'm considering an update to the draft, but as far as I can tell 
everything is in there.

> But as this seems a different proposal than Tony's, I'll remove the

Actually Tony's draft is just an addressing scheme which doesn't 
automatically solve routing. My geo proposal could use Tony's 
addressing scheme. (But Michel and me have written a draft of our own 
for this, you can find it under "GAPI".)

> Depends on the level of interconnection and how the routes are set up,

Fortunately, the days of transatlantic routing between two places in 
Europe are long gone. In other parts of the world this still happens, 
but I believe this will sort itself out for the most part before 
multihoming in those places becomes so popular the multihomed prefixes 
must be aggregated.

> route going through optimal paths and being in the full routing table 

If someone in Asia wants to connect through European ISPs, that's their 
problem.

I did some messing around with BGP yesterday. 4000 routes are 
responsible for 60% of all our traffic, 6000 routes for another 25%. So 
that means I can have optimal routing for 85% of all traffic with just 
10% of all routes. That's good enough for me.

> and ISP's which may or may not honor this model.

Cooperation from other ISPs is not needed: everyone can implement or 
not implement this as desired, just as long as the RIRs give out the 
addresses based on geography.

---

From: Pekka Savola <[email-masked]>
Date: Sun, 30 Mar 2003 22:40:25 +0300 (EEST)

> > just doesn't seem to fly.

Can you require all of these things from all the folks?

But really, the problem is that I'm having hard time (and based on the
lack of enthuasiasm from the others, the rest as well)  figuring out
whether the model could be useful and whether it could actually work.

It just doesn't seem to be enough fleshed out to say much other than
"ouch! I don't really understand the idea, but it still gives me creeps!".  
If you want to avoid that feeling, you must warm up the folks properly :-)

> view. 

If you note, I explicitly did not say that.  I said (or tried to, I was 
not precise), that:

"if you can't have the same routing table in all of your routers in a
single AS while being part of the DFZ, your architecture or the definition
of AS is broken"

In your proposal, what is the thing you're worried of?  International 
transits which are part of all the different continents and regions?

In the proposal, where are the more specific advertisements hidden?  
Being advertised between different AS's at peering points but hidden in
your that particular peering router by an internally advertised aggregate?

> I'd say it's the other way around: if your architecture requires 

I can agree with that (which is the addressing model Tony is proposing 
with his geo proposal, btw.) -- if it's required, but I'd like to avoid 
it.
 
> about what happens farther away?

Yep.  

.. Whether "geographic" split makes sense is a different thing though.
 
> I'm considering an update to the draft, but as far as I can tell 

Doesn't seem to be particularly unenlightened-friendly.. :-/
 
> automatically solve routing. 

I'd be very very careful of any mechanism which claimed would solve 
routing automatically.

> My geo proposal could use Tony's 

Yep, been through it.
 
> Europe are long gone.

.. with current topology-derived addressing...

> > Currently, there is no abstraction, so it's difficult to estimate..  
> > But if there was, people would probably very much like *their* pretty
> > little route going through optimal paths and being in the full routing
> > table of all the routers.
> 

Well, once the routes hit the dfz, it becomes your problem too.
 
> 10% of all routes. That's good enough for me.

Nice to get some figures, however, I can already hear the 10% yelling
quite loudly..

I also made a rough sketch on more specifics (it's in 5.3.1).  In an
exchange in Finland, 1661 prefixes were advertised some time ago.  50% of
them were /24's.  Those only covered 1.7% of all the advertised address
space though.

Interesting figure, eh?  I guess it's no wonder that less than 5% of folks
eat up at least 75% of the routing table entries.  I guess that shows
something about routing tables.
 
> addresses based on geography.

Forgive me for sounding skeptic, but I'm not as sure without further 
analysis that no cooperation would be required.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 31 Mar 2003 09:17:33 +0200

> whether the model could be useful and whether it could actually work.

It allows around ten million multihomed networks, so that answers the 
useful part. Nobody has ever pointed out any aspect that presumably 
wouldn't work (just stuff they aren't real comfortable with) so either 
I am as smart as I think or nobody wants to take the time to carefully 
read the draft.

> "ouch! I don't really understand the idea, but it still gives me 

What's so hard to understand??? In the US you create an aggregate route 
for the US address block and you accept /48s for US destinations. In 
Europe, you create an aggregate for the European address block and 
accept /48s for European destinations. So when you have a packet in the 
US that needs to go to a European destination, you don't have a /48 
route so the packet is routed to Europe by means of the aggregate. In 
Europe, the packet is routed further as per the /48 that is in the 
routing tables of European routers. Rinse, repeat.

> If you want to avoid that feeling, you must warm up the folks properly 

So how do I do that? Buy everyone drinks at the IETF meetings?

> single AS while being part of the DFZ, your architecture or the 

Ok, then OSPF and IS-IS explicitly support broken networks...

> transits which are part of all the different continents and regions?

I'm not worried.

> your that particular peering router by an internally advertised 

Not sure what you're getting at.

> with his geo proposal, btw.) -- if it's required, but I'd like to avoid

But just now you said you wanted to be able to have the same routing 
info in all routers! You can't have it both ways.

> .. Whether "geographic" split makes sense is a different thing though.

If you have a better idea, I'd like to hear it. The trouble with a 
situation where routers only hold a partial table is that the traffic 
has to physically move to the place where the routing information is. 
This is incredibly inefficient, _unless_ the place where the routing 
information is happens to be in the right direction of where the 
traffic has to go to anyway.

> Doesn't seem to be particularly unenlightened-friendly.. :-/

Tell me what's missing and I'll put it in.

> I'd be very very careful of any mechanism which claimed would solve

I don't mean solving all routing problems forever, just that Tony's 
scheme needs additional routing stuff to do something useful.

> Well, once the routes hit the dfz, it becomes your problem too.

Not if I filter them out.  :-)

> Nice to get some figures, however, I can already hear the 10% yelling

Your milage may vary. If you mainly host websites in the local language 
this is going to be very different from a situation where you connect 
the rooms of a big convention hotel.

> something about routing tables.

What really annoys me to no end is the people who announce 10, 20, 100, 
200 /20s that could easily be aggregated. Some of the people announcing 
/24s are doing this for good reasons, but people filter on the /20 
boundary and hurt exactly the wrong people.

I don't understand why the large networks don't do anything about this. 
I would simply de-peer.

> analysis that no cooperation would be required.

What exactly would another ISP have to do so I can aggregate within my 
own network??? If I want to accept all routes within 195.0.0.0/8 in 
Amsterdam, all routes within 196.0.0.0/8 in London and so on, I can do 
that without cooperation. (I have to deaggregate if I don't peer in 
London with someone who announces 196.12.34.0/24, though, but that's 
still something I can do on my own.)

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 31 Mar 2003 11:57:13 +0200

Iljitsch van Beijnum wrote:

...
> >> more specifics for people in the US, or vice versa?

Because of money, policy, and AUPs, traditionally. I doubt if this will
go away, but it shouldn't be visible outside the sites concerned.

   Brian

---

From: Tim Chown <[email-masked]>
Date: Mon, 31 Mar 2003 11:52:04 +0100

> go away, but it shouldn't be visible outside the sites concerned.

As a real example happening now(ish):

In the context of IPv6 experiments between Abliene and 6NET, we would
wish to have some experiments run over 6NET infrastructure and 6NET
partner (native) links to Abilene, while regular production IPv6 traffic
goes via GEANT (via its imminently native link to Abilene).  Thus we
would advertise a specific /64 (where the test equipment resides) to 
Abilene via the 6NET partner path, which would not be propogated outside 
Abilene.   So it's a private agreement between two networks which have 
a private link outside their regular production path.   

Tim