Subject: The state of IPv6 multihoming development

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 21 Oct 2002 23:24:10 +0200 (CEST)

In order to kickstart some activity in this wg, here is my view on the
current state of multihoming in IPv6 development. I hope this will start a
discussion on this here mailinglist, and also elsewhere, because I don't
think we can solve the entire problem right here, right now.

Since waiting for "someone else" to completely solve the problem hasn't
been very productive so far, I think everyone should assume responsibility
for that part of the solution space he or she has any control over. There
are things that can be done to help multihoming in operations, address
allocation policy and application, host stack and router development. Lack
of cooperation from even one of those will hurt the multihoming in IPv6
effort.

The problem
-----------

The user problem: the internet is unreliable. Parts of it fail from time
to time. The only way to really protect yourself against this is having
several connections to the internet over different ISPs (multihome).

The IETF problem: multihoming by announcing a globally visible address
block (as is current practice in IPv4) doesn't scale to a large enough
number of multihomed sites.

A number of new drafts and revised versions of existing drafts will be out
in time for Atlanta.

1. Application solutions
------------------------

One way to solve this would be by taking provider aggregatable addresses
(that don't polute the global routing table and thus scale well) from two
(or more) ISPs and give all systems addresses from both ranges. Then
applications simply try all addresses.

This has many drawbacks. For instance, sessions still break if there is a
problem, and source address selection is problematic, especially in the
presence of anti-spoofing filtering by ISPs. But the good part is it
actually works today, to some degree. For instance, I can list several IP
addresses for an FTP server, and when the first one is down the FTP
program simply tries the next. Unfortunatley, the applications must
support this, and most of the newer ones (especially web browsers) don't.

2. Transport protocol solutions
-------------------------------

It is also possible to solve the problem at the transport layer. This has
most of the drawbacks of solving the problem at the application layer,
such as the source address selection/filtering issue. But also the fact
that applications have to be changed: unfortunately, the IPv6 API as
described in RFC 2292 centers around a single IPv6 address. Therefore, an
application wishing to use an address-agile transport protocol must be
changed in order to provide the extra address or addresses to the
transport layer. Alternatively, the transport layer could incorporate a
mechanism to discover alternate addresses, but this leads to security
problems as applications would be unaware of the addresses they use.

Transport-layer solutions only solve the multihoming problem for
applications using the specific transport protocol so they only provide a
partial solution.

3. Routing implementation solutions
-----------------------------------

Current routers easily need upto a kilobyte or even more memory for each
individual prefix that is visible in the global routing table. In theory,
it should be possible to reduce this by one or two orders of magnitude by
removing information that doesn't directly lead to better routing
decisions.

However, this requires major changes to the internals of routers and the
potential gain is relatively limited.

4. Geographical aggregation
---------------------------

By confining the reachability information for multihomed prefixes to the
geographical region where the multihomed network is connected, it is
possible to keep the size of the routing tables in individual routers
small while the global routing table gets bigger.

The downside is that all aggregation implies loss of information, which
means less optimal routing. Also, aggregation across network boundaries is
complex. Geographical aggregation can only work in the presence of a
compatible address allocation policy.

5. Tunnelling or aliasing solutions
-----------------------------------

As a rule, current transport protocols do not have the capability to
switch soure/destination addresses when the addresses that are used no
longer provide connectivity, even if other addresses that are tied to
another physical or logical path that doesn't share the problem are
available. Such an alternate path can still be used by tunnelling the
original packets with the broken addresses.

An optimization to tunnelling is to simply rewrite the original addresses
with the alternate ones at (or close to) the source, and write back the
original addresses at (or close to) the destination. This is called
aliasing and has the advantage the size of the packet remains the same.
Aliasing has superficial similarities to NAT, but it is fundamentally
different as the destination sees the same addresses as the source. Note
the resemblance to mobility.

A major problem with tunnelling/aliasing solutions is the discovery of the
alternate addresses, and the security of these discovery mechanisms.

What have we learned
--------------------

In discussions on this mailinglist and elsewhere, I've come to the
following conlusions:

* The operations people really want/need multihoming. Current 6bone
  guidelines (which are treated as law in many circles) forbid announcing
  non-pTLA blocks with the result not even internet exchanges or regional
  internet registries can be provider independent.
* There is a lot of overlap between mobility and some of the multihoming
  solution space. It is likely that mobility can be extended to support
  host multihoming.
* GSE is interesting, but it doesn't provide much multihoming support.
* Applications behaving in a multi-address aware manner could very
  easily support some bare-bones host multihoming.
* Hosts having multiple addresses from different ISP PA blocks doesn't
  currently work in practice. This could be solved by having systems
  (mainly hosts, but possibly also routers) taking the source address into
  account when making routing decisions.
* Larg(er) organizations will do just about anything to avoid renumbering,
  this is NOT "free" or even particularly easy in IPv6.
* Very few organizations want or need to connect to the internet at
  geographical locations that are very far apart and provide "internal
  transit" for non-backup purposes.
* Some organizations that have a relatively large block of address space
  de-aggregate to avoid having to use their internal network for transit.

What we should do
-----------------

This is what I think this wg should do:

0. Move past the requirements. There is consensus--sort of. It's not going
   to get much better than this.

1. Make recommendations to developers and other wgs about what
   they can do to better support multihoming.

2. Several tunnelling/aliasing solutions have been proposed. It would be
   good if those could interoperate. This would make it possible for a
   multihomed host with (for instance) modified mobility support, to
   interact in a multihomed-meaningful way with a host sitting behind a
   cluster of big aliasing boxes. This would greatly relieve the
   chicken/egg problem for this type of solution and make experimentation
   with the really hard part (discovery and its security) easier. So we
   should come up with a protocol that makes this interoperation possible.

3. Open the lines of communication with people working on mobility.

4. Work on geographical aggregation, especially by getting an address
   allocation mechanism that supports this off the ground.

Iljitsch van Beijnum

---

From: "Tony Hain" <[email-masked]>
Date: Mon, 21 Oct 2002 18:48:24 -0700

Iljitsch van Beijnum wrote:
> ...

To which I submit:
http://www.ietf.org/internet-drafts/draft-hain-ipv6-pi-addr-03.txt
http://www.ietf.org/internet-drafts/draft-hain-ipv6-pi-addr-use-03.txt

Comments welcome...
Tony

---

From: Eliot Lear <[email-masked]>
Date: Tue, 22 Oct 2002 00:51:42 -0700

Dear all,

I submit that there are several reasons that this group has been (too) 
quiet.  The first is that the requirements document has not progressed. 
    I suspect the authors cannot agree on how to proceed, but I haven't 
actually discussed the matter with them.

More importantly, however, I believe the document is actually an 
impediment to moving forward.  One can view the document in two ways, 
and neither is positive.  Either the points made in the document are so 
obvious as to not be worth writing down, or they are so constraining as 
to make the resulting solution not worth implementing.

Here is an example of both:

3.2.5 Operations and Management

    It MUST be posssible for staff responsible for the operation of a
    site to monitor and configure the site's multihoming system.

I could envision a system where an upstream actually does the 
configuration.  Building a system that cannot be monitored is foolish 
(and therefore the requirement is obvious).  Who does that monitoring is 
at worst a matter of device authorization.

Similarly, I would want people to be able to reconsider the address 
format as well as the relationship between the EGP and the IGP.

The question then turns to this: should we revisit the requirements 
document, or simply discard it and place effort more along the lines of 
the note sent by Iljitsch van Beijnum?  It's not that the document is 
completely wrong.  I just don't want architects to be so bound by it 
that we end up with something less lasting or useful.

Let's have the engineering tradeoffs on the table.  Those can develop 
*after* solutions are proposed.

Eliot

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 22 Oct 2002 04:01:30 -0400

On Monday, Oct 21, 2002, at 17:24 America/Montreal, Iljitsch van 
>    allocation mechanism that supports this off the ground.

	At best, a different term needs to be found other than "geographical
aggregation".  At worst, this is a bad idea.

	The problem is that just because I am in city X, that still does not
mean that any of the ISPs that I connect to will interconnect inside X.
In all cases so far, I am on a tail circuit that goes to a *different*
geographical location (outside X in each case) for each provider that
I have.

	In short, the IP routing topology is NOT closely congruent with the
world's geographical topology.  If we assign addresses not congruent 
with
the actual IP routing topology, this tends to increase the number
of prefixes in the default-free-zone (which is generally understood
to be a bad property).

Ran
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 22 Oct 2002 12:01:13 +0200 (CEST)

> aggregation".  At worst, this is a bad idea.

Why a different term?

> mean that any of the ISPs that I connect to will interconnect inside X.

Currently interconnection isn't a game played on the city level. In
Europe, there is usually a good interconnect for each country. In the US,
it's more difficult, but it should be possible to divide the country
roughly into three or four parts and have two interconnects between all
the major players in each of those parts.

Also, this isn't a binary thing: if you can't interconnect with a peer in
the target region, you simply don't aggregate routes learned from this
peer.

Finally, it doesn't matter where the interconnect location is, as long as
you can limit the presence of more specifics to a subset of all routers.
So a US-only network could very well have all the routes towards Asian
destinations inside routers on the west coast and routes towards European
inside routers on the east coast. This is a very long way from the target
region, but it still makes sense: routers in Chicago get to forward
packets in the right direction until they reach a router that has more
specific information.

As the routing tables grow, it might even be necessary to have several
routers in one interconnect location that each handle a different target
region. For instance, a network could have two routers at Palo Alto: one
that has domestic west coast routes and one that has Asia/Pacific routes.

I'll have a draft on this ready within a few days.

> of prefixes in the default-free-zone (which is generally understood

Are you saying the IPv6 routing table should only contain PA routes (this
is current 6bone policy)? I don't think this is a sustainable position in
the long run. And if we are going to do PI, doing it in a way that at
least potentially supports aggregation would be good.

Iljitsch

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 22 Oct 2002 14:34:12 +0200

...
> least potentially supports aggregation would be good.

The last sentence is clearly true, but you are making assumptions
that multihoming will be solved in a particular way in the second sentence.
That presupposes a result from the multi6 debate, before that debate
has even started.

So for now, yes the BGP4+ table should only contain PA routes, until we
have a better solution to aggregation than that.

  Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 22 Oct 2002 14:38:59 +0200

> to make the resulting solution not worth implementing.

I think we need the document, but since it does indeed contain conflicting
requirements that will need tradeoffs, it clearly SHOULD NOT contain
pseudo-normative language. I would suggest that all the upper case normative
words should be replaced by shoulds and should nots in lower case, and then
just ship the thing as Informational.

   Brian

---

From: "Robert J. Rockell" <[email-masked]>
Date: Tue, 22 Oct 2002 08:55:36 -0400 (EDT)

->
->What we should do
->-----------------
->
->This is what I think this wg should do:
->
->0. Move past the requirements. There is consensus--sort of. It's not going
->   to get much better than this.

This 'sort of' could serve as a good indicator that the current solution
space is not well-received.    I think it is important that requirements
get documented, blessed, and put through the system in this working group.
There is already enough 'bad blood' in the IETF about IPv6.  If one were to
develop a solution to the multi-homing problem, without first having the
requriements document solidified, this just furthers the problem (ex. "the
solution stinks, it doesn't take into account X,Y, and Z").

Summary: Define the problem before you solve it. This has gotten us into
trouble with this nasty little protocol before. We should learn from
mistakes in the past.

->
->1. Make recommendations to developers and other wgs about what
->   they can do to better support multihoming.
->
->2. Several tunnelling/aliasing solutions have been proposed. It would be
->   good if those could interoperate. This would make it possible for a
->   multihomed host with (for instance) modified mobility support, to
->   interact in a multihomed-meaningful way with a host sitting behind a
->   cluster of big aliasing boxes. This would greatly relieve the
->   chicken/egg problem for this type of solution and make experimentation
->   with the really hard part (discovery and its security) easier. So we
->   should come up with a protocol that makes this interoperation possible.

Good for 6bone, bad for 'real' networks.  inter-provider SLA's are a long
way off... and this is requirement #1 for most of the tunneling stuff to
work.

->3. Open the lines of communication with people working on mobility.

...but they scale the same as this problem does... not sure. Perhaps
safer to circle back frequently, then try to develop some consistency
between the groups for the sake of similarity.

->
->4. Work on geographical aggregation, especially by getting an address
->   allocation mechanism that supports this off the ground.

I'm with RJA on this one.   that dog won't hunt.  You're making assumptions
about how interconnection occurs, and implementing a protocol BCP, that has
dependancies on what a network engineer can do/afford/justify/etc... a
dangerous game.   It is probably an ok assumption to say that every
geographic provider is not connected to each of it's 'peers' (in the
they-do-the-same-business sense of the word) in region.    Will the world
get better or worse?
Are we REALLY assuming filtering in iBGP? Really?  It
would seem that is a tertiary requirement out of every thread I see in this
solution.  I would like to see more operators weigh in on the feasibility of
these solutions.

rjr

->
->Iljitsch van Beijnum
->
->

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 22 Oct 2002 14:56:51 +0200 (CEST)

> That presupposes a result from the multi6 debate, before that debate

I think the debate has started some two years ago, when this wg was
formed. Unfortunately, the debate hasn't finished yet. The way I see it
is that we really need some level of working multihoming in IPv6 to get
it deployed, and probably in a way that doesn't require much
renumbering, preferably none, for large organizations. Guess what? PI
does that, and nothing else is even on the horizon. So either IPv6
remains a playpen for several more years, or we get some form of PI.

> have a better solution to aggregation than that.

So far, the people with the deep pockets haven't entered the v6 arena.
If and when they do, they'll get their routes in the DFZ, whether we
like it or not.

If there was another solution that could be deployed in (say) two years,
that would make things different...

---

From: "Michael H. Lambert" <[email-masked]>
Date: Tue, 22 Oct 2002 09:11:44 -0400

> support this, and most of the newer ones (especially web browsers) don't.

As a (perhaps) extreme case in this scenario, consider the situation of two 
sites which can communicate over either the commodity Internet or a 
Research and Education network (most likely high bandwidth and/or low 
latency).  If viewed only from a reachability standpoint, either path would 
work, but there are applications which demand the latter.  Simply returning 
a list of IP addresses to be tried in order is probably not sufficient in 
this case (in fact, both of these sites might have addresses, which, 
because of conditions of use on the network [ie, routing policy], cannot be 
reached from an arbitrary third site).

What is the validity of this statement:  All PA addresses (at least at RIR 
allocation boundaries) MUST appear in the DFZ.

Michael

+-----------------------------------------------------------------------+
| Michael H. Lambert, Network Engineer           Phone: +1 412 268-4960 |
| Pittsburgh Supercomputing Center               FAX:   +1 412 268-8200 |
| 4400 Fifth Avenue, Pittsburgh, PA  15213       [email-masked]        |
+-----------------------------------------------------------------------+

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 22 Oct 2002 15:35:39 +0200 (CEST)

On Tue, 22 Oct 2002, Michael H. Lambert wrote:

[Listing several addresses for a server and the application trying them
all in order to achieve a low degree of multihoming functionality]

> reached from an arbitrary third site).

This assumes a network/server/service has several addresses with each
address tied to a certain service quality. Is this the best way to go?
Obviously, at some point somewhere, the decision has to be made to
select the higher quality path. What would be the best place to do this?

The simple solution is to use the same addresses over both links and
simply route all traffic over the higher quality link.

A more advanced solution would be that the application or some kind of
classifyin device sets a diffserv code point that then makes sure the
appropriate path is chosen.

The solution where there are different addresses with "hidden meanings"
doesn't appeal much to me.

> allocation boundaries) MUST appear in the DFZ.

In the good old days many people requested routable (but not PA) space
with no intention to connect to the net. Having unique space is helpful
when interconnecting privately with others. There is even an RFC "RFC
1597 space considered harmful" or words to the same effect.

Also, what are you going to do with people that break this rule?
Disconnect them from the net?  :-)

Iljitsch

---

From: "Craig A. Huegen" <[email-masked]>
Date: Tue, 22 Oct 2002 10:32:52 -0500 (Central Daylight Time)

> have a better solution to aggregation than that.

I have to disagree here.  Adoption of IPv6 in enterprises REQUIRES
multihoming.  It's not a "nice-to-have".  Without it, IPv6 in enterprises
is limited to the labs.  If this means we have to accept some type of PI
addressing in the near-term, then so be it, with the condition that those
using it are given time frame <x> to migrate to the Ultimate Multihoming
Solution in the future.

As for the original multi-PA multihoming solution, it doesn't fly in an
enterprise.  The selection of source and destination addresses by the
originating host keeps enterprises from managing bandwidth on circuits,
because the routing infrastructure has no idea of possible alternate paths
for the packet.  BGP isn't perfect either, but it's much better than how
many bits a particular address has in common with another.  Managing n+1
prefixes per subnet where n is number of providers serving a site is a
nightmare.  The list of problems goes on...

If the goal was to design a theoretical protocol without end-user
operational considerations that can keep DFZ routing tables under 16k
prefixes, then we're doing a good job at designing multi-homing.  If the
goal is to gain adoption by and reliability for the end users of the IPv6
internet, we're failing.

/cah

---
Craig A. Huegen, Chief Network Architect      C i s c o  S y s t e m s
IT Transport, Network Technology & Design           ||        ||
Cisco Systems, Inc., 400 East Tasman Drive          ||        ||
San Jose, CA  95134, (408) 526-8104                ||||      ||||
email: [email-masked]       CCIE #2100      ..:||||||:..:||||||:..

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 22 Oct 2002 17:54:56 +0200

Craig,

I would love to be able to agree with you. Of course I do agree
that multi-homing is a must-have. But we just can't put the BGP4+
table on the exponential path. It's very unfortunate that this WG
has made no progress on resolving that dilemma.

New thinking is needed here if we are not to reproduce the old
problems.

   Brian

---

From: Shane Kerr <[email-masked]>
Date: Tue, 22 Oct 2002 17:58:46 +0200

On 2002-10-22 10:32:52 -0500, Craig A. Huegen wrote:
> 
> As for the original multi-PA multihoming solution, it doesn't fly in
> an enterprise.  The selection of source and destination addresses by
> the originating host keeps enterprises from managing bandwidth on
> circuits, because the routing infrastructure has no idea of possible
> alternate paths for the packet.  

A multi-PA multihoming solution can use several techniques to allow
the destination host to determine the addresses used, especially if it
is layer 4 or higher.  It seems like a lot of of end-sites would be
happy with redundancy first, speed second, and least-cost routing as
icing on the cake.

Do you think enterprises would you reject out of hand a solution that
uses DNS-style RTT optimisation for destination address selection?

> BGP isn't perfect either, but it's much better than how many bits a
> particular address has in common with another.  Managing n+1
> prefixes per subnet where n is number of providers serving a site is
> a nightmare.  The list of problems goes on...

Maybe you can point me to a URL of the specific draft you're
addressing here (I only see the hoary requirements document at the
IETF site).

-- 
Shane Kerr
RIPE NCC

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 22 Oct 2002 18:23:37 +0200 (CEST)

> has made no progress on resolving that dilemma.

I figured the whole geographical aggregation thing was a step in the
right direction...

I'll be sending the draft to the IETF later today or tomorrow (just
waiting for some last minute feedback) but I guess it's ready:

http://www.muada.com/drafts/draft-van-beijnum-multi6-geo-aggr-00.txt

(The title is now "Geographical Aggregation to Support Multihoming in
IPv6", this used to be "Geo For Now" (GFN).)

> New thinking is needed here if we are not to reproduce the old

Some face-to-face discussion would also be good. How do we get multi6
stuff on the agenda for Atlanta?

Iljitsch

---

From: "Craig A. Huegen" <[email-masked]>
Date: Tue, 22 Oct 2002 11:44:24 -0500 (Central Daylight Time)

> has made no progress on resolving that dilemma.

I wouldn't suggest PI as the long-term fix.  Rather, I would suggest that
we use it for adoption and support PI space for the short term, on
condition that it is returned in <x> timeframe when whatever solutions we
come up with are filled out.

> New thinking is needed here if we are not to reproduce the old

I agree that new thinking is needed.  At the same time, if any of the new
thinking requires router changes, or even worse, host stack changes, the
protocol continues to stagnate because there are few end-users beyond
research, and single-homed home use.  I'm not proposing that we just say
IPv4 multihoming is the solution and be done with it.  However, it does
give us a few years of breathing room to come up with a real solution.

/cah

---

---

From: Eliot Lear <[email-masked]>
Date: Tue, 22 Oct 2002 09:47:36 -0700

Brian E Carpenter wrote:

> I think we need the document

Why?

Eliot

---

From: "Craig A. Huegen" <[email-masked]>
Date: Tue, 22 Oct 2002 12:02:54 -0500 (Central Daylight Time)

> happy with redundancy first, speed second, and least-cost routing as

Enterprise end-sites want equivalent functionality to IPv4.  It also has
to be manageable, such that the cost of managing the multi-homing solution
is equal or less than that of managing multi-homing in IPv4 today.  They
need to be able to trust the infrastructure enough to make their AAAA RR
part of the records for www.<domain>.com.

Given the business environment of the IPv4 Internet, I don't think you can
trade off speed for redundancy.  Multi-homing in IPv4 today delivers both.
We need to deliver both.

> uses DNS-style RTT optimisation for destination address selection?

When the host selects the path by virtue of address selection, such that
the routing infrastructure can see no alternate paths to the destination,
then enterprise network operations teams have no capability to manage
traffic policy.  I believe the majority of enterprise network operations
teams would have an issue with that.

We should keep traffic routing within the network layer, IMO, or at least
give visibility to the network layer of alternate paths that can be taken
for a variety of policy reasons.

> addressing here (I only see the hoary requirements document at the

draft-ietf-ipngwg-default-addr-select-06.txt specifies the use of the pair
of addresses with most common bits (left-most) in a set of source and
destination address pairs.

The IPv6 specifications for aggregatable address architecture offer that
for multihoming, each segment could have multiple IP subnets from
different PA-assigned /48's for an end-user.  The host then uses the
selection mechanism in default-addr-select to select the source and
destination addresses to use.  An enterprise would have to assign, per
segment, a /64 from PA space from each provider plus perhaps a site-local
/64 as well, and manage that list of prefixes across the entire enterprise
network.

/cah

---

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 22 Oct 2002 22:27:30 +0200

> stuff on the agenda for Atlanta?

Persuade the chairs to schedule a meeting.

   Brian

---

From: Pekka Savola <[email-masked]>
Date: Tue, 22 Oct 2002 23:48:17 +0300 (EEST)

> Persuade the chairs to schedule a meeting.

Discuss what?  Solutions?  Forget about it...

I think we need some concrete proposals on how to proceed with the
requirements (it's a nice piece of work IMO but will probably be a bit
useless: no solution is likely to satisfy all the criteria..) or what.  
There has been some discussion on that already, good.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 22 Oct 2002 23:30:27 +0200 (CEST)

> Discuss what?  Solutions?  Forget about it...

Why not? Ultimately, what people need is solutions, not requirements.

> There has been some discussion on that already, good.

Allow me to repeat myself: the current requirements are as good as they
are going to get. We can officially adopt them and live with their
inherit problems, or reject them and live without requirements. Either
is a valid option. But spending more time on refining them will not
provide additional benefits. These requirements are all reasonably
obvious, and it is just as obvious that we're not going to arrive at
solutions that fullfill all of these requirements completely.

At some point, they'll have to be applied to parts of the solution space
to decide whether a certain type of solution is acceptable or not. We
might as well start with that part now. So:

- Does geographical aggregation meet the requirements to such a level it
  would be worth our while to develop it further?
- Do address agile transport protocols meet the requirements to such a
  level it would be worth our while to develop them further?
- Does application-based multihoming meet the requirements to such a
  level it would be worth our while to develop it further?
- Does dynamic aliasing/tunnelling meet the requirements to such a level
  it would be worth our while to develop it further?

---

From: Pekka Savola <[email-masked]>
Date: Wed, 23 Oct 2002 00:36:02 +0300 (EEST)

> Why not? Ultimately, what people need is solutions, not requirements.
[...]

Chairs didn't see this as necessary before, I doubt they see it now..

I agree on most of your points.

---

From: "Tony Hain" <[email-masked]>
Date: Tue, 22 Oct 2002 21:20:58 -0700

Iljitsch van Beijnum wrote:
> ...
> So far, the people with the deep pockets haven't entered the 
> v6 arena. If and when they do, they'll get their routes in 
> the DFZ, whether we like it or not.

This is the bottom line to:
http://www.ietf.org/internet-drafts/draft-hain-ipv6-pi-addr-use-03.txt

To a first order I don't care what PI mechanism we end up with, but it
is very clear that one will happen. If we do nothing it will simply be
random holes punched in the PA space. 

I have continued working on the particular proposal because even the
minimal implementation decouples the 120 degree regions that would cover
EMEA/AsiaPac/Americas. This has value and does not require a deployment
to cover the problem of an individual connection terminating outside its
natural geo region. At the same time, it is conceivable that the 200+
existing exchange points and private peering connections would allow the
scaling to be much better than the simple 3.

Tony

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 23 Oct 2002 13:46:30 +0200 (CEST)

> just ship the thing as Informational.

I second that commotion. After reading
http://www.ietf.org/internet-drafts/draft-ietf-multi6-multihoming-requirements-03.txt
again it seems to me that the problem isn't so much that the
requirements are unreasonable or clash (well, at least not all that
badly), but that many of the requirements aren't quantifiable. For
instance, a solution should be simple. Coulnd't agree more. But what is
simple? To me, configuring a router for BGP is much simpler than driving
a car, as I do the former on a weekly basis and don't have a driver's
license. For other people, this may very well be different.

In any event, the requirements as they are now should provide a basis to
prefer one solution over the other, if one solution consistently meets
the requirements to a larger degree than another.

So that would be:

YES to the current requirements, but without the normative language:

Brian E Carpenter <[email-masked]>
Iljitsch van Beijnum <[email-masked]>

NO to the current requirements:

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 23 Oct 2002 15:25:52 +0200 (CEST)

> random holes punched in the PA space.

Actually holes in PA space is perfectly fine for the network, as you can
filter the more specifics and still have connectivity. However, it is
not so great for the organization using the addresses since this doesn't
address all failure modes.

> scaling to be much better than the simple 3.

We did discuss this proposal at length on a different mailinglist. My
view was that it needs to address the problem of address density at the
poles and the fact that the aggregation boundaries don't fit either
topology or geography. This could be done by having several instances of
this address space and have two places that would otherwise be
aggregated together sit in a different instance. By rotating the axes
and limiting everything to (say) from -60 to +60 degrees you eliminate
the pole problem.

Still, there is one unanswered question: what problem is solved by
having such a close relationship between geography and address space?

---

From: Michael Richardson <[email-masked]>
Date: Wed, 23 Oct 2002 12:16:18 -0400

>>>>> "Eliot" == Eliot Lear <[email-masked]> writes:
    Eliot> I suspect the authors cannot agree on how to proceed,
    Eliot> but I haven't actually discussed the matter with them.

  I think that there is other stuff occuring as well.
  I know that some of the authors have changed jobs vastly.

  Sean, can you put us back on track?

    Eliot> The question then turns to this: should we revisit the
    Eliot> requirements document, or simply discard it and place effort more
    Eliot> along the lines of the note sent by Iljitsch van Beijnum?  It's
    Eliot> not that the document is completely wrong.  I just don't want
    Eliot> architects to be so bound by it that we end up with something less
    Eliot> lasting or useful.

  I would like to see things divided up into three categories:
  1) agreed to requirements. These are EXPLICITELY in scope.
  2) agreed to non-requirements. These are EXPLICITELY OUT OF SCOPE.

  3) requirements which to be controversial. These will be debated, and
likely it is based upon whether we ultimately rule it in scope or out
of scope that will determine the nature of the solution.

  Iljitsch has listed the possible solutions. The problem is that we don't
know how we are going to decide on which is best. 

  I think that I know the best solution, but I have learnt that not every
agrees, and some run away screaming. I would like to walk through the
proposals, and get a better understanding of the objections to them.

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [

---

From: Michael Richardson <[email-masked]>
Date: Wed, 23 Oct 2002 12:19:46 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "RJ" == RJ Atkinson <[email-masked]> writes:
    RJ> On Monday, Oct 21, 2002, at 17:24 America/Montreal, Iljitsch van
    >> allocation mechanism that supports this off the ground.

    RJ> At best, a different term needs to be found other than "geographical
    RJ> aggregation".  At worst, this is a bad idea.

    RJ> The problem is that just because I am in city X, that still does not
    RJ> mean that any of the ISPs that I connect to will interconnect inside
    RJ> X.  In all cases so far, I am on a tail circuit that goes to a

  Perhaps this is a problem!
  1) Maybe we can never solve the aggregation problem until we change this
     statement.

    RJ> In short, the IP routing topology is NOT closely congruent with the
    RJ> world's geographical topology.  If we assign addresses not congruent

  2) maybe geographical based addresses are always tunnelled. They are
     not intended to be efficiently routed, just efficiently aggregated.

  So, Ran, you are perfectly correct that geographical addresses are useless
in an IPv4 model of multihoming.

---

From: "Tony Hain" <[email-masked]>
Date: Wed, 23 Oct 2002 13:08:35 -0700

Iljitsch van Beijnum wrote:
> ...
> Actually holes in PA space is perfectly fine for the network, as you 
> can filter the more specifics and still have connectivity.

This statement is self contradictory. 

> However, it is not so great for the
> organization using the addresses since this doesn't address
> all failure modes.

The point I was making about punching holes in PA is that the more
influential organizations will get the full prefix routed globally.

> view was that it needs to address the problem of address density at 
> the poles

Over-engineering the approach doesn't accomplish the goal. 

> and the fact that the
> aggregation boundaries don't fit either topology or
> geography. 

By geography, I assume you mean geo-political agreements. It clearly
matches the global coordinate system those agreements are based on.

> This could be done by having several instances of
> this address space and have two places that would otherwise
> be aggregated together sit in a different instance. By 
> rotating the axes and limiting everything to (say) from -60 
> to +60 degrees you eliminate the pole problem.

I played with rotating the axis, but the only thing that made Europe
contiguous was to put the origin on a 22.5 degree mark. This would
unnecessarily complicate the rest of the world, for the simple gain of
reducing the London prefix set from 2 to 1. If it were 20 to 1 I would
reconsider, but I don't see how a single prefix makes it worthwhile.
Also, setting the boundaries at +/-60 does nothing useful in terms of
available address resolution. You would have to restrict it to +/-45 to
get a bit back, and that is clearly not a wide enough band.

> having such a close relationship between geography and address space?

I was not trying to force any particular alignment. The goals were: 
Uses current BGP protocol. 
Aggregate the basement-multi-homer, while recognizing the CNN's of the
world will always get a full prefix announced. 
Decouple the scale of aggregation, so it becomes a regional decision. 
Make it simple to derive by basing it on a globally consistent
reference.

The use of WGS84 was based on the fact that handheld receivers are cheap
and available, so the L1 installer can easily identify the reference
point. Bit-interleaving the result decoupled the aggregation scale
decision when using current BGP prefix exchanges.

Is it the most efficient use of address space? No, and the document
recognizes there is a clear trade-off being made between simplicity and
efficiency. Are there other ways to achieve the listed goals? Probably,
and when I see something that looks more operationally viable I will
back off. 

Tony

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 24 Oct 2002 11:08:49 +0200 (CEST)

> This statement is self contradictory.

How so?

> influential organizations will get the full prefix routed globally.

So then the question becomes: how do we limit the growth in
organizational influentialness?  :-)

This is bad for operators as they have to make complex filters. Also,
this "solution" still requires renumbering.

> contiguous was to put the origin on a 22.5 degree mark.

That would be shifting. What I mean is have at least three instances,
one with the poles at 90 north/south, one with the "poles" at the
equator at the 0 and 180 meridians and one with the poles at the
equator at the 90 and 270 meridians. Then you only need to go upto 45
degrees and still every part of the world would be covered by two
instances. This means 1.5 times more address space used, but 2 times as
much usable in the non-polar regions.

> Make it simple to derive by basing it on a globally consistent

You fail to address my question. This scheme uses an incredible amount
of address space, but still manages to come up short in very densely
populated areas. The aggregation properties are relatively poor. So what
is it that makes all these downsides worth it?

Basing a geographical address allocation scheme on population (x
addresses per person) rather than geography (x addresses per square
kilometer) makes more sense, as long as you're not going to use IP
addresses to aim your microwave antenna or something like that.

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 25 Oct 2002 00:08:45 +0859 ()

Huegen;

> for the packet.

According to your theory, the Internet today is not working, because
bandwidth of TCP is managed by individual hosts and the routing
infrastructure has no idea on it.

The reality is that L2/3 traffic engineering is a fallacy. It was
invented by people insisting on MPLS, even after they must admit
an obvious fact that MPLS routers are more expensive but no faster
than plain routers.

The real traffic engineering is that, if there is not enough link
capacity, add more fiber.

> BGP isn't perfect either, but it's much better than how

The nightmare is to manage n ASes where n is the number of multihomed
sites in the world.

							Masataka Ohta

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 25 Oct 2002 00:32:57 +0859 ()

Iljitsch;

> I figured the whole geographical aggregation thing was a step in the

A problem is that such geographical aggregation can not properly
handle multiple links between regions maintained by multiple entities.

Another problem is that it is helpless for a partitioned geographical
area.

							Masataka Ohta

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 24 Oct 2002 18:19:19 +0200 (CEST)

> handle multiple links between regions maintained by multiple entities.

Regions aren't "maintained" by "entities". You simply route traffic for
a region to a router inside your own network that has more specific
routing information for the region. Obvioulsy, it makes sense to put
that router inside that region, but you don't have to.

> Another problem is that it is helpless for a partitioned geographical

You can deaggregate. But automatic deaggregation is dangerous, so yes,
partitioning is bad. This means you must define your regions in such a
way that it is very unlikely that they will be partitioned internally,
from the rest of the network or from peers in the region. This makes it
hard to arrive at very small regions which in turn sets a practical
limit on the scalability of geographical aggregation. In theory, it can
scale to 1G multihomers. In practice, it won't be much fun anymore
beyond 25 - 100M.

---

From: "Tony Hain" <[email-masked]>
Date: Thu, 24 Oct 2002 10:22:02 -0700

> > This statement is self contradictory.

By definition, when there is filtering, connectivity is being
intentionally broken. 

> > > organization using the addresses since this doesn't address all 
> > > failure modes.
> 

You can't, so it becomes how do we scale to deal with them?

> 
> This is bad for operators as they have to make complex 
> filters. Also, this "solution" still requires renumbering.

This depends on the mechanism that is selected.

> > > aggregated together sit in a different instance. By rotating the 
> > > axes and limiting everything to (say) from -60 to +60 degrees you 
> > > eliminate the pole problem.
> 
> That would be shifting. What I mean is have at least three 
> instances, one with the poles at 90 north/south, one with the 
> "poles" at the equator at the 0 and 180 meridians and one 
> with the poles at the equator at the 90 and 270 meridians. 
> Then you only need to go upto 45 degrees and still every part 
> of the world would be covered by two instances. This means 
> 1.5 times more address space used, but 2 times as much usable 
> in the non-polar regions.

What is the real gain?? If it is only 2x, the size of the grid shrinks
from 10x10m to 7x7m. 

> > Uses current BGP protocol. Aggregate the 
> basement-multi-homer, while 
> > recognizing the CNN's of the world will always get a full prefix 
> > announced. Decouple the scale of aggregation, so it becomes 
> a regional 
> > decision. Make it simple to derive by basing it on a globally 
> > consistent reference.
> 
> You fail to address my question. This scheme uses an 
> incredible amount of address space, 

But you were arguing for using 1.5x the space???

> but still manages to come 
> up short in very densely populated areas. 

This is BS... Show me a space where 128 /64's per vertical meter is
insufficient. 

> The aggregation 
> properties are relatively poor. 

Based on what criterion? For the 200+ existing exchanges it would appear
to reduce the scope of the flat routed parts of the table. Also, the bit
interleave allows for varying aggregation scale based on local need.

> So what is it that makes all 
> these downsides worth it?

Show me a plan that works better and is simple enough to really deploy.

> 
> Basing a geographical address allocation scheme on population 
> (x addresses per person) rather than geography (x addresses per square
> kilometer) makes more sense, as long as you're not going to 
> use IP addresses to aim your microwave antenna or something like that.
> 
PA is an appropriate technology for X addr/person. What we are
discussing here is how we deal with multi-homed entities. What that
requires is a defined way to abstract without loss of utility. 

Tony

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 25 Oct 2002 08:30:50 +0859 ()

Iljitsch;

> Regions aren't "maintained" by "entities".

Oops, I mean links are maintained by entities.

But, regions not maintained by a entity will naturally be partitioned
into smaller regions, which makes the routing table explode.

> scale to 1G multihomers. In practice, it won't be much fun anymore

In theory and practice, it can scale to 100K.

							Masataka Ohta

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 25 Oct 2002 08:47:43 +0200 (CEST)

On Fri, 25 Oct 2002, Masataka Ohta wrote:

> > In theory, it can

> In theory and practice, it can scale to 100K.

What am I, stupid? I don't need to take this crap. Go irritate someone
else.

---

From: Simon Leinen <[email-masked]>
Date: 29 Oct 2002 10:32:36 +0100

Craig,

I absolutely agree with Ohta-san's response to yours - I see this
issue as fundamentally similar to the bandwidth management issue.

Bandwidth is obviously something that IT people want to manage, but
yet this is done on the end systems on the Internet (in TCP).

Of course some may argue that this has failed, because there sometimes
congestion somewhere on the Internet, and IT people are forced to buy
rate shapers and other middleboxes to gain the control back from end
systems, and lots of other people have spent a lot of effort to devise
various ways to improve the QoS of the network, most of them involving
making the network handle "less-important" traffic less well.

But the fact is that for many (most?) people the Internet works or
would work well even without all these things.

I think there are good ("end-to-end") arguments for considering to
leave the path selection decision to end systems too.  For instance,
applications run on end systems, and are in a good position to choose
a path that fits their requirement best (cheapest/quickest
response/highest bulk throughput...).

On Tue, 22 Oct 2002 12:02:54 -0500 (Central Daylight Time), "Craig
A. Huegen" <[email-masked]> said:
> When the host selects the path by virtue of address selection, such
> that the routing infrastructure can see no alternate paths to the
> destination, then enterprise network operations teams have no
> capability to manage traffic policy.  I believe the majority of
> enterprise network operations teams would have an issue with that.

Yes, but the question is how much we should care about that.  It's
true that IT people are the ones who have to buy our stuff, but in the
end our requirements are driven by business (in the large sense) needs
of "end users".

And you can rest assured that someone will sell those IT people boxes
that give them back some control over their vict^H^H^H^Hcustomers'
machines' path selection decisions, just like rate shapers "fix" TCP's
rate computation in many enterprises today.

> We should keep traffic routing within the network layer, IMO, or at
> least give visibility to the network layer of alternate paths that
> can be taken for a variety of policy reasons.

Or we push the policy information out to the edge, so that end systems
can select paths accordingly.
-- 
Simon.