Subject: RE: Preserving established communications (was RE: about draft-nordmark-multi6-noid-00)

From: Erik Nordmark <[email-masked]>
Date: Mon, 3 Nov 2003 10:45:26 +0100 (CET)

> IMHO You would want to use the source address based routing when you don't
> want to use source address rewriting.
> There are some situations where you don't want the rewrite to take place and
> you want source address based routing. I can think of the following:
> - The external host don't support M6
> - The M6 context has not been established yet
> - An ULP hint made the M6 layer to select a different path, so you want to
> let the host select the ISP and not the routing system

I understand why the first bullet doesn't want source rewriting, but from
that it doesn't follow that source address based routing is needed.

The second case can be made to handle rewriting by carrying the
identifiers explicitly in those packets. This has the cost of
additional overhead during the M6 context establishment, but might have
benefits in terms of more quickly being able to detect failures during
that establishment; need to understand the details in that tradeoff.

> So a possible approach would be that if the rewrite ok bit is set, then you
> perform destination address routing and if the rewrite ok bit is not set you
> do source address based routing.
> Does this makes sense?

I don't understand what pieces of the total routing system
needs to be changed to support source address based routing, so I'm not
sure this makes any sense.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Mon, 3 Nov 2003 15:05:22 +0100

> > IMHO You would want to use the source address based routing
> > when you don't want to use source address rewriting.
> > There are some situations where you don't want the rewrite to
> > take place and you want source address based routing.
> > I can think of the following:
> > - An ULP hint made the M6 layer to select a different path, so
> > you want to let the host select the ISP and not the routing system
>

Sorry, you are right.
You don't want source address rewriting and you need an ingress filtering
compatibility mechanism, which could be source address based routing. There
are other options for this as decribed in Christian's draft, such as
- a routing header (or tunnel) initiated from the host
- an icmp source address error from the exit routers back to the host
- tunnels between exit router (sort of source address based routing limited
to the exit router)

> that establishment; need to understand the details in that tradeoff.

Ok.

And the third case could be also addressed with some of the mechanisms
mentioned above, i guess that source address based routing or routing header
can be used for this case. ICMP and tunnel between exit router are not good.

So, backward compatibility with non M6 enabled host require a alternative
mechanism to source address rewriting. ULP hints would also require it. And
packets exchanged before context establishement would also benefit from it.
The identified mechanisms to provide this would be source address routing or
a routing header.
Do you agree with this summary?

[...]
> sure this makes any sense.

Ok.

The question here is whether every multi-homed ISP will be able to get an
allocation or it will have to obtain addresses form its upstream providers,
right?

Well let's assume that it will obtain addresses from their upstream
provider.
Let's asume that ingress filtering will be in place, ok?

Now, how would you deal with this without source address based routing?

Let's start with ingress filtering compatibility:
the options were:
- source address based routing. besides the multihomed site all isps that
don't have its own allocation would need to implement it
- routing header. this would require n addresses in the routing header, one
per upstream isp than don't have its own allocation, imposing extra
overhead, and besides, i am not quite sure how the end host would discover
the addresses of all the exit routers in the the path ( i am not sure how it
will find out n, neither)
- icmp back to the host. You have already mentioned some security concerns
about letting externally generated ICMPs influence the path selected by
hosts
- tunnel between exit router. well, i guess that this would be really
complex, since you would require tunnels between all exit router of the isp

If you want to support ULP hint mechanism you only have the first two
options.

So, IMHO even if not all the multihomed isps won't get a direct allocation
from the rirs, source address based routing seems the simplest option.

Regards, marcelo

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 3 Nov 2003 15:30:38 +0100

> - tunnels between exit router (sort of source address based routing 

Another way to handle this would be to bascially build two networks all 
the way down to the host:

ISP A     ISP B
    |        |
Router    Router
    |        |
Router    Router
      \   /
      Host

So the ISP selection happens inside the host, once a network is 
selected it is no longer possible to deviate from the path intended by 
the host.

> allocation or it will have to obtain addresses form its upstream 

Current rules say you must have 200 customers that take IPv6 addresses 
from you in two years. Now obviously there are going to be ISPs for 
which this is a problem, but I don't see how we can arrive at very 
large numbers of multihomed sites that use ISPs with less than 200 
customers.

An alternative here would be for the ISP to set up routing with such a 
multihomed customer over only one of its own ISPs.

> about letting externally generated ICMPs influence the path selected by

This isn't a solution as packets that belong to existing sessions can't 
be retransmitted with different source adddresses.

> complex, since you would require tunnels between all exit router of 

This is really the same thing as my "two separate networks" thing 
above, except that no actual hardware is used.

---

From: masataka ohta <[email-masked]>
Date: Mon, 03 Nov 2003 23:34:06 +0900

marcelo;

See my draft on how end systems know proper source addresses
corresponding to chosen destination addresses.

							Masataka Ohta

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 3 Nov 2003 17:21:54 +0100 (CET)

> Do you agree with this summary?

Another possible solution in some scenarios is that the ISP's ingress filtering
is relaxed to allow all the prefixes assigned to the site. That might not be
realistic for the low-end services for home users, but it might be a useful
approach in other cases.

> Now, how would you deal with this without source address based routing?

Not doing strict RPF ingress filtering between the ISPs seem to be the
current operational approach for such problems.

My gut feel currently is that one would need a really strong case
to make the routing system operate on source and destination addresses.
I've yet to see such a strong case.

But perhaps tweaking the routing system to do so is downright trivial
and I should sit down and shut up.

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 3 Nov 2003 17:50:59 +0100 (CET)

> selected it is no longer possible to deviate from the path intended by 

Isn't that what source address based routing implies conceptually?
It can be implemented while sharing the routers as long as the routing
table has the entries learned from ISPA as tagged to only apply to packets
with a source address from A's prefix and vice versa.

   Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Mon, 3 Nov 2003 20:15:55 +0100

> realistic for the low-end services for home users, but it might

Ok, as it is stated in Christian's draft, this will definitely be the case
for some sites for it is also certain that for other smaller sites this will
not be the case.

> and I should sit down and shut up.

Well, IMHO source address based routing is a good option for small sites (at
least).
you can easily implment it is this case, since you can use currenlty
available Policy based routing mannually configurated in the (few) routers
on the site.
For larger sites, this *may* be more difficult, and it also presents
additinal difficulties, such as the fact that policing is performed by the
hosts (which is seen as a problem by some site admins) (BTW the same admins
have claimed that multiaddressing is a problem for them too)

So we are again in the situation where one size fits all solution seems more
difficult than a per case situation.
Some people have expressed that big sites will continue doing multihoming as
they do now in IPv4 and that that is ok.
Do you think that we should look for different solutions depending on the
characteristics of the site?
and,do you think that for small sites source address based routing seems to
make sense?

Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 4 Nov 2003 11:21:19 +0100 (CET)

> and,do you think that for small sites source address based routing seems to

I actually currently trying to understand what will all be needed to provide
flexible source address based routing. For instance,
 - is there an impact on routing protocols needing to carry additional 
   information?
 - is there an assumption about who wins when the sender has specified
   a source address but the routing system knows that sending packets
   out that ISP doesn't work? is less likely to work than sending it
   out some other ISP?

Once I understand it a bit more maybe I can answer your question whether
we need different approaches for different types of sites.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Tue, 4 Nov 2003 11:56:40 +0100

>  - is there an impact on routing protocols needing to carry additional

Source address based routing needs as many routes as different prefixes it
has configured, ok?
For example, if a site is dual homed and it has two different prefixes, then
the source address based routing system requires two routes.
Moreover, these routes have to point to the site exit router corresponding
to the ISP that has delegated the prefix.
This don't need to be dynamic, since the isp that delegated the prefix
doesn't change
So i guess that there is no need for special routing protocols to support
source address based routing

>    out that ISP doesn't work? is less likely to work than sending it

I guess this is what we were discussing when we were considreing the rewrite
ok bit.
I think that the host should be able to decide if the packet will be routed
according the destiantion address (in which case rewriting is probably
needed to deal with ingress filtering) or it will be routed based on the
source address (in which case no rewriting occurs)
I see source address based routing as a mechanism to let the host source
address selection act as ISP selection mechanism that overrules the isp
selection performed by the routing system.
The host then would force the ISP selection in cases when it knows better
(ULP hints) or when it knowns it can't use source address rewriting (the
other end in not M6 enabled)
Regards, marcelo

---

From: masataka ohta <[email-masked]>
Date: Tue, 04 Nov 2003 20:48:14 +0900

marcelo;

> to the ISP that has delegated the prefix.

And the site exit router is determined by the destination address.

> This don't need to be dynamic, since the isp that delegated the prefix

Wrong. Source address is not an independent variable. Destination address
is.

> source address based routing

Yes, of course. We can just use an existing or a new routing protocol
to compute the proper source address for a destination address.

						Masataka Ohta

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 4 Nov 2003 13:00:56 +0100 (CET)

> source address based routing

So you would envision manually installing the rules in each internal router
that says "packets with source address prefix X gravitate towards exit
router Rx"?
Or do you envision internal routers infering this from information already
present in the IGP?
Or do you envision tunnels between the exit routers to get the packets
to the correct exit based on the source address prefix?

> source address (in which case no rewriting occurs)

In the latter case, if the packet arrives at an exit router and
the ISP matching the source address is known to be down, should the router
just drop the packet?

What if there never was a matching or preferred route for the destination
address through that ISP?

I think when source address based routing is used it is benefitial
to also look at the destination address when chosing routes.
One question is the details of the relationship between the source based
lookup and the destination based lookup.

> other end in not M6 enabled)

But the host will never know better than the routing system because it
is operating on aggregated information.
What the host might know better than the routing system is that the
last N packets sent using a given source and destination didn't result
in packets being returned from the peer, thus something might be
broken.

Thus for the host to tell the routing system "please try sending this packet 
over a different path than the default one" might be useful. But that is quite
different than having the routing system blindly honor the source address
for routing lookups.

  Erik

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Nov 2003 14:22:28 +0100

> Isn't that what source address based routing implies conceptually?

I suppose. But the important thing here is that only the host needs to 
have the SABR capability, in this picture the routers must do it:

ISP A   ISP B
    |      |
Router  Router
     \    /
     Router
        |
      Host

> with a source address from A's prefix and vice versa.

In essence we'd be doing type of service routing here, but rather than 
use different types of service we use different source prefixes. But as 
discussed before, simply bypassing normal routing and forward based on 
the source prefix exclusively is simpler.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Nov 2003 14:36:49 +0100

> is operating on aggregated information.

The host??? No, hosts only know about individual correspondent 
addresses, they don't aggregate under normal circumstances.

> different than having the routing system blindly honor the source 

So when we finally find that unused field in the IPv6 header (should 
have stuck to v4, there we can steal identification if DF is set which 
it nearly always is) we can use this field to say "I didn't care much 
for the connectivity provided by your previous routing decisions, 
please try something else". This field would be an index for finding 
the preferred route among the ones available. So if the packets don't 
make it using the best route (as determined by the router), the host 
increases the value of the field and the second best route is selected.

Hm, one way to do this would be to manipulate the hop limit field. 
Under normal circumstances, the hop limit a router sees in packets from 
hosts in the local network should be very consistent. The host could 
raise or lower the initial value for the hop limit to get the router to 
change its behavior.

---

From: "marcelo bagnulo" <[email-masked]>
Date: Tue, 4 Nov 2003 15:27:47 +0100

> that says "packets with source address prefix X gravitate towards exit

That was my initial idea, yes

Note that this is not a dynamic information, since it only changes when the
site changes ISP or when the ISP renumbers, and both events shouldn't be
very frequent (at least not so frequent to demand a dynamic protocol)

> Or do you envision internal routers infering this from information already

No

> to the correct exit based on the source address prefix?

Actually this is an alternative to source address based routing, but i find
it not very optimal, since it implies MTU reduction and sub optimal routing

> the ISP matching the source address is known to be down, should the router

Well if a router don't have a route to route a packet, it should discard it
but that case would depend on the behaviour of the host, let's see this
below...

[...]

> is operating on aggregated information.

Don't understand this...
The routing system is the one who works with aggregated information implying
information loss.
The host deals with no aggregated host information

> in packets being returned from the peer, thus something might be

Well, there are different approaches, here...
For instance when the host receives such an information it could just try
with all the source destiantion address cobination available for that
communication, and see which one returns faster. Such approach would not
only provide detection of available path but also some idea of the faster
path. Clearly it may be expensive
The other option would be just to change both source and destiantion address
and see what happens and if it doesn't work try another combiantion. The
problem here is response time.

> over a different path than the default one" might be useful.

This may work... ( i mean i don't see any reasons why not :-)

And what would be the benefits of such approach in wrt source address based
routing?
This feature is clearly not available in routers today, it requires a bit to
inform it to the routers and also that it would ony be usefull to deal with
M6 capable communications. We still have to provide some way to provide
ingress filtering compatibility for non M6 capable communications. The
benefit of source address based routing is that it also addresses both
situations (or kind of)

Regrds, marcelo

 But
> that is quite

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 5 Nov 2003 13:47:55 +0100 (CET)

> The host deals with no aggregated host information

I guess my statement wasn't very clear.
While the host has some information about reachability of src/dst locator
pairs, it does not have information on how this maps to the exit paths from
the site. For instance, if the host could choose between 3 destination
locators it wouldn't know that for instance the first two result in the same
exit path being used.
That level of information is known in the routing system.

> We still have to provide some way to provide

I'm upleveling this part of the discussion a bit.

I think the goal should be a workable and deployable multihoming solution.
Whether that also provides incremental benefits for steps along the way
(such as current ingress filtering issues for sites that have multiple prefixes
and multiple ISPs) doesn't look like the highest priority to me; if that bogs
us down so that we can't come up with a multihoming solution then I think we
need to ignore it. If we don't have that attitude we might become so entangled
by incrementalism that we can never design the necessary multihoming solution.

  Erik

---

From: masataka ohta <[email-masked]>
Date: Wed, 05 Nov 2003 22:22:29 +0900

Erik;

> That level of information is known in the routing system.

Why do you think host can not get infromation from the routing
system?

In good old days when IPv4 routing table was not bloated, it was
usual for UNIX box to run routed (in quite mode) even if a host
is singly homed.

						Masataka Ohta

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 5 Nov 2003 15:29:03 +0100 (CET)

> usual for UNIX box to run routed (in quite mode) even if a host

I recall reading a draft and slides which proposed using something like RIPv2
to distribute all the external routes from all the edges to the internal
routers and hosts (draft-ohta-e2e-multihoming).

Two issues with that aspect of that proposal: 
1. Folks seem to be want to put IP connectivity in smaller and smaller devices.
   Thus even though a laptop with a few hundered meg of memory can easily
handle
   the full routing table, it isn't obvious to me that it is a good thing
   assuming that all hosts that want to benefit from site multihoming will
   want to store the full routing table.

2. Efficiency/timeliness; when a different exit path should taken from
   a site for a given prefix/address this needs to be communicated rapidly
   and efficiently so that packets can use the new path.
   Sending all updates to the Internet routing table to all hosts is one way
   to convey this, but this means that updates to prefixes which nobody in
   the site care about might be communicated to hundereds of routers and 
   thousands of hosts.  And when a single or a few hosts indeed are interested
   in that update, it will potentially take some time to propagate, especially
   with a distance vector protocol as the distribution mechanism.

   A possible alternative is a packet driven approach at the border router
   where the arrival of a packet causes a signal back to the host.
   This signal could be either a source locator rewrite (which will return
   to the host in packets coming from the peer) or some new ICMP error
   directly to the host.

It would be good if the WG could explore the tradeoffs in this part of
the multihoming puzzle.

   Erik

---

From: masataka ohta <[email-masked]>
Date: Thu, 06 Nov 2003 07:47:48 +0900

Erik;

>    want to store the full routing table.

Smaller device? Read the draft. It says:

   But, with smaller full routing table and larger scale of integration,
   there is no reason not to have full routing table on every end
   system.

That is, my draft takes care of those who don't have much knowledge
on trend of integration scale. But, I can't help those who don't
read it. Smaller and smaller devices are having more and more memory.

In addition, the draft says:

   One may still be allowed, though discouraged, to have local
   configuration with dumb end systems and an intelligent proxy. But,
   such configuration should be implemented with a protocol for purely
   local use without damaging the global protocol.

that people who don't accept the trend of integration are also
taken care of.

Do read the draft.

>    thousands of hosts.

But it is non issue

   with smaller full routing table and larger scale of integration,

and faster network speed.

>    And when a single or a few hosts indeed are interested

A route can be used only after it is processed by the routing system.

So, it all depends on the routing protocol used.

Those who use RIPv2++ on networks with complex loop topology
will suffer a lot of delay.

Those who use OSPF (not OSPF++ but OSPF as is as it has fields to
carry information required for source address selection) on the
topology won't see much delay.

>    to the host in packets coming from the peer) or some new ICMP error

For the discouraged case, I prefer using an intelligent proxy server
to choose a source address for a dumb end system.

> It would be good if the WG could explore the tradeoffs in this part of

Before doing that, do you know how much memory can be put on a typical
chip? Do you know how much memory is consumed for typical code for
TCP/IP stack? Do you know how much memory is consumed for a typical
routing table entry?

These are the basic figures for constructive discussion.

							Masataka Ohta

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Thu, 6 Nov 2003 11:53:55 +0100

On måndag, nov 3, 2003, at 15:30 Europe/Stockholm, Iljitsch van Beijnum 

> large numbers of multihomed sites that use ISPs with less than 200 

This is being changed though. See the discussion in ARIN and RIPE.

- - kurtis -

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 08 Nov 2003 12:11:16 +0100

> very frequent (at least not so frequent to demand a dynamic protocol)

I'm sorry, if we are trying for a mass-market solution we can't assume
any ability to do manual config of routing tables. That doesn't exclude
this approach, but it requires automatic configs (a.k.a. a routing
protocol, or something like Router Renumbering on steroids).   

   Brian