Subject: host or border router

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 11 Dec 2002 16:10:58 +0100 (CET)

[Peter, I was going to reply to your message but decided against it in
order to more easily maintain my own train of thought.]

If we are going to create a multihoming solution that uses identifiers
that are usable as IPv6 adresess for end-to-end identification and
regular unicast IPv6 adresses that replace the identifiers in transit, I
see three ways to handle the address replacement:

1. In the end-host
2. In a site border router (SBR)
3. In some kind of middlebox

I believe doing it in the actual SBR is a non-starter. Sure, you can
build a box that can replace addresses and maintain the state that goes
with this. And sure, you can build a box that can forward IP real fast.
I'm just not sure you can do both at the same time, and even if it's
possible, it won't be cost effective. As far as I know, there are no
software-based routers that can handle IP(v6) forwarding at line rate
for gigabit interfaces. Now obviously this changes all the time (I've
done some tests recently and it looks like off-the-shelf PC hardware has
about reached this performance level not too long ago) but bandwidth
requirements also go up so the fundamental problem should remain the
same.

Anyway, not all that many people have more than 1000 Mbps worth of
bandwidth to the net, but those that do need hardware support. But
AFAIK, most of the really fast boxes are deployed either inside
corporate networks or in ISP networks, where address rewriting for
multihoming support isn't an issue. So this means router vendors have to
bake much more complex silicon for a very small group of customers. I
just don't see this happening, with the result that very large edge
networks (think Google, Yahoo, Amazon) won't be able to talk
multihomedly to their correspondents. That would be very bad.

There is little value in requiring that this happens in the real SBR
anyway. This type of processing can be done on middleboxes (ie existing
firewalls or new special purpose boxes) which either select the right
SBR or work together with the SBR(s) so the source/destination addresses
are compatible with the routing decisions made by the SBRs.

Doing this in middleboxes has the advantage that neither the routers nor
the hosts have to change.

Having said this, I think it is possible to do it on the hosts. This has
the advantage that the state the host has to maintain anyway can be used
to do quick failover to an alternate address. This would be
hard/expensive to implement in an external box.

Whether it's the hosts or middleboxes that do this, we need to support
some important policy/traffic engineering requirements. The ones that I
can think of are:

1. Always prefer link A, use link B only as a backup
2. Use link A for certain destinations/applications, link B for everything else
3. Send/receive x% of traffic over link A, 100-x% over link B
4. Select shortest path path for each destination
5. Combinations of the above

Currently, the routing system only supports 1. relatively well. The
others are doable to a certain degree, but the routing system doesn't
fully support them. To support these, we need:

1 outbound: statically configured policy, reachability information
1 inbound:  statically configured policy must be conveyed to
            correspondent, reachability information
2 outbound: same as 1 outbound
2 inbound:  same as 1 inbound
3 outbound: link usage information
3 inbound:  link usage information must be conveyed to correspondent
4 outbound: topology information or path attribute measurements +
            ability to measure both paths simultaneously
4 inbound:  will be determined by correspondent
5 outbound: 1, 3 and 4 outbound combined
5 inbound:  1 and 3 inbound combined

To make a long story short, we need:

- statically configured policies
- link usage information
- topology information and path measurements, preferably for both
  paths at the same time
- a way to negotiate session parameters with the correspondent
- a way to map identifiers to locators in order to bootstrap the
  negotiation process

Obviously querying the SBRs for link statistics isn't something we want
all hosts to do. So we need a small number of boxes that manage at least
part of the policy/traffic engineering. However, this doesn't mean these
boxes should also handle the traffic: it would be perfectly reasonable
to aggregate the P/TE info here and then distribute this to hosts. A
good way to do this would be "DNSng" where the host looks up a hostname
for a remote host and receives not only one or several IP addresses, but
also extra information such as the preference for each of those
addresses and the source address that goes with them, and DSCP or flow
label suggestions. Note that such a DNSng box wouldn't be able to
provide conclusive mapping information as it doesn't have the very
granular reachability information hosts or boxes that sit in the middle
of the traffic stream have.

Alternatively, all of this can be handled in middleboxes that also do
the actual mapping, so they are aware of which addresses work and which
don't. Then we probably want a protocol to distribute information
between those, but there is no rush there.

Iljitsch

---

From: Peter Tattam <[email-masked]>
Date: Thu, 12 Dec 2002 09:52:00 +1100 (EST)

> order to more easily maintain my own train of thought.]

That's ok.  I was going to expand further on the CCS as to how I'd typically
see it deployed.  However there has been little comment on my idea - just
waiting for feedback before expending more intellectual resources on the ideas.

> 3. In some kind of middlebox 

Whatever solution is deployed, of these 3, I would prefer 1. 

I think there will always be concerns as to whether a SBR will work.  The
problem with what I am propoing is that is isn't optional. it's an all or
nothing idea.  People can opt to use NAT and find a hardware solution that
works.  When they run out of steam, they have to move to real addressing and a
proper network.  Firewalls are much the same.  They function ok when they are
small, but when you get a monster site and you start having to make difficult
choices like making sure it isn't stateful and stuff. 

A middlebox is another point of failure and this would need to be thought
through carefully - can it be made redundant with rapid failover?  Is it
additional hardware cost in an IT world which is starting to count pennies more
than ever?

Since we are trying to design for the future, not for things as we see it now,
I would always prefer a solution that has room to scale.  We should be looking
for solutions where packets are not modified in transit - this is consistent
with IPv6 routing conventions.  That's why doing it in the hosts is starting to
look like the most scalable solution for the long term, but only if we can
constrain the lists of alternative addresses so that the host is not burdened
too much.  If we can get the CCS right, the address twiddling only needs to be
done at connection establishment time and when there is a path failure.  If the
CCS is rugged enough, the host would only need to keep two addresses - the
formal address and the actual address, and trust the CCX to return the right
information.  I still think it's a good idea to authenticate a new address when
it is to be switched over, and keeping the full set of potential MH addresses
is optional for the host.  Whichever way we go, it is important that the
complete list be able to be reconstructed at any point in the lifetime of a
connection - it's a matter of how much the host wants to trust the CCS and how
much MH information it wants to maintain itself.  Perhaps for regular work like
a large web site or somthing, the CCS is adequate, but for mission cirtical
stuff where you want to be absolutely sure you aren't being spoofed (e.g. 
linking exchanges - see later), you do the full MH exchange of addresses at
connection startup and keep the full address lists.

Some more on the CCS - I think that acronym is too overloaded so how about CCX
(Connection Cache eXchange system) but I will continue to use CCS for the time
being. I will refer to nodes in the CCS as exchanges.

I see it as a parasitic system that monitors the BGP system in a read only
manner.  Kind of like an autonomous feedback system for the network, and I'd
visualize it as an organism that has tentacles tapping into border and
transit routers.  An exchange could monitor as many BGP nodes as it feels
necessary and has authority to do so.  Very likely such monitoring will need to
be authenticated to prevent spoofing of CCS information.

Some additional characteristics.  

It would have to push network state information around in some manner and would
also have to be immune to some extent from path failures that it is trying to
prevent - would SCTP be a good candidate here?

Linking exchanges.  

Within a site if an exchange is 1 hop away, we can trust it to the same level
as we already do for ND.  The same applies for adjacent exchanges - if they are
1 hop away, then they can be trusted in much the same way as BGP trusts its own
peers (this assumes that a border router doubles as an exchange).  If an
adjacent exchange is more than one hop away, then I think the traffic should be
authenticated - I don't see this as difficulty because an exchange is a key
infrastructure item and so adding a level of security would prevent hostile
attacks on exchanges.

One advantage over the BGP system is that exchanges can be more highly meshed
than just its neighbors.  It would be possible to mesh exchanges so that a high
degree of path redudancy is maintained so that information can reliably sent. 
However one needs to take care that the right path topology is returned to the
site boundaries so that the correct MH prefixes can be used.

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: Peter Tattam <[email-masked]>
Date: Thu, 12 Dec 2002 10:38:16 +1100 (EST)

Some further thoughts on your comments....

Much of this MH debate is about pleasing as many interests as possible.  While
I think that no one solution is going to please everyone, I think that
compromise and hybrid solutions can be implemented in several ways might come
close. 

That said, one of the major criticisms of a host based solution is that the IP
stacks need changing and we've already deployed a lot of stuff.  I think that
the idea of a middlebox can deal with the legacy issues for the short term and
also serve to supply such infrastructure to machines that it is impractical to
impose such requirements on.

So I guess what I'm saying is that the same result can be achieved with either
host based or middlebox solutions and that we should pursue both directions
simultaneously, with the ultimate goal to have host based MH as the preferred
option for deploying such a system.

Also one possibility might be to combine the middlebox and the CCS exchange as
the two are likely to be strongly related.

There is a lot of mixing and matching that could take place and leave room for
exploration of various configurations of what I've proposed.

Peter