Subject: Transport multihoming

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 24 Oct 2002 21:14:51 +0200 (CEST)

Greg, Peter,

As I see it, the reason to have the multihoming functionality inside one
or more transport protocols is that the transport layer has end-to-end
knowledge that makes it possible to make better multihoming decisions.

Would it be possible to have a modified TCP talk to a non-modified TCP
through some kind of "mudem" (multihomer/demultihomer), without loss of
the core multihoming functionality, and without the "mudem" having to
keep long-term state?

This would create a much more attractive deployment path as people can
choose to either upgrade hosts or put them behind a box to provide the
multihoming functionality.

It also makes it possible to move the multihoming decision making to a
place where it can be better controlled if this is desired.

Iljitsch (or just call me dr. Frankenstein)

---

From: Peter Tattam <[email-masked]>
Date: Fri, 25 Oct 2002 07:14:32 +1100 (EST)

> knowledge that makes it possible to make better multihoming decisions.

My recent thoughts are that it need not be tied directly to the TCP protocol,
but can instead be done at the IP layer of the host stack.  However the TCP and
IP layers should be aware of each other in much the same way that PMTU is
facilitated.  If done that way, there would be benefits from caching the
multihoming information and sharing it over several connections.  Also because
of the strong aggregation, the cached information would be able to build a
multihoming hint tree more efficiently than would a flat list of IP addresses. 
For example, if a major link from one aggregator showed a problem, it would
take provide a hint that all aggregations from that provider would be
inaccessible and the stack could use this intelligence in advance. 

> the core multihoming functionality, and without the "mudem" having to

Depends what you mean by long term.  Please clarify.

This is perhaps close to what I alluded to by a decoupling process.  What is
fundamental to a reliable working solution using my concepts is that if the
prefix replacement is decoupled, it must still be done in a secure way so that
protocols like TCP which might depend on address immutability can have an iron
clad guarantee that the address selection is valid.  

It is clear that moving the process out of the host kernel enviromnent into an
ancilliary processor environment would imply using some kind of secure control
protocol to ensure that addresses are dealt with correctly, and this adds a
degree of complication which I believe to be excessive.

It is my conclusion then that it should be done on the host.  This also has the
advantage of distributing the processing and memory requirements for such a
protocol away from the core of the internet which fit the requirement for being
scalable.  Careful thought needs to be given to huge servers which are likely
to be servicing tens of thousands of connections though as the multihoming
process could add some burden to servers of that calibre.  It is perhaps sites
like these that could benefit from decoupling the multihoming process in a
controlled manner.

I strong suggest that such multihoming be restricted to prefix replacement
only, and not arbitrary address replacement, as there will be significant
advantage in exploiting the implied tree structure imposed by the strong
aggregation.

> place where it can be better controlled if this is desired.

Yes, I see your reasoning, but only if the immutability of end point addressing
and the prevention of connection hijacking can be guaranteed.

> Iljitsch (or just call me dr. Frankenstein)

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 25 Oct 2002 08:34:36 +0859 ()

Iljitsch;

"Transport multihoming" is a wrong terminology.

As long as multihoming is solved on end systems, it is irrelevent
whether it is solved by transport or application layer.

The separation between the transport and application layers is
an end system local issue and invisible from the network.

See my expired draft (available in ML log) for further explanation.

						Masataka Ohta

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 25 Oct 2002 22:23:28 +0200 (CEST)

> multihoming information and sharing it over several connections.

This may or may not be desireable; the best way to handle this is
probably some capabilities negotiation where each end tells the other: I
can do this and this and that and it applies to:

1. All hosts living in this prefix: ...
2. This host
3. This protocol (TCP, UDP, ...)
4. This session

> Also because

Hm, not sure if this would work very well as many different networks are
behind a single PA block, some of which may be down, some of which may
be operational. Also, I don't think many hosts will be donning a full
routing table. However, it would be nice to have. I've been thinking
about a protocol to share such information between hosts a while back,
maybe this would be a nice addition later on.

> Depends what you mean by long term.  Please clarify.

If a host sets up a connection that passes through a "mudem" this box
may intercept the setup packet and do some capability negotiation with
the other end. During this negotiation, there must be state in the mudem
box. That would be acceptable. But when the session is established, it
should be possible for the mudem to go down and come back up, or for
another to take over, without breaking the session. So for running
sessions, there must not be any state that can't be recovered by looking
at the session = state that would last as long as the session = long
time.

> clad guarantee that the address selection is valid.

I'm not sure what kind of bad things you want to protect against. If
someone has full access to the TCP packets, there is nothing you can do
anyway. (Other than SSL/IPSec, that is, but then they can still break
the session.)

> degree of complication which I believe to be excessive.

Hm, there is one thing that would be bad: a host thinks it talks with
host X, while in fact it talks to host Y. But that could happen with NAT
or some other man-in-the-middle thingy anyway, so is it worth the
trouble protecting against this? We could build something into the
protocol but obviously a corrupted multihoming processing box wouldn't
necessarily implement this and the fact that you _don't_ see such a box
doesn't mean anything.

> advantage in exploiting the implied tree structure imposed by the strong

I think it would be good to be able to replace the full address, that
way everyone can use it, no matter how limited their connectivity.

What I envision is an IP option that makes it possible to negotiate
capabilities at the start of a / the first session. This option can
either be added:

1. by the transport protocol without involvement from the IP layer
2. by the IP layer without involvement from the transport protocol
3. by the IP layer at the request of the transport protocol
   (= we need to extend the interface)
4. by some external entity

Which of 1 - 3 would you say is best? Don't forget this will probably
have to be implemented for several transport protocols.

It would of course also be possible to do this in a TCP option, but that
complicates things like the TCP checksum in the case of 4 and doesn't
address non-TCP protocols.

Iljitsch

---

From: Greg Maxwell <[email-masked]>
Date: Fri, 25 Oct 2002 16:39:48 -0400 (EDT)

> place where it can be better controlled if this is desired.

I could imagine a TCP extension that could interoperate with a translation
device that could handle adding the functionality of the extension to all
the non-supported hosts behind it.

			  /[network]\                    /[old host]
[enhanced host] - [router]           [router] - [magic] - [old host]
	      [magic]/    \[network]/
[old host] - /

The magic host would have to see all the traffic (no wraparound paths),
and I'd have to put a lot of thought into figuring out if it could be done
without tracking state in the magic host (initial though is, no, not
without an unreasonable amount of overhead).

This would represent a layering violation, but no worse then nat.

There are a few complications.. If one link was down, and the
locator service (DNS) was not TLM (transport layer multihoming) aware,
there would be complications in forming an initial connection.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 26 Oct 2002 01:09:31 +0200 (CEST)

> without an unreasonable amount of overhead).

Restoring the destination address should be simple, if we require that a
secondary destination address is recognizable as such: this would be a
simple prefix replacement. However, it becomes more complicated when the
source address changes. A translation box can't be configured with this
information. So either we require the source to always include the
original source address (so now we're tunnelling) or the magic box has
to keep source remapping state. However, it should be possible to
rebuild this state if the source includes the primary source address
whenever it feels this is appropriate, for instance, in retransmissions,
or when the magic box asks for this information because it isn't in the
cache.

> This would represent a layering violation, but no worse then nat.

I'm not sure it would... If we define this functionality to be in IP and
provide suitable new interfaces for TCP and other transport protocols to
interact with it, we should be fine. If those darn implementers then
decide it's easier to implement the whole thing in one go in TCP, well,
that's not our fault. I think being able to take advantage of the
end-to-end information that TCP has is worth a few minor infractions.

> there would be complications in forming an initial connection.

True. The simple solution would be to require the application to try
alternate addresses. (Note that a host would then have four addresses:

1. primary address in prefix A, listed in DNS
2. secondary address in prefix B, dynamically discovered
3. primary address in prefix B, listed in DNS
4. secondary address in prefix A, dynamically discovered

With just two addresses it would be impossible to easily determine when
addresses should be translated. The client must be smart enough to try
all addresses listed in the DNS, the magic box can't do this on behalf
of the client unless the translation state is already cached.

The other solution is that discovery of the alternate addresses happens
out of band. This is what Michel's MHAP does. This has the advantage the
primary address doesn't have to be reachable which in turn makes these
addresses portable. Unfortunately, this out of band discovery is a tough
nut to crack.

---

From: Peter Tattam <[email-masked]>
Date: Sat, 26 Oct 2002 10:13:06 +1100 (EST)

> necessarily implement this and the fact that you _don't_ see such a box

The real danger in all this is connection hijacking.  The set of addresses
negotiated must be kept immutable and also free from contamination during the
negotiation process.  Having a network in between the host and the mudem would
increase the risk of this contamination.

> have to be implemented for several transport protocols.

I think 2.  Then it is available to all transport protocols.  Maybe 4 if the
issues I outlined before can be adequately dealt with.

As I said in another thread selection of source address is also something that
has been largely overlooked.  The host based MH possibly has the advantage of
getting this right better than an external entity.

> address non-TCP protocols.

As discussed at the Tokyo meeting, there are limitations in the TCP header size
which may preclude this being done there.  Having it at the IP layer kind of
makes sense, and having the gleaned information cached beyond the life of
connections not only benefits multiple connections to the same host but should
also aid connectionless protocols as well.

At Tokyo, I think Steve Deering made a suggestion that we make this an IP layer
feature available to all protocols but it was so fresh in my mind that I had
not thought through all the possibilities.

So my latest thought would be that we could do it at the IP layer in a similar
way to that described in my preliminary draft.  

For TCP connections, the prefix/address set negotiation be done at syn/ack
time, and for any other protocols that have the same syn/ack semantics. 

For connectionless protocols, the IP layer would need to rely on cached state
to do its thing.  In such a scenario, it may be important to set an upper time
limit on the cached information being valid, the information being updated by
incoming traffic.  A nonce for the negotiation would have to be supplied by
both ends in this case.

If a nonce were to be added to the IP MH options, this could be made larger
than the TCP sequence number thereby increasing the security from seq number
attacks.

Another possibility is that we define a MH connection protocol which
establishes the MH characteristics of the connection between the two hosts. It
would be analogous to a TCP session and would have the same degree of security
as a TCP session.  It could be done directly at the IP layer or be an
ancilliary protocol on top of IP.  It would imply that there would be host-host
state information kept somewhere in the IP stack, but it is possible that this
could distributed amongst the control blocks associated with each connection.

My logic is that if this type of multihoming is done, state information has to
be kept, and that this state information is likely to be rather fine grained
and would very likely be a bad idea to place in a single box being used for a
whole organization whether that be a router or mudem or whatever.  The
connection states are on the hosts - these would closely map those states and
could best be managed on the hosts.

What we don't want is this thing looking like one humungous NAT box - in the
long term, that just won't scale, and it introduces another point of failure to
consider in the traffic flow.

Peter

---

From: Feico Dillema <[email-masked]>
Date: Sun, 27 Oct 2002 02:43:03 +0200

> inaccessible and the stack could use this intelligence in advance. 
Instead of letting hosts negotiate the prefixes why not
augment/generalize PMTU into a ``Path Capability (or MTU and Prefix) Discovery''
(or do something traceroute-like) that collects the information from
all nodes along the path. This might allow for some nice
aggregation and double-checking of information (assuming routers are
typically more trustworthy or better maintainedi than most hosts).

Feico.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 25 Oct 2002 16:14:26 +0200

Why would we do that rather than rolling out SCTP? I can imagine an
API trick to make SCTP look like TCP to a legacy app.

The notion that we can solve it without host stack mods is pretty
unlikely.

   Brian

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM 
On assignment at the IBM Zurich Laboratory, Switzerland