Subject: RE: Transport level multihoming

From: [email-masked] (Sean Doran)
Date: Sat, 4 Aug 2001 10:40:17 +0200 (CEST)

Yes, we welcome proposals and drafts, but the present focus
is on bringing to last call (i.e., please shout now) two
main documents:

	#1 - how multihoming is done today in the Internet
	#2 - requirements for multihoming in an IPv6 Internet
             (which probably should say "AT LEAST #1 and some more things")

After those two are agreed to, we can start examining individual
proposals against #2.  Until then, while the specific proposals
for protocols are an interesting way of converging on a reasonable
architecture in which to develop various specific protocols,
the freight train is a little before the foal...

So, while it's good to see interesting ideas from a number
of authors being put forward as drafts, I'd like to suggest
that the authors take a close look at the various present inputs 
to numbers one and two above.

Since it's hard to forsee us recommending a standard which
does not fully meet the requirements document, everyone
should be asking: is what is developing now too strict, too loose?
Are there pieces missing?  Are there pieces that should be missing?
Are we so on the wrong track that you are motivated to write a 
counter-proposal to numbers one or two above?  And so on.

Yours in wondrous TechniProcess,

	Sean.

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 18:54:19 +1000 (EST)

>              (which probably should say "AT LEAST #1 and some more things")

I pretty much agree with the two I-D's being presented at the IETF meeting
which I believe cover these two points.

> counter-proposal to numbers one or two above?  And so on.

My apologies.  I was picking up from where I was involved in IPng discussions a
couple of years back and was present at the interim meeting in Tokyo.  It had
gotten quite involved then and I presumed things were more advanced since then.

> Yours in wondrous TechniProcess,

Can I gently say that IPv6 is getting beaten up quite a bit by the technical
community, and it all centres on the issue of multihoming and strong
aggregation.  It's stalling deployment by major infrastructure, and a lot of
people are not buying the line that "we'll fix that bit later".

I'm a strong supporter of IPv6 but its very difficult to defend IPv6 on this
particular issue when I can't point to specific ways of how IPv6 solves the
multhoming issue.

I will suitably back off - I'd forgotten how carefully one needs to tread in
the working group environment.

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: Randy Bush <[email-masked]>
Date: Sat, 04 Aug 2001 10:06:03 +0100

>            (which probably should say "AT LEAST #1 and some more things")

just to make trouble, and not because i have a position on the question, is
the parenthetical above *really* a desirable/necessary condition?  i.e.  if,
by magic, we come up with a really sexy v6 solution, must it also support
current v4-style multihoming.

randy

---

From: Tony Li <[email-masked]>
Date: Sat, 4 Aug 2001 02:07:40 -0700

| It's stalling deployment by major infrastructure, and a lot of
 | people are not buying the line that "we'll fix that bit later".

Peter,

I hope you understand the rationale behind this.  Without a scalable
routing architecture that supports multihoming, it would be foolish to do
massive address assignments.  Been there, done that, got the IPv4 swamp to
show for it.  

Regards,
Tony

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 19:30:12 +1000 (EST)

> current v4-style multihoming.

Sorry, I missed something very important there.  Is it planned to support
multihoming exactly as Ipv4 does now?  The policy a couple of years back was
very firm.  IPv6 = Strong aggregation, small DFZ.  Current v4 style multihoming
can't fit that requirement.  Trying to shoehorn Ipv4 BGP practices into Ipv6
has very serious problems for scalability and long term stability of the core.

Has there been a backdown or crucial change in policy that I have somehow
missed because I had my head down on other work? 

Peter

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 19:22:24 +1000 (EST)

> massive address assignments.  Been there, done that, got the IPv4 swamp to

I do understand and it demonstrates the clear lack of confidence by the IPv6
community that the protocol will deliver what it promises.

That's all very well, but unfortunately the techies are digging their heels in
and implementing NAT based solutions to make things continue (sort of).

What you are saying could (and sometimes is) being interpreted as "well, we've
got Ipv6 to solve all your address space problems, but we haven't figured out
how to properly route it yet.  Come back in (X) months/years when we've figured
it out and then we'll tell you how to deploy it.".

> Tony

I challenge any of you to get on slashdot and successfully defend IPv6 against
its opponents - especially the issue of strong aggregation, core router bloat 
and multi homing.

Sorry to be off topic, but I'm trying to express the sense urgency to see a
solution and frustration of what's happeing in the real world.

Now I'll get off my soapbox. :)

Regards.

Peter

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 4 Aug 2001 18:31:07 +0859 ()

Peter;

> it out and then we'll tell you how to deploy it.".

Months?

According to the chairs, we shouldn't figure out how to properly route
it yet, until we finish our requirement draft.

While 3 multi6 meeting are being spent on it, discussions on specific
proposals are discouraged to make the requirement draft abstract
and separated from the real world.

But, it is still an improvement as 5 years are wasted waiting IPv6
deployed, insisting on keeping the specification as is (changes will
delay deployment of the perfect specification) and ignoring
real world requirements for changes.

							Masataka Ohta

---

From: Randy Bush <[email-masked]>
Date: Sat, 04 Aug 2001 11:06:13 +0100

> multihoming exactly as Ipv4 does now?  The policy a couple of years back
> was very firm.  IPv6 = Strong aggregation, small DFZ.

yes indeed.  unfortunately, to date, that rather idealistic policy has
lacked clear technical mechanisms with which to implement it.  and note
that, as this wg is in the ops area, folk here tend to be rather pragmatic
about utility and implementability.  

so, from that, you may be able to infer what this wg is about.

randy

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 20:29:44 +1000 (EST)

> so, from that, you may be able to infer what this wg is about.

I am beginning to infer that we are talking BGP style solutions much like
current IPv4 practice.

> randy

Could the chair please confirm this significant policy turnaround regarding
relaxation on indiscriminate advertising in the DFZ for IPv6?

If so, I question the credibility of what is being done and can only infer that
the BGP lobby has finally gotten their way.  I will politely leave because my
proposal is very likely no longer valid for the suggested direction.  It will
not be able to fit all the requirements that have been laid down. 

Thanks

I will lay down a wreath or two for IPv6.

Peter

---

From: Randy Bush <[email-masked]>
Date: Sat, 04 Aug 2001 11:35:19 +0100

you may find <http://www.ietf.org/html.charters/multi6-charter.html>
a bit helpful

randy

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 20:44:58 +1000 (EST)

Read it already.  My initial understanding of it was that it supported the
preexisting policy of small DFZ.   The only way this can be done is to enforce
strong aggregation policy.  If open the gate even slightly the traditional BGP
thinking will just push things back to what they were with IPv4.

I was on the side of weak aggregation policy until I "saw the light". Then when
I applied my lateral thinking skills, I stumbled onto a possible solution that
didn't involve complicated routing exchange protocols.  Any solution that
shifts the decision making towards the edges of the network is going to scale a
whole lot better than one at the core.

Peter

---

From: Randy Bush <[email-masked]>
Date: Sat, 04 Aug 2001 12:02:00 +0100

> I was on the side of weak aggregation policy until I "saw the light".

great!  send code.

randy

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 21:16:46 +1000 (EST)

> > I was on the side of weak aggregation policy until I "saw the light".

Is that a challenge for me to implement my protocol on our stack and distribute
it as standard with the next operating system release without peer review :) 
That's sure to make friends for me within the IETF and the rest of the
industry !!! 

The doc I sent the reference to was enough for someone to flesh out the details
and make it work - I believe there is a BSD implementation out before I even
got a chance to try it myself. What I've presented is a concept paper. Once
people are comfortable with the concepts, I'll put together a technical paper
with specifics.  But my time is too precious to spend on something that isn't
going to be used.  I need a strong green light before I take it any further. 

Peter

---

From: Feico Dillema <[email-masked]>
Date: Sat, 4 Aug 2001 13:30:53 +0200

> got a chance to try it myself. What I've presented is a concept paper. Once

One of our students implemented it for FreeBSD-3.2 with Kame IPv6
stack of 12-1999. His thesis and all source code can be found on-line
here:

http://www.vermicelli.pasta.cs.uit.no/ipv6/students/troels/index.html

Feico.

---

From: Peter Tattam <[email-masked]>
Date: Sat, 4 Aug 2001 21:49:59 +1000 (EST)

> http://www.vermicelli.pasta.cs.uit.no/ipv6/students/troels/index.html

Thanks, I will take a close look.

Peter

---

From: Bill Sommerfeld <[email-masked]>
Date: Sat, 04 Aug 2001 08:00:36 -0400

> current v4-style multihoming.

I really hope Sean meant:

problem #1: multihoming v4 in today's internet
problem #2: multihoming v6 in a future IPv6 Internet
 - requirements for #2 are a subset of the requirements to #1
 - possible solutions to #2 are a superset of the solutions to #1

					- Bill

---

From: Greg Maxwell <[email-masked]>
Date: Mon, 6 Aug 2001 08:09:39 -0400 (EDT)

[snip]
> so, from that, you may be able to infer what this wg is about.

Yes, it's about excluding new ideas which have strong technical merit
because some people believe that IPv6 has significant implementation
momentum and that the future cost of restructuring the IPv6 Internet to
make it scalable once it falls down is greater then the cost of
potentially slowing this supposedly huge IPv6 momentum.

---

From: Greg Maxwell <[email-masked]>
Date: Mon, 6 Aug 2001 08:18:28 -0400 (EDT)

> > I was on the side of weak aggregation policy until I "saw the light".

What code to you want? I've collected a few differnt transport hacks that
serve the purpose (such as SCTP) but they require minor (i.e. search and
replace) application level changes.

This working group has pretty much painted themselves into a corner by
concluding that application level changes for multihoming support are not
acceptable.

Because of this, the majority of the transport level multihoming
discussion has been about ugly hacks on TCP involving DNS kludges and
other things that are never going to work.

IPv6 itself requires application level changes of a simmlar extent then
moving to SCTP in TCP-like mode, multihoming *could* be left to apps which
have made the transition, and I've already posted what I believe to be a
fairly good high level set of ideas on how 'total aggregation' can by
maintained (via prefix flow-throughs).

---

From: Sean Doran <[email-masked]>
Date: Mon, 06 Aug 2001 15:40:50 +0200

>  - possible solutions to #2 are a superset of the solutions to #1

Yes.  Thank you for the improved wording.  (Although to
quibble, requirements for #2 are a set containing a subset
of the requirements to #1).

	Sean.

---

From: "Ben Black" <[email-masked]>
Date: Mon, 6 Aug 2001 09:54:05 -0700

> concluding that application level changes for multihoming support are not

I am part of this working group and I have definitely not concluded this.
I have concluded the exact opposite, and I believe rather strongly that
the reason this situation exists is layer violation (why does the
application
need to know the network and transport addresses?).

Ben

---

From: Peter Tattam <[email-masked]>
Date: Tue, 7 Aug 2001 09:43:47 +1000 (EST)

> potentially slowing this supposedly huge IPv6 momentum.

I don't quite know which way to read this :)

Peter

---

From: Peter Tattam <[email-masked]>
Date: Tue, 7 Aug 2001 09:48:05 +1000 (EST)

The problem with application methods is that they're prone to application
coding errors.  There's already a big hit ofr app developers to convert to IPv6
anyway.  Making it even harder can be seen to be counterproductive.

Peter

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 07 Aug 2001 15:11:03 -0500

> need to know the network and transport addresses?).

There is certainly a lot of evidence building up that the multi6 solution 
will require work at the transport layer; once we have agreed requirements 
we can be more precise about that conclusion. 

On the other hand we know that major restructuring of applications isn't going
to happen - that's the real world. So if we have to do work in the transport
layer, the transport layer is going to have to hide most of it below some
sort of socket API. I can't see any reason in principle why that API
would have to expose a bunch of addresses, even if it turns out that
multi6 requires more than one. A "primary" address per host would be enough
to expose to the middleware.

   Brian

---

From: Greg Maxwell <[email-masked]>
Date: Tue, 7 Aug 2001 16:22:15 -0400 (EDT)

On Tue, 7 Aug 2001, Brian E Carpenter wrote:

[snip]
> to expose to the middleware.

If the primary address is functional at connection setup time, then, yes
it could learn the other transports from the remote end. However, what do
you do if the primary is down? The only solutions I can think of either
require the applications to rotate the address passed to the API, or
require the transport to consult DNS (what an ugly kludge and layering
violation THAT would be!).

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 07 Aug 2001 17:01:51 -0500

> violation THAT would be!).

There is *always* an abstraction layer above which a single handle identifies
the other party in a session (and probably no handle at all identifies
"this" end of the session). So whatever layer is the highest layer to
be aware of N addresses, there is always a layer above that (conceptually
at least) that has a single handle for that set of N addresses.

All I am saying is that this abstraction layer may just as well be a socket
API, and that single handle may just as well be one of the N addresses - and
if you do it that way, you minimise the impact on existing apps layer logic.

It's probably sloppy to write as I did of a "primary" address being the
handle; it's just an arbitrary choice among the N addresses available.
It's the one that gethostbyname (equivalent) chooses to return to you;
it might conceal a bunch of others from you below the socket API,
and use them as needed for multihoming or whatever other cookery it
needs to do.

   Brian

---

From: Daniel Senie <[email-masked]>
Date: Tue, 07 Aug 2001 18:22:26 -0400

>and use them as needed for multihoming or whatever other cookery it

What you suggest could work, though the management of that new abstraction 
layer brings with it some level of trouble. Assuming some clean, fast and 
reliable mechanism is in place for that, and all applications use the API 
that automagically uses this capability, the application writers will be 
blissfully unaware. Diagnostic utilities could bypass that facility and 
learn what's actually going on. Interesting complications await for 
firewall vendors, as a single application may wind up with connections 
going out multiple links from a site if that's what the underlying 
multihoming stuff decides to do, but that's probably OK. Failover could be 
a bit messier, but I think the app writers are getting used to having to do 
reconnects on occasion.

Somehow, though all of this I get the same icky feeling I got with the 
design of IBM's source routing for 802.5 Token Ring. Seems like the answer 
then was the end stations had more processing power, so pushing all the 
info out to them was the way to go. Bridges were going to be hopelessly 
overworked, and so could handle nothing more than reading a few bits on the 
way by (man, this is starting to sound like MPLS). Well, along came 
intelligent Ethernet bridges which learned, and Token Ring Source Routing 
looked pretty bad. It was difficult for people to support and debug, made a 
total mess of the code in the end stations, and really didn't scale well.

I get nervous when folks call for Transport-level multihoming, it comes 
across as either:

"The routers in the core will never be able to handle the problem." (Refer 
to arguments for MPLS to save the core from overload, since they won't be 
able to forward at gigabit speeds, and look at present day hardware. Every 
time we claim hardware won't ever scale to do a thing, we appear to get it 
wrong).

or

"We can't come up with a good way to work out this problem, so let's push 
it up a level and it'll be someone else's problem".

I'm not ready to rule out transport-level as a bad idea, nor rule it in as 
a good idea, but it does give me a bad feeling of deja-vu.
-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.                    http://www.amaranth.com

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 08 Aug 2001 10:09:50 -0500

Daniel,

If I could avoid the conclusion that we are forced into transport level
gymnastics, I would be delighted. But remember that the scaling arguments
are several orders of magnitude worse than in the 802.5 case (potentially
many millions of entries in a flat routing table, as opposed to a few thousand
for LAN bridging tables.)

   Brian

---

From: Greg Maxwell <[email-masked]>
Date: Wed, 8 Aug 2001 11:33:53 -0400 (EDT)

> for LAN bridging tables.)

For transport level multi-homing, you'll need a higher level distributed
database to keep track of all the address->host mappings. Fortunately, we
already have one designed for just that purpose: DNS.

Since we don't want to make any application changes (i.e. make it gather
multiple addresses from DNS and pass them to the socket API), how about
this:

We make the existing API a userspace wrapper on an improved API that takes
multiple addresses. The resolver will return a bogus IP address which is
specific to that boot-session of the host, the socket API wrapper will
reconize the bogus address (because they will all be scoped out of a
special allocated area), and query the resolver again to retrieve all of
the real addresses then call the real API.

This would allow us to achieve application compatibility.

I don't see why anything more is needed. I'll code up an example of this
implementation for a GNU/Linux system once the SCTP TCP-like api support
is finished in the Linux kernel SCTP implementation.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 08 Aug 2001 11:15:25 -0500

> is finished in the Linux kernel SCTP implementation.

Yes, this is one way to do it (almost identical to the "bump in the API"
approach to v4/v6 interworking). I don't see any reason why the IP address
returned by the resolver needs to be bogus; surely it can simply be any one
of the real IP addresses, which might have some advantages.

   Brian

---

From: Daniel Senie <[email-masked]>
Date: Wed, 08 Aug 2001 12:30:58 -0400

>for LAN bridging tables.)

As I said, I remain open to transport level involvement if there's no other 
way. The arguments REALLY do remind me of the 802.5 ones. While I agree we 
are orders of magnitude larger in scope here, we also have significantly 
improved technology to work with. The reliance on "sky is falling" 
arguments to justify standards development which at best will take years is 
what I'm questioning. MPLS was started because of a belief we could never 
build routers that could handle the massive loads, yet routers CAN handle 
the loads. Yes, MPLS has other uses now.

A good test of the justification for undertaking a standardization effort 
would be to look back at the point at which a technology is ready for 
deployment, and see if the initial reasons are still valid.

We will not know for some years whether the decisions we make now are the 
correct ones. I do hope we can find ways to justify ourselves that stand 
the test of time.
-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.                    http://www.amaranth.com

---

From: Greg Maxwell <[email-masked]>
Date: Wed, 8 Aug 2001 12:41:43 -0400 (EDT)

> of the real IP addresses, which might have some advantages.

My only reasoning for not using one of the actual IP addresses is that
some hosts may advertise a different set of addresses based on application:

Consider a single host with two DNS names:

ftp.host.company.com = A,B,C,D (where the letters are addresses in
				different prefixes)
www.host.company.com = A,B

Why? C,D are high latency satellite links.

The host is smart enough to not include the high-latency links in any
assotiation binding in it's transport protocol for port 80. However, we
don't want remote systems to attempt to initiate across those slow links
either.

Of course, we could just declare that those kinds of smarts belong to
applications that are aware of the transport-level multihoming (in that
they pass all the addresses themselves) and go ahead and use one of it's
correct IP addresses.

This was my reasoning for using bogus identifyers.

---

From: Greg Maxwell <[email-masked]>
Date: Wed, 8 Aug 2001 12:57:46 -0400 (EDT)

> the loads. Yes, MPLS has other uses now.

What would be useful is a metric for defining the global computational and
storage complexity for transport-level and network-level multihoming.

Then we can make a comparison: Only if the global complexity of
network-level solution is much less then the transport-level solution
could it be an acceptable solution.

For example, if we accepted that people could only be multihomed to a
single multi-provider (i.e. a provider maintains multiple separate but
interconnected networks) so that we could maintain aggregation, this
solution would be preferable on a global routing complexity basis to
transport-level multihoming.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 08 Aug 2001 13:10:31 -0500

> they pass all the addresses themselves) and go ahead and use one of it's

Indeed. It's only a binary number after all. 

> This was my reasoning for using bogus identifyers.

I have a feeling that for diagnosis, using one of the actual values
to be found in DNS is probably better - a bogus (locally synthesised)
value has no diagnostic value. In any case it's a detail.

   Brian

---

From: Peter Tattam <[email-masked]>
Date: Thu, 9 Aug 2001 10:41:45 +1000 (EST)

> deployment, and see if the initial reasons are still valid.

That should not invalidate any research done on transport level solution in the
mean time.  There may be other important spin offs.

Peter

---

From: Peter Tattam <[email-masked]>
Date: Thu, 9 Aug 2001 10:47:07 +1000 (EST)

> transport-level multihoming.
It depends if the provider is supplying multiple completely independent paths
to the core or not.  If what you are implying is that a MH site can only have
multiple paths to its parent branch, or only partly up the tree to the core, 
this may be unnecessarily restrictive, and possibly defeat the purpose of
multihoming.

Peter

---

From: Peter Tattam <[email-masked]>
Date: Thu, 9 Aug 2001 10:58:48 +1000 (EST)

> already have one designed for just that purpose: DNS.

Not necessarily.  You just need enough address information to hook into the
peer.  With my idea, any additional addresses can be exchanged at SYN/ACK
time.

Peter

---

From: Greg Maxwell <[email-masked]>
Date: Thu, 9 Aug 2001 07:57:17 -0400 (EDT)

> peer.  With my idea, any additional addresses can be exchanged at SYN/ACK

If you only know one path to reach the other system, what happens if that
path goes down before you tack up a connection?

---

From: Peter Tattam <[email-masked]>
Date: Fri, 10 Aug 2001 12:45:48 +1000 (EST)

> path goes down before you tack up a connection?

It would be foolhardy to expect that such a scenario work well.  I would
suggest "sufficient" information for a connection to be visible at the time of
the exchange.  The lists need not be identical, but in practice may be.

Peter

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 13 Aug 2001 10:22:23 +0200 (CEST)

> This would allow us to achieve application compatibility.

If you are using bogus identifiers then the system, from the applications
perspective, looks very much like a NAT.

Thus multiparty applications which pass IP addresses to peer application
instances will completely fail. It isn't clear to me that even
the ftp port command can be made to work.
(Having implemented Bump-in-the-API with IPv4 applications on an IPv6
stack by making the applications believe they connect to addresses
in 0.0.0.0/8 I have a long list of things that can break. Doing this
for IPv6 "over" IPv6 has the same issues as far as I can tell.)

I don't yet fully understand the properties of using one of the real IP
addresses. It can cause some unexpected aliasing between IP address sets
as you pointed out in subsequent emails.

  Erik