Subject: comments on requirements draft

From: Steve Deering <[email-masked]>
Date: Mon, 6 Aug 2001 13:27:59 -0700

Here are some comments on the -01 version of the multi6 requirements
draft.

>             Requirements for IP Multihoming Architectures

Shouldn't this say "Site Multihoming" or "Enterprise Multihoming" or
something like that, to distinguish it from the other common uses of
the term "multihoming", such as host multihoming (i.e., having more
than one address or more than one interface on a host)?

>Abstract
>
>   Multihoming is an essential component of service for enterprises [3]
>   which are part of the Internet.  Existing IPv4 multi-homing...

"Multihoming" is used before it is defined.  How about something like
the following:

    Multihoming, that is, connecting to more than one IP service
    provider, is an essential component for enterprises...

>   Migration to IPv6, which will allow unprecedented scaling of the
>   number of potentially multihomed sites, will seriously exacerbate
>   this stress unless a substantially different approach to multihoming
>   is adopted.

It's not the move to IPv6 that will exacerbate the stress -- it's
the growth of the Internet and of multihoming, regardless of which
version of IP will be used.  It is often claimed that IPv6's larger
address will make the problem worse, as if the shorter length
of IPv4 addresses were acting as some sort of defense, which is
just not true.  The 32-bit space already accommodates "unprecedented
scaling of the number of potentially multihomed sites" way beyond
what can be handled with the current multihoming approach, as we
we move more and more towards giving sites only one or a small
number of public IPv4 addresses and expecting them to use NAT.
I.e., the current direction is intractable; IPv6 doesn't make it
"more intractable".

>   An "enterprise" is an entity autonomously operating a network using
>   TCP/IP and, in particular, determining the addressing plan and
>   address assignments within that network.  This is the definition of
>   "enterprise" used in [3].

I wonder why "TCP/IP" and not just "IP"?  Must an enterprise use TCP?

>3.1.1 Redundancy
>
>   By multihoming, an enterprise should be able to insulate itself from
>   certain failure modes within one or more transit providers, as well
>   as failures in the network providing interconnecting with one or more
>   transit providers.

"providing interconnecting with" is awkward.  How about "providing
interconnection among"?

>   The multihoming architecture must accommodate (in the general case,
>   issues of shared-fate notwithstanding) the following failure modes:

s/accommodate/survive/

>   o  Physical link failure, such as a fiber cut or router failure,

A router failure is not an example of a physical link failure.

>3.1.2 Load Sharing
>
>   By multihoming, an enterprise should be able to distribute both
>   inbound and outbound traffic between multiple transit providers.

s/between/across/

Also, change or add wording to make it clear that the requirement is
for potentially *concurrent* usage of the multiple connections, not
just the usage of one provider over one interval of time and another
provider over a different interval.

>3.1.3 Performance
>
>   By multihoming, an enterprise should be able to protect itself from
>   performance difficulties between transit providers.
>
>   For example, suppose enterprise E obtains transit from transit
>   providers T1 and T2, and there is long-term congestion between T1 and
>   T2.  The multihoming architecture should allow E to ensure that in
>   normal operation none of its traffic is carried over the congested
>   interconnection T1-T2.

How is the capability offered in the current multihoming approach,
given that BGP is not sensitive to congestion, short-term or long-term?

>   A multi-homed enterprise must also be able to distribute inbound
>   traffic particular multiple transit providers according to the
           ^
           from

>   particular network within their enterprise which is sourcing or
>   sinking the traffic.

How can a network within the enterprise source inbound traffic to that
enterprise via a particular transit provider?

This Performance section needs more elaboration and clarification.

>3.1.4 Policy
>
>   A customer may choose to multihome for a variety of policy reasons
>   outside technical scope (e.g.  cost, acceptable use conditions, etc.)
           ^
           the?

>   For example, customer C homed to ISP A may wish to shift traffic of a
>   certain class or application, NNTP, for example, to ISP B as matter
>   of policy.  A new IPv6 multihoming proposal must provide support for
>   multihoming for external policy reasons.

Inbound only?  Outbound only?  Both?  This requirement seems excessive
to me, e.g., if end systems ESP encrypt their traffic, how can the
network recognize NNTP traffic?

>3.1.5 Simplicity
>
>   As any proposed multihoming solution must be deployed in real
>   networks with real customers, simplicity is paramount.  The current
>   multihoming solution is quite straightforward to deploy and maintain.
>   A new IPv6 multihoming proposal must not be substantially more
>   complex to deploy and operate than current IPv4 multihoming
>   practices.

Complexity is in the eye of the beholder.  Also, the acceptability of
complexity may well depend upon whom the complexity is being imposed,
and on what benefits one gets in return for that complexity.  BGP and
OSPF are more complex than RIP, but their advantages are apparently
deemed worth their extra complexity.  So while simplicity is a very
important goal, and we each believe we know complexity when we see it
(even when others disagree), it's not feasible in this context to
measure complexity or agree on how much is too much, so it cannot
reasonably be made a "requirement".

>3.2.1 Scalability
>
>   Current IPV4 multihoming practices contribute to the significant
>   growth currently observed in the state held in the global inter-
                                               ^
                                   and information exchanged?

>   provider routing system; this is a concern both because of the
>   hardware requirements it imposes and also because of the impact on
>   the stability of the routing system.  This issue is discussed in
>   great detail in [9].
>
>   A new IPv6 multihoming architecture should scale to accommodate
>   orders of magnitude more multi-homed enterprises without imposing
>   unreasonable requirements on the routing system.

How many orders of magnitude?

>3.2.2 Impact on Routers
>
>   The solution may require changes to IPv6 router implementations, but
>   these changes must be either minor, or in the form of logically
>   separate functions added to existing functions.

This is a surprising requirement -- can you elaborate?   Also, the
nature and degree of acceptable change would presumably depend on
whether or not the change was required in the data plane (already being
cast in hardware for IPv6 on some routers) or the control plane
(still in software), and perhaps also on how many routers would have
to change, e.g., all routers vs. backbone routers vs. edge routers vs.
enterprise routers.

>3.2.3 Impact on Hosts
>
>   The solution must not destroy IPv6 connectivity for a legacy host
>   implementing RFC 2373 [5], RFC 2460 [7], RFC 2553 [8] and other basic
>   IPv6 specifications current in June 2001.  That is to say, if a host
>   can work in a single-homed site, it must still be able to work in a
>   multihomed site, even if it cannot benefit from multihoming.
>
>   It would be compatible with this requirement for such a host to lose
>   connectivity if the site's primary ISP connection failed.

Is the assumption that there's a "primary" ISP always valid?

What about newly-established sessions?

>   If the solution requires changes to the host stack, these changes
>   must be either minor, or in the form of logically separate functions
>   added to existing functions.

This needs more elaboration.

>   The multi-homing solution should allow host or application changes to
>   enhance session survivability.

s/to enhance/if that would enhance/

>3.2.5 Operations and Management
>
>   It must be posssible to monitor and configure the multihoming system.

Possible for whom?

>4. Security Considerations
>
>   Multihoming no doubt offers some attractive opportunites for denial
>   of service and spoofing attacks.  Multihomed sites must be protected
>   against such attacks at least as well as single-homed sites.

I.e., not at all?

- Steve

---

From: Geoff Huston <[email-masked]>
Date: Tue, 07 Aug 2001 21:37:28 +1000

nit upon nit...

>     provider, is an essential component for enterprises...

"essential"?

Not necessarily essential, but sometimes desirable.

How about:

"Multihoming, that is, connecting to more than one IP service provider for 
IP access and transit services is used by enterprises .....

Geoff

---

From: Geoff Huston <[email-masked]>
Date: Tue, 07 Aug 2001 21:42:38 +1000

one point at a time:

>version of IP will be used.

I would tend to suggest that growth in multi-homing is a cost and 
perception issue.
If the cost of multi-homing is considered to be sufficiently low and the 
perception
of service reliability is also sufficiently low then you will see increased 
activity
in multi-homing. Increase the perception of reliability, or make 
multi-homing more
difficult (cost, technology required, or anything else) and you reduce the
incidence of multi-homing. To ascribe growth to multi-homing to IPv6 is a rash
statement without adequate foundation.

Geoff

---

From: Peter Tattam <[email-masked]>
Date: Wed, 8 Aug 2001 00:21:28 +1000 (EST)

> statement without adequate foundation.

I was at a recent conference in Aus and the issue of Service Level Agreements
was being discussed in a forum panel.  Each of three very large
telco's/provider's (by Aus standards anyway) representatives were all highly
confident that they could promise extremely high levels of redundancy and there
was no need to have more than one provider.

However by the end of the panel discussion and after several war stories were
shared by members of the audience and even one member on the panel, it
was eventually conceded that for an effective SLA, redundancy was crucial, and
in particular to multiple providers.  The forum concluded with an independent
consultant's reflection that they always recommended complete redundancy to
their clients especially at the last mile.

I also find that the further you are away from the core of the internet in your
region, the higher is the risk of things going bang.  We are multply homed to 3
providers even though we are a small ISP, and I have had occusion to depend on
each of them for large outages over the past year.  Some outages were
configuration problems in networks, while others were meltdowns in the
providers infrastructure including major cable breaks, and even one
international fibre break.  Had it not been for our multihoming, our service
would have been out for a minimum of several hours, and in one worst case
scenario we suffered, 2-3 days.

This risk of things going wrong being higher at the edge is a curious
observation as it flies in the face of the conventional wisdom that says that
the smaller you are, and further from the core, the less need one should have
for multihoming.  I would suggest the converse may actually be true.  The
smaller or perhaps more remote a site or network is, the higher is the risk
that the quality of their service is going to be degraded in one way or
another.

I guess my point is that to maintain any kind of quality of service, a small
provider has to go that extra mile (pardon the pun) and provide a better
service to differentiate their brand from the big providers.  One method of
differentiation is redundancy by multihoming, even if the perceived cost may be
high.

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: Joe Abley <[email-masked]>
Date: Tue, 7 Aug 2001 13:36:46 -0400

Hi Steve,

Thanks for your comments! A couple of small comments on a couple of
them to start off with:

> I wonder why "TCP/IP" and not just "IP"?  Must an enterprise use TCP?

The origin of this (surprisingly contentious!) paragraph is an idea
that was put forward in response to -00, which was that since
"enterprise" was already defined in an existing document (RFC1918),
it might be nice to re-use the definition.

The definiton as shown is that included in RFC1918, and that is where
the reference to "TCP" comes from.

I agree with the point you are making, particularly as people will be
reading this draft with transport-layer solutions in mind; I just wanted
to explain why "TCP" featured in that paragraph.

> given that BGP is not sensitive to congestion, short-term or long-term?

That capability is provided by the fact that T[i] will apply a higher
local preference to the prefixes operated by E that they learn directly
from E, rather than those they learn from T[3-i]. That means in normal
operation, traffic from T[i] destined for E will never be routed
towards T[3-i].

> enterprise via a particular transit provider?

"network" here refers to a prefix; an enterprise may advertise more
than one prefix to its transit providers. By advertising a particular
prefix in different ways to different transit providers (e.g. path
prepending, setting provider-specific community strings), the enterprise
and change the way in which inbound traffic to that prefix is handled.

We were trying to avoid terms with routing connotations in the reqs
draft, and unfortunately that has introduced some ambiguity. We
certainly need to make sure we define our terms rigourously.

> >   outside technical scope (e.g.  cost, acceptable use conditions, etc.)

Mmm. I don't think so -- perhaps s/outside/beyond/ ?

> Inbound only?

Sure.

> Outbound only?

Sure.

> Both?

Sure.

> This requirement seems excessive

In current practice, this can be achieved so long as the traffic
shifting is accommodated at the IP layer -- i.e. stick all your news
servers on one subnet, and advertise that subnet in different ways.

This is done in the real world, horrible though it may seem. A common
example is in far-flung networks where under-sea cable is expensive,
rare and low-latency, and satellite circuits are cheaper, more readily
available and high-latency. Shifting non-interactive traffic such
as mail and news onto satellite circuits is good for business. If
we can't accommodate this kind of thing, there are lots of operators
in (apparently) the fastest-growing parts of the network who will not
be very happy.

> This is a surprising requirement -- can you elaborate?

That's Brian's paragraph -- perhaps he can comment on it?

> s/to enhance/if that would enhance/

s/if // :)

> >   It must be posssible to monitor and configure the multihoming system.

Possible for the operator of the multi-homed enterprise. I think I'm
not seeing the ambiguity -- who else could that be read as referring
to?

> >   against such attacks at least as well as single-homed sites.

I have also been harbouring some doubts about that paragraph. I'm glad
I'm not alone :)

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 07 Aug 2001 14:53:17 -0500

Steve,

Cherry-picking among your comments:

...
> network recognize NNTP traffic?

Joe Abley already responded and I agree with him. Like any traffic classification
policy, this one would have restricted applicability in the case of ESP traffic.
In fact the issues are essentially if not totally identical to those for
diffserv classification. If you can't classify for diffserv reasons, you
can't classify for multi6 reasons either and default policy will apply. There
is nothing special here - multi6 is simply added to the list of modules
that look at policy (i.e. a local config table) as part of their decision
process.

...
> This is a surprising requirement -- can you elaborate?  

You work for a router company so you tell us :-)

The intention is to *require* a solution that does not cause router
vendors to need to throw away their IPv6 routing engines and start again.
IMHO such a solution would have no chance of getting implemented
across all routers, and therefore is not something the IETF can endorse.

> Also, the

Indeed. Of course not all hardware architectures are so rigid that
the data plane is carved in the silicon. 
> Is the assumption that there's a "primary" ISP always valid?

In the sense of providing backwards compatibility, for pre-multi6
hosts, I think it is (even if it is an arbitrary choice among the
site's ISPs). 

> What about newly-established sessions?

You mean can a pre-multi6 host switch to a backup ISP for new sessions? 
I intended *not* to require that, since that would require today's 
(BGP-expensive) solution still to be in place. Thus in this respect the 
pre-multi6 host gets *worse* service from multi6, and there's the incentive 
to upgrade it.

> This needs more elaboration.

As for routers, the intention is to *not* require stack rewrites, which I can
tell you are very unlikely to happen. Multi6 needs to be an add-on. I don't know 
how to say it more clearly.
...
> >   against such attacks at least as well as single-homed sites.

:-)

Well, you write a better security section!

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 07 Aug 2001 23:22:19 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "Geoff" == Geoff Huston <[email-masked]> writes:
    Geoff> I would tend to suggest that growth in multi-homing is a cost and
    Geoff> perception issue.  If the cost of multi-homing is considered to be
    Geoff> sufficiently low and the perception of service reliability is also
    Geoff> sufficiently low then you will see increased activity in
    Geoff> multi-homing. Increase the perception of reliability, or make

  As a consumer of multihoming...

  Small (100 person and less) organizations have it very difficult.

  The current service reliabilities are just far too low. We can easily
consume more than 1 unit of bandwidth (whether that was a 56K, a T1 or
now a 10Mb ATM VC), so it makes a lot of sense to buy a second unit from
another provider.
  But, as a small organization we did not get provider independant space
when we started (likely private network space, but not always), and ISPs
willing to deal with us are either:
	1) too small to have the multiple connections
	2) too big to care

  Almost every 100 person organization that I know of with these kinds
of needs finds that multihoming is too expensive, but no amount of money gets 
us good service.
  We go for ad-hoc solutions: e.g. a pair of web caches with different
default routes with ICP between them. The assumption that everything is HTTP
often bites here as well...

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy");  [

---

From: "marcelo" <[email-masked]>
Date: Mon, 13 Aug 2001 17:14:55 +0200

> network recognize NNTP traffic?

Perhaps it may be interesting to distinguish between internally initiated
comunications (comunications iniiated by a host from the MH site) and
externally initiated comunication (comunications initated by a host not
belonging to the MH site)

marcelo