Subject: Fwd: Re: note from the iesg plenary

From: Margaret Wasserman <[email-masked]>
Date: Fri, 10 Aug 2001 22:40:43 -0400

Hi All,

Sean Doran and Thomas Narten asked that I send a summary of my routing 
issues to the multi6 list.  

This is not a very well-developed list.  I'm working on a draft for the 
IPv6 group that will answer these questions as well as I can, and hopefully 
serve as a discussion tool for closing the other questions.

In the meantime, please let me know if you have any of the answers.
I don't think that any of these particular issues related directly to
multi-homing, but I could be wrong -- everything seems to related to
multi-homing these days. :-).

Thanks,
Margaret

>>Basically, I have four issues:
>>
>>         (1) Boundaries for regular route lookups.
>>         (2) Site-local routing.
>>         (3) Anycast routing.
>>         (4) Routing table MIBs.
>>
>>(1)  The current addressing architecture defines all routable 
>>unicast addresses to include a 64-bit identifier.  Does that mean
>>that it is reasonable for a router to only look at the first 64
>>bits when making a routing decision?  Or should routers still treat
>>these addresses as CIDR-like?
>>
>>(2) Site-local routing is particularly confusing on a site-border router.
>>I don't know if anyone has really implemented one, but it seems likely
>>that it will need to have multiple routing tables (one global table
>>and one per attached site).  Even an intra-site router may need two 
>>conceptual routing tables (one global, one site).
>>
>>Also, some of the scoped addressing architecture assumptions only work
>>if sites are "convex" (Steve's word).  Unfortunately, that assumption
>>is not yet documented and the word "convex" has no currently defined
>>meaning in terms of routing -- even though we understand what it means.
>>Also, topologies change.  So, what happens if a site becomes 
>>"concave"?
>>
>>There is wording in a variety of IPv6 specs talking about how
>>routers should (and shouldn't) pass site-local information, but I haven't
>>looked to see if the current IPv6 routing protocols reflect this 
>>information (or even know that they may pass site-local routes).
>>
>>Since there doesn't have to be a one-to-one mapping between sites
>>and ASes, I need to figure out if there are any problems with propagating
>>site information when a site contains two (or more) ASes or when two
>>(or more, or parts of multiple) sites are contained in a single OSPF
>>AS.  I have no clue how this would work yet...
>>
>>(3) IPv6 anycast is substantially different from the anycast hacks
>>currently used in IPv4.  One big difference:  a global IPv6 anycast
>>address would (probably?) not appear to be inside of anyone's AS.
>>I don't know if this causes problems, as I haven't had time to look
>>at how OSPF would handle a host route injection for a router outside
>>the AS.  
>>
>>And, despite the fact that we are specifying the use of site-local
>>anycast addresses in documents (like Dave Thaler's DNS stuff), we
>>don't actually know what a site-local anycast would look like.  The
>>latest verbal possibilities from Steve seem to indicate that we might
>>artificially reserve one of the subnet numbers (all zeros or all 
>>ones?) as an area to assign site-local anycast addresses.  If so,
>>the routing might be okay...?
>>
>>However, the IPv4 stuff has only been tried with BGP.  Do we know for
>>sure that the OSPF tree will converge if it thinks that one host is
>>actually located at two different places in the topology?  Maybe this
>>would just work -- I don't know OSPF well enough to know for sure 
>>without closely reviewing the spec (or maybe trying it).  
>>
>>Also, since a site doesn't actually map to an AS, and may not be 
>>handled by a single routing protocol (i.e. it may be running OSPF
>>between some RIP networks, or something?), then it isn't clear
>>exactly where and how you would "inject" a route into the routing
>>system (or whatever loose wording we're using) and be sure that
>>it would propogated to the entire site...
>>
>>(4) The current IPv6 routing table MIBs are based on the IPv4
>>routing MIBs.  It isn't clear if they will be able to properly
>>handle IPv6 routing tables, as they may be different.  Once we
>>figure out the architecture and conceptual datastructures, I will
>>closely review the MIBs to see if they match.
>>
>>So, those are basically my issues.  Some of these may be non-issues,
>>but I won't know for sure until I investigate (much) further.  I had
>>been hoping that some actual routing person was looking at these 
>>issues, but alas...I guess that I will be forced to play a routing 
>>person on TV (or on the mbone, anyway).  I will try to write something 
>>up in the next several weeks for the IPv6 WG.
>>
>>Sean, I doubt that this answers any of your questions.  But, feel
>>free to let me know if you have specific issues or questions.  I'll
>>also try to read the multi6 list next week.
>>
>>I'm glad that someone was listening...  :-).
>>
>>Margaret
>>
>>
>>
>>
>>At 06:28 AM 8/10/01 , Thomas Narten wrote:
>>>Hi Sean.
>>>
>>>Margaret Wasserman was the person who raised the issues from the
>>>floor during the plenary..
>>>
>>>Margaret, are you on the multi6 list?
>>>
>>>Thomas
>>>
>>>[email-masked] (Sean Doran) writes:
>>>
>>> > it was noted that there has been little attention payed in the ietf
>>> > to scoped addresses of various types in the ipv6 standard (e.g., link
>>> > or anycast addresses).  
>>>
>>> > for our wg (from me, not from the floor): can these different address
>>> > varities, when a host uses them, be considered as a form of multiple-address
>>> > multihoming?    likewise, those approaches which change the routing system
>>> > in some fashion: how can/do they deal with such addressing types?
>>>
>>> >       Sean.
>>>
>>> > ps - several questions are coming up and unfortunately i am too slow a typist:
>>> > how does one choose and forward addresses for anycast objects, how does
>>> > one deal with sites which are not convex, how does one scope addresses
>>> > beyond a site boundary.  there were others.  if someone can identify
>>> > the questioner from the floor, i would like to invite her to ask the
>>> > same questions here.   as noted, with my other hat, irtf-rr is a reasonable
>>> > place to ask (not necessarily ipv6-specific) similar questions, but
>>> > note well that the irtf is *NOT* the ietf, whereas this is.

---

From: Pekka Savola <[email-masked]>
Date: Sat, 11 Aug 2001 12:14:36 +0300 (EEST)

I've added ipng to Cc: list as this has never been discussed there, and is
IMO mostly ipng-stuff.

> >>these addresses as CIDR-like?

I think this was discussed very recently in 'prefix length used by
backbone routers' thread in multi6.  I'm not sure if all issues raised
here were brought up there.

> >>conceptual routing tables (one global, one site).
[snip]

Why would there be a need for two routing tables?  It might help, but I
don't see it as strictly necessary.

Wouldn't "practically" some equivalent of an IPv6 access list, blocking
site-local addresses from spreading outside of the site and in the site be
sufficient?

Also, all routing EGP protocols should have route-map's or equivalent to
eliminate the possible spread of site-local, and link-local, address
announcements.

It's legal to use different AS numbers inside the site (e.g. one or two
primary ones and some from private space), the default behaviour perhaps
should be:

 1) in eBGP (or anything between two AS numbers), filter site-local
announcements by default
 2) if extra keyword "same-site" is used even with different AS's, this
would not be done.

> >>at how OSPF would handle a host route injection for a router outside

Another huge difference (from usage point of view): IPv6 anycast address
MUST not be used as a source address.

I doubt host route injection will work due to "martian" filtering of
/128's, or other such routes.  What will work, is the similar thing as it
is done with IPv4 anycast:

Consider:

6to4 IPv4 anycast prefix is 192.88.99.0/24.
6to4 IPv4 anycast address is 192.88.99.1.

If someone injected 192.88.99.1, that would surely be filtered.

But as IPv6 anycast is defined the same as IPv4 anycast, you can just
inject routes to to some /48, /64, etc. networks -- no different than
current route injections.

(btw, nobody is advertising 6to4 anycast prefix in DFZ yet.  Shows the
current ipv4 anycast deployment rate mentioned in ngtrans at IETF51 wrt.
isatap discovery IMO.. :-)

> >>without closely reviewing the spec (or maybe trying it).

We cannot be sure.  That is, if OSPF costs (or other protocols' metrics)
to different anycast advertisements are equal, there will be
load-balancing.  But then again, this might be designed behaviour (for
unicast, probably not for anycast) too.

By designing the site topology properly, and adjusting the last-hop (ie.
router to host providing anycast service; this might not always be doable
if routers themselves act as anycast servers) OSPF costs by adding big
values there, e.g. "500" or "1000" so there would be no overlap no matter
what, this should converge quite nicely.

This is a policy issue.  One might want to (in intra-site policy) either
(or some others):

 - select the closest anycast server (ospf cost should not be adjusted,
loadbalancing would be considered a feature)
 - select always the default anycast server if it's responding, if not,
select a different one (preferring the service with ospf cost)

> >>closely review the MIBs to see if they match.

As you know, Bill Fenner is revising these.  Now's the time...

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

---

From: "Brian Haberman" <[email-masked]>
Date: Mon, 13 Aug 2001 09:10:37 -0400

Hi Margaret,
     Let me take a crack at answering some of these.

> >>these addresses as CIDR-like?

They still need to be treated as CIDR-like.  A longest prefix match
lookup may find a 16-bit or 48-bit entry, especially if the address
being looked up is outside that router's domain.

> >>conceptual routing tables (one global, one site).

I have begun implementing one.  And it does require having multiple
"conceptual" routing tables.  The reason for this is that on a site
boundary router, the zone IDs per interface determine which interfaces
are in each zone.  Since the site-local addresses do not contain any
of this information, the routing table needs to enforce the separation.

For example, a router that is a border between site A and site B may
find that it has entries for FEC0:0:0:ABCD:: in both sites.  If that
is the case, they can only be distinguished via the zone IDs assigned
to sites A and B.  This can be implemented as either separate route
tables per site or by including the zone ID in the route table
lookup.

> >>Also, topologies change.  So, what happens if a site becomes

So, I am partly at fault for not getting that text into the architecture
draft.  My working definition for a convex site is that for any two
points within the site, the shortest path between the points never
leaves the site.

As for topology changes, an event that turns a site from convex to
concave can be treated in the same manner as an event that fragments
a network.

> >>information (or even know that they may pass site-local routes).

To my knowledge, none of the v6 routing protocol specs have been
updated to reflect the information in the scoped address architecture.
I think that once the architecture is solidified, we can go and look
to see what needs to be done in the routing specs.

> >>at how OSPF would handle a host route injection for a router outside

From discussions I have had, I don't think there is much interest
in global anycast.  The proposals I have heard have been for using
anycast within domains, so that the anycast address gets aggregated
into the global prefix for that domain.

> >>closely review the MIBs to see if they match.

There are updates in the works on the v6 MIBs.  One of the topics
being discussed is the structure of the routing table MIB.  Take a
look at http://www.aciri.org/fenner/mibs/v6/

Brian

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 16 Aug 2001 17:08:29 +0200 (CEST)

> points within the site, the shortest path between the points never

Brian,

I can see odd cases where using global addresses the traffic would
indeed leave the site, whereas using site-local address it might not.
So shouldn't the definition be related to the addresses used?

Or is there a concern related to a global destination (routed on
a path that goes outside) and a site-local source, which would cause the
packet to be dropped at the border?

  Erik

---

From: "Brian Haberman" <[email-masked]>
Date: Thu, 16 Aug 2001 13:35:01 -0400

Erik,

> So shouldn't the definition be related to the addresses used?

Correct, it should be in terms of the scope of the addresses being
used.

> packet to be dropped at the border?

I know that this subject has been discussed several times.  I believe
the scenario that Margaret is concerned about is based on this
diagram she posted to IPNG several months ago (correct me if I am
wrong, Margaret):

==========================================================
 SITEA
             Host1                      Host2
               |                         |
       ______._|_________    ___.______._|_____________
       Link1 |                  |      |       Link2
             |                (down)   | 
             |                  |      |
             |+-----------------+      |
             ||                        |
============R1========================R3===================
 SITEB       |                         |
             |                         |
        _____|_________________________|________________
                                                 Link3

===========================================================

If Host1 is using site-locals, then Host2 can't be reached and R1
cannot return any info that is very useful to Host1 other than a
Destination Unreachable message.  If Host1 uses a global destination
address and a site-local source address, R1 returns a Scope
Exceeded message even though Host2 is in the same site as Host1.

Brian

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 17 Aug 2001 10:29:26 +0200 (CEST)

> diagram she posted to IPNG several months ago (correct me if I am

But I think there is another issue even when the path isn't down.
Suppose the metric/preferences on the routes are such that when sending to
the global address of H2 from H1 that the packets pass through site B.
Then it is even worse since H1 and H2 have both a path inside the site
and one outside, but if H1 doesn't have a global source address they
will fail to communicate.

Thus I think this needs to be captured in the notion of convex - packets
sent to a global address in the site can not leave the site.

  Erik

---

From: Margaret Wasserman <[email-masked]>
Date: Sat, 18 Aug 2001 16:03:19 -0400

Hi Brian,

>diagram she posted to IPNG several months ago (correct me if I am

This is one case that I am concerned about, yes.

You and Steve resolved this case by stating that a site must be
"convex", and I am sure that the text explaining this will be in the next 
version of the document -- I do understand what it means, and I agree
that it will solve the problem.

However, this does place several constraints on the concept of a site.
For example, consider two office of the same company, both connected to 
the Internet, that are connected by a high-cost link (higher cost than
traversing the Internet between the two locations).  These two 
locations could not be configured as a single site, although they 
would probably cross-pollinate their global addresses (for redundancy)
and might be treated as a single routing AS.  This adds some complexity
for routing purposes, as it clearly decouples the concept of sites
(which must be convex) and routing ASes (which do not have that
restriction).  

Also it is possible that all of the subnets of a single global address 
allocation may not be in the same site, right?  So, it would probably
not be acceptable for renumbering to be constrained by site boundaries.

Some other issues have been raised in my previous mail, but I forgot to
raise two additional issues:

Default Routes:  Will multi-interface-hosts and non-default-free site border 
routers potentially have several different default routes (one for global 
traffic, one for each site, etc.)?

And in general, will we allow multi-site hosts?  Or, can only a router have
interfaces in more than one site?

Just for the record, I think that I know the answers to many of the
questions that I have posed.  I'm sure that several of you know the answers
as well.  But, do we know the same answers?  

And, how can we standardize this behaviour well-enough for people to implement 
site-border, anycast-compatible, IPv6 hosts and routers without investing 
years of effort to understand IPv6?

One of the most disturbing issues (to me) is the widespread misconception
that it should be possible to route IPv6 packets by looking at only the
first 64-bits.  This may be true for the small number of addresses that
we've already defined, but the IESG has been quite clear that we shouldn't
rely on any hard internal boundaries in the addresses.  And this certainly
wouldn't work for host routes.

Also, site-local addressing and anycast addressing are the biggest changes in
IPv6 from IPv4.  Basically, addressing == routing.  And, I think that it
would be worth the investment of our time to make sure that we understand
all of the routing implications of site-local and anycast addressing.  Also,
we need to make sure that the appropriate semantics are included in the
current IPv6 routing protocols (OSPFv3, BGP-4+, RIPv6, ??) to handle site-local
addressing and the injection of anycast host routes (as we are suggesting in
some drafts).

I am not actually disagreeing with any of the work in this area, to date.
It just isn't complete.

Margaret

---

From: "Brian Haberman" <[email-masked]>
Date: Mon, 20 Aug 2001 09:22:39 -0400

Hi Margaret,

> that it will solve the problem.

Good.  Now if I could get some cycles to actually edit the doc...

> (which must be convex) and routing ASes (which do not have that

This is true and I would state that it is a "good thing" to have the
routing ASes and the sites decoupled.  In your scenario, if the
sites had a cheaper connection via the Internet, I would not even
consider putting them in one site.

If it was necessary to have both locations in the same site, it
would still be possible with the use of IPSec tunnels and some
tweaked routing parameters.

> not be acceptable for renumbering to be constrained by site boundaries.

I agree.  Renumbering is not constrained by site boundaries, it is
constrained by delegation of the prefix being renumbered.

> traffic, one for each site, etc.)?

Sure.  I have that today in my v6 lab network.  There is a default
route per scope (global, site, etc.).  Works fine.  There may be an
issue of having multiple default routes per zone ID.

> interfaces in more than one site?

I would not make that restriction.

> as well.  But, do we know the same answers?

Don't know.  What are your answers to these questions?

> years of effort to understand IPv6?

I hope so.

> wouldn't work for host routes.

That is correct.  I hope people start forgetting that misconception.

> addressing and the injection of anycast host routes (as we are suggesting in

Agreed.  I happen to be working on both of these areas.  I am hoping
that once we get the scoped addressing architecture done, we can go
and look at making necessary changes to the routing docs.  The anycast
will take a little longer, but may not affect the routing RFCs.

> I am not actually disagreeing with any of the work in this area, to date.

Agreed.

Brian

---

From: Randy Bush <[email-masked]>
Date: Sun, 26 Aug 2001 09:56:49 -0700

> currently used in IPv4.

this is not generally appreciated, in both senses of the word.

> One big difference: a global IPv6 anycast address would (probably?) not
> appear to be inside of anyone's AS.

this part i missed.  or by 'AS' do you mean OSPF area?  in this gang, 'AS'
is a bgp thing.

> don't actually know what a site-local anycast would look like.

why would those of us who want an anycast interface to appear to the world
as no different from a unicast interface not think it should be the same as
a site local unicast anycast address?

> However, the IPv4 stuff has only been tried with BGP.

not true at all.

> Do we know for sure that the OSPF tree will converge if it thinks that
> one host is actually located at two different places in the topology?

hmmm.  i will admit to not having checked the multi-area issue.

> I'm glad that someone was listening...  :-).

my apologies for not listening well enough.

> (which must be convex) and routing ASes (which do not have that

i think things are getting very crufty here, and we will regret trying to
create these complex little sub-definitions of site etc.

> questions that I have posed.  I'm sure that several of you know the
> answers as well.  But, do we know the same answers?

and, as you say, have these questions and their answers been clearly and
simply spelled out in the published specs?

From: "Brian Haberman" <[email-masked]>
> into the global prefix for that domain.

aiiiii!!!!!  global v4 anycast is used significantly.  let's not further
break v6 anycast, please.

> In your scenario, if the sites had a cheaper connection via the Internet,
> I would not even consider putting them in one site.

what is 'putting them in one site'?  do you mean considering them as a site
in terms of things such as site-local addressing/routing?  or are you
starting to mandate operational restrictions?

> If it was necessary to have both locations in the same site, it would
> still be possible with the use of IPSec tunnels and some tweaked routing
> parameters.

please explain.

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 28 Aug 2001 05:26:41 +0800

> First, no one that I have talked to on the subject of anycast feels
> that global anycast is all that useful.

can't help the limits of your circle of friends, but since you say "no one"
and i am disagreeing, the problem may be on the receiving, not the sending,
end <grin>.

> someone can't offer a global anycast-based service, as long as
> the servers are all based out of that network's prefix so that the
> anycast address gets aggregated in the global routing table.

you may want to look at the aroot experiments.

> Second, can you describe the current v4 use of anycast where anycast
> servers are using globally routable anycast addresses from outside
> their administrative domains?

ibid

randy

---

From: "Brian Haberman" <[email-masked]>
Date: Mon, 27 Aug 2001 18:23:15 -0400

Randy,

> and i am disagreeing, the problem may be on the receiving, not the sending,

Not at all.  I was just pointing out that you are the first person
who has expressed much of an interest in generic, global anycast
support.

> you may want to look at the aroot experiments.

So, has this been actually deployed in an operational network?
I recall seeing the slides on experimentation, but have not heard
any results of those experiments nor heard of operational use.  The
aroot mailing list archive seems to be unreachable at this time.

> > their administrative domains?

Would that be the PolyVision whiteboard product?  There is no info
on their website as to how they utilize IP anycast.

Brian

---

From: Randy Bush <[email-masked]>
Date: Tue, 28 Aug 2001 07:52:05 +0800

> So, has this been actually deployed in an operational network?

yup, for a year or so now, and is about to expand.

> any results of those experiments nor heard of operational use.

works boringly well.

> The aroot mailing list archive seems to be unreachable at this time.

ohhh?  what url did you try?

randy

---

From: "Brian Haberman" <[email-masked]>
Date: Tue, 28 Aug 2001 09:14:47 -0400

Randy,

> > any results of those experiments nor heard of operational use.

Can you give me a definition of works boringly well?  Would that
entail no calls to the NOC as the measuring stick?

> ohhh?  what url did you try?

www.psg.com/lists/aroot/

I was consistently getting a DNS resolution error last night.  It
seems to work this morning.

So now that you have given me a use of global anycast services, do
you expect that injecting 128-bit host routes into the global route
table, that can't be aggregated, is going to scale?

Brian

---

From: Randy Bush <[email-masked]>
Date: Wed, 29 Aug 2001 07:50:03 +0800

> entail no calls to the NOC as the measuring stick?

that's one.  also had measurements.

>> ohhh?  what url did you try?

dunno where you made up that url.  but try <http://ops.ietf.org/lists/aroot/>

> table, that can't be aggregated, is going to scale?

i would offer my abject apologies for suggesting that, except i didn't.

randy