Subject: Re: An idea: GxSE

From: [email-masked] (Sean Doran)
Date: Wed, 20 Jun 2001 23:50:09 +0200 (CEST)

Mohan Parthasarathy <[email-masked]>

| > Why not just use SCTP?  Why reinvent the wheel?
|
| In SCTP, i think you have to know all the addresses at the beginning
| of the connection. Here, the new address is not known until
| renumbering happens.

My interpretation of Ohta-san's draft proposal (see list archives)
understands that it calls for the addition and subtraction of
addresses *known inside the host* during the lifetime of a connection,
based on learned information about the behaviour of nonlocal topology.
In other words, Mohan is correct if "here" refers to Ohta-san's proposal.

Although his draft is the only one we've gotten so far, I do not
think we would want to exclude other GSE-like proposals, should
the arrive upon us in the form of a draft (hint, hint, hint to everyone).

By contrast, Paul Francis's idea (GxSE) proposes that things inside
multihomed sites do not need to know their entire set of addresses,
but that a nameserver will ("typically") know all of them.   To me,
this is Ohta-san's proposal with the topology knowledge pushed one
or more hops away from the host, and as a result of which introduces
an architecture which requires some small changes by host-developers
(e.g., don't make assumptions about the end-to-endianness of the 
GR (RG in Mike O'Dell's GSE) when doing things like checksums or
encodings of host addresses in upper-layer protocols).   PF notes this,
Ohta notes the opposite in his draft.

Without making any sort of judgement, I'd like to suggest that
although our initial volunteer team hasn't finished the draft of
a requirements doc yet, at least a couple of them read the list.
I suspect that (if it's not there already), the reqs will indicate
a desire to support a wide variety of "transport-layer" protocols.

Thus, while we're waiting on the reqs, I think it'd be particularly
nice if proposals (however informal) somehow could be "fitted-into" --
or at least take into consideration -- TCP, UDP, and so forth,
and ideally make implementing/managing IGPs (in the sites themselves
or in their respective inverse trees of providers) very much more
difficult than today (in v4 or v6).    Shim approaches (like SCTP)
are not the only -- or even necessarily the best (though maybe they are) --
way of taking these possible-requirements into consideration.

In other words, it's far too early to exclude prima facie any new
ideas, or any old ones, either.

Yours wishy-washily,

	Sean.

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 20 Jun 2001 17:27:18 -0500 (CDT)

> difficult than today (in v4 or v6).    Shim approaches (like SCTP)

I KNOW you meant the opposite here, Sean :)  You want IGPs to be easier,
not difficult, RIGHT? :)

> way of taking these possible-requirements into consideration.

SCTP isn't a shim, but some people want to use it that way.  It's really
good for some connections, but I'm tired of hearing people advocate it in
bad situations :P  However, reinventing the wheel is bad sometimes, too.

Part of the idea GxSE induced in my cluttered head was the notion that
this doesn't have to be at the border routers only.

Border routers are often very heavily loaded.  Announcing valid GR/SK
pairings via BGP or the like and then propagating them into an IGP
(OSPF/IS-IS) would allow you to make translations further into the
network, without renumbering.  Basically, you could make the network
renumber itself.  If this is done properly, you could keep the
intelligence at the edge, completely avoid renumbering, AND get better
routing/multihoming performance overall.  This COULD be a win-win
situation, I just think we have to look at it from all angles and proceed
very carefully.

Not quite so wishy-washily yours,
-Taz

-- 
        "Be liberal in what you accept,
      and conservative in what you send."
--Jon Postel (1943-1998) RFC 1122, October 1989

---

From: Ben Black <[email-masked]>
Date: Tue, 26 Jun 2001 10:32:58 -0700

> situation, I just think we have to look at it from all angles and proceed

The problem I ran into when thinking up something very similar to GxSE
was that _only_ the border routers know the actual prefix at the egress
point (barring end to end virtual circuits of some sort).  

My solution was to distribute "valid" prefixes in the IGP and have them
communicated to end hosts using an RIP-like mechanism to simply 
broadcast the set of prefixes hosts on a given segment might be
translated into (does that make sense?).  Hosts would then inform
remote ends of sessions (TCP, UDP, what have you), of the prefixes
from which they might communicate.  

Given the hop by hop nature of IP, I suspect it will be rather difficult
to have the translations occur anywhere but at the border.

Ben

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Tue, 26 Jun 2001 13:22:20 -0500 (CDT)

> from which they might communicate.  

This is similar to the idea I had expressed.

> to have the translations occur anywhere but at the border.

Not at all.  As I explained in a previous post, having the valid GR/SK
pairs announced in BGP or the like and propagating them through the
internal network via an IGP or a proptocols specifically designed for this
purpose would let you translate in the distribution layer or even access
layer (although in most cases the distribution layer routers would be the
ones beefy enough to handle it of the two).

This moves the intelligence pretty close to the edge.  If you use my idea
of using SK addresses that are globally unique as the actual endpoints of
the connection, your hosts can be dumb as well.  Therefore, it is possible
to do GxSE with multiple remappings where only routers are aware of the
remappings, thus making it backwards compatible with vanilla v6, except
between your border router and the router in your network that is
remapping.  For connections not remapped, it has no effect.

In fact, I think this is very important to note.  It is possible to do
this where remappings are done invisibly to the opposite end if there's
not a GxSE router in the path between your GxSE remapper and the opposite
endpoint of the connection.  While you get a far less effective scenario,
you still get SOME of the benefits, while allowing for vanilla v6.

-Taz

---

From: Ben Black <[email-masked]>
Date: Tue, 26 Jun 2001 22:14:43 -0700

> ones beefy enough to handle it of the two).

This doesn't address the concern I noted above which is that the interior
routers simply don't know the egress router, and hence the egress prefix
(assuming you always map into space provided by a given upstream, which
is how I would expect to avoid rapid table growth).  This information
_could_ be communicated to the interior routers, but it would require them
to carry full reachability information, which seems a rather onerous
requirement.

Ben

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 27 Jun 2001 02:16:42 -0500 (CDT)

I understand your point, but with careful construction this shouldn't be
an issue.  Assuming you use a link state protocol for the interior portion
and you allow the border routers to notify the interior routers of the
egress point, you could move the rewrite closer to the edge.  In fact, if
you let the initial rewrite occur at the egress router, and then move the
rewrite backwards into the network, with the egress router indicating
validity of all recently used routes back into the IGP/other protocol for
this purpose, you would know where the packets are going.

Basically, if there is a topology change, you just need to make sure
your path is still valid.  Another behavior to control this may be to stop
rewriting at the edge if you detect a topology change that may affect your
internal routing, and let the egress router rewrite and re-alert you.

I had another point, but I forgot it...serves me right for stepping away
to deal with something else.

-Jon

---

From: Ben Black <[email-masked]>
Date: Wed, 27 Jun 2001 00:16:03 -0700

> egress point, you could move the rewrite closer to the edge.  In fact, if

In order to do that, the interior routers carry full tables.  Whether the
information is distributed via BGP, an IGP (an unpleasant thought), or by
some mechanism specific to this purpose, the issue remains.

> internal routing, and let the egress router rewrite and re-alert you.

I shudder to consider the troubleshooting implications of this.

Ben

---

From: Joe Abley <[email-masked]>
Date: Wed, 27 Jun 2001 11:21:17 -0400

> this purpose, you would know where the packets are going.

This may be a trivial point, but I know many "edge enterprises" (i.e.
customers) who have extrordinary difficulty in running an IGP, and
either wind up statically-routing everything within their network
or use a fantastic mixture of RIP, EIGRP and OSPF corresponding to
the various waves of contractors that have been brought in to fix
things at different times.

These people can currently multi-home just fine, since all the multi-
homing smarts are constrained to one or two BGP-speaking border routers,
which is a nice black box that "our BGP guy" can handle without impinging
on the hoardes of other junk connected in the network.

This is just a tentatively-raised red-tinged warning flag: requirements
involving IGPs may prove onerous to end-users.

Joe

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 27 Jun 2001 10:23:05 -0400

At 03:16 27/06/01, Jon (Taz) Mischo wrote:
>egress point, you could move the rewrite closer to the edge.  

        If there are multiple egress points, then either one is
carrying full routes in the link-state IGP that you postulate
xor the modification happens at the site border router only.
The site border router must be listening to EBGP anyway if it
is multihomed, so it already would have the data needed for reasonable
path selection.

        The latter is a lot simpler.  IMHO, simpler is better.

>In fact, if

        Operational folks probably won't generally tolerate putting
all that extra stuff into the IGP, because it makes a complex system 
a great deal more complex.  This is particularly true in multi-homed
enterprises, most of which just run a plain old IGP (e.g. RIPv2, OSPF)
and are unlikely to deploy protocols common in ISPs (e.g. I-BGP, IS-IS).

        An alternative would be a new kind of ICMP message from
the border router back to the originating host.  This could be
authenticated (e.g. using AH) in situations where such authentication
was a concern.

>internal routing, and let the egress router rewrite and re-alert you.

        The latter is simpler and consequently more palatable, IMHO.

Ran

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 27 Jun 2001 13:44:00 -0500 (CDT)

> is multihomed, so it already would have the data needed for reasonable

I was talking about a design that specifically didn't require route
redistribution.  Except in NSPs and larger running IS-IS, you generally
don't have that sort of route distribution.  Your IGP will generally
reliably get you to the same border router.  It may send you to a
different border router due to IBGP, but these behaviors are fairly
static.  As such, it's pretty easy to alert back towards the host that a
rewrite needs to happen, and let the nearest capable router take care of
it.

>         The latter is a lot simpler.  IMHO, simpler is better.

I agree that simplicity is often better.

> and are unlikely to deploy protocols common in ISPs (e.g. I-BGP, IS-IS).

"All that extra stuff" is actually 1 thing if you use the ICMP message you
talk about below.  The one thing is the ability to signal that it has
routing table stability or doesn't have routing table
stability.  Furthermore, you would only want to move rewriting into the
network if you are running a link state protocol, as I said before.  This
means OSPF, IS-IS, or EIGRP (which is really a hybrid and non-standard,
but it IS used in lieu of IS-IS sometimes).

> authenticated (e.g. using AH) in situations where such authentication

An ICMP message sent back towards the host with rewriting directions
would, indeed, be a good way to move the rewrite further back in the
network.

>         The latter is simpler and consequently more palatable, IMHO.

Agreed.  But it means either an indication into the IGP that there is some
sort of route instability, or another ICMP message towards the host
indicating that rewriting should be done at the egress point again.

Another point is that if there's some route flap going on, you may want a
rewrite indication dampening period.  Even if you are incorrectly
rewriting near the edge, your border routers are still capable of
rewriting, since they'll recognize the SK and decide that the GR is
incorrect and rewrite it.  If there's instability between core routers,
this could creat an ICMP storm or delayed convergence in an IGP, so a
dampening period might be wise.

The main reason for moving the rewrite back is that you don't want to be
putting all of the work on the border routers all of the time.

-Taz

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 27 Jun 2001 13:47:46 -0500 (CDT)

> involving IGPs may prove onerous to end-users.

Perhaps it will comfort you to know that the only time you actually NEED
something in the IGP, or need to use new ICMP messages is when you
actually ARE moving the rewrite into the network.  If you don't have the
expertise to run an IGP, you shouldn't be moving the rewrite to the edge,
you should just let the border routers do it.

Moving it towards the edge simply reduces the load of the border
routers.  Any enterprise that doesn't have capable network engineers
probably doesn't have any edge routers capable of anything more than
dropping every other packet, if you get my drift.

-Taz

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 27 Jun 2001 14:58:51 -0400

>rewrite needs to happen, and let the nearest capable router take care of

        We need an approach that handles large end-user organisations
(consider: GE, HP, NRL) where not all upstream trunks have interfaces
in a single router.  NRL's campus, which has ~3000 users at one
site, used to have 3-4 separate routers with upstream connections
(don't know what their topology is these days).

        It is precisely that sort of site that is more likely to be 
multi-homed, IMHO.

>but it IS used in lieu of IS-IS sometimes).

        Right, so I'd much rather use that ICMP message, work with any
IGP, not change any IGP, and not put any extra gorp in the IGP.

        A multihoming approach ought to work equally well if one is 
using RIPv2.  Lots of sites can and do use RIPv2 as their IGP.  RIPv2 
is more common when there is a single border router with 1 or more 
uplinks to different service providers, IMHO.  It is fine provided
the scale of the site is reasonable and the site has been structured
in a thoughtful manner.  RIP is also much easier to configure than 
a typical link-state IGP.

        Note that my objection is to putting any additional goop
into any of the IGPs (I don't consider IBGP to be an IGP).  Perhaps
I misunderstood your note ?

>would, indeed, be a good way to move the rewrite further back in the

        Or just tell the host to change its own source address
for that session and continue to let the site border router(s)
rewrite the destination if appropriate...

>indicating that rewriting should be done at the egress point again.

        Or the egress point just does the rewrite and also throws a clue
back to the origin host, without putting any gorp into the IGP.

>putting all of the work on the border routers all of the time.

        I don't object to letting folks move the rewrite back
or to letting the host participate in such decisions.  

        I will note that I'm truly not worried about a router 
performing the rewrite over say 10 Gbps Ethernet uplink to an ISP 
at full line rate.  Forwarding ASICs are your friend.  Other folks 
mileage might vary of course...

Ran

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 27 Jun 2001 14:32:57 -0500 (CDT)

> (don't know what their topology is these days).

You need to re-read some of what I've said about how this rewriting is
actually done.  You could have 20,000 border routers.  The behavior is
still the same.  A border router merely looks at the SK address of a
packet when it is going into an egress queue, and makes sure the GR is
appropriate.  If it's not, it rewrites the GR.  This may not happen on the
queue, this may happen in an asic on the port...there are many ways to
implement it.

>         It is precisely that sort of site that is more likely to be

They are a type of multi-homed site.

> IGP, not change any IGP, and not put any extra gorp in the IGP.

As a former Network Engineer for a few large networks (including one of
the largest ISPs in the history of the US), I can tell you I'd tolerate a
new directive in an IGP better than a whole slew of ICMP messages to every
host every time Joe Blow's route flaps.

> a typical link-state IGP.

RIP is used in some cases, yes.  It is good for very simple
networks.  GxSE works better with a link state protocol because it has
faster convergence and notices something is wrong a lot faster.  GxSE
would work just fine with RIP, it just wouldn't be as efficient.  That
is just a basic of routing, however.

> I misunderstood your note ?

I think you may not have seen why I was talking about link state
IGPs.  The additional goop actually would HELP, in that it would prevent
packet storms.  Most network engineers dislike packet storms.

> rewrite the destination if appropriate...

ummm?  I don't think you understand GxSE.  Source addresses get rewritten,
not destination.  The point is NOT to have to do what you just
said.  That's what we're trying to avoid.

> back to the origin host, without putting any gorp into the IGP.

Origin host doesn't need a clue.  My approach is for a stupid host.  
Doesn't need to know, unless it specifically asks because it is doing
something like IPsec.

> mileage might vary of course...

Heh.  The point is that ASIC changes mean hardware changes.  That's even
more slowly adopted.  How you choose to implement it is your business, not
mine.  I could tell you 10 different approaches.  The point here is mainly
that you should be able to move it away from the core, especially if your
core is overworked.

-Taz

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 27 Jun 2001 18:01:08 -0400

>host every time Joe Blow's route flaps.

        I believe that you believe that.  

        I also think that Enterprise network engineers often view 
network issues VERY differently than backbone network engineers.
Neither is right or wrong necessarily.  The network contexts are really
different between a small enterprise, a large enterprise, and any
backbone.

>ummm?  I don't think you understand GxSE.  

        Remind me again, what is the filename on your Internet Draft ? :-)

>Heh.  The point is that ASIC changes mean hardware changes.  
>That's even more slowly adopted.  

        I'd be startled if multiple vendors couldn't do address
rewriting already.  It uses roughly the same hardware logic as NAT,
so if one has a partially programmable forwarding engine and 
thought about NAT, one might well have the capability in the
ASICs already deployed.  Different folks mileage will vary of course.

Ran
[email-masked]

---

From: "Jon (Taz) Mischo" <[email-masked]>
Date: Wed, 27 Jun 2001 17:15:41 -0500 (CDT)

> different between a small enterprise, a large enterprise, and any

I've dealt with all three, rather extensively.  I've noticed that often
enterprises whose engineers don't "think big" are also the ones whose
networks are constantly broken.

>         Remind me again, what is the filename on your Internet Draft ? :-)

Touche.  This really is just conversation, though.  A draft will come
later, after there are requirements (unless Paul has other ideas?  the
idea did start with him).

> ASICs already deployed.  Different folks mileage will vary of course.

This may or may not be true.  Just becase rewriting is possible doesn't
mean it is possible in this manner, especially since it's not a one to one
or many to one relationship, but a decision tree relationship.

-Taz

---

From: "Paul Francis" <[email-masked]>
Date: Fri, 29 Jun 2001 12:34:38 -0700

>         Remind me again, what is the filename on your Internet Draft ? :-)

Taz, I think it would be good if you come up with a different name for what
you are thinking about (GxSE+R or something  :-), because for me there is a
qualitative jump between involving routing and not involving routing.  This
is not to say that what you are talking about is bad, I just don't want
people to lose the distinction.

PF