Subject: Re: tunnels within tunnels and MPLS/RSVP ERO

From: Curtis Villamizar <[email-masked]>
Date: Wed, 01 Mar 2000 17:57:49 -0500

In message <[email-masked]>, "Ashish 
Monga" writes:
> How is tunneling handled in  case we use CR-LDP with ER-HoP TLV having LSPID
> of LSP thru which new LSP is to be tunneled.
> How are label request and label mapping messages processed in that case.

From the standpoint of the pair of customer routers each router has a
virtual interface.  If you send traffic into the virtual interface on
one customer router it comes out on the virtual interface on the
other.  If the customer wants to run LDP over such an interface that
is fine.  In each direction, the customer router adds two labels.  One
gets the traffic across the ISP who may be running RSVP, not LDP, and
the other label identifies the source and gets the traffic to come out
the correct virtual interface on the other customer router (and is
entirely administered by the customer).

As I've described it, there is no change to any of the signaling
protocols or encapsulation required to do this except the conventions
for processing the ERO last hop that I suggested earlier.  This is
simply a usage of MPLS.

Curtis

> -----Original Message-----
> From: Kireeti Kompella <[email-masked]>
> To: Curtis Villamizar <[email-masked]>
> Cc: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
> Date: Wednesday, March 01, 2000 8:07 PM
> Subject: Re: tunnels within tunnels and MPLS/RSVP ERO
> 
> 
> >Hi Curtis,
> >
> >> I have another question.  This may be more of a usage question.  This
> >> time regarding how the ERO should appear when supporting tunnels
> >> within tunnels.
> >
> >There are at least two issues here.  Suppose we wish to tunnel LSPa
> >(access LSR to access LSR) through LSPb (backbone).
> >
> >1) What does the ERO for LSPa look like?
> >2) How does the head end of LSPb know when it gets the path message
> >   for LSPa to tunnel it through LSPb?
> >
> >Both are addressed in draft-kompella-mpls-optical, section 4.3.
> >Issue 1) has two subcases: a) LSPb doesn't exist when LSPa is first
> >signalled -- the signalling for LSPa creates LSPb; and b) LSPb exists
> >when LSPa is signalled.
> >
> >> One way to do this is specify loose hops.  The initial ERO would
> >> contain from the ingress (a-LSR1) would contain a set of one or more
> >> strict hops to the near end backbone LSR (b-LSR1), a single loose hop
> >> specifying the far end backbone LSR (b-LSR2), and a single loose hop
> >> to the egress LSR (a-LSR2).
> >
> >Note that in the above draft, we suggest that LSPb be advertised as
> >a 'forwarding adjacency' as far as traffic engineering goes, so the
> >hop from b-LSR1 to b-LSR2 can in fact be a strict hop.  Yes, RSVP
> >would need to see TE links as well as physical links for this to work.
> >
> >>                              One interpretation of this would create
> >> separate tunnels across the backibone from each access LSR to each far
> >> end access LSR.  An alternate is for b-LSR1 to push a label on the
> >> stack, and place the traffic within a tunnel with b-LSR2 at the
> >> egress.
> >
> >How does b-LSR1 know which to do, i.e. to which LSPs the former applies,
> >and to which the latter?
> >
> >We solve this problem by marking each link with an LSP hierarchy number.
> >All links (unidirectional) on access LSRs are PSC-1.  All links from
> >backbone LSRs to access LSRs are also PSC-1.  Links from a backbone LSR
> >to another backbone LSR are PSC-2.
> >
> >If an LSR sees an ERO for LSPa such that the previous hop is over a
> >PSC-1 link and the next hop over a PSC-2 link, the LSR knows that
> >LSPa needs to be tunneled through another.  The LSR determines the
> >end point of the enclosing LSP, checks if there is an LSP going that
> >way with enough bandwidth; if so, the LSR tunnels LSPa through it.
> >Otherwise the LSR creates the enclosing LSP.  Again, RSVP would have
> >to have access to the TE database to do all this.
> >
> >>          The ERO would need to be modified, replacing the loose hop
> >> with SOMETHING maybe followed by a "MPLS label switched path
> >> termination" subobject in the ERO.  The backbone router is free to
> >> reroute the backbone tunnel and all the access tunnels within it.  The
> >> far end backbone LSR (b-LSR2) would do a similar replacement on the
> >> next loose hop.
> >>
> >> My question is what is that "SOMETHING".  It would seem that a strict
> >> hop would be needed but the type IPv4 might not be as appropriate as a
> >> type which indicates an LSP and provide the LSP-ID of the backbone
> >> tunnel in which the tunnel is encapsulated.
> >
> >Our approach was to IP-number the enclosing LSP (forwarding adjacency),
> >so normal IPv4 strict hops just work.
> >
> >Also, the "MPLS label switched path termination" subobject would not
> >be necessary in our framework, as the egress of the enclosing LSP sees
> >a transition from PSC-2 to PSC-1 link in the ERO.
> >
> >Note that an ERO-based scheme for determining where to start and end
> >a tunnel (such as yours) provides a finer granularity of control than
> >one that marks interfaces (such as ours).
> >
> >>                                              Alternately, the IP
> >> address of the egress of the tunnel can be put in the ERO and sent
> >> through the tunnel.
> >>
> >> Either way it seems that this requires that an RSVP adjacency be
> >> formed through the tunnel.  This means that in a network of 100
> >> routers where each has 5 physical adjacencies, 104 (99+5) RSVP
> >> adjacencies would be needed.  That doesn't seem like a good solution
> >> but I don't know of an alternative.
> >
> >New topic: is it worth tunneling LSPs within LSPs?
> >
> >Let's take your example: 100 backbone routers with an out-degree of
> >5.  The average number of hops between router pairs is about 3.
> >Let's say the typical path across the backbone looks like: A-B-C-D.
> >
> >Suppose there are N access-to-access LSPs passing through A-B-C-D.
> >If there was no forwarding adjacency, A, B, C and D all see 5 RSVP
> >adjacencies and RSVP state for N LSPs.
> >
> >If there was a forwarding adjacency A->B->C->D, and all N LSPs went
> >over this adjacency, then for the cost of one new LSP and one new
> >RSVP adjacency, the RSVP state has nearly been halved: A and D have
> >state for N+1 LSPs while B and C have state for 1 LSP.
> >
> >If the average hops between LSRs (diameter?) goes up, the state
> >savings increase.  If the number of tunneled LSPs per forwarding
> >adjacency (N) increases, the overhead decreases.
> >
> >My humble opinion is that the size of the RSVP state is of greater
> >significance than the number of RSVP adjacencies, but I'm willing
> >to be educated.
> >
> >Going back to the question "is it worth tunneling LSPs within LSPs?"
> >I would say, "Probably".
> >
> >> I'd be very happy if someone told me that the answer was already
> >> clearly spelled out in one of the drafts and that I just somehow
> >> missed it.
> >
> >We'd like to think the our draft spells this out, although perhaps
> >not clearly.  Note that the draft is focused on MPL(ambda)S, but
> >the tunneling of electronic LSPs through electronic LSPs falls out
> >fairly naturally in the paradigm of electronic LSPs tunneled through
> >SONET trails tunneled through optical trails ...
> >
> >Kireeti.
> >
> 
>

---

From: "J. Noel Chiappa" <[email-masked]>
Date: Sat, 4 Mar 2000 20:58:05 -0500

> From: Vijay Gill <[email-masked]>

    > On Sat, 26 Feb 2000, Tony Li wrote:

    >> However, I'm concerned that LSPs will not be evenly distributed. In
    >> particular, the backbone routers would seem to be a concentration
    >> point. ... this implies that the backbone router ends up knowing about
    >> each and every LSP that crosses the backbone/region boundary for that
    >> particular region. ... that's on the order of 50,000-90,000 LSPs that
    >> the backbone router needs to support.

I think it was Vadim who first pointed this problem out to me, several years
ago. I've been pondering it ever since; my thoughts below.

    > An approach might be to break the regions down into a n level hierarchy
    > ... and construct n hierarchical domains. One set of routers that
    > aggregate circuits that span regions and act as purely label switching
    > boxes, and another set of routers that aggregate regional LSPs and have
    > a dual mesh with some other set of routers in another region that
    > perform the same function.

    >> I'm suggesting that individual LSPs not transit the backbone router.
    >> Instead, they terminate at the backbone and that the only LSPs in the
    >> backbone are region-to-region. This would imply that the backbone
    >> router would make a layer 3 forwarding decision for all transit packets.

There are really several different possible solutions to doing "large-scale"
label-based packet switching, none of them perfect. Which, of course, means
that in real life we'll probably use all of them, depending on which fits
the exact circumstances best.

The first is the one you mention - terminate the LSP (so that no LSP's cross
a "label-switching-region" boundary), and re-examine the internetwork level
address. This way, there's something of a match between the graph of the
routing table entries to X (which form a tree), and the LSP entries; because
as you get close to X, routes to X disappear anyway, to be replaced by routes
to X.1, X.2, etc - at that kind of boundary you *have* to terminate an LSP
for X.

The second is to maintain N^2 LSP's, including in the core - not a pretty
sight.

The third is to do what has been under discussing for a long time, which is
flow aggregation. In other words, local site L1 does have a flow F1 set up to
local site L2, via large core ISP's I1 and I2, but going across the core,
that flow is aggregated into flow A1 (along with some other flows, F2...Fn,
which happen to be congruent with A1 inside I1 and I2), and the core only
knows about A1. As A1 exits the core, it is broken up into its consituent
flows, F1...Fn. The mechanism to do this is simple; when F1 gets to the
last-hop router before it enters A1, the A1 label is pushed onto the front of
the packet, hiding the F1 label; and when the packet exits A1, the A1 label
is popped, revealing F1, and the packet continues on.

This takes less state, but a little more bandwidth - a common tradeoff when
you start thinking about flow-based packet handling. This is probably an OK
tradeoff as long as you're not on some hyper-expensive link. There, one of
the other alternatives is probably the way to go.

This is a very brief gloss on a much more complex situation with aggregation,
for a number of reasons. For one, the chances of F1, where F1 is a flow from
one site to another, being congruent with other flows along its entire length
is not so great, so you wind up doing multiple levels of aggregation. This in
turn leads you into what I call assymetric aggregation [don't ask :-], which
you can fix with still more labels.

So it really comes down to whether you want to spend state (for N^2 flows),
or bandwidth (aggregation, with pushed labels), or processing power (to
re-parse the internetwork level address).

Depending on the particular technological base you're working with, one may
look better than others. E.g. in an optical substrate where you have lots of
bandwidth, and complex processing is hard, aggregation may look good. If
you're on a hyper-expensive undersea link, you may prefer to pay for the
processing power. Etc, etc...

	Noel

---

From: "J. Noel Chiappa" <[email-masked]>
Date: Sat, 4 Mar 2000 21:08:42 -0500

> From: Tony Li <[email-masked]>

    > The point of all of this is that the scalability of the signaling is a
    > concern ... I can't help wondering if there is a real need to provide
    > entry-to-exit LSPs and if the architecture could not be greatly
    > simplified by using concatenated LSPs instead.

Oh, duh, I forgot one option (which reading this reminded me of) - which is to
use concatenated LSP's - but not via having the router at the end of one LSP
make an internet-level forwarding decision. Rather, the packet can contain a
source route (in the form of a stack of labels), so that when it exits one
LSP, it is immediately guided down another LSP by the next label.

(Feel free to do a vector product of this with flow aggregation, but the
mental model gets pretty complicated, there's now ay I'm gogin to try and
write it down! :-)

Anyway, this kind of source routing, with a label stack, is just another
different kind of space/bandwidth/etc tradeoff - although in this case the
extra space in the header (when the packet starts off) is gone by the time the
packet gets to the destination.

	Noel

---

From: Jim Boyle <[email-masked]>
Date: Tue, 07 Mar 2000 00:51:57 -0700

I like Noel's analysis, which says there are several ways to approach this, 
which means of course that all will be used as appropriate.

Here is another approach which probably is not appropriate to networks like 
curtis describes which have congestion issues to manage within a "region".

If one wants to avoid 1M LSPs below, but retain end2end MPLS capability 
(for VPN, for instance), and can manage congestion in a "region" via other 
mechanisms.....

You can run LDP out into the region to give, in curtis's example, a-LSR1 a 
binding to use to get through the core to a-LSR2.  All backbone routers 
would run RSVP-TE to manage congestion/utilization in the high capacity 
WAN.  They also have to run directed LDP sessions so when the penultimate 
backbone router pops the RSVP-TE learned label, the ultimate b-LSR knows 
how to forward.

Want to add an a-LSR? no problem, no need to touch all the other 
a-LSRs.  It can also be argued that LDP is an easier barrier for 
regional/edge routers than RSVP, but to be sure that would be one one would 
have to argue.

The numbers this design should work for is on the order of 100 or so 
"b"/backbone routers w/ 10-20 "a"/ regional routers per region.

regards,

Jim

>Lets make the example more concrete and slightly smaller.  There are a
>doxen or so backbone routers covering "regions" and on the order of 50
>routers in a given region.  The backbone routers are all MPLS capable.
>They RSVP neighbor with their adjacent routers.  They have a full mesh
>of tunels among themselves.  They also have a tunnel to each MPLS
>capable access router within their region (assume a reasonable subset
>are).
>
>The access routers could set up tunnels that go end to end but to keep
>the number of tunnels down in the backbone, hierarchical tunnels are
>used.  The access router (same naming convention) creates an ERO with
>"hop hop .. hop b-SLR1 b-LSR2 a-LSR2".  b-LSR2 must send RSVP through
>the tunnel and therefore has a RSVP neighbor b-LSR2.  b-LSR2 has
>traffic engineered a path to a-lSR2 and therefore has a RSVP neighbor
>a-LSR2.  This would mean that each backbone router has 1) on the order
>of 4-5 RSVP neighbors that are directly connected on the backbone, 2)
>at least a few that are directly connected on the access side, 3)
>about a dozen RSVP neighbors that are the loopbacks (router-ID) of
>other backbone routers, 4) and 50-100 RSVP neighbors that are the MPLS
>capable routers in the same region.
>
>In this scenario there may be 500-1000 routers (maybe more).  If they
>are exhaustively connected that's up to 1,000,000 LSPs.  Multiple
>services and that's still more.  That would involve a lot of setup on
>a link failure.
>
>The network as a whole can still be quite scalable if hierarchical
>tunnels are used.  For example, the backbone LSR may have 1,000,000
>tunnels, but when a physical link is lost in the backbone, in this
>example less than a dozen need to rerouted (those in the backbone that
>were affected).  The "link" being used by the hierarchical tunnels is
>not considered down unless the underlying tunnel can't be rerouted (if
>its local-protected or has a precomputed alternative, this is
>particularly a non-issue).
>
>
> > I apologize that this is half-baked and I apologize further if I'm
> > accidentally reinventing parts of PNNI.  The important meta point here is
> > that there are reasonable alternatives.
> >
> > Tony
>
>
>It sounds like you agree that RSVP neighbors must be formed through
>the tunnels but don't see any need for a new subobject.
>
>The only question to you and other implementers is then what to put in
>the ERO.  I would imagine that if we are strictly using the existing
>ERO subobjects it would be two type 1 "IPv4 prefix" subobject, with a
>prefix length of 32 and the router-Id of the two ends of the tunnel.
>
>For example, with the tunnel setup going left to right where 10.1.1.1
>is an interface in a router N-1 and 10.1.2.2 is an interface in router
>N+2.
>
>              ------ router N ------        ---- router N+1 ----
>                   Id = 10.128.1.1          Id = 10.128.1.2
>   10.1.1.1 -- 10.1.1.2         ===== LSP =====       10.1.2.1 -- 10.1.2.2
>
>   the ERO would have:
>
>     ... 10.1.1.1 10.1.1.2 10.128.1.1 10.128.1.2 10.1.2.1 10.1.2.2 ...
>
>If this is how others interpret how to form the ERO for tunnels in
>tunnels then all is fine and I can be quiet an go back to coding.
>
>Best Regards,
>
>Curtis