Subject: [RAM] Number of DFZ routers

From: Robin Whittle <[email-masked]>
Date: Mon, 28 May 2007 08:35:29 +1000

Can anyone provide an estimate for the number of routers in
the DFZ?

  Transit routers  (DFZ)

  Multihomed border routers  (DFZ)

  Singlehomed border routers

Can anyone suggest the range of number of interfaces these
routers have?  For instance, what is the largest number of
interfaces on an operational router?

What would be a typical range of interface numbers for
transit routers?

In addition to direct forwarding of Internet traffic, and
handling the communications paths between the routers
themselves, do transit routers often perform other functions,
such as handling MPLS?

I would also be curious about typical transit router models and
interface speeds.

Please respond off-list if this is too off-topic.

  - Robin

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Ted Seely <[email-masked]>
Date: Mon, 28 May 2007 12:10:12 -0400 (EDT)

> Can anyone provide an estimate for the number of routers in

That will be a different number for every provider.  Are you proposing
canvassing all the Tier 1s for this number?  Some (most) may not be
comnfortable providing that number, but with a little hardwork and
homework you could discern this.

>   Multihomed border routers  (DFZ)

Again, SP or customer based?  Most all SPs access or gateway routers (pick
your semantics of choice, basically where customers terminate) are all
dual homed upstream to the backbone routers, some may even be in a
different ASes.  As for true border routers as in Tier 1 SP A talking to
Tier 1 SP B, probably safe to assume all of them.

>   Singlehomed border routers

Not in 1239s network.

> interfaces on an operational router?

Backbone or customer facing?  If customer, can approach 1000's, are they
all BGP, no.

> What would be a typical range of interface numbers for

Again, P or PE?

> themselves, do transit routers often perform other functions,

Not for 1239, but probably safe to assume so for other providers.  Comes
down to architecure koolaid flavor of choice.

> I would also be curious about typical transit router models and

Model as in 12416, CRS-1, M160, T640etc...?

Curious as to where you may be taking this?

-ted

> Please respond off-list if this is too off-topic.

Ted Seely
Principal Network Design Engineer
Internet Engineering - SprintLink
(W) 703.689.6425
(M) 703.967.3289
AIM - wanpro00
Yahoo IM - tseely01

"can be maintained, upgraded, enhanced, and scaled without requiring
service interruption"

---

From: "Ricardo V. Oliveira" <[email-masked]>
Date: Mon, 28 May 2007 10:29:30 -0700

>   Singlehomed border routers
These numbers are very difficult (if not impossible) to guess at this  
stage. If you want to decrease the granularity to autonomous systems  
instead of routers, you can find some answers.

--Ricardo

---

From: Tony Li <[email-masked]>
Date: Mon, 28 May 2007 15:32:14 -0700

> Curious as to where you may be taking this?

Working backwards, these are exactly the questions that you have to  
ask if you're trying to determine how large a lookup result needs to  
be.  See Robin's previous work on SRAM lookups.

> interfaces on an operational router?

Chris and others have pointed out a limitation in IOS.  Note that  
this is not shared by XR and presumably JunOS.  In particular, from a  
router architecture viewpoint, what's even more relevant is the  
potential number of interfaces that COULD be required from a system.

Now, the currently largest system that I know of that's being  
publicly discussed is the CRS-1.  That's 97Tbps, although other,  
larger systems have been discussed and will almost certainly be  
necessary in the distant future.  Not everyone agrees with this.

> What would be a typical range of interface numbers for

Today, I'd say that interfaces in the DFZ range from DS-3 to OC-768.   
For the architectural bound, we can look at the CRS with T-3's,  
giving us around 97Tbps/45Mbps ~= 2e6 interfaces.

It should be noted that some interface types (Ethernet) are multi- 
access and require a router to store not only an interface, but next  
hop information.  A reasonable guess right now is to provision for  
about 35e3 next hops per Ethernet interface.  This allows the same  
hardware to be sold into most enterprise accounts.

Net that leaves you at around 7e10 next hops.  Based on these types  
of numbers, many folks have pointed out that distributed systems may  
be more scalable if they perform multiple lookup stages.  Not  
everyone agrees with this.  The cost of going to multiple lookups is  
that it requires additional high-speed lookup hardware, right at the  
point of the highest bandwidth in the architecture, the interface  
between the internal fabric and a chassis.

> themselves, do transit routers often perform other functions,

As noted, yes, this is required.  Also, a slew of other features that  
have forwarding time impact are also practical requirements in this  
market space.  For example, ACLs, policy based forwarding,  
accounting, and QoS all have non-trivial impacts on the forwarding  
plane and may directly impact your lookup.  There are a number of  
considerations that must be taken into account.

Tony

---

From: Robin Whittle <[email-masked]>
Date: Tue, 29 May 2007 13:41:15 +1000

Thanks Ted and others for your responses.  I recognise there
is no precise way of estimating these things.  Here are some
reasons I asked these questions.

Apart from the shortage of IPv4 addresses, I think all the
routing and addressing problems we are trying to solve are
due to the demands placed on DFZ routers.

Even the IPv4 shortage would not exist, or would be much
less of a problem, if these routers could happily handle
flat routing to single IP addresses.  Then there would be
no need for route aggregation and IPv4 address space could
be assigned in much smaller pieces, as needed, so it could
be used much more efficiently than at present.

That will never be practical, but I think it is practical to
make the FIBs of the next generation of routers do "flat
routing" to /24 granularity with less expense and trouble
than any current proposal such as LISP.  This only makes
sense is if BGP can cope with ten or so times the number of
advertised prefixes as today.  Also, I think LISP would have
many uses, including for TE and for multihoming to finer
granularity than /24.

So if BGP could be radically improved, RAM-based FIBs would
make the global routing system support finer-grained address
assignment with greater efficiency of use, while supporting
existing BGP multihoming practices to a granularity of 256
IP addresses.  This would reduce the size and scope of the
problems which remain to be solved by LISP etc.

Unfortunately, no BGP experts think that such improvements
are feasible.  Maybe some unconventional ideas from
non-experts might be helpful.

Here are my rough guesses about some aspects of the total
cost burden of the Internet's DFZ routers.

Firstly, there is a current number of routers in the DFZ.
With a proposal such as mine, this would continue to grow
for a long time, as more and more SPs and AS end-user
organisations connect to the Net - and as they use more
routers per AS, for instance at more and more sites.

Secondly, the FIB cost scales with the number of interfaces
and/or total bandwidth of each router, since there is
usually one FIB per interface, per four interfaces, or per
X Gigabit/sec bandwidth.  On a border router, not all of
these will be BGP interfaces, but on a multihomed border
router, I guess the FIB of each interface needs to handle
the routes for the BGP system.

Thirdly, each router needs RAM and CPU power to cope with
all the messages received from and sent to its BGP peers.
This scales directly with the number of interfaces using
BGP, and not with the speed of those interfaces.

Finally, once the router has booted and received and
processed all the routes from its BGP peers, including
transmitting its best routes to those peers, I think the
burden on each router also scales with:

1 - The rate of updates received from peers.  Average and
    peak would be very different.  All peak load messages
    need to be handled, but delays in handling them will
    slow local and global convergence times, perhaps
    making some kinds of inherent BGP instability patterns
    harder to manage.

2 - The rate of these updates which actually change the best
    route for a prefix for this router.  This scales roughly
    with (1), but for a router with dozens of BGP peers, I
    guess most of the updates wouldn't change the current
    best route, whereas for a router with only 3 peers, a
    larger proportion of updates would cause a change to the
    best route.

3 - The number of peers to send out updates to.

4 - The CPU time spent deciding whether the update results
    in a new best route.

5 - CPU time and especially any downtime for the FIBs as
    they are updated for the new best route.

Factor (1) is broadly proportional to the to total number of
advertised prefixes, but at present, a large proportion of
updates are probably the result of bad management, rather
than things the BGP network really should handle.  See
the middle of:

  http://www1.ietf.org/mail-archive/web/ram/current/msg01117.html

I wanted to get some guesstimates of total router numbers and
some actual case reports of transit routers and the number
of interfaces they have, the speeds of those interfaces etc.

As long as there are no solid proposals to improve the
capacity of BGP in general, or routers, in the future, I
think that the solutions we are considering are intended to
either reduce the demand for more DFZ routers, more
advertised prefixes etc. and/or are intended to work with
some administrative pressure to reduce these two figures.

I think any such pressure or limits will lead to long-term
problems with competition, since the limited number of
ISPs and AS end-users who connect to the DFZ will be like
a "top tier", with many others excluded.  (Unless perhaps
the ability to have a DFZ router and/or to advertise a
prefix is handled by a market-based system with tradeable
licenses to do so.  Then anyone can do it if they pay a
price which makes others happier to achieve their goals
some other way.)

All this is a lot of trouble to go to - all because we
can't figure out a way of improving routers' capacity to
handle BGP and/or how to resolve problems with poor
convergence in the BGP system.

One line of argument might be that the costs of these
proposed architectural changes is so high that we should
consider not making them, or at least postponing them or
reducing their scope, by the simple expedient of throwing
more money at every DFZ router.  (Maybe this is a good
time to buy the stock of companies which produce TCAM and
QDR-II SRAM.)

That is one reason I asked these questions - to get some
kind of idea on how many billions of dollars the current
DFZ router system costs.

It would be good to have a similar estimate of the number
of routers which would need to be upgraded for LISP,
depending on what volume of traffic was to be handled
by LISP in the future.  I think the ITR role is quite
demanding, especially in the FIB or whatever is used
to figure out if an incoming packet is handled by
one of a probably very large number of LISP prefixes.

 - Robin

---

From: "Chris L. Morrow" <[email-masked]>
Date: Tue, 29 May 2007 05:05:03 +0000 (GMT)

> many uses, including for TE and for multihoming to finer

keep in mind that /24 is 'globally visible' but there are cases inside an
ISP where /32's may need to be routed in the same way as /16's (same speed
atleast). So, if you mean by /24 "16 million routes in teh FIB flat
routed" then 'ok'... if you mean 'flat routing stops at /24 anything more
specific needs another stage of lookups' then, probably that needs to be
re-thought :(

---

From: Robin Whittle <[email-masked]>
Date: Tue, 29 May 2007 22:37:31 +1000

> specific needs another stage of lookups' then, probably that needs to be

Following Jari's guidance, I have responded to this on the Routing
Research Group list:

  http://www1.ietf.org/mailman/listinfo/ram
  http://www1.ietf.org/mail-archive/web/ram/current/

with the subject: "RAM lookup FIB & prefixes > /24"

  - Robin

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 29 May 2007 14:48:25 +0200

On Tue, May 29, 2007 at 10:37:31PM +1000,
 Robin Whittle <[email-masked]> wrote 

>   http://www1.ietf.org/mail-archive/web/ram/current/

Isn't it http://psg.com/lists/rrg/ instead? Your post is
http://psg.com/lists/rrg/2007/msg00126.html

---

From: Robin Whittle <[email-masked]>
Date: Tue, 29 May 2007 23:07:16 +1000

Thanks Stephane, I posted the wrong URLs.  The Routing Research Group's
homepage and archives are:

  http://www.irtf.org/charter?gtype=rg&group=rrg
  http://psg.com/lists/rrg/2007/maillist.html

   - Robin