Subject: RE: Scalability of Netconf

From: "Sharon Chisholm" <[email-masked]>
Date: Mon, 4 Jul 2005 13:19:57 -0400

hi

These would be interesting numbers to get. I've heard related (but
opposite?) concerns around being able to get the most out of each
netconf connection to ensure that more complex deployments don't max out
on connections. This might eventually require us to continue the
channelization discussions which was mainly deferred, but I think that
can wait a bit. First step would be to determine if this is really an
issue in practice.

Sharon

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Juergen Schoenwaelder
Sent: Saturday, July 02, 2005 12:27 AM
To: Kenneth V. Crozier; [email-masked]
Subject: Re: Scalability of Netconf

On Sat, Jul 02, 2005 at 06:05:35AM +0200, Juergen Schoenwaelder wrote:
> On Fri, Jul 01, 2005 at 11:13:20AM -0700, Kenneth V. Crozier wrote:
> 
> > What i'm concerned with is how many connections a NM station would 
> > have
> > to maintain, and by this are we raising the bar for storage and 
> > processor requirements of the NM station.
> 
> If I buy a machine for say 1.000 Euros and put Linux or BSD on it. 
> What do you think how many TCP connections I can maintain with such a 
> box?

Actually, I should have said ssh connections. BTW, this is not a 
terrible difficult experiment to do so I might even do this excercise
next week if anyone really cares about numbers.

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen,
Germany

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 5 Jul 2005 16:03:08 +0200

On Tue, Jul 05, 2005 at 04:01:17PM +0200, Juergen Schoenwaelder wrote:

> P.S. Attached is the ugly quick and dirty script - I am sorry for
>      using Tcl and not Perl - I am sure Perl will make this faster. :-)

Sorry, as usual I forgot to attach...

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 5 Jul 2005 16:01:17 +0200

> can wait a bit. First step would be to determine if this is really an

I am not yet done with my experiments. So far, I have writen a script
which connects to ssh subsystems I called cat:

Subsystem       cat     /bin/cat

The script kicks off a standard ssh client to connect to a remote cat 
subsystem and then starts a loop where it periodically sends a message
and checks whether that is echoed back correctly. Nothing fancy. (Note
that this script is rather heavy weight since it requires a new ssh
process for each connection - a more serious implementation would
require some C code so that you can keep multiple connections in one
process.)

I am running this script on a 3GHz Pentium with 512 MB Ram (so really
nothing high end for a management system) running Linux 2.6.10. I 
keep 300 connections open and the funny thing is that this box does 
not at really get stressed by this. The load never went above 0.1% 
and was most of the time significantly below. The memory used excluding
buffers was ~235 MB.

Given that the script is really a crude resource hog, the box performed
much better than expected. The next step is probably to shoot for
1000 ssh connections with this crude script. I just have to setup
more cat subsystems to talk to...

/js

P.S. I admit that my Mac OS X notebook does not behave that friendly
     since I was simply running out of slots in the process table. ;-)

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 05 Jul 2005 09:04:58 -0700

>>>>> On Tue, 5 Jul 2005 16:01:17 +0200, Juergen Schoenwaelder <[email-masked]> said:

Juergen> I am running this script on a 3GHz Pentium with 512 MB Ram
Juergen> (so really nothing high end for a management system) running
Juergen> Linux 2.6.10. I keep 300 connections open and the funny thing
Juergen> is that this box does not at really get stressed by this. The
Juergen> load never went above 0.1% and was most of the time
Juergen> significantly below. The memory used excluding buffers was
Juergen> ~235 MB.

I'd expect the limiting fact in that to be the memory not the CPU
speed.  Machines of that caliber can handle huge numbers of web
connections without blinking much.  The number of TCP connections
won't be the limiting factor at all, I don't think.  The CPU speed
will certainly be affected by the continuous stream of cryptographic
connections, but unless you're doing something like using multiple
clients connecting at once you have no way to be sure that the server
is not the limiting factor there and not your client.

-- 
Wes Hardaker
Sparta, Inc.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 5 Jul 2005 19:00:32 +0200

> is not the limiting factor there and not your client.

I totally agree - but we continue to hear stories of boxes dying with
slightly more than a dozen IPsec associations. And I do know from
experience that some OSes are not as scalable as some other OSes. 
Some OSes are rather good these days in dynamically allocating TCP 
control blocks and process table entries when needed. In the not
so good old days, these things often had static limits you could
run into. But this is mostly history now.

I am actually not even sure memory is the issue - filling 512 MB (or
you can easily and cheaply get 1GB or more these days) with ssh state
data also requires rather poor implementations (like my stupid script 
for example).

/js

---

From: Phil Shafer <[email-masked]>
Date: Tue, 05 Jul 2005 15:20:00 -0400

Juergen Schoenwaelder writes:
>filling 512 MB (or

Or the use of java ;^)/2

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 6 Jul 2005 17:25:38 +0200

> and was most of the time significantly below. The memory used excluding

Today I was running 1200 ssh connections using the same basic script
on the same box. The load again never went above 0.1 while the memory 
usage went up to ~494 MB. Shall I try 10.000 tomorrow? With 2GB of
memory this should be possible. The two Linux XEON servers that play 
the remote end for 600 connections each do not seem to bother much 
about all this.

/js

---

From: Eliot Lear <[email-masked]>
Date: Wed, 06 Jul 2005 18:15:24 +0200

Yeah, I just had this funny thought of 1200 NMSes trying to configure a
router.  That's the sort of locking contention we certainly did NOT
optimize for!

Eliot

---

From: Wes Hardaker <[email-masked]>
Date: Wed, 06 Jul 2005 09:51:13 -0700

>>>>> On Tue, 5 Jul 2005 19:00:32 +0200, Juergen Schoenwaelder <[email-masked]> said:

Juergen> I totally agree - but we continue to hear stories of boxes
Juergen> dying with slightly more than a dozen IPsec associations.

FYI, these are figures showing the number of IPsec SAs that a linux
box handled a number of years ago (I forget the CPU speed of the
laptop, but it had about 512Mb of memory). The IPsec stack was NISTs
running on a 2.4 kernel.

http://sbsm.hardakers.net/sadb-insert.png
http://sbsm.hardakers.net/sadb-speed.png

In short, just the creation of SAs (and these were manually created,
not via IKE) didn't affect much till you got out into the >10000
range.

(sorry about the fuzzy pictures...  converted quickly from postscript)

-- 
Wes Hardaker
Sparta, Inc.