Subject: RE: [Isms] Wehn the network doesn't.

From: "Blumenthal, Uri" <[email-masked]>
Date: Thu, 12 May 2005 12:05:18 -0400

> 1) If the network is not working well, central services like DNS may
>   not be available to any party.

Which is why one can configure SNMP engine notification destinations
etc. as IP addresses rathr than host names. 

> 2) I constructed the example so that it was the command responder who
>   needed to contact both the AAA server and the OCSP server.  So it
>   is the command responder that needs all those facilities.

When the network is healthy and the device that hosts SNMP command
responder has enough memory and cycles to spare for these extras,
there's no problem in adding all those facilities. 

One of the postulates of the original SNMP design was "Device is small,
limited and dumb - in addition to struggling with doing its main job; it
has few resources to spare on 'parasitic' management overhead." I don't
know how much of it is still true. Intuitively though - even for the
biggest router: the more cycles it spends on SNMP, the less of CPU is
left for doing its main task (shoveling IP packets around). Why don't
those who make the darn things, speak up and tell how much overhead
they're willing to suffer for convenience such as deployable SNMPv3 that
can plug into their Kerberos (or RADIUS, or SSH) infrastructure.

_______________________________________________
Isms mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/isms

---

From: "David B Harrington" <[email-masked]>
Date: Thu, 12 May 2005 14:51:21 -0400

Hi Uri,

> Why don't

You're confusing two targets here - those who "make the darn things"
are not the same as those who deploy them and have Kerberos (or
RADIUS, or SSH) infrastructure.

>From the generic standpoint of those who make the darn things, we want
to use as little overhead at as little cost as possible without
jeopardizing our ability to make the "short list." That's why many MIB
modules never get implemented in products - the lack of the MIB module
will not prevent the sale of the product. The lack of SNMPv3 in a
product generally does not keep the product off the short list. The
lack of SNMPv3 in SNMP management applications definitely does not
force the application off the short list.

>From the generic standpoint of those who deploy networks, as far as I
can tell, they want the most important functionality for their
business requirements for the least amount of cost. If a vendor's
hardware is missing, say 10G Ethernet, they won't make the short list
even if they support SNMPv3. If a vendor's product does support 10G
Ethernet but doesn't support SNMPv3, the lack of SNMPv3 generally
won't keep the vendor off the short list. 

The economic realities lead many vendors to not bother implementing
SNMPv3 or a full complement of MIB modules.

David Harrington
[email-masked]

---

From: Kaushik Narayan <[email-masked]>
Date: Thu, 12 May 2005 13:16:01 -0700

Hi All,

I didn't want to say this because the consensus has chosen TLS,
but since we are discussing SNMP design principles I wanted to
bring an issue about typical SNMP usage patterns and TLS

SNMP is predominantly used for data collection, this includes
operational state that is monitored periodically at very short intervals,
bulk data such as performance data that is collected with far lesser
frequency but entails massive amount of data and inventory data that
is collected on change (periodic polling done to detect any missed
change). Each of these data types requires different types of protection.

With protection at the transport layer, SNMP engines, for e.g. will need to 
open
new connections even all they might be doing is collecting operStatus
every minute, the other option is to have long  lived connections and 
typically
with a ratio of 10,000:1 between managed elements and management systems
that would be a lot of open connections. Long lived connections also have the
side effect that TLS cipher suites negotiated during connection establishment
will apply for the length of the connection. This would result in same 
protection
for all SNMP data transported during the lifetime of a connection irrespective
of whether it is configuration, operational or performance data (isn't this 
why
there was securityLevel in every SNMPv3 message).

In the past when we tried to run SNMPv2c over IPSec and it failed miserably
since management systems typically hosted on desktops with 500MB of
RAM could not scale to even 10s of devices and in most cases operators
are managing tens of thousands.

regards,
   kaushik!

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 12 May 2005 23:09:58 +0200

On Thu, May 12, 2005 at 01:16:01PM -0700, Kaushik Narayan wrote:

> open new connections even all they might be doing is collecting operStatus
> typically with a ratio of 10,000:1 between managed elements and management 
> systems that would be a lot of open connections. Long lived connections 
> also have the side effect that TLS cipher suites negotiated during connection 
> establishment will apply for the length of the connection. This would 
> result in same protection for all SNMP data transported during the 
> lifetime of a connection irrespective of whether it is configuration, 
> operational or performance data (isn't this why there was securityLevel 
> in every SNMPv3 message).

Are your products really choosing a different securityLevel depending on
whether they operate on configuration, operational or performance data?
Is this the default setup or something someone could in principle 
configure?
 
> are managing tens of thousands.

Do you really poll 10,000 boxes from a single centralized manager? And
does your management software running on a 500MB desktop really scale 
to tens of thousands of boxes with SNMPv3 security enabled?

What I recall from other big companies selling management products is
that they usually ask customers for some quite powerful hardware even 
to manage modest networks and that these vendors are usually very 
proud of their distributed data collection engines...

But I do agree with you that scalability in the number of devices that 
can be polled is a critical point to have in mind when moving to a TLS 
transport. For situations where you poll a single variable from a very
large number of nodes, TLS will raise overhead. On the other hand, if 
the data you like to poll does not fit into a few tiny PDUs anymore
(and that is often already the case if you poll a few ifTable
columns from an IEEE 802 switch these days), then a TCP based transport
will at some point become more efficient due to larger message sizes 
and all the nice things that TCP does for you. (And I assume you are 
aware of the various proprietary ways to ship larger SNMP data sets 
via protocols such FTP to overcome SNMP's relatively poor performance 
for larger data sets.)

I do agree that scalability needs to be taken into account and that 
this is a real issue (while repairing broken networks with SNMP
really is a myths in my view a non-issue).

My personal take on this is that SNMPv3/USM is out there and going to 
stay so in cases where you need high end small-data lots-of-devices 
polling engines, you can still use SNMPv3/USM (and you have to pay
the price for the key management or you resort to noAuth/noPriv to 
avoid lengthy computations). For the more bulky-data medium-number-of-
devices situations, I do not think TLS will be unfeasible. Note 
that I am reading my email using imaps and I you can be sure that I 
am not the only imap reader on our IMAP server polling mail boxes, 
which BTW is just one of these many Intel PCs.

/js

P.S. If you have contacts to customers who are polling 10,000+ devices
     from a single centralized server, please ask them whether they 
     are willing to support research by capturing traces, running them
     through analyzers and and giving condensed information to the NMRG. 
     I like to see some more substance added to scalability discussions 
     such as this one by having some reliable data sources and traffic 
     models.

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

---

From: Sam Hartman <[email-masked]>
Date: Thu, 12 May 2005 17:30:58 -0400

>>>>> "Kaushik" == Kaushik Narayan <[email-masked]> writes:

    Kaushik> Hi All, I didn't want to say this because the consensus
    Kaushik> has chosen TLS, but since we are discussing SNMP design

Er, we've chosen encapsulation not TLS.

Your points might be very important when discussing what we want to
encapsulate.

---

From: Kaushik Narayan <[email-masked]>
Date: Thu, 12 May 2005 17:58:59 -0700

Hi Juergen,

Please find my reply inline.

>Is this the default setup or something someone could in principle

Our products in the field currently supporting only SNMPv3 authNoPriv since
DES is the only privacy option for most devices in the field, we will be 
shipping
agents with SNMPv3 AES support soon.

As we are trying to support SNMPv3 privacy with AES, we are trying to 
define default
out-of-the box rules for use of privacy and we did want to make sure that 
we don't use
privacy protection for our performance collection use cases. We are trying 
to selectively
use two securityLevels (authPriv and authNoPriv) in addition to 
noAuthNoPriv used by
SNMP engine discovery.

>to tens of thousands of boxes with SNMPv3 security enabled?

Most of customers use a single server (typical config is around 500MB to
1 Gig of RAM) to manage up to 5000 devices with a majority of them greater
than 1000 devices.

>via protocols such FTP to overcome SNMP's relatively poor performance

We do have mechanisms to ship performance data via bulk files, again
this is not an option available universally and a large set of our customers
are still using SNMP to periodically collect statistics using MIB walks.

>which BTW is just one of these many Intel PCs.

My concern is that essentially if a part of the usage pattern of SNMP
is not met by ISMS there is really too much overhead for doing both
USM and ISMS. Also, the bulky-data use cases require least
amount of protection.

The success of ISMS will be in introducing security in the least intrusive
fashion and we need to be careful of the overhead ISMS will add both in
terms of requiring additional credentials provisioned on SNMP engines
and in terms of performance implications for managers and agents. For
deployment of ISMS, operators will compare the overhead to SNMPv2c
(which is what is used universally) and not SNMPv3/USM.

regards,
  kaushik!

>      such as this one by having some reliable data sources and traffic

A lot of these customers are very sensitive about providing
traces for public consumption. Let me see what I can do.

---

From: Wes Hardaker <[email-masked]>
Date: Mon, 16 May 2005 15:14:32 -0700

>>>>> On Thu, 12 May 2005 23:09:58 +0200, Juergen Schoenwaelder <[email-masked]> said:

Juergen> I do agree that scalability needs to be taken into account and that 
Juergen> this is a real issue (while repairing broken networks with SNMP
Juergen> really is a myths in my view a non-issue).

I agree too.  I think it's important to leave to the operators, when
possible, about which solutions to pick from.  Documentation for a
product will hopeful discuss the pros and cons of certain deployment
scenarios.

One thing to consider when thinking about the problem of how many
connections a manager can handle will not just be the raw number, but
*where* the impact will be.  One of the reasons that TCP didn't do
really well back in the days of yore with small amounts of memory is
that the TCP stack is implemented in the kernel.  Thus, expansion to a
huge number of connections there suffers from the fact that generally
today's OSes can't swap out kernel memory.  Thus, a huge increase in
TCP connections meant DoSing the machine pretty easily.

Some of the connection types we are considering will have vastly
different impacts based on the underlying transport.  EG, a TLS
connection certainly will have ramifications that affect kernel tables
since it's based on TCP.  However, I think that the majority of the
data for the security half of TLS will be in application space not in
kernel space which is good.  Even better, in the case of DTLS where
the underlying protocol is UDP based the ratio of kernel to user space
requirements will be even lower I think.  Thus, picking the right
scalable solution will certainly require selecting the best transport
in an EK approach.  But don't blame the problem on EK itself.
Somewhere, somehow data must be stored for a secure connection and of
course there is no way around that.  Instead you have to compare the
size and where it is likely to be stored.  EG, if you compare DTLS and
SNMPv3/USM I think you'd find that the kernel requirements will be
identical.  The size of the application space might be larger for
DTLS, however, but I have zero data to back that up with and I think
the difference will likely be fairly small.

-- 
Wes Hardaker
Sparta, Inc.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 17 May 2005 08:43:27 +0200

Wes,

Which do you think is more likely to be optimized for performance? 
TLS/TCP or *anything*/UDP?  People spend huge amounts of money just on 
the former...

Eliot

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 17 May 2005 09:43:16 +0200

> TCP connections meant DoSing the machine pretty easily.

I think this problem has been recognized and solved about 10 years
ago when the Web took off and persistent HTTP connections did not
yet exist.

/js

---

From: "Tom Petch" <[email-masked]>
Date: Tue, 17 May 2005 11:40:06 +0200

There is an assumption buried in this thread that I would challenge, namely that
all SNMP sessions will be the same and will be equally secure.  From a security
standpoint alone, I think this wrong.  The more a security technology is used,
the less secure it is, so I believe the security should be used only when it is
needed.
The SNMP manager polling the interface table on n,000 agents every 30 second
does not need and should not be using security.  Rather, the secure sessions
should only be the ones that need it - eg configuration change, anything related
to
security and such like - which I would expect to be much fewer in number.

Tom Petch

 Original Message -----
From: "Wes Hardaker" <[email-masked]>
To: "Kaushik Narayan" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, May 17, 2005 12:14 AM
Subject: Re: [Isms] TLS / SSH scalability concerns

---

From: Eliot Lear <[email-masked]>
Date: Tue, 17 May 2005 13:20:57 +0200

> the less secure it is, so I believe the security should be used only when it is

Perhaps definition of terms is in order, but I will tend to trust 
mechanisms that are both widely deployed and frequently used more so 
than I would ones that are niether widely deployed nor frequently used, 
simply due to the eyeball factor.  Now that's not the only factor, but 
it's an important one.

> does not need and should not be using security.  

Let's just agree that the operators of a system may wish to decide the 
sensitivity of their information.  I think VACM designers had this in 
mind.  I, however, am concerned about an operator's ability to make this 
decision, especially since attacks can develop more rapidly than an 
operator's ability to respond to them.  Also, the level of device 
configuration required increases to cover each OID tree requiring 
coverage.  Finally, as you begin to nitpick along these lines, the 
ability of management applications to actually do their thing will 
diminish if we end up with a situation where the application is 
configured to retrieve information using "insecure" while all of a 
sudden the security administrator has made the information accessible 
via only "secure" means in response to the latest attack.

[Ok, I myself am tempted to end that paragraph with Imminent Death of 
the Net Predicted. ;-]

> Rather, the secure sessions

An ever-increasing swamp if ever there was.

Eliot

---

From: "David B Harrington" <[email-masked]>
Date: Tue, 17 May 2005 09:02:39 -0400

Hi,

I understand the concern about the overhead of applying security to
everything, and wanting to selectively apply security based on the
data involved. However, I think the amount of work to configure
different levels of security for different data requests from
different users will be significant, and might be a show-stopper. 

VACM provides a static configuration of data access policy, and I know
of nobody that is actively utilizing it in their *operations* (please
speak up if you know of environments where it is being actively used).
I think that VACM suffers a similar problem to USM - it needs to be
dynamically configurable from a central authorization server (or
something similar), and/or management applications need to make it
easy to configure across a heterogenous network.

Let me ask this: when operators connect to the CLI, do they open
multiple connections, SSH for "show" commands for sensitive
information and to change configurations, and Telnet when they are
just monitoring a device? I suspect they just use SSH for both
purposes, and eat the overhead. Now SNMP exaggerates the problem
compared to CLI, but I think the result will be similar - they'll just
have a single secure connection for their SNMP, if we design a
reasonably efficient secure connection.

Of course, one approach is to use SNMPv3 proxy at a distributed
polling engine (mid-level manager) - if a section of the network can
be secured via non-SNMP means adequately (mgmt VLANs, source address
filtering, etc.) then they might use AuthNoPriv to gather large tables
of not-necessarily-sensitive data, such as statistics from the
interface tables, then aggregate the tables, and send them over a
secure AuthPriv channel to a centralized manager of managers.

David Harrington
[email-masked]

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Tom Petch
> Sent: Tuesday, May 17, 2005 5:40 AM
> To: Wes Hardaker; Kaushik Narayan
> Cc: [email-masked]

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 17 May 2005 10:20:03 -0700

>>>>> On Tue, 17 May 2005 08:43:27 +0200, Eliot Lear <[email-masked]> said:

Eliot> Which do you think is more likely to be optimized for performance? 
Eliot> TLS/TCP or *anything*/UDP?  People spend huge amounts of money just on 
Eliot> the former...

I don't know.  That was my point.  I don't have data for this
particular case.

-- 
Wes Hardaker
Sparta, Inc.

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 17 May 2005 10:21:29 -0700

>>>>> On Tue, 17 May 2005 11:40:06 +0200, "Tom Petch" <[email-masked]> said:

Tom> The SNMP manager polling the interface table on n,000 agents
Tom> every 30 second does not need and should not be using security.

Err...  Riiiigght.

I'm glad those ever important SETs will be done securely at least.
Even though the decision about whether to send them might be based on
insecure data.

-- 
Wes Hardaker
Sparta, Inc.

---

From: Marcus Leech <[email-masked]>
Date: Tue, 17 May 2005 13:28:56 -0400

Tom Petch wrote:

>...  From a security
With the exception of certain badly-engineered cryptosystems, I can't 
support
  this statement at all.

I don't think any of us is proposing to deploy a system that 
deliberately ignores
  60 years of deep thought in security and cryptography.  But I could be 
wrong :-)

-- 
Marcus Leech                            Mail:   Dept 1A12, M/S: 04352P16
Security Standards Advisor        Phone: (ESN) 393-9145  +1 613 763 9145
Advanced Technology Research
Nortel Networks                          [email-masked]

---

From: Sam Hartman <[email-masked]>
Date: Thu, 26 May 2005 21:58:09 -0400

>>>>> "Tom" == Tom Petch <[email-masked]> writes:

    Tom> There is an assumption buried in this thread that I would
    Tom> challenge, namely that all SNMP sessions will be the same and
    Tom> will be equally secure.  From a security standpoint alone, I
    Tom> think this wrong.  The more a security technology is used,
    Tom> the less secure it is, 

Long term keys are sometimes valuable to protect.  For example I don't
use signed mail except when I need to because keeping my signing key
available in a way taht would be useful for sending all my mail would
increase the probability that the key would be compromised.

Similarly you might want to protect the key you use for sets to the
OrbitalAdjustment MIB on your satellite more than you protect the key
you use to read the IFTable.

However limiting the use of confidentiality and integrity keys in an
encapsulation security mechanism is unlikely to improve the security
of anything.  Remember that you generate a fresh key for each session.
Also, remember that we typically design AES-based symmetric security
systems to allow a key to be used for 2^64 blocks (2^64*16 bytes).
There are a few ways you could reasonably design things that would
only give you 2^32 blocks on each session key.

But the general thing to take away is that the traffic encryption for
encapsulation is likely to support as much use of security as you
need.  If you exceed some lifetime then generating a new key is
relatively easy.