Subject: [Hipsec] HIP usage in a large SmartGrid/SCADA environment

From: Robert Moskowitz <[email-masked]>
Date: Tue, 02 Feb 2010 10:14:21 -0500

I really don't know if this is the way such a scenario would be 
implemented, but talking to the Power Engineering people, they SEEM to 
think so...

Situation:

10 Million or so meters all controlled from a single server.  Now I 
don't believe it would be a single server, I suspect that the meters 
will be regionally grouped by location on the grid.  But still and all, 
lets work with the PES people on this one.  It still could be 100K 
meters per server....

Meters do a HIP BEX at setup, and may never do another, as the key 
entropy will well handle the low volume of data.  So the MK is stored in 
protected memory in the meter and a secure store on the server (that 
would survive server reboot, imagine forcing 10M meters to rekey becuase 
the server rebooted).

Can HIP, will actually ESP SAs, scale to this large in a reasonable 
implementation?  For now don't add in mobility (RVS server) and meter 
server redundancy (HIP/ESP SA fate sharing through some backend 
protocol?).  Hopefully the meter app will be implemented over UDP, so a 
single packet comes in on an ESP SPI and that has to be looked up, 
processed and tossed up to the app.  Then perhaps there will be a 
response back to the meter.  The round trip needs to be in 100ms.  How 
many SAs can we expect a single server to be able to support?

Then there is the downstream alerts.  Say a rate change notice taking 
affect in 5 min to 50K customers.  That data packet with the rate 
information will have to be sent to each meter (this customer's meter is 
currently at IP address (HIT) n) with ACKs coming back that the rate 
information was accepted.  Round trip for this is in seconds.

Anyone want to tackle a scaling modeling challenge?  Would HIP scale 
better than other security tools or about the same?

---

From: Oleg Ponomarev <[email-masked]>
Date: Tue, 2 Feb 2010 23:04:30 +0200 (EET)

Hi! On Tue, 2 Feb 2010, Robert Moskowitz wrote:

There are the following bottlenecks:

1) number of ESP SAs in RAM
2) number of packets per second
3) bandwidth

Let us take a look at them:

1) We have to store in RAM
2 * 128 bit - remote HIT, remote IP addr
2 * 128 bit + 2 * 32 bit - incoming enc and auth key, SPI, sec #
2 * 128 bit + 2 * 32 bit - outgoing -//-

total about 112 bytes per meter, if we round to it 128 bytes (sorted tree 
structure etc), we need 10*10^6*128/2^30 ~= 1.2 GB of RAM

It should fit multiple times to a modern server

2) one core of Intel Xeon E5440 CPU using OpenSSL-1.0.0-beta2 can perform
91,142 encryptions per second of 1024-byte blocks with 128-bit AES key
157,053 sha256 per second of 1024-byte blocks

Assuming decryption and verification takes about the same time, it should 
send (or receive) up to 57,672 1024-byte packets per second

So it should take 2*10*10^6/57672 ~= 347 seconds to "ping" every meter

3) This data exchange should generate about 57672*1500*8/2^20 =~ 660 Mbps 
in/out traffic and ~120 kPPS through the router.

So the task should be viable using modern server with multicore CPU(s) and 
good network connection. Of course the initial BEX'es with all meters 
would take 10*10^6/145/4/3600 ~= 5 hours using quad-core Xeon with 
RSA1024/DH1536 (10*10^6/639/4/3600 ~= 1 hour with ECDSA160/ECDH192).

This estimations are quite optimistic, the reality might be much worse. I 
guess, the scalability should be about the same with other solutions, but 
it may be more/less convenient to setup such meters from various vendors 
using other tools.

> 10 Million or so meters all controlled from a single server.  Now I don't 
> believe it would be a single server, I suspect that the meters will be 
> regionally grouped by location on the grid.  But still and all, lets work 
> with the PES people on this one.  It still could be 100K meters per 
> server....
>
> Meters do a HIP BEX at setup, and may never do another, as the key entropy 
> will well handle the low volume of data.  So the MK is stored in protected 
> memory in the meter and a secure store on the server (that would survive 
> server reboot, imagine forcing 10M meters to rekey becuase the server 
> rebooted).
>
> implementation?  For now don't add in mobility (RVS server) and meter server 
> redundancy (HIP/ESP SA fate sharing through some backend protocol?). 
> Hopefully the meter app will be implemented over UDP, so a single packet 
> comes in on an ESP SPI and that has to be looked up, processed and tossed up 
> to the app.  Then perhaps there will be a response back to the meter.  The 
> round trip needs to be in 100ms.  How many SAs can we expect a single server 
> to be able to support?
>
> Then there is the downstream alerts.  Say a rate change notice taking affect 
> in 5 min to 50K customers.  That data packet with the rate information will 
> have to be sent to each meter (this customer's meter is currently at IP 
> address (HIT) n) with ACKs coming back that the rate information was 
> accepted.  Round trip for this is in seconds.
>
> Anyone want to tackle a scaling modeling challenge?  Would HIP scale better 
> than other security tools or about the same?

-- 
Regards, Oleg.