Subject: [IPsec] Ikev2 HA message Id Issue

From: "Kalyani Garigipati (kagarigi)" <[email-masked]>
Date: Thu, 3 Sep 2009 18:36:58 +0530

Hi ,

In Ikev2 HA, there is an issue with the message Id and window size.

Standby device-----------------------active
device----------------------------------Peer device

The active device participating in the exchange with the peer will
update its message id counters as per the exchanges done.

This info cannot be synced to the stand-by device for every exchange
done since that would take up all the bandwidth and is not an efficient
way.

The stand-by device when it becomes active will start with the message
Id as 1 and this will not be accepted by the peer, since its message Id
counters are different.

Hence a solution is required to sync the message Id counters to the
standby device.

1. A solution for this is to get the required info from the peer device
since it maintains all these counters.

The abstract details of how this can be done are given in the attached
document.

2. An alternative solution for this could be to send a new notify called
(RESET_MESSAGE_ID) to the peer device as soon as the standby comes up.
But this may lead to 

Reuse of message Id's within the same SA which is not desirable.

I think solution 1 should be implemented with Ikev2. Please give your
comments

Regards,

Kalyani

---

From: Yoav Nir <[email-masked]>
Date: Thu, 3 Sep 2009 18:06:52 +0300

Hi  Kalyani

Of the two, I prefer the 2nd solution, as it is simpler. Reusing message IDs is not that bad, and you can decrease the change by including (in the RESET_MESSAGE_ID notification) a random number as the starting message ID.

What I'm not so sure, is that there is a real problem here that needs to be solved now.

The IKEv2 documents totally ignore both high-availability solutions and load-sharing solutions.  They are just out of scope.  So the documents don't specify what data needs to be synched, or how is failover detected and accomplished on the LAN or the WAN.

To get there, you'd need to address issues of routing, signaling (between the peers) and AAA for both IKE and IPsec traffic.  That's a tall order that you probably don't want to tackle just now (maybe the WG would want this as the next "big" document)

So to have a high-availability or load-sharing solution that participates in IKE/IPsec, an implementation needs to somehow pretend that both of these are actually one gateway.  This can be done with smart switches, multicast addresses and synchronization links, which are out of scope for the WG (for now)

I can think of three levels of synchronizing IKE data between the peers.
 1. Synchronize just the IKE SA at creating and deletion
 2. Synchronize the IKE SA whenever the counters are updated
 3. Synchronize both IKE and Child SAs whenever a packet is sent.

#3 is obviously impractical.  #1 doesn't work, because after fail-over, the redundant gateway cannot take over the IKE SA, because it doesn't know the message ID counters. #2 can be made to work, although you need to either immediately rekey all the child SAs, or else skip ahead in the IPsec retrans counters.  This solution is practical enough, because most gateways have very few Child SAs per IKE SA. With IKEv2 you can have complex traffic selectors that allow one SA to cover all the domain protected by a gateway. So you might have a few SAs because of different QoS classes, and maybe a few if your implementation prefers to deal with whole ranges or subnets, but really, a single IKE SA with thousands of concurrent Child SAs is more of a lab thing than a practical thing.

Both your solutions ask peer gateways to assist the HA pair with their fail-over process. To the other gateway it seems as if the (single) peer is requesting information about current message ID numbers (and windows) or else for a reset. It seems strange that the first thing we would do for HA support is to help a private extension to the architecture work better, when that private extension is not really documented.

What do you plan to do in your cluster, if the peer does not support this extension?

You might also want to ask Paul and Yaron to present this on the Interim meeting on 22-Sep.

Yoav

On Sep 3, 2009, at 4:06 PM, Kalyani Garigipati (kagarigi) wrote:

Hi ,

Standby device-----------------------active device----------------------------------Peer device

The active device participating in the exchange with the peer will update its message id counters as per the exchanges done.
This info cannot be synced to the stand-by device for every exchange done since that would take up all the bandwidth and is not an efficient way.

The stand-by device when it becomes active will start with the message Id as 1 and this will not be accepted by the peer, since its message Id counters are different.
Hence a solution is required to sync the message Id counters to the standby device.

1. A solution for this is to get the required info from the peer device since it maintains all these counters.
The abstract details of how this can be done are given in the attached document.

2. An alternative solution for this could be to send a new notify called (RESET_MESSAGE_ID) to the peer device as soon as the standby comes up. But this may lead to
Reuse of message Id’s within the same SA which is not desirable.

I think solution 1 should be implemented with Ikev2. Please give your comments

Regards,
Kalyani

---

From: <[email-masked]>
Date: Thu, 3 Sep 2009 18:28:02 +0200

One obvious approach would be not to sync after every exchange (that could be a lot of messages), but sync, say, every N seconds (say, N=5)  in one big batch (for all IKE_SAs that changed in the last N seconds). Most of the time, almost all IKE_SAs are just sitting there idle (so IKEv2 message ID counters don't change). In case of failure, the stand-by device would have out-of-date information for some small percentage of IKE_SAs (those whose counters changed since last sync) , but that's always going to be the case (for exchanges where something more happened just before/during the failure).

I haven't done the math, though, so I don't know what value of N would result in both acceptable bandwidth and acceptable failure rate for the stand-by (depends on how many messages your typical IKE_SAs have per hour on average)...

Best regards,
Pasi

From: [email-masked] [mailto:[email-masked]] On Behalf Of ext Kalyani Garigipati (kagarigi)
Sent: 03 September, 2009 16:07
To: [email-masked]
Subject: [IPsec] Ikev2 HA message Id Issue

Hi ,

Regards,
Kalyani

---

From: "Kalyani Garigipati (kagarigi)" <[email-masked]>
Date: Thu, 3 Sep 2009 22:51:27 +0530

Hi Pasi,

Please find replies inline.

Regards,

Kalyani

________________________________

From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, September 03, 2009 9:58 PM
To: Kalyani Garigipati (kagarigi); [email-masked]
Subject: RE: Ikev2 HA message Id Issue

One obvious approach would be not to sync after every exchange (that
could be a lot of messages), but sync, say, every N seconds (say, N=5)
in one big batch (for all IKE_SAs that changed in the last N seconds). 

<Kalyani> If  sync is done in batches and if active device crashes
between the interval sync of the batches, then we again see the same
message Id issue.       

If dpd is enabled then ikev2 counters keep updated frequently. Hence we
cannot rule out the possibility of out of sync between stand by and
active device with the above approach.

Most of the time, almost all IKE_SAs are just sitting there idle (so
IKEv2 message ID counters don't change). In case of failure, the
stand-by device would have out-of-date information for some small
percentage of IKE_SAs (those whose counters changed since last sync) ,
but that's always going to be the case (for exchanges where something
more happened just before/during the failure).

<Kalyani> With HA,  we want to ensure the maximum avoidance of out of
sync. In any case of out of sync , the retransmission of messages should
take care of the exchanges. 

In the worst case the SA will have to deleted (which is the case
Currently now for IKEV2 when windowing is used and some requests are
lost )

I haven't done the math, though, so I don't know what value of N would
result in both acceptable bandwidth and acceptable failure rate for the
stand-by (depends on how many messages your typical IKE_SAs have per
hour on average)...

<Kalyani > we might like the solution to work in all cases of exchange
frequency, hence I think we cannot fix N.

Best regards,

Pasi

From: [email-masked] [mailto:[email-masked]] On Behalf
Of ext Kalyani Garigipati (kagarigi)
Sent: 03 September, 2009 16:07
To: [email-masked]
Subject: [IPsec] Ikev2 HA message Id Issue

Hi ,

Regards,

Kalyani

---

From: "Kalyani Garigipati (kagarigi)" <[email-masked]>
Date: Thu, 3 Sep 2009 22:51:32 +0530

Hi Yoav,

I agree that 2nd solution is easier to implement, but the peer will have
to delete the impending requests it might retransmit to the standby
device. 

 This will also not catch the message replay attacks which is the major
advantage of using message Id's

 (once the window is reset, earlier windows messages might be replayed
which will fall in this window at some point of time). 

Using a random number as starting message Id after reset, may also cause
the same issue

with 1st solution we can start from the requests where the active device
has stopped . This would ensure a smooth continuation of message Id's .

If the peer participating in the HA does not support this extension,
then the ordinary method of updating the counters for every exchange has
to be followed even though its inefficient.

I don't see any other solution with the current ikev2 protocol for this
message Id problem. Hence I feel that for efficient updation of
counters, this extension should be supported .

In case of site to site there will be some cases of single IKE SA and
multiple IPSEC SA's but we cannot rule out the case of multiple IKE SA's
depending on the deployment.

In addition , in case of remote access there might be many . In such
cases if dpd is also enabled then solution #2 would create lots of
message updations from 

Active to standbydevice. The concern is not only abt the bandwidth
between stand by and active , but the active device would be involved in
only doing these updations to the stand-by device.

Regards,

kalyani

From: Yoav Nir [mailto:[email-masked]] 
Sent: Thursday, September 03, 2009 8:37 PM
To: Kalyani Garigipati (kagarigi)
Cc: [email-masked]
Subject: Re: [IPsec] Ikev2 HA message Id Issue

Hi  Kalyani

Of the two, I prefer the 2nd solution, as it is simpler. Reusing message
IDs is not that bad, and you can decrease the change by including (in
the RESET_MESSAGE_ID notification) a random number as the starting
message ID.

What I'm not so sure, is that there is a real problem here that needs to
be solved now.

The IKEv2 documents totally ignore both high-availability solutions and
load-sharing solutions.  They are just out of scope.  So the documents
don't specify what data needs to be synched, or how is failover detected
and accomplished on the LAN or the WAN.

To get there, you'd need to address issues of routing, signaling
(between the peers) and AAA for both IKE and IPsec traffic.  That's a
tall order that you probably don't want to tackle just now (maybe the WG
would want this as the next "big" document)

So to have a high-availability or load-sharing solution that
participates in IKE/IPsec, an implementation needs to somehow pretend
that both of these are actually one gateway.  This can be done with
smart switches, multicast addresses and synchronization links, which are
out of scope for the WG (for now)

#3 is obviously impractical.  #1 doesn't work, because after fail-over,
the redundant gateway cannot take over the IKE SA, because it doesn't
know the message ID counters. #2 can be made to work, although you need
to either immediately rekey all the child SAs, or else skip ahead in the
IPsec retrans counters.  This solution is practical enough, because most
gateways have very few Child SAs per IKE SA. With IKEv2 you can have
complex traffic selectors that allow one SA to cover all the domain
protected by a gateway. So you might have a few SAs because of different
QoS classes, and maybe a few if your implementation prefers to deal with
whole ranges or subnets, but really, a single IKE SA with thousands of
concurrent Child SAs is more of a lab thing than a practical thing.

Both your solutions ask peer gateways to assist the HA pair with their
fail-over process. To the other gateway it seems as if the (single) peer
is requesting information about current message ID numbers (and windows)
or else for a reset. It seems strange that the first thing we would do
for HA support is to help a private extension to the architecture work
better, when that private extension is not really documented.

What do you plan to do in your cluster, if the peer does not support
this extension?

You might also want to ask Paul and Yaron to present this on the Interim
meeting on 22-Sep.

Yoav

Hi ,

Regards,

Kalyani

---

From: <[email-masked]>
Date: Thu, 3 Sep 2009 20:12:32 +0200

Kalyani Garigipati wrote:

>> One obvious approach would be not to sync after every exchange
>> (that could be a lot of messages), but sync, say, every N seconds
>> (say, N=5)  in one big batch (for all IKE_SAs that changed in the
>> last N seconds).
>

Yes; so N can't be very large.

> If dpd is enabled then ikev2 counters keep updated frequently. 

This depends on how often you do DPD... Obviously, you want dead
IKE_SAs to go away eventually, but e.g. 30 minute DPD interval would
be sufficient for that. If your DPD interval was close to the value 
of N, that would not work well... but on the other hand, if you have 
lot of traffic going back and forth, IKEv2 DPD won't get triggered..

> Hence we cannot rule out the possibility of out of sync between
> stand by and active device with the above approach.

We cannot rule out this possibility with any approach, I think.

>> Most of the time, almost all IKE_SAs are just sitting there idle
>> (so IKEv2 message ID counters don't change). In case of failure,
>> the stand-by device would have out-of-date information for some
>> small percentage of IKE_SAs (those whose counters changed since
>> last sync) , but that's always going to be the case (for exchanges
>> where something more happened just before/during the failure).
>
> <Kalyani> With HA,  we want to ensure the maximum avoidance of out
> of sync. In any case of out of sync, the retransmission of messages
> should take care of the exchanges.  In the worst case the SA will
> have to deleted (which is the case Currently now for IKEV2 when
> windowing is used and some requests are lost )

Yes. But this worst case cannot be avoided (since the worst case
can occur due to other reasons than message IDs) -- it can be 
just made less likely to happen.
 
>> I haven't done the math, though, so I don't know what value of N
>> would result in both acceptable bandwidth and acceptable failure
>> rate for the stand-by (depends on how many messages your typical
>> IKE_SAs have per hour on average)...
>
> <Kalyani > we might like the solution to work in all cases of
> exchange frequency, hence I think we cannot fix N.

I agree; clearly N would be a configurable parameter, not something
fixed in some spec.

Best regards,
Pasi

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 4 Sep 2009 12:40:10 +0300

> lot of traffic going back and forth, IKEv2 DPD won't get triggered..

You should not really have fixed timer for DPD. You should base your
DPD interval depending on the other things, i.e. if there is ESP
traffic coming from the other end to your site, there is no point of
doing DPD based on timer unless something else says otherwise.

If you start suspecting there might be something wrong with IKEv2 SA
(i.e. you receive ICMP or network goes down and comes up again etc),
then you might trigger DPD once to see if the other end is still
there.

If you only trigger timer based DPD when there is no ESP traffic at
all (i.e. the both IKEv2 SA and IPsec SA are completely idle) then
there is no point of trying to use too short DPD timers as the SA is
idle anyways, and in such cases you do not need very fast recovery
from other ends crashes...

Only case where you might need more frequent timer based DPD is when
your traffic is unidirectional, i.e. you are sending ESP traffic to
other end but other end is not sending anything back. As this is not a
common case in normal operation, that is good indication there might
be something wrong and in such cases you should trigger DPD to verify
it the other end is up.

In general I consider syncing HA boxes after each IKEv2 Message (or
once per second etc) not too big problem. HA boxes are usually
directly connected with fast network cable (usually at least as fast
as their traffic in), and every single IKEv2 message requires some
cryptographic operations anyways, and is bigger than what it would be
to send short cleartext message to other HA telling "I finished
processing my request message id XXX at IKE SA YYY" or "I finished
processing my reply to message id XXX at IKE SA YYY and packet sent
was ZZZ" (you need to sync the reply packet data you sent to other end
just in case the packet was lost and other end didn't get it, so you
can retransmit it from HA pair). 

In any case you will loose all IKE SA which are in the middle of
exchanges when one of the devices goes down, as syncing intermediate
state from one device to other would be way too complex.
-- 
[email-masked]