Subject: [IPsec] AH ICV calculation

From: Sean Lawless <[email-masked]>
Date: Wed, 02 Apr 2008 12:08:33 -0700

Hi all,

We are in the process of implementing static keyed IPSec for the IPv6 
portion of a commercial TCP/IP stack and are having problems computing 
the AH ICV correctly.  Using NULL as the authentication type there are 
no problems communicating with other hosts (Windows XP) so I have ruled 
out problems with packet decoding, SPI assignment, etc.  However, using 
any HMAC algorithm (MD5-96 or SHA1-96) the calculated ICV never 
matches.  I should also note that the HMAC algorithm we are using works 
correctly with our TLS 1.0 implementation and so I feel we can rule out 
HMAC and MD5/SHA1 algorithms as the problem.

The ICV calculation matches if I test one Windows XP machine against 
another or one of our development boards running our stack against 
another.  This means our ICV calculation is consistent (but incorrect) 
upon inbound and outbound processing.  I have read through the NetBSD 
code (ah_core.c) and cannot find any difference in how they are muting 
the IPv6 header fields vs. how we are...

I have walked us through an example below that uses MD5 as the AH 
algorithm with our implementation receiving an ICMPv6 ping from a 
Windows XP host with outbound SPI 3001 (0x00000BB9), configured with 
HMAC-MD5-96 and key file "0123456789ABCDEF":

Complete ICMPv6 packet captured with Ethereal:

0000   00 00 00 54 85 00 00 1d 60 0d 97 3e 86 dd 60 00  ...T....`..>..`.
0010   00 00 00 40 33 80 fe 80 00 00 00 00 00 00 02 1d  ...@3...........
0020   60 ff fe 0d 97 3e fe 80 00 00 00 00 00 00 02 00  `....>..........
0030   00 ff fe 54 85 00 3a 04 00 00 00 00 0b b9 00 00  ...T..:.........
0040   00 07 31 02 0c 3f 1d 6d ee 58 9d f1 fb 1a 80 00  ..1..?.m.X......
0050   59 33 00 00 00 07 61 62 63 64 65 66 67 68 69 6a  Y3....abcdefghij
0060   6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 61 62 63  klmnopqrstuvwabc
0070   64 65 66 67 68 69                                defghi

Packet after mutable field hop limit and ICV are zero'd:

0000   00 00 00 54 85 00 00 1d 60 0d 97 3e 86 dd 60 00
0010   00 00 00 40 33 00 FE 80 00 00 00 00 00 00 02 1D
0020   60 FF FE 0D 97 3E FE 80 00 00 00 00 00 00 02 00
0030   00 FF FE 54 85 00 3A 04 00 00 00 00 0B B9 00 00
0040   00 07 00 00 00 00 00 00 00 00 00 00 00 00 80 00
0050   59 33 00 00 00 07 61 62 63 64 65 66 67 68 69 6A
0060   6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 61 62 63
0070   64 65 66 67 68 69

The manual key being used is "0123456789ABCDEF" which has a key length 
of 16.  After the HMAC init function is called with the key and key 
length, the packet is processed with HMAC-MD5.  There are 104 bytes 
processed from packet offset 000E to 0075:

60 00 00 00 00 40 33 00 FE 80 00 00 00 00 00 00
02 1D 60 FF FE 0D 97 3E FE 80 00 00 00 00 00 00
02 00 00 FF FE 54 85 00 3A 04 00 00 00 00 0B B9
00 00 00 07 00 00 00 00 00 00 00 00 00 00 00 00
80 00 59 33 00 00 00 07 61 62 63 64 65 66 67 68
69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 61
62 63 64 65 66 67 68 69

After the HMAC calculation is finished, the first 96 bits of the hash of 
the above data is:

70 14 6d d1 15 4c 85 2c 26 a7 31 68

Which does not match the ICV in the packet:

31 02 0c 3f 1d 6d ee 58 9d f1 fb 1a

I have double checked and the HMAC-MD5-96 hash we calculated is correct 
for the key and data being processed, therefore I can only assume that 
the data we are processing is incorrect (or the manual key requires 
preprocessing?).  I've reread the RFCs many times and am at a loss for 
what could be the problem.  Can anyone spot what I am doing wrong?  Also 
it would be very helpful to me if anyone could point to an example of AH 
processing on an actual packet.  Something similar to the steps outlined 
above?

Best Regards,

Sean Lawless
Blunk Microsystems LLC
_______________________________________________
IPsec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipsec

---

From: Scott C Moonen <[email-masked]>
Date: Thu, 3 Apr 2008 09:40:41 -0400

Hi Sean.  Please take the following with a grain of salt, because it is 
relayed secondhand, from a year or two ago, and the original tester is 
actually no longer with our company.  Our system test group was able to 
successfully implement dynamic SAs between our platform and Microsoft 
Windows.  One of our testers had some problems, however, configuring 
manual SAs between our platform and Windows.  We got the impression at the 
time that Windows doesn't directly use the key data that you enter in its 
configuration, but that it somehow hashes it to produce the actual key for 
the SA.  My recollection (could be wrong) is that he was able to enter key 
values longer than 16 bytes on Windows for various algorithms (both HMAC-x 
and encryption algorithms), and this may have been what led us to conclude 
that the key was derived from what he entered rather than being exactly 
what he entered.

We couldn't find any documentation on how the keys are derived, and I 
don't know whether he pursued an answer from Microsoft support.  I suspect 
that he never got manual keys working.  It wasn't a high priority test in 
his case since we were able to successfully interact with numerous other 
platforms using manual SAs.  At very least, we don't currently have any 
manual SAs in our testbed between our platform and Windows (though we do 
have dynamic SAs).

So you might consider trying your test with another platform than Windows; 
perhaps Linux or Cisco.  Alternatively, you might consider contacting 
Microsoft (or perhaps a Microsoft representative will comment in response 
to this) to understand how their implementation derives keys for manual 
SAs.  Lastly, you might consider dynamic SAs, though I realize that this 
may not be a trivial exercise. :)

For what it's worth, with a quick seat-of-the-pants calculation I did 
arrive at the same ICV value for your sample data that you obtained.  That 
doesn't mean that you and I aren't making the same oversight. :)

Scott Moonen ([email-masked])
z/OS Communications Server TCP/IP Development
http://scott.andstuff.org/
http://www.linkedin.com/in/smoonen

From:
Sean Lawless <[email-masked]>
To:
[email-masked]
Date:
04/02/2008 03:16 PM
Subject:
[IPsec] AH ICV calculation

Hi all,

Best Regards,

Sean Lawless
Blunk Microsystems LLC

---

From: Sean Lawless <[email-masked]>
Date: Thu, 03 Apr 2008 11:52:18 -0700

Thanks Scott, your response was helpful.  I previously read this thread 
on the web about Win2003 and Kame and then assumed that Windows was not 
doing anything special and would interoperate:

http://atm.tut.fi/list-archive/snap-users/msg03001.html

It sounds like this may not be a safe assumption but I sure wish someone 
at Microsoft would comment about this.  Has anyone successfully 
interop'ed static keyed Window IPv6 
(http://msdn2.microsoft.com/en-us/library/ms737602.aspx) with Kame or 
another IPSec implementation?  If not my next step must be to test with 
another implementation.

Thanks for the help,
Sean

Scott C Moonen wrote:
>
> Hi Sean.  Please take the following with a grain of salt, because it 
> is relayed secondhand, from a year or two ago, and the original tester 
> is actually no longer with our company.  Our system test group was 
> able to successfully implement dynamic SAs between our platform and 
> Microsoft Windows.  One of our testers had some problems, however, 
> configuring manual SAs between our platform and Windows.  We got the 
> impression at the time that Windows doesn't directly use the key data 
> that you enter in its configuration, but that it somehow hashes it to 
> produce the actual key for the SA.  My recollection (could be wrong) 
> is that he was able to enter key values longer than 16 bytes on 
> Windows for various algorithms (both HMAC-x and encryption 
> algorithms), and this may have been what led us to conclude that the 
> key was derived from what he entered rather than being exactly what he 
> entered.
>
> don't know whether he pursued an answer from Microsoft support.  I 
> suspect that he never got manual keys working.  It wasn't a high 
> priority test in his case since we were able to successfully interact 
> with numerous other platforms using manual SAs.  At very least, we 
> don't currently have any manual SAs in our testbed between our 
> platform and Windows (though we do have dynamic SAs).
>
> So you might consider trying your test with another platform than 
> Windows; perhaps Linux or Cisco.  Alternatively, you might consider 
> contacting Microsoft (or perhaps a Microsoft representative will 
> comment in response to this) to understand how their implementation 
> derives keys for manual SAs.  Lastly, you might consider dynamic SAs, 
> though I realize that this may not be a trivial exercise. :)
>
> arrive at the same ICV value for your sample data that you obtained. 
>  That doesn't mean that you and I aren't making the same oversight. :)
>
>
> From: 	Sean Lawless <[email-masked]>
> To: 	[email-masked]
> Date: 	04/02/2008 03:16 PM
> Subject: 	[IPsec] AH ICV calculation
>
>
> ------------------------------------------------------------------------
>
>
>
> Hi all,
>
> ------------------------------------------------------------------------
>

---

From: Sean Lawless <[email-masked]>
Date: Fri, 04 Apr 2008 13:54:12 -0700

Hi,

To conclude this thread, I tested our implementation with FreeBSD (Kame) 
today and found they interoperate just fine with static keys.  Thus the 
issue I encountered is Windows specific.  So if anyone else asks, the 
ipsec6.exe command on Windows cannot be used to configure static keys 
that inter operate with other IPSec implementations.  Maybe the next 
person who goes down this road will see this thread and avoid this.

Regards,
Sean