Subject: [IPsec] leading versus trailing ICV

From: William Allen Simpson <[email-masked]>
Date: Thu, 30 Jul 2020 22:13:57 -0400

The comments thus far seem to be mixed.  This is a perennial topic.
We spent much time on it in PIPE/SIPP/IPv6.

We agreed on leading for AH and trailing for ESP.

When I wrote the KA9Q NOS code implementing Van Jacobson's packet
buffers that eventually was ported to Linux by Alan Cox, the code knew
it had an incoming Ethernet or PPP frame, and offset the head on a
16-bit or 32-bit boundary as needed with enough space at the tail for
all trailing bytes.  The IP header was always on a 64-bit boundary.
Hopefully, that code is still present.

In modern CPUs, there's always an issue with cache lines.  But for a
parallel implementation, it really isn't going to matter.  The CPU
that finishes last and needs to check the ICV isn't particularly
likely to be the CPU that processed the initial header anyway.

As a matter of historical record, this was also a long debate for TCP.
The default is leading, and there is a TCP option for trailing checksum.

Might it be a non-default option negotiated per SPI?

---

From: Steffen Klassert <[email-masked]>
Date: Fri, 31 Jul 2020 10:32:59 +0200

On Thu, Jul 30, 2020 at 10:13:57PM -0400, William Allen Simpson wrote:
> all trailing bytes.

On Linux, it dependes a bit on the NIC driver how that is handled.
The default headroom is max(32 bytes, L1_CACHE_BYTES), space for a
trailer is not reserved.

> The IP header was always on a 64-bit boundary.

The default alignment of the IP header is on a 16 byte boundary.
It looks like this:

NET_IP_ALIGN(2) + ethernet_header(14) + IP_header(20/40) + L4_header(8)

Architectures can change the 2 byte NET_IP_ALIGN if they prefer
DMA alignment over IP alignment.

> likely to be the CPU that processed the initial header anyway.

While that would be possible for some algorithms, I've never seen that
a single cipher request is handled by multiple CPUs. I guess that
would lead to cacheline bouncing, and for GCM an atomic synchronization
of the counter would be needed.

Usually parallelization is achieved by using AVX registers/instructions
where multiple cipher blocks can be handled simultaneously with a single
instruction.

So it might make sense to have the ICV at the end because it is
likely cache hot when needed.

---

From: Michael Rossberg <[email-masked]>
Date: Fri, 31 Jul 2020 10:44:48 +0200

> likely cache hot when needed.

This is not what we have seen in measurements. Caches are usually much
larger than a single packet. This is especially true for small packets, where
current software en/decrypters have performance issues with. Since the packet
is passed further through the stack the header needs to be "hot” anyways.

However, what we have seen: appending at front and end leads to situations
where a new cache line is required at both ends.

More cumbersome in my option is the whole zero-copy/dealing with segment
lists issue, when having to add things in front and at the end.