Subject: Re: [IPsec] leading versus trailing ICV

From: William Allen Simpson <[email-masked]>
Date: Sat, 1 Aug 2020 06:47:39 -0400

On 7/31/20 4:32 AM, Steffen Klassert wrote:
> Architectures can change the 2 byte NET_IP_ALIGN if they prefer
> DMA alignment over IP alignment.
> 

As I mentioned in an earlier email, my most recent RFC work is RDMA.

> While that would be possible for some algorithms, I've never seen that
> a single cipher request is handled by multiple CPUs. I guess that
> would lead to cacheline bouncing, and for GCM an atomic synchronization
> of the counter would be needed.
> 

Currently, one CPU cannot keep up with encryption/decryption of RDMA at
merely 10 Gbps, let alone 100 Gbps.

This changes over time.  In the beginning, we were struggling to keep up
with 56 Kbps, and upgrading the backbone from 1.5 Mbps T1 to 54 Mbps T3.
Then for while, CPUs outran network speeds.  Now we're back again to link
speeds outpacing CPUs.

As I'd written PPP over SONET/SDH in 1992 (RFC 1619 published in 1994),
when designing ESP circa 1992-1993 we were cognizant that link speeds would
increase.  After all, I'd specified the minimum speed as 156 Mbps, and up to
2.5 Gbps.  There were only a few places in the world that could support
those speeds, but we anticipated them.

Photuris was able to run on the Intel 186 (admittedly, it took about 3
seconds), but Rivest estimated it would handle DoS attacks up to 4 Tbps.

> Usually parallelization is achieved by using AVX registers/instructions
> where multiple cipher blocks can be handled simultaneously with a single
> instruction.
> 

I'm also personally less interested in AES-GCM than ChaCha20-Poly1305.
We really need to design for at least 2 algorithms.

In the beginning, we worried about designing for unknown future algorithms.
That's why it had very flexible transforms.

> So it might make sense to have the ICV at the end because it is
> likely cache hot when needed.
> 

But after removing padding for these stream algorithms, then the ICV is
very likely not aligned.  For zero-copy RDMA, it is rather inconvenient.
And the IP header cache lines are likely still hot.

Anyway, that's why I'd like to consider at least a negotiated option --
as long as it's possible to implement efficiently in Linux and others.
We need to hear from more implementers.

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 6 Aug 2020 20:22:28 +0300

> We need to hear from more implementers.

I think it would be a bad idea to have such option. Yes, it might
offer small gains on environments where the option is picked exactly
when both ends like it, but when one ends pick it in a way which other
end does not like, we usually end up big performance penalties.

And also it again multiples the testing effort as now you need to add
to your test suites this combined with all posible ciphers etc.

This is was one of the main problems with IKEv1, there were so many
different combinations of different options that to be able to test
all of them required thousands or tens of thousands of test cases.

Example of similar optimzation causing problems was found during
interop events when some version (I think it was Linux) was sending
fragmented packets in reverse order, so the first network packet
sent/received was the last fragment. The idea was that when receiver
saw that last fragment it can immediately know how big the final
packet will be and it can allocate big enough buffer for the packet.

Then when you combined that with IPsec with per flow policy, meaning
each TCP/UDP flow might be using different SA, that meant that SGW
required to store all fragments in memory until it got the last packet
from the network, which was the first fragment, and only after that it
could check whether this packet is allowed to pass, and which SA it
needs to use.

Then it sent the fragments out, but there was lots of added latency
because of this. Actually I think there were also implementations
which did not even store the later fragments, they simply checked the
later fragments, and found out that they have not seen the first
fragment that would allow them to be passed, so they dropped them.

The SGW of course then sent the frames out in order, so that the
receiving SGW can efficently do exit tunnel checks (i.e., check from
the first fragment that this packet should be allowed, and match later
fragments with same fragment id to that, and allow them to passed),
and it then did not need to do same buffering. Of course the final
destination host now did not benefit at all from this optimization as
it was negatied by the SGWs in the middle.

So immediately when the options to use start to depend on the
platform, implementation etc it gets harder and harder to find out
which will be the optimal combination between two devices, and they
might end up using suboptimal feature set. And all of those add more
complexity and combinations that needs to be tested.

Of course if this option is added, it should be something like IPCOMP
or transport mode, meaning it is off by default, and everybody MUST
implement case where it is not enabled, then you would only implement
(and propose) it on cases where it actually benefits you, and
everybody else would never ever implement it.
-- 
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Fri, 07 Aug 2020 23:52:43 -0400

William Allen Simpson <[email-masked]> wrote:
    > In modern CPUs, there's always an issue with cache lines.  But for a
    > parallel implementation, it really isn't going to matter.  The CPU
    > that finishes last and needs to check the ICV isn't particularly
    > likely to be the CPU that processed the initial header anyway.

SGI did some extensive analysis on their multiprocessors around 1995.
I have the references somewhere if you want them.
I did a lot of analysis and reading (on paper, in the library...) at the time
about architectures for parallel firewalls.

SGI's conclusion (born out by architectures today) was that layer-by-layer
parallelism (i.e. Solaris STREAMS) was a total fail.  You have to keep the
packet with the same CPU all way up.
The Linux kernel has always tried to do this, and most of the focus in
hardware systems is to do the inverse-demux as early as physically possible,
and then stick with it.

SGI also concluded, at the time of 100Mhz CPUs and no hardware assist, that
the cost of doing IPsec (or SSL at the time), made the difference between
these vertical and horizontal parallelism architectures unmeasureable.  So
might as well go with the one that works best for non-encrypted traffic, and
is easier to code/debug.

    > As a matter of historical record, this was also a long debate for TCP.
    > The default is leading, and there is a TCP option for trailing checksum.

    > Might it be a non-default option negotiated per SPI?

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-