Subject: Re: [IPsec] ESP next payload number [Re: Some thoughts regarging draft-hopps-ipsecme-iptfs-01]

From: Christian Hopps <[email-masked]>
Date: Sun, 1 Dec 2019 03:52:30 -0500

I think it's important for this discussion to recognize that we have 2 orthogonal issues.

1) How does IP-TFS work on the wire with IPsec/ESP - this is where we need to make sure we don't unnecessarily restrict uni-directional use.

2) What are the changes to IKEv2 to support IP-TFS - this is where we need to decided if we need the ability to negotiate uni-direction cases, we don't believe we need to do this. I think this is inline with what people are expecting here (i.e., a pair of SAs with IP-TFS enabled or not).

More inline..

> On Dec 1, 2019, at 2:41 AM, Paul Wouters <[email-masked]> wrote:
> 
> On Fri, 29 Nov 2019, Christian Hopps wrote:
> 
>>> ESP SAs are always created in pairs. And whether IP-TFS is enabled or not is common for both SAs.
>>> Even if you use only one SA for IP-TFS traffic, the other will have IP-TFS enabled too.
>>> How are you going with your spec to create a pair of SAs where IP-TFS is enabled in one direction and
>>> disabled in the other? Am I missing something? Even if you are able to create such a pair of SAs, is it justified?
>> 
>> Understood that SAs are created in pairs. The intent here (and throughout the draft) is not to unnecessarily restrict possible uses. Having IP-TFS enabled on only one of a pair of SAs can make sense when you're only trying to protect traffic sent in one direction from traffic analysis (again e.g., telemetry). The use of IP-TFS does involve the allocation of a fixed amount of bandwidth so the justification for not enabling it in one direction is to save that bandwidth.
> 
> It seems unwise to protect traffic one way but not the other way. Are
> endusers really able to make the right decision based on their generated
> traffic? If you are that hungry for resources, perhaps this isn't an
> option for you to use?

There is no traffic to protect in the reverse direction. Consider telemetry where one is simply sending un-acked UDP data using to monitors.

> 
> Also, if an inbound or outbound SA is not using IP-TFS, couldn't the
> memory be deallocated, or the memory allocation could be postponed
> until the first IP-TFS type packet arrives?
> 
> The architecture is really symmetrical for in/out bound SA's, so I
> would prefer we do not change that.

We are not looking to change the architecture, only to use the existing one (see orthogonal issues). The telemetry example is one case of uni-directional traffic; however, multicast is another area where you can get non-"TCP flow-like" unidirectional configurations.

> 
>> The spec is (trying to be) very careful in not restricting the unidirectional TFS use case b/c there's no reason to do so, and so there should not be anything in the document that restricts having only one of the SA pairs used by IPsec/ESP running with IP-TFS.
> 
> The question is, does it need to be negotiated or not or can a well
> implemented implementation postpone all memory requirements if this
> feature is unused in one direction?

The standard case of using IKEv2 (i.e., what most users will use) assumes bi-directional use which seems appropriate. The draft also allows for not starting the IP-TFS constant send immediately. Finally, the draft does not restrict other less-traditional use cases which fall outside the scope of IKEv2 use.

For IKEv2 we believe bi-directional only is fine and that's what the draft has defined.

> 
>> Setting up such an SA pair is orthogonal to being able to run with that configuration. While the vast majority of IPsec utilizes IKEv2 for SA creation/management, this is not a requirement of IPsec/ESP. The unidirectional case can be covered with local configuration (along with IKEv2), or by using another method entirely for the SA management.
> 
> Non-IKE can negotiate whatever it want and hack in custom IPsec SA's
> that we would find completely illegal. That's out of scope of this WG,
> but we should also not bring it into scope as "others might want that".
> 
>> The intent in the end is to specify what we need to for IP-TFS operation, but not too over-specify and unnecessarily restrict possible future (or outside) uses. IOW, less is more.
> 
> But I am still not convinced endusers can make these decisions of
> bidirectional vs unidirectional and whther or not their traffic
> becomes more susceptible to traffic analyses attacks.

Which users are we talking about here? We agree, for the vast majority of end-users the standard use case will be bi-directional, and the IKEv2 changes reflect this (see 2).

We're not making it *easy* to run uni-directional we're simply not outright prohibiting this use case for users who may have more complex requirements and/or know what they are doing. 

We've got the restriction in IKEv2, but what is the value in further restricting things at the IPsec/ESP level (1)? There is real value lost in making this restriction, so I think we need to justify it if we choose to do so.

Thanks,
Chris.

>> FWIW, I did explore adding machinery to IKEv2 to allow setting up/negotiating unidirectional use cases, but it adds a lot of complexity, and so we decided that while unidirectional TFS protection is useful it will still be a relatively uncommon configuration and was best left for local configuration for now. If in the future we see a large call for on-demand/negotiated (vs local config) uni-directional TFS we can always come back and add it to IKEv2. What's important in the context of the IKEv2 changes for now is that we don't do anything to block doing this in the future.
> 
> I have not yet been convinced of this.
> Paul
>

---

From: Christian Hopps <[email-masked]>
Date: Sun, 1 Dec 2019 03:55:27 -0500

> On Nov 30, 2019, at 1:35 AM, Paul Wouters <[email-masked]> wrote:
> 
> 
> On Nov 29, 2019, at 21:01, Valery Smyslov <[email-masked]> wrote:
> 
>> A lot of things in IKEv2 is negotiated by using notifications. With regard to child SA they are - 
>> using transport mode, using IPcomp, using WESP, using ROHC. 
>> IP-TFC looks to me like a natural continuation of this line.
> 
> I agree. This is easier to implement than using traffic selectors and seems more appropriate as it is an “optional” thing to enable - similar to transport mode and compression.
> 

Just sending a note so others don't need to +1 this, that I'll be looking into these changes for the next rev, :)

Thanks,
Chris.

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 1 Dec 2019 18:28:59 -0800

> There is no traffic to protect in the reverse direction. Consider telemetry where one is simply sending un-acked UDP data using to monitors.

I feel like I must be missing something; if there's no traffic in the
reverse direction why does it matter if we assign semantics to the reverse
SA or not?

-Ben

---

From: Steffen Klassert <[email-masked]>
Date: Mon, 2 Dec 2019 09:01:55 +0100

On Thu, Nov 28, 2019 at 04:49:36PM +0300, Valery Smyslov wrote:
> Hi,
> 
> after reading through draft-hopps-ipsecme-iptfs-01 I have some thoughts.
> 
> 1. I think it's a wrong decision to support tunnel mode ESP only. IP-TFS for transport mode ESP
>     is equally important because one of the widely used scenario is to combine general purpose
>     tunneling (like GRE) with transport mode ESP. In this case traffic flowing over such SA
>     will in fact be tunnel traffic from several hosts, but the SA is created in transport mode.
>     For this reason I think that IP-TFS must support transport mode SA either.

I'd like to agree here. It does not add much more complexity and there are valid usecases
for transport mode (and even for BEET mode).

> 4. I'd like to see more text in the draft regarding reassembling of incoming packets.

Yes, I think some words on how to reassemble the fragments are really
needed.

>     It seems to me that it can be done pretty easy by linking the reassembly logic
>     with replay protection window.

While it looks like doing the reassembling based on ESP sequence numbers
might be an easy approach, it could be also dangerous.

Consider a system that encapsulates two flows on different cpus
with the same SA. This system can TX packets in the following
order:

TX cpu0 inner flow0 SA0:

      Offset: 0                               Offset: 100        
      [ ESP1  (1500) ]                        [ ESP3  (1500) ]  
      [--800--][--800-                        -][-----1400---] 

--------------------------------------------------------------------------------------
TX cpu1 inner flow1 SA0:
                          Offset: 0                                Offset: 100   
                          [ ESP2  (1500) ]                        [ ESP4  (1500) ]
                          [--800--][--800-                        -][----1400----]    

On the receive side, it is not that clear how to reassemble the fragments
from ESP3 and ESP4 into the fragments from ESP1 and ESP2. Maybe some
packet ID in the IP-TFS header could help to identify related fragments.

Steffen

---

From: "Valery Smyslov" <[email-masked]>
Date: Mon, 2 Dec 2019 11:28:16 +0300

Hi Steffen,

>       [ ESP1  (1500) ]                        [ ESP3  (1500) ]

I'm probably missing something here, but I think that sending side assigns 
every outgoing IP packet to some SA. Then the packet is added to the ESP message 
(that may already contain previous packets). If the packet cannot fit into the
left space, it is split and the rest of the packet is sent in the next
ESP message of the same SA. In this case there must not be situations
when one part of the IP packet is sent over one SA and the rest of
the packet is sent over different SA. Am I missing something?

Regards,
Valery.

---

From: Steffen Klassert <[email-masked]>
Date: Mon, 2 Dec 2019 09:49:30 +0100

Hi Valery,

On Mon, Dec 02, 2019 at 11:28:16AM +0300, Valery Smyslov wrote:
> ESP message of the same SA.

All packets are sent over the same SA, but on different cpus. This means
that the 'rest' might not be in the next ESP message. The other cpu
could have TXed some ESP packets before, it is a race.

In this example, flow0 is encapsulated on cpu0, flow1 is encapsulated on cpu1,
both on the same SA.

ESP1 contains flow0, but ESP2 contains flow1. The 'rest' from flow0 is
encapsulated in ESP3, the 'rest' from flow1 is encapsulated in ESP4. 
So I think it is not clear how to do a correct reassembling here.

Steffen

---

From: "Valery Smyslov" <[email-masked]>
Date: Mon, 2 Dec 2019 12:02:42 +0300

> So I think it is not clear how to do a correct reassembling here.

Ah, I see now. It seems to be a real problem and I think that in this case fragmentation 
must be disabled. And this must be discussed in the draft.

Regards,
Valery.

---

From: Christian Hopps <[email-masked]>
Date: Mon, 2 Dec 2019 05:37:45 -0500

> On Dec 1, 2019, at 9:28 PM, Benjamin Kaduk <[email-masked]> wrote:
> 

B/c an IP-TFS enabled SA is sending a constant stream of packets either filled with the actual traffic or pad if no data is available.

Thanks,
Chris.

---

From: Christian Hopps <[email-masked]>
Date: Mon, 2 Dec 2019 06:22:26 -0500

> On Dec 2, 2019, at 3:01 AM, Steffen Klassert <[email-masked]> wrote:
> 

Indeed the code mustn't fragment this way. :)

We could add a bit of text that one should avoid this mistake.

One way to implement this is using a queue of packets to order the transmission. The flows (using any number cpus) add their packets to the queue. The IP-TFS ESP encapsulation code (again using any number of CPUs) remove packet fragments, in order, from the queue when constructing the IP-TFS ESP packets. The SA ESP sequence number stored with the queue.

Another way to do this (the way we are coding it) is to queue pre-built IP-TFS payloads directly onto a queue. In this case you have an "in-progress" payload that you add the next packet to while the queue is locked. If you fill this in-progress payload you simply move it to the send queue and start a new in-progress payload, repeat if need be until the whole packet is present in the queue, then you unlock the queue. The send routine (again running on any number of CPUs) dequeues, encrypts and sends the payloads.

For the second method the payload construction is very quick as we are not doing buffer copies, but simply using buffer chaining/indirection, so the queue lock is not problematic.

Thanks,
Chris.

> Steffen
> 
> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec
>

---

From: Steffen Klassert <[email-masked]>
Date: Mon, 2 Dec 2019 15:11:45 +0100

> We could add a bit of text that one should avoid this mistake.

I'm not so sure whether the receiver should rely on that
the sender did the fragmentation right. I think a packet
ID, like IP fragments have, would just solve the problem.
The implementation would not even need to care about this
multicore race then.

Steffen

---

From: Christian Hopps <[email-masked]>
Date: Mon, 2 Dec 2019 10:57:59 -0500

> On Dec 2, 2019, at 9:11 AM, Steffen Klassert <[email-masked]> wrote:
> 

The receiver can do any number of wrong things with what it sends, but I'd normally call those bugs. :)

Technically though, attaching a packet ID to the fragments to allowing sending them in any order saves only a little on code complexity (i.e., not using an ordering queue) on the sender side; however, it seems to add a disproportionate amount of complexity to the receiver/reassembly (which could e.g., be aggregating VPN server). Adding a packet ID also means that you can't just chain the inner traffic buffers together to form the IP-TFS payload as you must now insert an extra header between each of the inner packets, this is going to affect performance and memory use on whitebox/software based deployments as well as reduce available bandwidth on the tunnel.

I think we should try and keep fragmentation and reassembly as simple as possible so that it is easy to implement and get right. Having coded this using just the ESP sequence numbers to correct-order the received packets, I can say it's an easy-to-moderate complex function that performs well, it took a few iterations on the code to get it right and well tested. Adding another level to this receive complexity should only be done if we absolutely have to. I don't think we are there yet, given that the use of a simple ordering queue on the sender side is all we are talking about.

Again it's easy to add some text to the document to highlight what needs to be paid attention to if one is using multiple cores to send on a single IPsec SA.

Thanks,
Chris.

---

From: Steffen Klassert <[email-masked]>
Date: Tue, 3 Dec 2019 09:10:08 +0100

> The receiver can do any number of wrong things with what it sends, but I'd normally call those bugs. :)

Yes, that's true. But if the protocol allows to do things wrong, it is
a bug in the protocol :)

Maybe you can just make it clear at the sender side by saying something
like 'Fragments must be sent ordered and ESP encapsulated with consecutive
sequence numbers.'

> 
> Technically though, attaching a packet ID to the fragments to allowing sending them in any order saves only a little on code complexity (i.e., not using an ordering queue) on the sender side;

I hoped to avoid such an ordering/serialization queue as I fear
this will become a bottleneck. With the current design, I don't
see how to do this without a queue. I know, it is an implementation
detail, but implementation matters too :)

> however, it seems to add a disproportionate amount of complexity to the receiver/reassembly (which could e.g., be aggregating VPN server).

Yes, if you know that the next fragment comes with the next ESP
packet, things are much easier. So we have the complexity either
at the sender or the receiver side, not so sure what performs better.

> Adding a packet ID also means that you can't just chain the inner traffic buffers together to form the IP-TFS payload as you must now insert an extra header between each of the inner packets, this is going to affect performance and memory use on whitebox/software based deployments as well as reduce available bandwidth on the tunnel.

Good point. You can always chain extra headers and inner packets with
a scatter-gather list, but it will have some performance impact.

> I think we should try and keep fragmentation and reassembly as simple as possible so that it is easy to implement and get right.

I absolutely agree here.

> Having coded this using just the ESP sequence numbers to correct-order the received packets, I can say it's an easy-to-moderate complex function that performs well, it took a few iterations on the code to get it right and well tested. Adding another level to this receive complexity should only be done if we absolutely have to. I don't think we are there yet, given that the use of a simple ordering queue on the sender side is all we are talking about.
> 

As said above, just make it clear how the sender must order the
fragments if you don't want a packet ID.

Steffen

---

From: "Valery Smyslov" <[email-masked]>
Date: Tue, 3 Dec 2019 14:02:57 +0300

> like 'Fragments must be sent ordered and ESP encapsulated with consecutive

I concur (modulo s/must/MUST). And in addition: "If this requirement cannot be met, the sender MUST NOT fragment packets."

> > Adding a packet ID also means that you can't just chain the inner traffic buffers together to form the IP-TFS
> payload as you must now insert an extra header between each of the inner packets, this is going to affect
> performance and memory use on whitebox/software based deployments as well as reduce available
> bandwidth on the tunnel.
> 

If support for transport mode is added then an extra header seems anavoidable...

> > I think we should try and keep fragmentation and reassembly as simple as possible so that it is easy to

Strongly agree.

Regards,
Valery.

---

From: Christian Hopps <[email-masked]>
Date: Tue, 3 Dec 2019 07:34:08 -0500

> On Dec 3, 2019, at 3:10 AM, Steffen Klassert <[email-masked]> wrote:

Absolutely.

> at the sender or the receiver side, not so sure what performs better.

FWIW, we will have some implementation/operational experience here to look at, as a WG, prior to making any final choices. I'm currently working to code this for running over a 100G link in a whitebox setup, and I will report back to the WG when we have some numbers. :)

Thanks,
Chris.

---

From: Christian Hopps <[email-masked]>
Date: Tue, 3 Dec 2019 07:36:57 -0500

> On Dec 3, 2019, at 6:02 AM, Valery Smyslov <[email-masked]> wrote:
> 

I suspect you are correct. For the transport case it should be enough to just replicate the ESP sequence number method (i.e., add it to the IP-TFS header once) too though.

Thanks,
Chris.