Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

From: Antony Antony <[email-masked]>
Date: Thu, 1 Apr 2021 15:36:15 +0200

On Wed, Mar 31, 2021 at 23:38:01 +0000, Bottorff, Paul wrote:
> Hi Antony:
> 
> Below,
> 
> Cheers,
> 
> Paul
> 
> 
> 
> -----Original Message-----
> From: IPsec [mailto:[email-masked]] On Behalf Of Antony Antony
> Sent: Wednesday, March 31, 2021 3:32 AM
> To: Bottorff, Paul <[email-masked]>; IPsec <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07
> 
> Hi,
> 
> This is an interesting draft. I would love to see a generic solution for network paths and receiver use cases, such as RSS.
> 
> PB>> Can you explain your use case for RSS a little more? I'd guess you are looking at LB around the RSS queues to get better distribution for the decodes.
> <<

We are using muliple SAs, with identical Traffic Selectors, to
load balance IPsec traffic between IPsec peers.
https://datatracker.ietf.org/doc/draft-pwouters-multi-sa-performance/

This should improve multi-core-cpu utilization and IPsec throughput. The receiver
should distribute the flows, based on SPI, to different CPUs for decryption.
Ideally the NIC should support a ntuple flow distribution using SPI.
It is not yet widely supported, and also not compatible with older NICs.
Now we are thinking of ESP in UDP with different source ports. Just like the draft
proposed for ECMP and LAG.
When implementing it I ran into issues with NAT and SAD remapping messages which update IKE.
I am using strongSWAN for IKE and Linux XFRM for ESP.

> The RFC3948 specifies one pair of UDP ports 4500-4500.
> Both the IKE flow and the ESP in UDP flow should use the same UDP flow.
> The draft seems to suggest new destination port and source ports are only for ESP? How would this change work with IKE?
> May you are not intending to use IKE?

PB> Our use cases use IKE, however as stated in RFC3948 ESPinUDP does not have to be
PB> tied to IKE, it is just advantageous to do so for the NAT case since this allows
PB> a single mapping for both at the NAT rather than two mappings.

PB> I've wondered why we could not use the RFC3948 encoding for ESPinUDP, but allow 
PB> the source port to be chosen differently than IKE. Perhaps Xu has some thoughts on this. 

In my experience it would work well when there is no NAT. When there
there is NAT the IKE and ESP in UDP should use same ports, otherwise
IKE will get established and ESP packets could get dropped in one
direction. When there is NAT it would look more like a client-to-server
than a peer-to-peer.

> How would the new ESP flow work when there is a NAT gateway along the path?
> I ran into issues with both sides choosing different source ports.
> It would cause SADB mapping changes which force changes IKE flows. One coul disable
> SADB mapping changes. However, that would break real NAT changes.
>
PB> We are mostly interested in data centre use cases which don't have intervening NATs,
PB> however I believe SD-WAN cases could have NAT and FW traversals between tunnel end
PB> points. As it stands draft-xu-ipsecme-esp-in-udp-lb does not specify how the source port PB> value is determined. It seems like it could be based on a hash value within the ESP or
PB> based on the SPI and IPs.

I implemented a proof concept - UDP source port set to 16bit hash of out going SPI.
Then it occurred to me the hash could collide with other well
known UDP ports e.g. hash of an SPI or/and IP address could be 53(DNS).
Some admins may not be happy to see source port 53 for ESP in UDP traffic.

If you choose ephemeral source ports both sides should know the source
ports otherwise SADB remap will be generated. If we disable that real NAT
would break...
So far I don't have a clean solution which would work with NAT and without NAT
for our use case.

---

From: Paul Wouters <[email-masked]>
Date: Thu, 1 Apr 2021 10:10:51 -0400 (EDT)

> direction. When there is NAT it would look more like a client-to-server

One idea could be to use MOBIKE probes (without UPDATE_SA) by the
initiator behind NAT to get multiple NAT mappings? It might require
kernel changes depending on how it implements NAT updates :/

> Some admins may not be happy to see source port 53 for ESP in UDP traffic.

I seen this in the wild already with a large VPN provider blocking port
19 (chargen) and NAT gateways picking port 19 as source port. While this
could be an attack to try and trigger chargen responses from a spoofed
packet, I don't think it is. It might work for TCP, but not really for
UDP as a DoS attack. I think in the end, sadly, we have to expect every port
to be used by badly designed NAT gateways.

However, it would still be a good design to keep UDP source ports in the
ephemeral ranges. This also helps against local policy enforcements,
such as Linux's SElinux policies.

> So far I don't have a clean solution which would work with NAT and without NAT

That is indeed the real problem to solve. I have to do some more
testingwith MOBIKE probes on the same IP with only port differences
and see if the kernel acts like these are port updates or not if it
is only IKEinUDP and not ESPinUDP.

Paul

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 2 Apr 2021 01:40:48 +0300

> The RFC3948 specifies one pair of UDP ports 4500-4500.

No it does not. It says you must use same ports than what you do for
IKE traffic.

> Both the IKE flow and the ESP in UDP flow should use the same UDP
> flow. The draft seems to suggest new destination port and source
> ports are only for ESP? How would this change work with IKE? May you
> are not intending to use IKE?

The reason NAT-T uses same ports for both ESP and IKE is to make sure
that responder will know the port mapping from the IKE packets for ESP
packets too. If they would use different port numbers, then responder
would need to wait for first ESP packet to come in from the initiator
before it can send any ESP packets back, as it would be able to know
which port numbers to use for ESP traffic. Also keepalives would be
needed to do for both IKE and ESP separately. 

> PB>>Our use cases use IKE, however as stated in RFC3948 ESPinUDP does not have to be tied to IKE, it is just advantageous to do so for the NAT case since this allows a single mapping for both at the NAT rather than two mappings.
> PB>>I've wondered why we could not use the RFC3948 encoding for ESPinUDP, but allow the source port to be chosen differently than IKE. Perhaps Xu has some thoughts on this. 
> <<

As there is no way for the responder to know whether the initiator
sent the packet out using source port of 4500 and then NAT changed it
to port number of xxxx, or whether the initiator sent the packet out
using port number of xxxx himself, the RFC7296 do require that
responder MUST accept incoming packets regardless of the source port,
and MUST always respond back to exactly same IP and port than what was
in the source packet of the request:

>From the 2.11 of the RFC7296:

						An implementation MUST
   accept incoming requests even if the source port is not 500 or 4500,
   and MUST respond to the address and port from which the request was
   received.  It MUST specify the address and port at which the request
   was received as the source address and port in the response.

There is text in 2.23 of the RFC7296 which do say:

								For this
   reason, even though IKE packets MUST be sent to and from UDP port 500
   or 4500, they MUST be accepted coming from any port and responses
   MUST be sent to the port from whence they came.

Which is bit funny, as the "even though" claims that IKE packets must
be sent to and from UDP port 500 or 4500, and there is no such claim
anywhere else in the RFC7296. I think it was supposed to say that
"even though IKE packets are normally sent ...", as this is not
correct place to make this kind of new requirement.

So making IKE implementation which uses some other source port than
500 or 4500 is common way to force NAT-T traversal and UDP
encapsulation on, and in that way allow user mode IKE + IPsec to be
implemented without affecting the in kernel version of the IPsec in
the host. I.e., user mode application wanting to use IPsec in usermode
without affecting host IPsec in any ways can use any random port as
source port and port 4500 as the destination port, and make sure that
NAT_DETECTION_SOURCE_IP does not match, so responder will enable udp
encapsulation.

The responder MUST work even with that setup, so there is no reason
why there should be requirement for the oritinal sender should be
mandated to use source port 4500...

> PB>>We are mostly interested in data centre use cases which don't have intervening NATs, however I believe SD-WAN cases could have NAT and FW traversals between tunnel end points. As it stands draft-xu-ipsecme-esp-in-udp-lb does not specify how the source port value is determined. It seems like it could be based on a hash value within the ESP or based on the SPI and IPs.
> <<
> 
> Should both sides use the same source port?

For the load balancing I think it is enough for just one of the ports
to be different, thus initiator could simply allocate n random source
port numbers, and initiate IKE from each of them to responder, and
then create SAs for each of them separately, thus allowing load
balancing using UDP encapsulation using existing hardware.

The real issue is that if you do not want to create n separate IKE
SAs, then you need to do something special, and there are other things
that needs to be considered then. 
-- 
[email-masked]

---

From: "Valery Smyslov" <[email-masked]>
Date: Fri, 2 Apr 2021 09:08:13 +0300

Hi Tero,

> balancing using UDP encapsulation using existing hardware.

RFC 7791 + MOBIKE can be used to clone IKE SA  and move 
it to a different local IP+port.

Regards,
Valery.

> --
> [email-masked]
> 
> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec

---

From: "Bottorff, Paul" <[email-masked]>
Date: Fri, 2 Apr 2021 21:59:12 +0000

Hi Valery:

Agreed that LB only needs control of the source port to provide entropy. 

Our application is for traversal of highly meshed data centre fabrics. We encapsulate and de-encapsulate at the server using smart NICs and so don't have any impact on RSS or host software (nor improvement over their standard operation). We perform the encapsulation/de-encapsulation after the ESP packet is formed. Since encapsulation occurs after and de-encapsulation occurs before the IPsec stack, IPsec does not see any of the new port assignments so we don't have any issues with the SADB or IKE.

Cheers,

Paul

-----Original Message-----
From: Valery Smyslov [mailto:[email-masked]] 
Sent: Thursday, April 1, 2021 11:08 PM
To: 'Tero Kivinen' <[email-masked]>; Bottorff, Paul <[email-masked]>
Cc: 'IPsec' <[email-masked]>; [email-masked]
Subject: RE: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

Hi Tero,

> balancing using UDP encapsulation using existing hardware.

RFC 7791 + MOBIKE can be used to clone IKE SA  and move it to a different local IP+port.

Regards,
Valery.

> INVALID URI REMOVED
> man_listinfo_ipsec&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=CCwOcKkISkWxd8Ymy11M8VW3U6Peq8aJ_DDlgVbQW5E&m=ykseXYzNH5MG1guNwTPMGiGby4o46mBhv92vwoSpb0U&s=nCqbPzmEc1xdTkL0jPmKmNgH252j3dURPVnH8bt4OtE&e=