Subject: [IPsec] PLMTUD probes for IPsec

From: Michael Richardson <[email-masked]>
Date: Wed, 04 Apr 2018 17:35:21 -0400

> Dynamic IPsec PMTU PLPMTUD - Shibu Piriyath, Ron Bonica
    > https://datatracker.ietf.org/meeting/101/materials/slides-101-ipsecme-packetization-layer-path-mtu-discovery-01

    > Problem: IPsec Tunnel has an PMTU as any other tunnel.
    > Solution in Transport Area: Inband Path discovery

I spoke to Ron afterwards, and I'm very enthusiatic about getting PLPMTUD
widely deployed.  We didn't get to this slides, so we didn't figure
out what he needs. Slides talk about an IP-level probe that IPsec
encapsulators can use to get estimate for tunnel inner MTU.

But, if we can get PLMTUD deployed for all traffic, then the end-nodes can
do appropriate PMTU probing and can figure out what the inner MTU is.
i.e. just get everyone to enable plpmtu (4821). It's a knob on most
systems, which has been left in the off state because of lack of evidence
that it isn't harmful.

There seems to be some sticking points among the high-speed about CDNs:
they hate all PMTUD as it screws with tx scheduling into hardware.
(TCP segment offload issues)

So they just use 1280 is what I was told.  That's good for the network, but
it removes them as strong allies for getting PLPMTUD enabled by default
everywhere.   It would be nice if we could get a BCP out of them. Such
BCP could also bless PLPMTUD for everyone else.  I was pushing for this
with RFC8200/STD86 but there was insufficient evidence of deployment.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: "Valery Smyslov" <[email-masked]>
Date: Thu, 5 Apr 2018 14:40:11 +0300

Hi Michael,

> encapsulators can use to get estimate for tunnel inner MTU.

Why IKE messages cannot be used for this purpose?

Regards,
Valery.

---

From: Paul Wouters <[email-masked]>
Date: Thu, 5 Apr 2018 08:08:52 -0400 (EDT)

> Why IKE messages cannot be used for this purpose?

IKE messages might not take the same path, eg ESP goes through hardware
offload or other things, or intermediary routers might have different
rules for UDP vs ESP.

Paul

---

From: "Valery Smyslov" <[email-masked]>
Date: Thu, 5 Apr 2018 15:48:04 +0300

Hi Paul,

> offload or other things, or intermediary routers might have different

True, unless UDP encapsulation is used...

Regards,
Valery.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 05 Apr 2018 11:08:47 -0400

> Why IKE messages cannot be used for this purpose?

I think that possibly it can, and this is the kind of discussion that
I think we should have.   The advantage of doing that is that it requires
no new code on the responder!  That's a big win for deploying.
It means that this can be done unilaterally, no new specification,
just implementation advice.

The slight implementation challenge is making sure that the IKE traffic
is going along the same path as the ESP traffic.  I'd like to hear from
Ron about whether this is an issue.

I also thought about using having some plpmtud on each end make a TCP
connection *within* the tunnel and use the already defined plpmtu for
TCP.  That might be really easy for systems without user/kernel divisions
(some router kernels). It would require some notifications to indicate that
the responder has a TCP port open.  And of course, the traffic would
have to fit into the tunnel as well.

---

From: Ron Bonica <[email-masked]>
Date: Thu, 5 Apr 2018 20:01:11 +0000

Folks,

In the first version of this draft, we leveraged IKE to exchange messages. And provided with a good enough reason, we might go back to that position!

We moved away from IKE for the following reasons:

- The path between the encrypting and decrypting nodes might include ECMP. If so, IKE messages might take a different path than actual encrypted data
- The current method provides the same level of protection as IKE and possibly better performance
- The current doesn't require changes to IKE

Are these reasonable assumptions. If not, we would be happy to return to the IKE solution.

                                                           Ron

> -----Original Message-----
> From: Michael Richardson <[email-masked]>
> Sent: Thursday, April 5, 2018 11:09 AM
> To: Valery Smyslov <[email-masked]>
> Cc: [email-masked]; Ron Bonica <[email-masked]>
> Subject: Re: [IPsec] PLMTUD probes for IPsec
> 
> 
>     >> > https://datatracker.ietf.org/meeting/101/materials/slides-101-
> ipsecme-packetization-layer-path-mtu-
>     >> discovery-01
>     >>
> It means that this can be done unilaterally, no new specification, just
> implementation advice.
> 
> The slight implementation challenge is making sure that the IKE traffic is going
> along the same path as the ESP traffic.  I'd like to hear from Ron about
> whether this is an issue.
> 
> connection *within* the tunnel and use the already defined plpmtu for TCP.
> That might be really easy for systems without user/kernel divisions (some
> router kernels). It would require some notifications to indicate that the
> responder has a TCP port open.  And of course, the traffic would have to fit
> into the tunnel as well.
> 
> 
> 
> 
>     > Regards,
>     > Valery.
>

---

From: "Valery Smyslov" <[email-masked]>
Date: Fri, 6 Apr 2018 10:23:47 +0300

Hi Ron,

> Folks,
> 
> In the first version of this draft, we leveraged IKE to exchange messages. And provided with a good enough
> reason, we might go back to that position!
> 
> - The path between the encrypting and decrypting nodes might include ECMP. If so, IKE messages might take a
> different path than actual encrypted data

That's a valid concern. The only time when you can be sure that the paths are the same 
is when NAT traversal is in use (either because some NAT is in between, or you force it on the peers).

> - The current doesn't require changes to IKE

True. But with your current proposal some changes to the IPSec in kernel are needed too
(or you need to have standalone client and server applications to exchange probes).
Not sure what is easier.

> Are these reasonable assumptions. If not, we would be happy to return to the IKE solution.

I think one problem with the suggested approach is that the tunnel selectors might
not allow sending packets over UDP. Consider you have "narrow" ESP SA that only
allows TCP packets to go through and drops all UDP. In this case your approach
won't work, or some additional heuristics would be needed.

Regards,
Valery.

---

From: Ron Bonica <[email-masked]>
Date: Mon, 9 Apr 2018 17:22:43 +0000

Valery,

Both good points. So, it appears that we have the following choices:

- leverage IKE for exchanging probes and acks  (But risk sending probes and acks over a different path than encrypted data)
- send probes and acks in-band. Try UDP probes first. If that doesn't work, try TCP probes.

Which do you think is better?

                                                                       Ron

> From: Valery Smyslov <[email-masked]>
> Sent: Friday, April 6, 2018 3:24 AM
> To: Ron Bonica <[email-masked]>; 'Michael Richardson'
> <[email-masked]>
> Cc: [email-masked]
> Subject: RE: [IPsec] PLMTUD probes for IPsec
> 
> Hi Ron,
> 
> > Folks,
> >
> > In the first version of this draft, we leveraged IKE to exchange
> > messages. And provided with a good enough reason, we might go back to
> that position!
> >
> > - The path between the encrypting and decrypting nodes might include
> > ECMP. If so, IKE messages might take a different path than actual
> > encrypted data
> 
> That's a valid concern. The only time when you can be sure that the paths are
> the same is when NAT traversal is in use (either because some NAT is in
> between, or you force it on the peers).
> 
> True. But with your current proposal some changes to the IPSec in kernel are
> needed too (or you need to have standalone client and server applications to
> exchange probes).
> Not sure what is easier.
> 
> I think one problem with the suggested approach is that the tunnel selectors
> might not allow sending packets over UDP. Consider you have "narrow" ESP
> SA that only allows TCP packets to go through and drops all UDP. In this case
> your approach won't work, or some additional heuristics would be needed.
> 
> Regards,
> Valery.
> 
> >                                                            Ron
> >
> >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.iet
> > > f.org_meeting_101_materials_slides-2D101-
> 2D&d=DwICAg&c=HAkYuh63rsuhr
> > > 6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcA
> > >
> wrDThKP8&m=QoSMQregi2cS6qaSUZ1ttnd_izUvFwYYoIA80K3xgI8&s=YXPwE
> w5OTeG
> > > sOP8sS0P463jWQ-OhKpYEK9LUnAWcQgA&e=
> > > The slight implementation challenge is making sure that the IKE
> > > traffic is going along the same path as the ESP traffic.  I'd like
> > > to hear from Ron about whether this is an issue.
> > >
> > > I also thought about using having some plpmtud on each end make a
> > > TCP connection *within* the tunnel and use the already defined plpmtu
> for TCP.
> > > That might be really easy for systems without user/kernel divisions
> > > (some router kernels). It would require some notifications to
> > > indicate that the responder has a TCP port open.  And of course, the
> > > traffic would have to fit into the tunnel as well.
> > >
> > >
> > >
> > >
> > >     > Regards,
> > >     > Valery.
> > >
> > >     >> There seems to be some sticking points among the high-speed
> > > about
> > > CDNs:

---

From: "Valery Smyslov" <[email-masked]>
Date: Tue, 10 Apr 2018 09:57:13 +0300

Hi Ron,

> - send probes and acks in-band. Try UDP probes first. If that doesn't work, try TCP probes.

What about ICMP-only SAs (yeah, it's weird, but possible)?

> Which do you think is better?

Both don't look ideal. I slightly prefer the former, as it looks simpler to implement
(at least for me), but the issue with different paths  can outweigh all. One potential
solution to this issue would be to always use UDP encapsulation, but I'm not sure
we may impose such a requirement... Your opinion?

Regards,
Valery.

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 10 Apr 2018 13:51:58 +0300

Valery Smyslov writes:
> Both don't look ideal. I slightly prefer the former, as it looks
> simpler to implement (at least for me), but the issue with different
> paths can outweigh all. One potential solution to this issue would
> be to always use UDP encapsulation, but I'm not sure we may impose
> such a requirement... Your opinion?

I would just use IKE packets, and ignore the cases where ESP and IKE
get different routes which have different MTUs. I would expect that in
most cases if there is something in the middle using different MTUs
then the routes are not equal cost anymore, thus routing will use only
one of the routes. Usually the issues with MTU comes with road
warriors connecting from random locations around the world, and in
most of the cases there will be NAT for IPv4 in the middle, which will
move all traffic to UDP anyways.

Do we have any real world examples where the ESP and IKE packets
consistently take different routes, and those different routes have
different MTUs? And does all ESP traffic follow always one route, and
all IKE traffic follow another route, or is it more random or based on
the SPI or something?

I do not know enough to really say whether such cases exists or do not
exists, but before we start to make complicated protocols to cope with
cases which are very rare, I want to get more information.

If those cases are very rare, I might still want to make sure IPsec
works somehow, even if it is not as efficient as it could be (i.e.,
there might be extra fragmentation happening or finding proper MTU
might take more time). 
-- 
[email-masked]

---

From: Ron Bonica <[email-masked]>
Date: Wed, 11 Apr 2018 14:56:42 +0000

Hi Valery,

I am thinking like this.......

- If we do nothing, tunnel performance  is acceptable but suboptimal. We can prevent blackholing by statically configuring the tunnel MTU to a sufficiently low value. However, we cannot take advantage of tunnels with larger PMTUs.

- If we use IKE to exchange probes and acks, tunnel performance may become totally unacceptable. In the situation where a) IKE messages traverse a different path than encrypted payloads and b) the PMTU associated with the IKE path is greater than the PMTU associated with encrypted payload path, we may produce an inflated estimate of the Tunnel MTU. This may lead to black holing.

So, our probe and acks have to be exchanged over the IPSEC tunnel. At first I thought that the best way to do this was by exchanging UDP messages. But you have a point. If we exchanged encrypted ICMP Echo / Echo Response messages instead of UDP messages, there would be slightly less code to write. Furthermore, we wouldn't need to allocate a new UDP port.

                                                                                 Ron

> Sent: Tuesday, April 10, 2018 2:57 AM
> > - leverage IKE for exchanging probes and acks  (But risk sending
> > probes and acks over a different path than encrypted data)
> Both don't look ideal. I slightly prefer the former, as it looks simpler to
> implement (at least for me), but the issue with different paths  can outweigh
> all. One potential solution to this issue would be to always use UDP
> encapsulation, but I'm not sure we may impose such a requirement... Your
> opinion?
> 
> Regards,
> Valery.
> 
> >
> > Ron
> >
> >
> > > > - The path between the encrypting and decrypting nodes might
> > > > include ECMP. If so, IKE messages might take a different path than
> > > > actual encrypted data
> > >
> > > That's a valid concern. The only time when you can be sure that the
> > > paths are the same is when NAT traversal is in use (either because
> > > some NAT is in between, or you force it on the peers).
> > >
> > > True. But with your current proposal some changes to the IPSec in
> > > kernel are needed too (or you need to have standalone client and
> > > server applications to exchange probes).
> > > Not sure what is easier.
> > >
> > > > Are these reasonable assumptions. If not, we would be happy to
> > > > return to
> > > the IKE solution.
> > >
> > > I think one problem with the suggested approach is that the tunnel
> > > selectors might not allow sending packets over UDP. Consider you
> > > have "narrow" ESP SA that only allows TCP packets to go through and
> > > drops all UDP. In this case your approach won't work, or some additional
> heuristics would be needed.
> > >
> > > Regards,
> > > Valery.
> > >
> > > >                                                            Ron
> > > >
> > > >
> > > > > It means that this can be done unilaterally, no new
> > > > > specification, just implementation advice.
> > > > >
> > > > > traffic is going along the same path as the ESP traffic.  I'd
> > > > > like to hear from Ron about whether this is an issue.
> > > > >
> > > > > I also thought about using having some plpmtud on each end make
> > > > > a TCP connection *within* the tunnel and use the already defined
> > > > > plpmtu
> > > for TCP.
> > > > > That might be really easy for systems without user/kernel
> > > > > divisions (some router kernels). It would require some
> > > > > notifications to indicate that the responder has a TCP port
> > > > > open.  And of course, the traffic would have to fit into the tunnel as
> well.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >     > Regards,
> > > > >     > Valery.
> > > > >
> > > > >     >> do appropriate PMTU probing and can figure out what the
> > > > > inner
> > > MTU is.
> > > > >     >> There seems to be some sticking points among the
> > > > > high-speed about
> > > > > CDNs:
> > > > >     >> BCP could also bless PLPMTUD for everyone else.  I was
> > > > > pushing for
> > > this

---

From: Ron Bonica <[email-masked]>
Date: Wed, 11 Apr 2018 15:36:39 +0000

Hi Tero,

In 99.9% of cases you are probably correct. If there is an ECMP between the encrypting node and the decrypting node, all ECMPs have the same PMTU.

And because this is true in such a vast majority of cases, none of us have seen a situation where one ECMP has a larger PMTU than then next.

However, we still need to be prepared for that rare situation where one ECMP does have a greater PMTU that the next. Although black swans are rare, they bite very hard!

                                                                                      Ron

> From: Tero Kivinen <[email-masked]>
> Sent: Tuesday, April 10, 2018 6:52 AM
> Cc: Ron Bonica <[email-masked]>; 'Michael Richardson'
> <[email-masked]>; [email-masked]
> > paths can outweigh all. One potential solution to this issue would be
> > to always use UDP encapsulation, but I'm not sure we may impose such a
> > requirement... Your opinion?
> 
> I would just use IKE packets, and ignore the cases where ESP and IKE get
> different routes which have different MTUs. I would expect that in most
> cases if there is something in the middle using different MTUs then the
> routes are not equal cost anymore, thus routing will use only one of the
> routes. Usually the issues with MTU comes with road warriors connecting
> from random locations around the world, and in most of the cases there will
> be NAT for IPv4 in the middle, which will move all traffic to UDP anyways.
> 
> consistently take different routes, and those different routes have different
> MTUs? And does all ESP traffic follow always one route, and all IKE traffic
> follow another route, or is it more random or based on the SPI or something?
> 
> exists, but before we start to make complicated protocols to cope with cases
> which are very rare, I want to get more information.
> 
> If those cases are very rare, I might still want to make sure IPsec works
> somehow, even if it is not as efficient as it could be (i.e., there might be extra
> fragmentation happening or finding proper MTU might take more time).
> --
> [email-masked]

---

From: "Valery Smyslov" <[email-masked]>
Date: Thu, 12 Apr 2018 14:34:05 +0300

Hi Ron,

> - If we do nothing, tunnel performance  is acceptable but suboptimal. We can prevent blackholing by statically
> configuring the tunnel MTU to a sufficiently low value. However, we cannot take advantage of tunnels with
> larger PMTUs.
> 
> - If we use IKE to exchange probes and acks, tunnel performance may become totally unacceptable. In the
> situation where a) IKE messages traverse a different path than encrypted payloads and b) the PMTU
> associated with the IKE path is greater than the PMTU associated with encrypted payload path, we may
> produce an inflated estimate of the Tunnel MTU. This may lead to black holing.

Then it is possible to couple PLMTUD with NAT traversal. In other words - send probes
over IKE, but only if NAT was detected in between (or NAT traversal is forced by local
configuration). In case of NAT traversal all ESP packets will be encapsulated in UDP
and sent over the IKE connection (same UDP ports). So in this case we are sure that
IKE and ESP paths are the same.
 
> So, our probe and acks have to be exchanged over the IPSEC tunnel. At first I thought that the best way to do
> this was by exchanging UDP messages. But you have a point. If we exchanged encrypted ICMP Echo / Echo
> Response messages instead of UDP messages, there would be slightly less code to write. Furthermore, we
> wouldn't need to allocate a new UDP port.

Yes. but if your tunnel allows only TCP (or UDP), then sending ICMP Echo won't work. 
You still need to be able to send probes over all major transport protocol,
otherwise some SPD configurations would prevent this to work.

Regards,
Valery.

---

From: Paul Wouters <[email-masked]>
Date: Thu, 12 Apr 2018 09:51:54 -0400 (EDT)

> - If we use IKE to exchange probes and acks, tunnel performance may become totally unacceptable. In the situation where a) IKE messages traverse a different path than encrypted payloads and b) the…

Using IKE also has a disandvantage for for those implementations that
only support a window size of one. If those IKE messages are lost -
which is highly likely because we are trying out larger window sizes
until we find something that works - things get tricky (even trickier
then the current liveness + mobike situation)

Paul

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 13 Apr 2018 11:14:20 +0300

Ron Bonica writes:
> - If we do nothing, tunnel performance  is acceptable but
>   suboptimal. We can prevent blackholing by statically configuring
>   the tunnel MTU to a sufficiently low value. However, we cannot
>   take advantage of tunnels with larger PMTUs. 
> 
> - If we use IKE to exchange probes and acks, tunnel performance may
>   become totally unacceptable. In the situation where a) IKE
>   messages traverse a different path than encrypted payloads and b)
>   the PMTU associated with the IKE path is greater than the PMTU
>   associated with encrypted payload path, we may produce an inflated
>   estimate of the Tunnel MTU. This may lead to black holing. 

And same happens, if the ESP packet path gots broken, but IKE path
does not. This will lead to black holing. Or if the IKE path is broken
but ESP works, the IKE will tear down the whole IKE SA (along with all
Child SAs) and start over.

Also as we are running TCP or similar inside the IPsec tunnel, that
will most likely also notice that packets do not go through with that
big MTU and will make packets smaller, or does something here disable
end to end PMTUD...

> So, our probe and acks have to be exchanged over the IPSEC tunnel.
> At first I thought that the best way to do this was by exchanging
> UDP messages. But you have a point. If we exchanged encrypted ICMP
> Echo / Echo Response messages instead of UDP messages, there would
> be slightly less code to write. Furthermore, we wouldn't need to
> allocate a new UDP port. 

I am not really convinced about that yet.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 13 Apr 2018 11:18:00 +0300

Ron Bonica writes:
> In 99.9% of cases you are probably correct. If there is an ECMP
> between the encrypting node and the decrypting node, all ECMPs have
> the same PMTU. 

Is it 99.9%, or 99.9999%? 

> And because this is true in such a vast majority of cases, none of
> us have seen a situation where one ECMP has a larger PMTU than then
> next.

If none has not yet been seen it is much closer to 100% than 99.9%.
Depending of course how many setups people have seen... 

> However, we still need to be prepared for that rare situation where
> one ECMP does have a greater PMTU that the next. Although black
> swans are rare, they bite very hard! 

There is also option to say that network is so broken that we fall
back to IPsec over TCP, and in that case both ESP and IKE packets go
inside same TCP stream and MTU discovery is done simlarly for both, so
things work. I.e., that might be acceptable solution for those rare
really broken cases.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 13 Apr 2018 11:20:56 +0300

> then the current liveness + mobike situation)

That is good reason for requiring bigger window size if we do this in
IKE for implementations supporting this. Actually if you do mobike,
you most likely also want to use bigger window sizes...

And of course the protocol needs to be designed to work with IKE
packets. 
-- 
[email-masked]

---

From: Paul Wouters <[email-masked]>
Date: Sun, 15 Apr 2018 22:21:30 -0400 (EDT)

> you most likely also want to use bigger window sizes...

Perhaps such a requirement could help. I still feel the whole msgid
handling in IKEv2 requires some clarification document, or perhaps even
a change in the specification.

Paul