Subject: [rtcweb] SCTP issue: Datachannel MTU seems to be problematic.

From: Harald Alvestrand <[email-masked]>
Date: Mon, 1 Mar 2021 20:17:54 +0100

This for info:

We have recently discovered some problems with SCTP on small-MTU networks.

Turns out we underestimated the max size of the encapsulating headers.

Current calculation:

// The biggest SCTP packet. Starting from a 'safe' wire MTU value of 1280,
// take off 85 bytes for DTLS/TURN/TCP/IP and ciphertext overhead.
//
// Additionally, it's possible that TURN adds an additional 4 bytes of 
overhead
// after a channel has been established, so we subtract an additional 4 
bytes.
//
// 1280 IPV6 MTU
//  -40 IPV6 header
//   -8 UDP
//  -24 GCM Cipher
//  -13 DTLS record header
//   -4 TURN ChannelData
// = 1191 bytes.

Which is somewhat smaller than the number we assumed in the RFC of 1200 
bytes.

We didn't discover the issue until we encountered a situation with large 
packets being sent on IPv6-only networks.

Harald

---

From: Nils Ohlmeier <[email-masked]>
Date: Mon, 1 Mar 2021 14:53:42 -0800

Hi Harald,

> On 1Mar, 2021, at 11:17, Harald Alvestrand <[email-masked]> wrote:
> 
> This for info:
> 

Interesting problem you encountered there.

> // Additionally, it's possible that TURN adds an additional 4 bytes of overhead
> // after a channel has been established, so we subtract an additional 4 bytes.
> //
> // 1280 IPV6 MTU
> //  -40 IPV6 header
> //   -8 UDP
> //  -24 GCM Cipher
> Which is somewhat smaller than the number we assumed in the RFC of 1200 bytes.

What I don’t understand here is that this calculation is only done for IPv6, which has a significantly higher minimum MTU compared to IPv4.

> We didn't discover the issue until we encountered a situation with large packets being sent on IPv6-only networks.

So this IPv6 only network operated at 1280 bytes MTU?
But you would have encountered the same issue on an IPv4 network with an MTU of ~1200 bytes as well, right?

In other words if would do the safe thing here and use the way smaller IPv4 minimum MTU, we would end up with super small values.
So I think the sane, but painful solution to this problem is to actually adjust the data channel MTU based on actual facts about the link (IP version, cipher in use, TURN channel vs data indication) rather then use a hard coded value.

Best regards
  Nils Ohlmeier

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 2 Mar 2021 00:09:22 +0100

> On 1. Mar 2021, at 23:53, Nils Ohlmeier <[email-masked]> wrote:
> 
> Hi Harald,
> 
I was assuming that an IPv6 network is used with an MTU of 1280 to provide IPv4 connectivity via a TURN service.
So the IPv4 level MTU would be 1191, which is smaller than 1200 as mentioned in the RFC... Or am I wrong?

Best regards
Michael
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 2 Mar 2021 08:42:01 +0100

> So I think the sane, but painful solution to this problem is to actually adjust the data channel MTU based on actual facts about the link (IP version, cipher in use, TURN channel vs data indication…

IPv4 supports in-network fragmentation of overlarge datagrams (as long 
as you don't use the DF flag). So it would cause a performance penalty, 
but would not break communication.

And the practical IPv4 MTU on the Internet is 1480, not 572 (Ethernet 
MTU - typical tunneling overhead).

There's an outstanding bug in usrsctp 
(https://github.com/sctplab/usrsctp/issues/205) about SCTP path MTU 
discovery; the short version seems to be "it doesn't work".

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 2 Mar 2021 08:53:03 +0100

> So the IPv4 level MTU would be 1191, which is smaller than 1200 as mentioned in the RFC... Or am I wrong?

The actual problem we discovered in production was in a case without 
TURN, but with a client that did IPv6 only. (It may also be client OS 
specific - we're not sure about that.)

So we could probably have gotten away with an MTU of 1195 bytes.

I'm also not sure if the 24 bytes of GCM cipher overhead is the right 
number for all ciphersuites that we permit; we may have to keep an eye 
on that.

But yes, we need Path MTU discovery to work, including a way to recover 
from too-large initial assumptions.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 2 Mar 2021 11:58:41 +0100

> On 2. Mar 2021, at 08:42, Harald Alvestrand <[email-masked]> wrote:
> 
> 
> IPv4 supports in-network fragmentation of overlarge datagrams (as long as you don't use the DF flag). So it would cause a performance penalty, but would not break communication.
> 
> And the practical IPv4 MTU on the Internet is 1480, not 572 (Ethernet MTU - typical tunneling overhead).
> 
> There's an outstanding bug in usrsctp (https://github.com/sctplab/usrsctp/issues/205) about SCTP path MTU discovery; the short version seems to be "it doesn't work".
The version which is implemented controls the DF bit and relies on ICMP feedback.
That doesn't work with the SCTP/DTLS stack. I'm also not sure if WebRTC implementations
turn off PMTUD.

We do have code for doing PLPMTUD without relying ICMP feedback, which this is not
merged yet and Timo is looking into different search strategies. However, all of them
assume a safe lower bound of the PMTUD. so knowing that 1200 is not a good lower
bound is interesting...

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 2 Mar 2021 12:00:52 +0100

> On 2. Mar 2021, at 08:53, Harald Alvestrand <[email-masked]> wrote:
> 
> 
> The actual problem we discovered in production was in a case without TURN, but with a client that did IPv6 only. (It may also be client OS specific - we're not sure about that.)
But in that case the specification says the IP level PMTU is at least 1280. The
overhead related to the protocol between IPv6 and SCTP is not specified in the RFC.

Best regards
Michael
> I'm also not sure if the 24 bytes of GCM cipher overhead is the right number for all ciphersuites that we permit; we may have to keep an eye on that.
> 
> But yes, we need Path MTU discovery to work, including a way to recover from too-large initial assumptions.
> 
> 
> 
> 
>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 8 Mar 2021 10:19:54 +0100

> overhead related to the protocol between IPv6 and SCTP is not specified in the RFC.

It's implied by the stack in RFC 8835 section 3.5 (rtcweb-transport); we 
just never worked out the right number from that.