1699230 - Hard coded MTU of 1200 causes problems

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, enhancement, P3)

Description

[Nils Ohlmeier [:drno]]

Apparently Google has encountered scenarios where the IPv6 minimum MTU of 1280 bytes is used in the wild and with all the extra over head for TURN, DTLS etc the data channel packets arrive truncated https://bugs.chromium.org/p/webrtc/issues/detail?id=12495

I think simply lowering the SCTP MTU here https://searchfox.org/mozilla-central/rev/b7b156e53643f0237f3e98a76e5fc7fa9e3b4e71/netwerk/sctp/datachannel/DataChannel.cpp#869 is not a good solution. As seen in the above link a proper solution should start by looking at the link MTU, and then add TURN overhead if needed (our overhead is going to be bigger then Chromes because we don't support TURN channels yet), DTLS cipher over head.

It is not clear to me if IPv6 automatic path MTU results in the link local MTU reflecting the path MTU.

The best solution would be to use proper path MTU. But to have that working across everything we would have to implement something like https://tools.ietf.org/html/draft-ietf-tram-stun-pmtud-18 but I don't anyone has implemented this yet.

Another important aspect about this is that the final MTUs for the datachannel and SRTP are going to different, because SRTP doesn't need to take the DTLS cipher overhead into account.

The other part which I don't like about Chrome's "solution" is that they lower the MTU for everything based on the IPv6 minimum MTU. AFAK IPv4 doesn't have such a limit, or it is way lower at ~500 bytes. And I don't fully understand if the Google folks have not encountered any problems with IPv4 MTU at all (which I find unlikely) or are not "seeing" them.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

FYI chrome's code: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/media/sctp/usrsctp_transport.cc;l=71

Comment 2

[mohkar]

Attachment: Screenshot from 2022-12-08 15-23-20.png

Comment 3

[mohkar]

This bug is still present in Firefox and is consistently reproducible. We reproduce this by sending a ~1300 bytes large data channel message on an IPv6 network. Wireshark shows the first packet sent has a 1249 bytes large UDP payload. Adding 8 bytes of UDP header and 40 bytes of IPv6 header adds up to a 1297 bytes IPv6 packet, which exceeds the minimum MTU of 1280 bytes. This leads to those packets being dropped in some situations.

This commit has more details on how we ended up with a 1191 bytes MTU.
https://webrtc-review.googlesource.com/c/src/+/208642

This can be fixed by reducing the hard coded MTU or using path discovery. In the former case, switching from libusrsctp to dcSCTP (also used in Chrome) could be considered.

I have attached a screenshot from Wireshark. Let me know if I can help with anything.

Comment 4

[Byron Campen [:bwc]]

(In reply to mohkar from comment #3)

This bug is still present in Firefox and is consistently reproducible. We reproduce this by sending a ~1300 bytes large data channel message on an IPv6 network. Wireshark shows the first packet sent has a 1249 bytes large UDP payload. Adding 8 bytes of UDP header and 40 bytes of IPv6 header adds up to a 1297 bytes IPv6 packet, which exceeds the minimum MTU of 1280 bytes. This leads to those packets being dropped in some situations.

This commit has more details on how we ended up with a 1191 bytes MTU.
https://webrtc-review.googlesource.com/c/src/+/208642

This can be fixed by reducing the hard coded MTU or using path discovery. In the former case, switching from libusrsctp to dcSCTP (also used in Chrome) could be considered.

I have attached a screenshot from Wireshark. Let me know if I can help with anything.

So, it looks like this field:

https://searchfox.org/mozilla-central/rev/5349e0db4db34dabecced362a59ea7528dcc7afc/netwerk/sctp/datachannel/DataChannel.cpp#870

is misnamed:

https://searchfox.org/mozilla-central/rev/5349e0db4db34dabecced362a59ea7528dcc7afc/netwerk/sctp/src/netinet/sctp_usrreq.c#6336

That code is adding the size of the packet overhead to the value we supply, and configuring that number as the MTU. It appears that this field is really just the maximum SCTP payload size we want. So what we really want (assuming Google's calculations are accurate) is:

1280
-12 (sctp header)
-24 (GCM cipher)
-13 (DTLS record header)
-8 (UDP header)
-4 (TURN ChannelSend)
-40 (IPV6 header)
= 1179

Comment 5

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/jobs?repo=try&revision=6ee7f019b5767f6df51e06c26edaadb6b904e83b
https://treeherder.mozilla.org/jobs?repo=try&revision=6183f9228a4beea7a751031dab67f262254ab5cb

Comment 6

[Byron Campen [:bwc]]

Attachment: Bug 1699230: Decrease the SCTP MTU. r?ng

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/981fa0861c53
Decrease the SCTP MTU. r=ng

Comment 8

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/981fa0861c53