1912626 - Crash in [@ syscall | quinn_udp::imp::recvmmsg_with_fallback]

Status: RESOLVED FIXED

(Core :: Networking, defect, P3)

Description

[BugBot [:suhaib / :marco/ :calixte]]

Crash report: https://crash-stats.mozilla.org/report/index/321a62c5-1e6c-4af9-971d-4128a0240804

Reason: SIGSYS / SYS_SECCOMP

Top 10 frames of crashing thread:

0  libc.so.6  syscall  /usr/src/debug/glibc-2.39-17.fc40.x86_64/sysdeps/unix/sysv/linux/x86_64/syscall.S:38
1  libxul.so  quinn_udp::imp::recvmmsg_with_fallback  third_party/rust/quinn-udp/src/unix.rs:453
1  libxul.so  quinn_udp::imp::recv  third_party/rust/quinn-udp/src/unix.rs:391
1  libxul.so  quinn_udp::imp::UdpSocketState::recv  third_party/rust/quinn-udp/src/unix.rs:188
2  libxul.so  neqo_udp::recv_inner::{{closure}}  third_party/rust/neqo-udp/src/lib.rs:64
2  libxul.so  std::thread::local::LocalKey<core::cell::RefCell<T>>::with_borrow_mut::{{closure}}  library/std/src/thread/local.rs:508
2  libxul.so  std::thread::local::LocalKey<T>::try_with  library/std/src/thread/local.rs:286
2  libxul.so  std::thread::local::LocalKey<T>::with  library/std/src/thread/local.rs:262
2  libxul.so  std::thread::local::LocalKey<core::cell::RefCell<T>>::with_borrow_mut  library/std/src/thread/local.rs:508
2  libxul.so  neqo_udp::recv_inner  third_party/rust/neqo-udp/src/lib.rs:61

By querying Nightly crashes reported within the last 2 months, here are some insights about the signature:

• First crash report: 2024-08-04
• Process type: Socket
• Is startup crash: No
• Has user comments: No
• Is null crash: Yes - all crashes happened on null or near null memory address

Comment 1

[Max Inden]

Thank you for the ping. I am taking a look now.

Comment 2

[Max Inden]

Some context:

• This has not shown up on manual nor automated testing on Linux x86-64.
• quinn-udp has been introduced in https://bugzilla.mozilla.org/show_bug.cgi?id=1901295.
• The long term plan is to replace NSPR with quinn-udp for QUIC I/O. See https://bugzilla.mozilla.org/show_bug.cgi?id=1901292.
• It is currently only enabled on Firefox Nightly.

    # Use NSPR for HTTP3 UDP IO
  - name: network.http.http3.use_nspr_for_io
    type: RelaxedAtomicBool
    # Always use NSPR on Android x86. On Android x86 sendmsg and recvmmsg, both used
    # by quinn-udp, are disallowed by seccomp. Future fix and further details
    # tracked in https://github.com/quinn-rs/quinn/issues/1947.
    #if defined(ANDROID) && !defined(HAVE_64BIT_BUILD)
    value: true
    #else
    value: @IS_NOT_NIGHTLY_BUILD@
    #endif
    mirror: always
  

  https://searchfox.org/mozilla-central/rev/03258de701dbcde998cfb07f75dce2b7d8fdbe20/modules/libpref/init/StaticPrefList.yaml#13590-13601
• Setting network.http.http3.use_nspr_for_io to true would prevent the Bug reported here.
• This might be related to a previous seccomp failure on Android x86.
  • See Android x86 CI failure reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1910594
  • See upstream bug report https://github.com/quinn-rs/quinn/issues/1947.
  • See proposed solution upstream https://github.com/quinn-rs/quinn/pull/1964.

I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set network.http.http3.use_nspr_for_io to true?

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

This is a moderately significant signal from Nightly; it doesn't require an immediate rollback, but I'd be cautious about moving it to beta with this. I wonder if this can be isolated down more in terms of platform/distro/etc

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

https://crash-stats.mozilla.org/signature/?signature=syscall%20%7C%20quinn_udp%3A%3Aimp%3A%3Arecvmmsg_with_fallback&date=%3E%3D2024-05-13T10%3A32%3A00.000Z&date=%3C2024-08-13T10%3A32%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=install_time&_columns=platform_pretty_version&_sort=-date&page=1
Looks like Arch, Fedora 40, Ubuntu Oriole development branch. Might want to check kernel versions/etc for those.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

With OS versions:
https://crash-stats.mozilla.org/signature/?signature=syscall%20%7C%20quinn_udp%3A%3Aimp%3A%3Arecvmmsg_with_fallback&date=%3E%3D2024-05-13T10%3A32%3A00.000Z&date=%3C2024-08-13T10%3A32%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=install_time&_columns=platform_pretty_version&_columns=platform_version&_sort=-date&page=1

Comment 6

[Kershaw Chang [:kershaw] PTO until Jan 2]

I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set network.http.http3.use_nspr_for_io to true?

No, I think those crashes are fine. Mostly because all of these crashes are from the socket process. Basically, these crashes in the socket process mean that users have manually set the preference network.http.network_access_on_socket_process.enabled to true (it's false by default). When the preference is false, the parent process makes the HTTP/3 connection, not the socket process.

For what it's worth, the reason for these crashes might be caused by sandboxing rules, but I'm not sure.

Comment 7

[Max Inden]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #3)

This is a moderately significant signal from Nightly; it doesn't require an immediate rollback, but I'd be cautious about moving it to beta with this. I wonder if this can be isolated down more in terms of platform/distro/etc

Thank you Randell for the additional context!

---

(In reply to Kershaw Chang [:kershaw] from comment #6)

I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set network.http.http3.use_nspr_for_io to true?

No, I think those crashes are fine. Mostly because all of these crashes are from the socket process. Basically, these crashes in the socket process mean that users have manually set the preference network.http.network_access_on_socket_process.enabled to true (it's false by default). When the preference is false, the parent process makes the HTTP/3 connection, not the socket process.

For what it's worth, the reason for these crashes might be caused by sandboxing rules, but I'm not sure.

This is a great hint. Thank you Kershaw! I can reproduce the bug now.

I will work on a patch next. I assume extending SocketProcessSandboxPolicy::EvaluateSocketCall needs a Some(Allow()) for sendmsg and recvmmsg.

https://searchfox.org/mozilla-central/source/security/sandbox/linux/SandboxFilter.cpp#2035-2045

Comment 8

[Max Inden]

Attachment: Bug 1912626 - allow sendmsg and recvmmsg in SocketProcessSandboxPolicy r=kershaw

https://bugzilla.mozilla.org/show_bug.cgi?id=1901295 introduced optional
HTTP3/QUIC UDP IO via quinn-udp instead of NSPR, see
network.http.http3.use_nspr_for_io pref. NSPR uses sendto and recvfrom
syscall, quinn-udp uses sendmsg and recvmmsg syscall.

With network.http.network_access_on_socket_process.enabled true and
network.http.http3.use_nspr_for_io false Firefox panics due to seccomp
disallowing a recvmmsg syscall.

This commit allows sendmsg and recvmmsg in the SocketProcessSandboxPolicy
EvaluateSocketCall function.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5f3a2655d2f4
allow sendmsg and recvmmsg in SocketProcessSandboxPolicy r=gerard-majax

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5f3a2655d2f4