Crash in [@ syscall | quinn_udp::imp::recvmmsg_with_fallback]
Categories
(Core :: Networking, defect, P3)
Tracking
()
Tracking | Status | |
---|---|---|
firefox-esr115 | --- | unaffected |
firefox-esr128 | --- | unaffected |
firefox129 | --- | unaffected |
firefox130 | --- | disabled |
firefox131 | --- | fixed |
People
(Reporter: release-mgmt-account-bot, Assigned: mail)
References
(Blocks 2 open bugs, Regression)
Details
(Keywords: crash, regression, Whiteboard: [necko-triaged])
Crash Data
Attachments
(1 file)
Crash report: https://crash-stats.mozilla.org/report/index/321a62c5-1e6c-4af9-971d-4128a0240804
Reason: SIGSYS / SYS_SECCOMP
Top 10 frames of crashing thread:
0 libc.so.6 syscall /usr/src/debug/glibc-2.39-17.fc40.x86_64/sysdeps/unix/sysv/linux/x86_64/syscall.S:38
1 libxul.so quinn_udp::imp::recvmmsg_with_fallback third_party/rust/quinn-udp/src/unix.rs:453
1 libxul.so quinn_udp::imp::recv third_party/rust/quinn-udp/src/unix.rs:391
1 libxul.so quinn_udp::imp::UdpSocketState::recv third_party/rust/quinn-udp/src/unix.rs:188
2 libxul.so neqo_udp::recv_inner::{{closure}} third_party/rust/neqo-udp/src/lib.rs:64
2 libxul.so std::thread::local::LocalKey<core::cell::RefCell<T>>::with_borrow_mut::{{closure}} library/std/src/thread/local.rs:508
2 libxul.so std::thread::local::LocalKey<T>::try_with library/std/src/thread/local.rs:286
2 libxul.so std::thread::local::LocalKey<T>::with library/std/src/thread/local.rs:262
2 libxul.so std::thread::local::LocalKey<core::cell::RefCell<T>>::with_borrow_mut library/std/src/thread/local.rs:508
2 libxul.so neqo_udp::recv_inner third_party/rust/neqo-udp/src/lib.rs:61
By querying Nightly crashes reported within the last 2 months, here are some insights about the signature:
- First crash report: 2024-08-04
- Process type: Socket
- Is startup crash: No
- Has user comments: No
- Is null crash: Yes - all crashes happened on null or near null memory address
Updated•1 month ago
|
Some context:
- This has not shown up on manual nor automated testing on Linux x86-64.
quinn-udp
has been introduced in https://bugzilla.mozilla.org/show_bug.cgi?id=1901295.- The long term plan is to replace NSPR with
quinn-udp
for QUIC I/O. See https://bugzilla.mozilla.org/show_bug.cgi?id=1901292. - It is currently only enabled on Firefox Nightly.
https://searchfox.org/mozilla-central/rev/03258de701dbcde998cfb07f75dce2b7d8fdbe20/modules/libpref/init/StaticPrefList.yaml#13590-13601# Use NSPR for HTTP3 UDP IO - name: network.http.http3.use_nspr_for_io type: RelaxedAtomicBool # Always use NSPR on Android x86. On Android x86 sendmsg and recvmmsg, both used # by quinn-udp, are disallowed by seccomp. Future fix and further details # tracked in https://github.com/quinn-rs/quinn/issues/1947. #if defined(ANDROID) && !defined(HAVE_64BIT_BUILD) value: true #else value: @IS_NOT_NIGHTLY_BUILD@ #endif mirror: always
- Setting
network.http.http3.use_nspr_for_io
totrue
would prevent the Bug reported here. - This might be related to a previous seccomp failure on Android x86.
- See Android x86 CI failure reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1910594
- See upstream bug report https://github.com/quinn-rs/quinn/issues/1947.
- See proposed solution upstream https://github.com/quinn-rs/quinn/pull/1964.
I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set network.http.http3.use_nspr_for_io
to true
?
Updated•1 month ago
|
Comment 3•1 month ago
|
||
This is a moderately significant signal from Nightly; it doesn't require an immediate rollback, but I'd be cautious about moving it to beta with this. I wonder if this can be isolated down more in terms of platform/distro/etc
Comment 4•1 month ago
|
||
https://crash-stats.mozilla.org/signature/?signature=syscall%20%7C%20quinn_udp%3A%3Aimp%3A%3Arecvmmsg_with_fallback&date=%3E%3D2024-05-13T10%3A32%3A00.000Z&date=%3C2024-08-13T10%3A32%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=install_time&_columns=platform_pretty_version&_sort=-date&page=1
Looks like Arch, Fedora 40, Ubuntu Oriole development branch. Might want to check kernel versions/etc for those.
Comment 5•1 month ago
|
||
Comment 6•1 month ago
|
||
I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set
network.http.http3.use_nspr_for_io
totrue
?
No, I think those crashes are fine. Mostly because all of these crashes are from the socket process. Basically, these crashes in the socket process mean that users have manually set the preference network.http.network_access_on_socket_process.enabled
to true (it's false by default). When the preference is false, the parent process makes the HTTP/3 connection, not the socket process.
For what it's worth, the reason for these crashes might be caused by sandboxing rules, but I'm not sure.
Updated•1 month ago
|
(In reply to Randell Jesup [:jesup] (needinfo me) from comment #3)
This is a moderately significant signal from Nightly; it doesn't require an immediate rollback, but I'd be cautious about moving it to beta with this. I wonder if this can be isolated down more in terms of platform/distro/etc
Thank you Randell for the additional context!
(In reply to Kershaw Chang [:kershaw] from comment #6)
I am new to Mozilla. Thus far I see a total of 18 reports. Is this reason enough to role back the feature on Nightly, i.e. set
network.http.http3.use_nspr_for_io
totrue
?No, I think those crashes are fine. Mostly because all of these crashes are from the socket process. Basically, these crashes in the socket process mean that users have manually set the preference
network.http.network_access_on_socket_process.enabled
to true (it's false by default). When the preference is false, the parent process makes the HTTP/3 connection, not the socket process.For what it's worth, the reason for these crashes might be caused by sandboxing rules, but I'm not sure.
This is a great hint. Thank you Kershaw! I can reproduce the bug now.
I will work on a patch next. I assume extending SocketProcessSandboxPolicy::EvaluateSocketCall
needs a Some(Allow())
for sendmsg
and recvmmsg
.
https://searchfox.org/mozilla-central/source/security/sandbox/linux/SandboxFilter.cpp#2035-2045
https://bugzilla.mozilla.org/show_bug.cgi?id=1901295 introduced optional
HTTP3/QUIC UDP IO via quinn-udp instead of NSPR, see
network.http.http3.use_nspr_for_io
pref. NSPR uses sendto
and recvfrom
syscall, quinn-udp
uses sendmsg
and recvmmsg
syscall.
With network.http.network_access_on_socket_process.enabled
true
and
network.http.http3.use_nspr_for_io
false
Firefox panics due to seccomp
disallowing a recvmmsg
syscall.
This commit allows sendmsg
and recvmmsg
in the SocketProcessSandboxPolicy
EvaluateSocketCall
function.
Updated•1 month ago
|
Comment 10•1 month ago
|
||
bugherder |
Updated•1 month ago
|
Description
•