1916629 - Crash in [@ mio::sys::windows::named_pipe::read_done]

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

6 crashes from 6 installs of Firefox 130.0 on Windows, 4 crashes for 130.0b9. 2 isolated crash reports for versions <130.

Paul: Do you know what changed in v130 what could trigger this?

Crash report: https://crash-stats.mozilla.org/report/index/9c71526a-047c-4906-bc57-9fa840240904

MOZ_CRASH Reason: internal error: entered unreachable code

Top 10 frames:

0  xul.dll  MOZ_Crash(char const*, int, char const*)  mfbt/Assertions.h:317
0  xul.dll  RustMozCrash(char const*, int, char const*)  mozglue/static/rust/wrappers.cpp:18
1  xul.dll  mozglue_static::panic_hook(core::panic::panic_info::PanicInfo*)  mozglue/static/rust/lib.rs:102
2  xul.dll  core::ops::function::Fn::call<void (*)(ref$<core::panic::panic_info::PanicInf...  /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250
3  xul.dll  alloc::boxed::impl$50::call()  library/alloc/src/boxed.rs:2036
3  xul.dll  std::panicking::rust_panic_with_hook()  library/std/src/panicking.rs:799
4  xul.dll  std::panicking::begin_panic_handler::closure$0()  library/std/src/panicking.rs:656
5  xul.dll  std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_...  library/std/src/sys_common/backtrace.rs:171
6  xul.dll  std::panicking::begin_panic_handler()  library/std/src/panicking.rs:652
7  xul.dll  core::panicking::panic_fmt()  library/core/src/panicking.rs:72

 8 	xul.dll	core::panicking::panic()	library/core/src/panicking.rs:146
9 	xul.dll	mio::sys::windows::named_pipe::read_done(windows_sys::Win32::System::IO::OVERLAPPED_ENTRY*, enum2$<core::option::Option<ref_mut$<alloc::vec::Vec<mio::sys::windows::event::Event, alloc::alloc::Global> > > >)	third_party/rust/mio/src/sys/windows/named_pipe.rs:871
10 	xul.dll	mio::sys::windows::selector::SelectorInner::feed_events(alloc::vec::Vec<mio::sys::windows::event::Event, alloc::alloc::Global>*, ref$<slice2$<mio::sys::windows::iocp::CompletionStatus> >)	third_party/rust/mio/src/sys/windows/selector.rs:503
10 	xul.dll	mio::sys::windows::selector::SelectorInner::select2(ref_mut$<slice2$<mio::sys::windows::iocp::CompletionStatus> >, alloc::vec::Vec<mio::sys::windows::event::Event, alloc::alloc::Global>*, enum2$<core::option::Option<core::time::Duration> >)	third_party/rust/mio/src/sys/windows/selector.rs:463
11 	xul.dll	mio::sys::windows::selector::SelectorInner::select(mio::sys::windows::event::Events*, enum2$<core::option::Option<core::time::Duration> >)	third_party/rust/mio/src/sys/windows/selector.rs:436
11 	xul.dll	mio::sys::windows::selector::Selector::select(mio::sys::windows::event::Events*, enum2$<core::option::Option<core::time::Duration> >)	third_party/rust/mio/src/sys/windows/selector.rs:359
11 	xul.dll	mio::poll::Poll::poll(mio::event::events::Events*, enum2$<core::option::Option<core::time::Duration> >)	third_party/rust/mio/src/poll.rs:435
12 	xul.dll	audioipc2::ipccore::EventLoop::poll()	third_party/rust/audioipc2/src/ipccore.rs:198
13 	xul.dll	audioipc2::ipccore::impl$7::new::closure$0(audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>)	third_party/rust/audioipc2/src/ipccore.rs:706
13 	xul.dll	std::sys_common::backtrace::__rust_begin_short_backtrace<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > >(audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/sys_common/backtrace.rs:155
14 	xul.dll	std::thread::impl$0::spawn_unchecked_::closure$2::closure$0(std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > >)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/thread/mod.rs:540
14 	xul.dll	core::panic::unwind_safe::impl$25::call_once(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > > >)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panic/unwind_safe.rs:272
14 	xul.dll	std::panicking::try::do_call(unsigned char*)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:559
14 	xul.dll	std::panicking::try(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > > >)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:523
14 	xul.dll	std::panic::catch_unwind(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > > >)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs:149
14 	xul.dll	std::thread::impl$0::spawn_unchecked_::closure$2(std::thread::impl$0::spawn_unchecked_::closure_env$2<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > >)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/thread/mod.rs:541
14 	xul.dll	core::ops::function::FnOnce::call_once<std::thread::impl$0::spawn_unchecked_::closure_env$2<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > >, tuple$<> >(std::thread::impl$0::spawn_unchecked_::closure_env$2<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum2$<core::result::Result<tuple$<>, std::io::error::Error> > >*)	/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250
15 	xul.dll	alloc::boxed::impl$48::call_once()	library/alloc/src/boxed.rs:2022
15 	xul.dll	alloc::boxed::impl$48::call_once()	library/alloc/src/boxed.rs:2022
15 	xul.dll	std::sys::pal::windows::thread::impl$0::new::thread_start()	library/std/src/sys/pal/windows/thread.rs:52 

Comment 1

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 20 desktop browser crashes on release
• Top 10 content process crashes on release

:jimm, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 2

[Dianna Smith [:diannaS]]

:jimm any luck investigating here? Looks like the volume went up a little since first reported.

Comment 3

[Matthew Gregan [:kinetik]]

No major changes in AudioIPC between Firefox 129 and 130, but mio was updated from 0.8.8 to 1.0.1 in bug 1910523 (AudioIPC dependency change) and bug 1910513 (actual mio update) which landed in 130. That mio update includes this fix to the named pipe code, which is likely related to the crash spike. I'm not sure if that change caused the spike or revealed an existing bug in AudioIPC; I'll investigate further.

Comment 4

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1910523

Comment 5

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1910523

Comment 6

[Donal Meehan [:dmeehan]]

:kinetik have there been any updates since Comment 3?

Comment 7

[Matthew Gregan [:kinetik]]

Apologies for the delay. I'm working through the process of getting a fix upstreamed to mio for this, so will hopefully have a fix available soon.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Can you please add a See Also to the mio fix you're trying to get landed?

Comment 9

[Gabriele Svelto [:gsvelto]]

FYI in case it might: we collect the value that would have been returned by GetLastError() in nightly builds and all the crashes in the past six months had that value set to ERROR_PIPE_NOT_CONNECTED with the exception of one which had ERROR_BROKEN_PIPE (see the "reports" tab in this query). The latter is also the only 32-bit crash we have on file so it's likely the same issue being reported differently by Windows.

Comment 10

[Matthew Gregan [:kinetik]]

It's likely this is fixed by bug 2024485, but I'll keep an eye on the crash reports to confirm.

Comment 11

[Matthew Gregan [:kinetik]]

Confirmed: bug 2024485 fixed the internal error: entered unreachable code
panic this bug tracked.

Last 90 days on Firefox 150 release:
150.0: 2 crashes (both non-panic, see below)
150.0.1: 0
150.0.2: 0
150.0.3: 0

Nightly (last 60 days): 0.
Beta (last 60 days): 5, all on pre-fix builds (140.0b1, 149.0b9) reporting
late.

The 2 remaining 150.0 crashes are different failure modes, not the
unreachable!() panic:

7be8a678-d2da-4c40-93a7-55b700260424: EXCEPTION_ILLEGAL_INSTRUCTION at
0x00007fffe7cc9dff with no Rust panic message - a hard CPU fault.

dd317b66-fe00-4afc-8386-338f70260422: EXCEPTION_BREAKPOINT with panic
message "called Result::unwrap() on an Err value: PoisonError { .. }" -
a secondary failure from a poisoned mutex, not the original panic.

Both will continue to be reported under this signature since the top stack
frame is unchanged, but they represent different problems. Will file
separately if volume grows.

ESR 140 was not uplifted (wontfix) and continues to produce ~2500+
read_done panics per dot release - users there remain affected.