1621360 - Crash in [@ trunc | cubeb_backend::capi::capi_stream_init<T>]

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P1)

Description

[[:philipp]]

This bug is for crash report bp-13691418-d971-4aba-b7db-93b590200310.

Top 10 frames of crashing thread:

0 xul.dll RustMozCrash mozglue/static/rust/wrappers.cpp:16
1 xul.dll mozglue_static::panic_hook mozglue/static/rust/lib.rs:89
2 xul.dll core::ops::function::Fn::call<fn src/libcore/ops/function.rs:232
3 xul.dll std::panicking::rust_panic_with_hook src/libstd/panicking.rs:475
4 xul.dll std::panicking::begin_panic_handler src/libstd/panicking.rs:375
5 xul.dll core::panicking::panic_fmt src/libcore/panicking.rs:84
6 xul.dll trunc 
7 xul.dll cubeb_backend::capi::capi_stream_init<audioipc_client::context::ClientContext> third_party/rust/cubeb-backend/src/capi.rs:155
8 xul.dll cubeb_stream_init media/libcubeb/src/cubeb.c:335
9 xul.dll mozilla::AudioStream::OpenCubeb dom/media/AudioStream.cpp:322

this content crash signature is newly showing up in the 75 cycle - so far it only seems to hit 32bit users on windows.

MOZ_CRASH Reason is commonly showing calledResult::unwrap()on anErrvalue: Error(Io(Os { code: 0, kind: Other, message: "The operation completed successfully." }), State { next_error: None }) or translated variations thereof...

Comment 1

[Matthew Gregan [:kinetik]]

This must be fallout from bug 1432303, otherwise cubeb_backend::capi::capi_stream_init wouldn't be in the stack. Unfortunately, the stack has nothing useful after that to narrow down where this is failing.

Comment 2

[Matthew Gregan [:kinetik]]

trunc seems to be bad unwinding, I can't see how that's a real possible stack frame at that point. cubeb_backend::capi::capi_stream_init@155 means we're within audioipc's ClientStream::init when we hit the unexpected unwrap().

The error implies we turned an ERROR_SUCCESS (0x0) into an std::io::Error somewhere, which seems wrong but is unclear. Given this is 32-bit only, it could be related to handling of shmem and hitting virtual address space limits - although in each of the 21 crashes, the virtual memory is > 550MB and usually > 800MB, so perhaps not.

Comment 3

[Julien Cristau [:jcristau]]

This crash is pretty bad on 75 (#2 top content crash), I don't think we can ship with it.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

This bug is tracked by a release manager but with a small severity so change it to major.
For more information, please visit auto_nag documentation.

Comment 5

[Matthew Gregan [:kinetik]]

Working on reproducing and debugging locally.

Worst case plan: we can pref off AudioIPC for 32-bit Windows only to buy time, but suspect that's not ideal for sandboxing team's plans.

Comment 6

[Matthew Gregan [:kinetik]]

Attachment: GitHub Pull Request

I couldn't reproduce this locally with a 32-bit Windows build during normal/heavy browser use. Following my guess from comment 2 on virtual address space limits, I can reproduce this by repeatedly loading a page of 200 autoplaying videos.

After the third or fourth attempt to load the page, the content tab will crash with a similar stack to the crash reports. Caught in a debugger with better stack unwinding, it's clear that the panicking unwrap is during shmem setup for a client stream. Inside memmap-rs, MapViewOfFile returns nullptr (with a GetLastError of 8, which is ERROR_NOT_ENOUGH_MEMORY).

The linked PR resolves this by treating shmem setup failure on the client as non-fatal, instead aborting the audio stream setup.

I'm still not sure why the unwrapped error in the crash report shows the incorrect error number and name. I originally thought the last error was clobbered by an interleaved CloseHandle, but since verified that was not the issue. The Error constructed after MapViewOfFile fails contains an OsRepr(8), as expected, but by the time the error has been converted/returned/unwrapped (possibly because std::io:Error treats this type as an unknown "Other" error and suffers a lossy conversion?) it has an internal error code of 0 which results in the error message string being "The operation completed successfully." when formatted.

Comment 9

[Julien Cristau [:jcristau]]

Matthew, would you mind filing a bug to disable audioipc on 32-bit windows in 75? That seems like the safest course at this point.

Comment 10

[Matthew Gregan [:kinetik]]

(In reply to Julien Cristau [:jcristau] from comment #9)

Matthew, would you mind filing a bug to disable audioipc on 32-bit windows in 75? That seems like the safest course at this point.

https://bugzilla.mozilla.org/show_bug.cgi?id=1623793

Comment 11

[(PTO June 19-July 7) Ryan VanderMeulen]

AFAICT, we haven't seen any crashes on Nightly since bug 1623799 landed, and the crashes went away on Beta as expected after bug 1623793 landed there. How long did you want to give this bug before declaring victory?

Comment 12

[Matthew Gregan [:kinetik]]

I think we can mark this fixed. There is a possibility that this fix converts crashes into non-functioning audio for users that would've crashed, as the low address space explanation for the crash doesn't seem to fit the memory reports for all crashes, but I'm unsure how reliable those stats are. I'll keep an eye on reports for Nightly.

Comment 13

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.