1694671 - Crash in [@ rayon_core::job::{{impl}}::execute<T>]

Status: RESOLVED FIXED

(Core :: Layout, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

[Tracking Requested - why for this release]: Frequent crash

50+ crashes in a single day on release, all on Windows 7 with Firefox 32-bit executable. The signature was already active for previous releases but often <50 crashes for the whole cycle and those crashes also included amd64.

Emilio, can you investigated what's causing these crashes?

Crash report: https://crash-stats.mozilla.org/report/index/fde4fbe4-0f6a-4c77-8b24-546400210224

MOZ_CRASH Reason: called `Option::unwrap()` on a `None` value

Top 10 frames of crashing thread:

0 xul.dll RustMozCrash mozglue/static/rust/wrappers.cpp:16
1 xul.dll mozglue_static::panic_hook mozglue/static/rust/lib.rs:89
2 xul.dll core::ops::function::Fn::call<fn ../e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/ops/function.rs:227
3 xul.dll std::panicking::rust_panic_with_hook ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/std/src/panicking.rs:597
4 xul.dll std::panicking::begin_panic_handler::{{closure}} ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/std/src/panicking.rs:497
5 xul.dll std::sys_common::backtrace::__rust_end_short_backtrace<closure-0, !> ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/std/src/sys_common/backtrace.rs:141
6 xul.dll std::panicking::begin_panic_handler ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/std/src/panicking.rs:495
7 xul.dll core::panicking::panic_fmt ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/core/src/panicking.rs:92
8 xul.dll core::panicking::panic ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/core/src/panicking.rs:50
9 xul.dll rayon_core::job::{{impl}}::execute<closure-0> third_party/rust/rayon-core/src/job.rs:168

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Rayon was updated in 86 (bug 1683294). Probably a rayon regression?

Comment 2

[Emilio Cobos Álvarez (:emilio)]

I filed https://github.com/rayon-rs/rayon/issues/827 to get some more eyes on this.

Comment 3

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1694671 - Downgrade rayon-core to 1.8.1, rayon to 1.4.1. r=nical,jrmuizel

This should help us isolate what rayon changes could've caused this
stability issue on 32-bit windows.

Automatically generated with:

$ cargo update -p rayon --precise 1.4.0 && cargo update -p rayon-core --precise 1.8.0 && ./mach vendor rust

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/689dd6a2949c
Downgrade rayon-core to 1.8.0, rayon to 1.4.0. r=jrmuizel

Comment 5

[Noemi Erli[:noemi_erli]]

Backed out changeset 689dd6a2949c (Bug 1694671) for causing RustMozCrash crashes CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=689dd6a2949c151a646595778fe1ddf371dc9de9&selectedTaskRun=GLKpg4ZuS4ypvz69dLXcFg.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=332918294&repo=autoland&lineNumber=8709
https://treeherder.mozilla.org/logviewer?job_id=332935942&repo=autoland&lineNumber=125268

Backout: https://hg.mozilla.org/integration/autoland/rev/6e30de47d7008b9f21d1cf24fd179710b2750d02

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a31f5ee91094
Downgrade rayon-core to 1.8.1, rayon to 1.4.1. r=jrmuizel

Comment 7

[Sandor Molnar]

https://hg.mozilla.org/mozilla-central/rev/a31f5ee91094

Comment 8

[Release mgmt bot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:emilio, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 9

[Emilio Cobos Álvarez (:emilio)]

We still don't know if/how this will affect the crash rate.

Comment 10

[Julien Cristau [:jcristau]]

Still seeing crashes on 88 beta (https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Drayon_core%3A%3Ajob%3A%3A{{impl}}%3A%3Aexecute<T>&product=Firefox&version=88.0b). On 87 release this is again one of the top crashers.

Comment 11

[Julien Cristau [:jcristau]]

Is there anything else we can do here?

Comment 12

[Emilio Cobos Álvarez (:emilio)]

We can keep bisecting / downgrading I guess. Other than that, we could really use the help of somebody with windows knowledge to see what might be going on, but that might be hard.

Trying to build a win32 rayon stress test and see whether we can reproduce the crash is another thing to do, but it will probably take a bit of time. I can try though.

Comment 13

[Emilio Cobos Álvarez (:emilio)]

I believe bug 1716028 has helped quite a lot here, crash rate since 92 seems way down.

Comment 14

[Emilio Cobos Álvarez (:emilio)]

I think I'll call this fixed for now. We can dig into it if it shows up again.