1524951 - Cranelift: Crash in js::WasmMemoryObject::volatileMemoryLength

Status: RESOLVED FIXED

(Core :: JavaScript: WebAssembly, defect, P3)

Description

[Gabriele Svelto [:gsvelto] (PTO)]

This bug is for crash report bp-071b4df6-565f-4f52-9e09-6b6f70190203.

Top 5 frames of crashing thread:

0 xul.dll js::WasmMemoryObject::volatileMemoryLength js/src/wasm/WasmJS.cpp:1798
1 xul.dll js::wasm::Instance::currentMemory_i32 js/src/wasm/WasmInstance.cpp:330
2  @0x3e7e04d1d4d 
3  @0x1c27b9496ff 
4 xul.dll js::LiveSavedFrameCache::~LiveSavedFrameCache js/src/vm/Stack.h:1314

The stacks on this one are unhelpful so I'm unsure if it's a valid bug or just a new signature due to flaky hardware.

Comment 1

[Marcia Knous [:marcia]]

Here is the potential regression range based on the Build ID: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9ee54a21a22ab5beab264bcabe3c8039a27a32e8&tochange=024bef408a8896f365789759f6a3b6e00daf5aca. It appears there are several wasm changes in there.

Comment 2

[Lars T Hansen [:lth]]

Hm, the most interesting change in that range is the resumePoint fix, this is probably correct by itself but may have exposed other bugs of course...

Comment 3

[Benjamin Bouvier [:bbouvier] (inactive)]

Crashes URLs shared by release managers show the recent spike happened in addon content, and these addons are not known by AMO, so it's a dead-end.

Comment 4

[Luke Wagner [:luke]]

The correlations tab shows:
(100.0% in signature vs 37.84% overall) Addon "uBlock Origin" = true
uBlock Origin does indeed use wasm:
https://github.com/gorhill/uBlock/blob/master/src/js/wasm/hntrie.wat#L301
and the linked to line is indeed a memory.size.

It's also interesting that all the crashes are amd64, although this could simply be a single user (given the low total volume).

Comment 5

[Luke Wagner [:luke]]

I installed uBlock origin 1.18.4, see the wasm in about:memory in the extension process. No crashes though (yet).

Comment 6

[Neha Kochar [:neha]]

All crash signatures are from 2 days only - Feb 7 & 8 and all on Windows NT and all crashing on the day of install too.
Luke, did you install new when you tried to reproduce in comment 5?

Comment 7

[Luke Wagner [:luke]]

No; and I also tried on Linux. Do we have a QA flag that we can use to request someone attempt to repro under these conditions?

Comment 8

[[e-mail]]

uBO will use localStorage.getItem('FilterHostnameDict.trieDetails') to remember the size to use next time it launches, so as to avoid repeated memory growing operations.

If the issue here is related to growing the WebAssembly.memory from a call originating from the wasm code itself, than it might be useful to delete the entry FilterHostnameDict.trieDetails in uBO's own localStorage before restarting the extension when trying to reproduce the issue.

Comment 9

[Marcia Knous [:marcia]]

Ovidiu is this something you or your team can help test? Thanks.

Comment 10

[ovidiu boca[:Ovidiu]]

Hi Marcia,
Yes, we can help you test this issue, but we want some STR if it's possible. For the moment we tested this on Windows 10 using the latest Nightly build 20190220040540, and uBlock Origin but we didn't have any crash until now. Please let us know if you have more details about this issue.

Comment 11

[Neha Kochar [:neha]]

Ovidiu, did you test this with a new install? Luke, do you have other ideas to help Ovidiu try to repro?
This is a new crash in 67.

Comment 12

[Marcia Knous [:marcia]]

It looks as if starting with the 2-23 nightly, we haven't had any further crashes. If we do, perhaps Luke can help Ovidiu with a good plan to test this.

Comment 13

[Luke Wagner [:luke]]

Well, I'd suggest installing specifically version 1.18.4 of uBlock and trying to do what comment describes.

That being said, if we can't repro after a few different tries, then given the very low/zero crash rate, I'd be inclined to wait for someone to report with STR.

Comment 14

[ovidiu boca[:Ovidiu]]

I retested this using the latest FF Nightly 67.0a1(2019-03-03) and uBlock version 1.18.4, there are no crashes. Please NI? me if there is something else that I can do.

Comment 15

[Neha Kochar [:neha]]

Closing as no more crashes are seen with this.

Comment 16

[(no longer active)]

Just got this on the latest Firefox Nightly: https://crash-stats.mozilla.org/report/index/a5d1500b-c31c-428a-beb8-c54ef0190328 when viewing https://bmcresnotes.biomedcentral.com/articles/10.1186/s13104-019-4179-2 (just this one page, so pretty sure that's the culprit.)

Comment 17

[Luke Wagner [:luke]]

Interesting, thanks for reporting! The crash report doesn't show uBO installed and the link doesn't seem to contain any wasm usage. Does the crash reproduce for you?

I noticed the profiler addon was installed, and the profiler now does use wasm (when analyzing a profile). Were you by chance profiling near the time of the crash?

Incidentally, I looked back over the url column of the crash reports linked above and many have moz-extension::// URLs.

Comment 18

[Neha Kochar [:neha]]

Adding ni for Ehsan to respond to comment 17.

Comment 19

[(no longer active)]

(In reply to Luke Wagner [:luke] from comment #17)

Interesting, thanks for reporting! The crash report doesn't show uBO installed and the link doesn't seem to contain any wasm usage.

I have uBO installed but disabled in this profile. Hopefully it doesn't get to run any code...

Does the crash reproduce for you?

It does not. :-(

I noticed the profiler addon was installed, and the profiler now does use wasm (when analyzing a profile). Were you by chance profiling near the time of the crash?

Fairly confident that I wasn't.

Comment 20

[Neha Kochar [:neha]]

Luke, ideas for next steps?

Comment 21

[Luke Wagner [:luke]]

Nope, still no STR or obvious clues from crash stack.

Comment 22

[Daniel Bodea [:danibodea]]

I have attempted to reproduce this issue manually with Nightly v69.0a1 (2019-06-06); I set the xpinstall.signatures.required pref to false to be able to install uBlock v1.18.4. Then I have scrolled around a bunch of random sites that have ads to try and reproduce a crash. No luck.
Furthermore, I attempted to install this add-on on Beta v68.0b7, but it would not install even after setting the xpinstall.signatures.required to false.

Unless some sort of STR is discovered, a regression cannot be performed for this bug.
If an STR is found, please remove the [qa-regression-triage] tag. Thank you.

Comment 23

[(no longer active)]

I got bp-e72b9bce-cb56-4efd-8d65-7f0cc0190705, bp-f295ca3c-d179-443e-9504-59b780190705 and bp-e8f60c62-27f0-4f3f-82ba-e6a6f0190705 with the following STR:

1. On the latest Windows Nightly, go to https://www.phoronix.com/forums/forum/hardware/general-hardware/1099221-hands-on-with-the-atomic-pi-as-a-35-intel-atom-alternative-to-the-raspberry-pi.
2. Open Devtools.
3. Open the Inspector tab.
4. Next to the <html> element, click on the event button to open the list of event listeners. Firefox will crash shortly after the menu is opened.

Comment 24

[Lars T Hansen [:lth]]

I can't repro locally, but I run without extensions installed, and I see the crash reports lists a number of them.

Comment 25

[Lars T Hansen [:lth]]

Installed gecko profiler and the web compat extension and about:profile shows the presence of the other extensions (monitor, screenshots, autofill) and I have not others, but unable to reproduce. For step 2, I've tried opening the devtools from the menu (hamburger > Web Developer > Inspector) and via a keystroke combination (Ctrl-Shift-I).

My Windows 10 install is probably not 100% current but I think I refreshed it within the last six months, at least.

Comment 26

[Daniel Bodea [:danibodea]]

Attachment: event listeners next to the html tag.png

I have attempted to reproduce a crash using the steps in comment 23, but I did not succeed. I used a Windows 10 machine and Nightly v69.0a1 (2019-07-07), Beta v68.0 and Release v67.0.4.
On Nightly, I could have uBlock v1.18.4 installed by setting pref "xpinstall.signatures.required" to false, but on Beta and Release versions, only the latest version could be installed (signed version).

All considered, I still cannot reproduce this crash on any of the latest main Firefox versions. Thanks.

Comment 27

[(no longer active)]

Attachment: prefs.js file used to reproduce the crash in comment 23

Comment 28

[(no longer active)]

Regression range: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=7bff7a09483d3390388033324f69ebe4b7248a85&tochange=00d55a99831bd9ce0e4e5fc58fb06d27feff6fbc

Candidates: bug 1510216 or bug 1505768

Comment 29

[(no longer active)]

(In reply to :Ehsan Akhgari from comment #23)

I got bp-e72b9bce-cb56-4efd-8d65-7f0cc0190705, bp-f295ca3c-d179-443e-9504-59b780190705 and bp-e8f60c62-27f0-4f3f-82ba-e6a6f0190705 with the following STR:

1. On the latest Windows Nightly, go to https://www.phoronix.com/forums/forum/hardware/general-hardware/1099221-hands-on-with-the-atomic-pi-as-a-35-intel-atom-alternative-to-the-raspberry-pi.
2. Open Devtools.
3. Open the Inspector tab.
4. Next to the <html> element, click on the event button to open the list of event listeners. Firefox will crash shortly after the menu is opened.

Note that after further testing I realized that these steps do not reproduce the bug on a new profile on trunk, but with the prefs.js file attached in comment 27 they do. I used that prefs.js file using mozregression in order to obtain the regression range.

Comment 30

[(no longer active)]

I'm pretty sure setting javascript.options.wasm_cranelift to true is enough to reproduce this bug.

Comment 31

[Lars T Hansen [:lth]]

I can repro a crash when cranelift is enabled, 22e41ec7-82e7-4563-9499-44a2a0190710. And this is true even without the wasm baseline compiler disabled, so the problem is probably not related to tiering down to debug code (assuming we even do such a thing; I suspect we don't). So it sounds like a run-of-the-mill cranelift bug.

Comment 32

[Liz Henry (:lizzard) (relman/hg->git project)]

Given that this is marked P3 and is very low volume crash I'm also marking it fix-optional for 70 to remove it from weekly regression triage.

Comment 33

[Benjamin Bouvier [:bbouvier] (inactive)]

Considering the massive source of Windows-only Cranelift bugs that is discussed in bug 1573550, it is very likely the same cause; we'll see if the crash rate goes to 0 once this lands. Crashes are one thing, in the worst case this could lead to infinite loops (observed on Riot.im) or incorrect behavior (not sec-sensitive though, because this is Nightly-only and pref'd off by default).

For what it's worth, the STR in comment 23 are still useful, when enabling Cranelift; I just needed to open devtools to get a crash. So that'll be one way to verify that this bug fixes it.

Comment 34

[Benjamin Bouvier [:bbouvier] (inactive)]

Ehsan, can you confirm it's fixed in latest Nightly builds, please? The patch landed in bug 1573550 should have fixed this.

Comment 35

[(no longer active)]

(In reply to Benjamin Bouvier [:bbouvier] from comment #34)

Ehsan, can you confirm it's fixed in latest Nightly builds, please? The patch landed in bug 1573550 should have fixed this.

The steps to reproduce in comment 23 no longer reproduce a crash for me!

Comment 36

[Benjamin Bouvier [:bbouvier] (inactive)]

Thanks! Calling this fixed by bug 1573350, since there's no crash report with this signature since bug 1573350 has landed in central (august 20th).