Closed Bug 1986900 Opened 10 months ago Closed 10 months ago

Firefox crashes at startup because of invalid instruction

Categories

(Core :: Machine Learning: General, defect)

Firefox 144
defect

Tracking

()

RESOLVED FIXED
144 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox-esr128 --- unaffected
firefox-esr140 --- unaffected
firefox142 --- unaffected
firefox143 --- unaffected
firefox144 blocking fixed

People

(Reporter: viktor_jaegerskuepper, Assigned: padenot)

References

(Regression)

Details

(Keywords: perf-alert, regression)

Crash Data

Attachments

(3 files)

The latest Firefox Nightly crashes at startup on my system (x86-64, Intel Core 2 Duo) because of an "invalid instruction" (translated from German).

I ran mozregression and got:
Last good revision: 8e649e2c1476f102a2bbe699c1cf4949e557640b
First bad revision: 4c630ff1b3255fb7e6cb13eb8018652aa6703bef
Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=8e649e2c1476f102a2bbe699c1cf4949e557640b&tochange=4c630ff1b3255fb7e6cb13eb8018652aa6703bef

Bug 1971158 seems relevant because it has a patch with the description "Compile llama.cpp w/ AVX2 on x86_64 ..." and my CPU doesn't support AVX2.

Sandor, please check if this is a regression of Bug 1971158 too and/or move it to the appropriate component.

Flags: needinfo?(smolnar)

(In reply to 08xjcec48 from comment #1)

Sandor, please check if this is a regression of Bug 1971158 too and/or move it to the appropriate component.

talked with @padenot, will fix it

Flags: needinfo?(smolnar) → needinfo?(padenot)

Viktor, sorry about this, I have put in some code specifically for this. Do you mind getting us a link to a crash report? You can likely open another firefox version with the same profile, and find it in about:crashes.

Flags: needinfo?(viktor_jaegerskuepper)
Assignee: nobody → padenot
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Unfortunately there aren't any reports for this particular crash in about:crashes because the crash occurs too early (?). I will find out how to get a backtrace or if I can get this information from a coredump which was saved by systemd.

Viktor, what OS are you running? I've added a patch that could fix it, but I'm quite surprised the code we've added runs at startup.

Flags: needinfo?(padenot)

I'm running Arch Linux.

Superb, then you can probably just do gdb /path/to/firefox-nightly and it will crash, then bt can print a stack trace properly, potentially unsymbolicated (but I can symbolicate afterward). Are you running our builds, or something arch is building maybe?

Attached file backtrace-full.txt

I am running the Mozilla builds. I ran gdb with debuginfod enabled (I guess to use debug info for Arch Linux packages) and then entered bt, I'm attaching the output.

Flags: needinfo?(viktor_jaegerskuepper)

Thanks, it's very clear, I'll get a patch in asap.

08xjcec48: It's better to set a needinfo to the assignee of the bug which one thinks might be the regressor. The landing (merge) of the change to mozilla-central / firefox-main is done by people not involved with the code which landed in the bug.

Keywords: regression
Regressed by: 1971158
Component: Untriaged → Machine Learning
Product: Firefox → Core
Crash Signature: [@ std::_Construct_in_place] [@ std::_Mutex_base::_Mutex_base]
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → 144 Branch

Viktor, thanks a lot for taking the time to open a bug. I think you'll have to download a new nightly (when it is published), but it should be fixed. Please let us know here, or directly to me at padenot@mozilla.com if there's any problem, and sorry for the inconvenience.

Flags: needinfo?(viktor_jaegerskuepper)

A Nightly update with this fix is now available.

The issue was fixed by the second patch (Remove more static initializers from llama.cpp) and the new Nightly build is running fine. Thanks!

Flags: needinfo?(viktor_jaegerskuepper)
See Also: → 1987003
See Also: → 1987107

(In reply to Pulsebot from comment #14)

Pushed by padenot@mozilla.com:
https://github.com/mozilla-firefox/firefox/commit/ad918290746f
https://hg.mozilla.org/integration/autoland/rev/09242258e16e
Remove more static initializers from llama.cpp. r=atossou

Perfherder has detected a build_metrics performance change from push 09242258e16ef379e696a42afecf69f23ea84fac.

If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
4% compiler_metrics num_static_constructors osx-cross 96.00 -> 92.00
4% compiler_metrics num_static_constructors linux64 106.00 -> 102.00
4% compiler_metrics num_static_constructors linux64-shippable 106.00 -> 102.00
4% compiler_metrics num_static_constructors linux64 110.00 -> 106.00
4% compiler_metrics num_static_constructors linux64-shippable instrumented 112.00 -> 108.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46614

The following documentation link provides more information about this command.

Keywords: perf-alert
QA Whiteboard: [qa-triage-done-c145/b144]
Blocks: 1987453
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: