1514965 - Enable clang -ftrivial-auto-var-init=pattern 0xAA in debug builds

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, enhancement, P2)

Description

[Chris Peterson [:cpeterson]]

clang trunk now has a -ftrivial-auto-var-init option to initialize automatic variables with either a pattern or with zeroes. The expected runtime overhead is 3-5%.

https://reviews.llvm.org/rL349442

Excerpt from the option's commit message:

Pattern initialization's goal is to initialize automatic variables with values which will likely transform logic bugs into crashes down the line, are easily recognizable in a crash dump, without being values which programmers can rely on for useful program semantics.

- Integers are initialized with repeated 0xAA bytes (infinite scream).

- Vectors of integers are also initialized with infinite scream.

- Pointers are initialized with infinite scream on 64-bit platforms because it's an unmappable pointer value on architectures I'm aware of. Pointers are initialize to 0x000000AA (small scream) on 32-bit platforms because 32-bit platforms don't consistently offer unmappable pages. When they do it's usually the zero page.

- Floating point values and vectors are initialized with a negative quiet NaN with repeated 0xFF payload (e.g. 0xffffffff and 0xffffffffffffffff).

- Arrays are initialized to their homogeneous elements' initialization value, repeated.

- Structs are initialized to their heterogeneous element's initialization values.

Comment 1

[Nathan Froyd [:froydnj]]

I think this would be an interesting option to enable.  We'd have to see what the performance impacts were, though.

Comment 2

[Chris Peterson [:cpeterson]]

Attachment: Bug 1514965 - Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r?froydnj

... and DevEdition and local opt builds.

Comment 3

[Chris Peterson [:cpeterson]]

The -ftrivial-auto-var-init flag is now supported in:

• Apple Xcode 10.2's clang
• Homebrew's clang 8.0
• mozilla-central's clang version for Android and macOS (but not Linux or Windows??)

This patch proposes to enable -ftrivial-auto-var-init in:

• Debug builds on Android and macOS: local, Nightly, Dev Edition, Beta, and Release
• opt/PGO builds on Android and macOS: local, Nightly, and Dev Edition

btw, Windows builds use clang-cl, but moz.configure's add_gcc_flag() (and thus not check_and_add_gcc_flag() or check_and_add_gcc_warning()) seem to affect clang-cl's CXXFLAGS. Is that intentional?

Comment 4

[Chris Peterson [:cpeterson]]

Here's a Perfherder comparison of some Raptor benchmarks (Speedometer, tp6m page load tests on Android, MotionMark and WebGL graphics on macOS). I can run some other benchmarks, if you like.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=e26c541ff689e8e5560d4f4134b081626005f3b8&framework=10&selectedTimeRange=172800

Android page load regressed 0-20%, typically about 5%.

Speedometer regressed about 0-5% on both Android and macOS.

WebAudio regressed about 12% on macOS.

This performance impact might be too big for Nightly Android builds or local opt builds (Android and macOS), if Gecko developers will be profiling their local opt builds.

Comment 5

[Chris Peterson [:cpeterson]]

tp6 page load tests on macOS regress similarly to Android: 0-10% with one outlier (Instagram) at 25%.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=651680f0f59140644d957b68472edb224f8fd22f&framework=10&selectedTimeRange=172800

Comment 6

[Nathan Froyd [:froydnj]]

I find performance comparisons of try vs. mozilla-central unreliable; the better way to compare is to push to try with your chosen baseline revision (and --rebuild 10 or whatever) and then push to try with your changes (again, --rebuild 10), and compare those pushes. Could you please do that instead?

Comment 7

[Chris Peterson [:cpeterson]]

Here is a try-vs-try --rebuild 10 comparison with just the -ftrivial-auto-var-init change. The results don't look as bad.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=ff8e223faef88e2a1e650ced462db465077e36e9&newProject=try&newRevision=18ca6d5e43b08b2c3315a32d8a5154068172f9fb&framework=10

Do we have a threshold for performance regressions from diagnostic changes in Nightly or local opt builds? Who would decide whether the trade-off is worthwhile? I can change this patch to only enable -ftrivial-auto-var-init in debug and fuzzing builds or on macOS Nightly but not Android. It would be interesting to see how the change fares with Nightly user testing.

Comment 8

[Chris Peterson [:cpeterson]]

macOS regressions:

-8.41% WebAudio
-2.62% MotionMark Animometer
-1.00% MotionMark HTML Suite
-1.93% Stylebench
-1.20% Speedometer
-0.78% wasm-godot
-0.71% SunSpider

Average tp6 regression = +2.65% but Yahoo! News got 5% faster?!

+6.86% tp6: Reddit
+5.98% tp6: IMDb
+5.54% tp6: Google Sheets
+5.39% tp6: PayPal
+5.07% tp6: Yandex
+5.03% tp6: Apple
+4.85% tp6: Tumblr
+4.69% tp6: Wikipedia
+3.90% tp6: Google Docs
+2.98% tp6: Wikia
+2.89% tp6: Microsoft
+2.89% tp6: eBay
+2.75% tp6: Pinterest
+2.52% tp6: Bing
+2.40% tp6: Imgur
+1.59% tp6: Gmail
+1.57% tp6: Twitter
+1.54% tp6: Amazon
+1.45% tp6: Google Search
+1.17% tp6: YouTube
+1.05% tp6: Instagram BinAST
+0.95% tp6: Instragram
-0.04% tp6: Google Slides
-0.84% tp6: Facebook
-5.86% tp6: Yahoo! News

Comment 9

[Chris Peterson [:cpeterson]]

Android ARMv7 on Moto G5 regressions:

-2.86% Speedometer
-0.57% Unity WebGL

Average tp6m regression = +4.27% but eBay search got 6% faster?!

+13.50% tp6m: web.de
+12.57% tp6m: Amazon search
+7.26% tp6m: A-Frame
+6.81% tp6m: Google search
+6.78% tp6m: Amazon home page
+6.43% tp6m: Google Maps
+5.72% tp6m: YouTube home page
+4.81% tp6m: eBay home page
+4.76% tp6m: ESPN
+4.48% tp6m: Stack Overflow
+4.35% tp6m: Facebook user page
+4.05% tp6m: Instagram
+3.84% tp6m: IMDb
+3.23% tp6m: Jianshu
+2.97% tp6m: Booking
+2.71% tp6m: Facebook home page
+2.51% tp6m: Bing home page
+2.12% tp6m: Bing restaurants search
+2.05% tp6m: YouTube watch page
+1.73% tp6m: All Recipes
+1.22% tp6m: Reddit
+0.98% tp6m: Wikipedia
-6.65% tp6m: eBay search

Comment 10

[Chris Peterson [:cpeterson]]

Android ARM64 on Pixel 2 regressions:

-2.55% Speedometer

Average tp6m regression = +1.36% but eBay search got 26% faster?!!

+10.35% tp6m: web.de
+7.67% tp6m: Amazon home page
+5.82% tp6m: Wikipedia
+5.25% tp6m: A-Frame
+5.23% tp6m: Amazon search
+4.11% tp6m: Stack Overflow
+3.53% tp6m: IMDb
+2.79% tp6m: Facebook user page
+2.75% tp6m: Facebook home page
+2.66% tp6m: eBay home page
+2.52% tp6m: ESPN
+2.06% tp6m: Google search
+1.99% tp6m: Jianshu
+1.93% tp6m: Booking
+1.69% tp6m: Instagram
+1.57% tp6m: Bing home page
+1.55% tp6m: YouTube watch page
+0.80% tp6m: Google Maps
+0.19% tp6m: Bing restaurants search
-1.11% tp6m: YouTube home page
-1.14% tp6m: Reddit
-4.66% tp6m: All Recipes
-26.16% tp6m: eBay search

Comment 11

[Chris Peterson [:cpeterson]]

Attachment: Bug 1514965 - Part 2: Don't enable -ftrivial-auto-var-init on Windows. r?froydnj

Neither clang-cl nor mingw-clang support -ftrivial-auto-var-init yet.

Depends on D26437

Comment 12

[Chris Peterson [:cpeterson]]

Attachment: Bug 1514965 - Part 1: Refactor mingw_clang checks for reuse. r=froydnj

Depends on D27357

Comment 13

[(Away)]

Chris, if I understand correctly, the only platforms that won't have the flag are beta and release, right? I'm somewhat worried about the initialization masking bugs that only get unmasked at beta when we don't have much time left. I understand that the scream pattern is designed to reveal bugs earlier, but I still don't feel great about doing a large behavior change when a train reaches beta. I think I'd feel more comfortable if more channels had a mix of both initializing and non-initializing builds.

Comment 14

[(Away)]

Another concern is that this would make it harder for developers profiling on Nightly to understand the perf-behavior of their code when it ships to users.

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to David Major [:dmajor] (low availability) from comment #13)

Chris, if I understand correctly, the only platforms that won't have the flag are beta and release, right?

Yes, that's what I've currently proposed.

I still don't feel great about doing a large behavior change when a train reaches beta. I think I'd feel more comfortable if more channels had a mix of both initializing and non-initializing builds.

Another concern is that this would make it harder for developers profiling on Nightly to understand the perf-behavior of their code when it ships to users.

I see what you mean. Some other options to increase the channel mix and reduce the impact on developer profiling:

• Enable in official Nightly builds but not in local opt builds.
• Enable in official Dev Edition builds but not in Nightly or local opt builds.
• Enable in local opt builds but not in official channel builds.
• Enable in fuzzing opt builds but not in any official channel or local opt builds.

These days, Dev Edition is built from mozilla-beta with some different flags. I don't know if QA monitors Dev Edition crashes separately from Beta crashes. Any 0xAA... crashes in Dev Edition might be drowned out by the 5x more Beta users on the same Firefox version. Neither QA's Mission Control stability dashboard or Socorro's top crash list separate Dev Edition from Beta users.

Comment 16

[Chris Peterson [:cpeterson]]

I removed my review requests because I have a new patch that adds support for clang-cl and only enables -ftrivial-auto-var-init in debug builds (the least controversial option).

Comment 17

[Chris Peterson [:cpeterson]]

I'm able to build Firefox for Windows with clang-cl.exe -clang:-ftrivial-auto-var-init, but all Try tests fail with marionette.py timeouts because firefox.exe was terminated with exit code 3221225477 (0xC0000005 aka access violation):

07:59:55     INFO - REFTEST INFO | Application command: Z:\task_1555572909\build\application\firefox\firefox.exe -marionette --wait-for-browser 
-profile c:\users\task_1555572909\appdata\local\temp\tmpl5zqae.mozrunner
08:02:55    ERROR - TEST-UNEXPECTED-FAIL | None | application terminated with exit code 3221225477

But I can download the x64 Try build and run it locally without crashing. Interestingly, the windows10-aarch64 opt build passes the tests fine. These test failures only affect the x86 and x64 builds, opt or debug.

When enabling -clang:-ftrivial-auto-var-init for debug builds, the debug tests fail but the opt tests pass:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdf22b20feeb3493dc6d1eddcd3d4a5541383592

When enabling -clang:-ftrivial-auto-var-init just for opt builds, the opt tests fail and the debug tests pass:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=96b42c960293bb656913cdaedb0821e9ecfea899

I don't have a Windows build environment set up, so that's probably the next step.

Comment 18

[(Away)]

I get a crash in patched_NtMapViewOfSection in the "main" process (the thing created by the launcher stub) because the compiler has inserted calls to memset, and we're so super-early in this newborn process that the loader hasn't yet resolved the import of memset from vcruntime140.dll.

At a minimum we'd want to disable var-init on that function (or file), but if that's not enough we might have to disable var-init in the sandbox for similar reasons (see also bug 1437452 comment 41).

Comment 19

[Chris Peterson [:cpeterson]]

(In reply to David Major [:dmajor] (low availability) from comment #18)

I get a crash in patched_NtMapViewOfSection in the "main" process (the thing created by the launcher stub) because the compiler has inserted calls to memset, and we're so super-early in this newborn process that the loader hasn't yet resolved the import of memset from vcruntime140.dll.

Thanks! I saw debug breaks in TargetNtMapViewOfSection when I ran my -ftrivial-auto-var-init firefox.exe build in WinDbg, but I saw the same debug breaks with the official Nightly firefox.exe build so I assumed they were expected.

Disabling -ftrivial-auto-var-init for DllBLocklistWin.cpp appears to fix the startup crash. clang allows -ftrivial-auto-var-init to be disable for individual variable definitions, but not for whole functions.

Here is a Try test run with -ftrivial-auto-var-init disabled for DllBLocklistWin.cpp. There are some oranges, but none of them appear to be related to -ftrivial-auto-var-init.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=85c9c667a7b725745978774eef6786b31fe20a97&selectedJob=241739783

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b36a9dda426
Part 1: Refactor mingw_clang checks for reuse. r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/7a6e07feb572
Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj

Comment 21

[Chris Peterson [:cpeterson]]

My sense here is "yes", but we might want to talk to the Linux distro people that are active on Bugzilla (SuSE and RedHat, at least; I guess @glandium would speak for Debian?)

I think that enabling should be a separate bug.

I filed bug 1546873 if anyone wants --enable-hardening to also enable clang -ftrivial-auto-var-init.

Comment 22

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 2 changesets (Bug 1514965) for linux build bustage

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=242510983&revision=7a6e07feb5722ba601a04efeea2e96811429450e

Backout link: https://hg.mozilla.org/integration/mozilla-inbound/rev/81efffda1929913443e758020a7d627b7af4497c

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242510983&repo=mozilla-inbound&lineNumber=39121

[task 2019-04-25T07:00:11.549Z] 07:00:11 INFO - ../python/mozbuild/mozpack/test/test_files.py::TestMercurialNativeRevisionFinder::test_recognize_repo_paths SKIPPED
[task 2019-04-25T07:00:11.549Z] 07:00:11 INFO - ===================== 48 passed, 3 skipped in 4.79 seconds =====================
[task 2019-04-25T07:00:15.856Z] 07:00:15 INFO - /builds/worker/workspace/build/src/testing/xpcshell/selftest.py
[task 2019-04-25T07:00:15.856Z] 07:00:15 INFO - ============================= test session starts ==============================
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - collecting ... collected 55 items
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskRunNextTest PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkip TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkipAll TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskStackTrace TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestFailureInside PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestMultiple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestRejected PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestSingle TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestFailing PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestSimple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejection PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejectionJSM PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAssertStack TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAsyncCleanup TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChild TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildFail TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildHang TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildMozinfo TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseExplicit TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseInManifest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseNotExplicit TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportForeignObject TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportNonSyntaxError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportRefError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportSyntaxError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowForeignObject TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowString TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testHangingTimeout <- ../../../../../usr/lib/python2.7/unittest/case.py SKIPPED
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testKnownFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testLogCorrectFileName TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMissingHeadFile PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMozinfo TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskMultiple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestEmptyTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPassFail TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testRandomExecution TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testReturnNonzero PASSED
[task 2019-04-25T07:00:15.863Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkip PASSED
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSyntaxError PASSED
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejection TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejectionJSM TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnexpectedPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.865Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnicodeInAssertMethods TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - =================================== FAILURES ===================================
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - ______________________ XPCShellTestsTests.testAddTaskSkip ______________________
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self = <selftest.XPCShellTestsTests testMethod=testAddTaskSkip>
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - def testAddTaskSkip(self):
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self.writeFile("test_tasks_skip.js", ADD_TASK_SKIP)
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self.writeManifest(["test_tasks_skip.js"])
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - > self.assertTestResult(True)
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py:1068:
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py:528: in assertTestResult
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - """ % ("passed" if expected else "failed", self.log.getvalue()))
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E AssertionError: Tests should have passed, log:
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E ========
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E Found node at /builds/worker/workspace/build/src/node/bin/node
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E Found moz-http2 at /builds/worker/workspace/build/src/testing/xpcshell/moz-http2/moz-http2.js
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E Running tests sequentially.
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E SUITE-START | Running 1 tests
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E profile dir is /tmp/xpcshell/xpcshellprofile
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E TEST-START | test_tasks_skip.js
[task 2019-04-25T07:00:15.867Z] 07:00:15 WARNING - E TEST-UNEXPECTED-FAIL | test_tasks_skip.js | xpcshell return code: -11

Comment 23

[Chris Peterson [:cpeterson]]

Nathan, my clang -ftrivial-auto-var-init patches got backed out for xpcshell/selftest.py failures on linux32. The same tests pass on linux64. Other xpcshell tests, and all the regular Firefox tests, also pass on linux32. It's only xpcshell/selftest.py on linux32 that fails (with xpcshell return code -11).

Do you know why xpcshell/selftest.py might behave differently on linux32 and linux64 with clang -ftrivial-auto-var-init? If these failures were similar to the Windows problem dmajor described above about early use of memset during process startup, I would imagine both linux32 and linux64 would be affected.

Comment 24

[Nathan Froyd [:froydnj]]

(In reply to Chris Peterson [:cpeterson] from comment #23)

Nathan, my clang -ftrivial-auto-var-init patches got backed out for xpcshell/selftest.py failures on linux32. The same tests pass on linux64. Other xpcshell tests, and all the regular Firefox tests, also pass on linux32. It's only xpcshell/selftest.py on linux32 that fails (with xpcshell return code -11).

Do you know why xpcshell/selftest.py might behave differently on linux32 and linux64 with clang -ftrivial-auto-var-init? If these failures were similar to the Windows problem dmajor described above about early use of memset during process startup, I would imagine both linux32 and linux64 would be affected.

I don't have any good ideas. It certainly sounds like we're taking some kind of different path for 32-bit vs. 64-bit, but there could be a number of places that could be happening. Sure would be nice if selftest.py reported intelligent errors...

Comment 25

[(Away)]

Whenever I break selftest.py, the only way I can get a meaningful diagnosis is to run the test locally and see what my debugger complains about. Want to give it a try, Chris?

Comment 26

[Chris Peterson [:cpeterson]]

If I am honest with myself, I know I'll never get around to setting up a Linux32 dev environment to debug my patch's Linux32-only xpcshell test failure. And AFAICT the only Linux32 tests we bother running on mozilla-central are xpcshell and wpt. I'd like to resubmit my -ftrivial-auto-var-init patch but disable it on Linux32. There's little reason to not let other platforms benefit from clang's -ftrivial-auto-var-init.

Since I last submitted my patch, we are now using a version of mingw-clang that understands -ftrivial-auto-var-init=pattern. That removes one special condition.

Comment 27

[Chris Peterson [:cpeterson]]

Attachment: Bug 1514965 - Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj

Disable -ftrivial-auto-var-init=pattern for DllBLocklistWin.cpp with clang-cl because the file's interceptions happen so early in the main process that the loader hasn't yet resolved the import of memset (used by -ftrivial-auto-var-init) from vcruntime140.dll.

Disable -ftrivial-auto-var-init on Linux32 because it causes some xpcshell test failures.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5ad6f4f0edeb
Part 1: Refactor mingw_clang checks for reuse. r=froydnj

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5babe33486a1
Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj

Comment 30

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/5ad6f4f0edeb
https://hg.mozilla.org/mozilla-central/rev/5babe33486a1

Comment 31

[Thomas Nguyen (:tnguyen)]

Hi Chris,
I got build failed only for Debug bug after rebasing new source code
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'
0:07.21 DEBUG: configure: failed program was:
0:07.21 DEBUG: #line 4529 "configure"
0:07.21 DEBUG: #include "confdefs.h"
0:07.21 DEBUG:
0:07.21 DEBUG: int main() {
0:07.21 DEBUG: return 0;
0:07.21 DEBUG: ; return 0; }
0:07.21 DEBUG: configure:5077: checking for valid debug flags
0:07.21 DEBUG: configure:5088: /usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -std=gnu99 -c -fstack-protector-strong -ftrivial-auto-var-init=pattern -fno-strict-aliasing -g -Qunused-arguments -fstack-protector-strong -ftrivial-auto-var-init=pattern conftest.c 1>&5
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

I guess that this is related to this change. Running ./mach bootstrap does not help.
I tried to checkout the last version without this commit (8a08eb947133d0be8d5eeb8a9d9a3774e99eae61) then I could build without any error.
I guess that we may update some dependencies in bootstrap. Could you please take a look?

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Which version of clang do you have?

Comment 33

[Thomas Nguyen (:tnguyen)]

clang --version

Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Thanks

Comment 34

[(Away)]

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

Comment 35

[(Away)]

Thomas, if you need to get unblocked in the meantime, you can temporarily use ac_add_options --disable-hardening.

Comment 36

[Thomas Nguyen (:tnguyen)]

(In reply to :dmajor from comment #35)

Thomas, if you need to get unblocked in the meantime, you can temporarily use ac_add_options --disable-hardening.
0:13.68 js/src> checking for -framework ExceptionHandling... no
0:13.71 js/src> checking for -dead_strip option to ld... no
0:13.74 js/src> checking for valid debug flags... no
0:13.74 js/src> configure: error: These compiler flags are invalid: -g
0:13.74 js/src> ERROR: old-configure failed

That does not work, the lastest version works for me is 8a08eb947133d0be8d5eeb8a9d9a3774e99eae61

Comment 37

[Chris Peterson [:cpeterson]]

(In reply to Thomas Nguyen from comment #33)

Apple LLVM version 9.1.0 (clang-902.0.39.2)
...
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Thomas, is there any specific reason you are using Xcode's clang instead of the clang installed by mach bootstrap in $HOME/.mozbuild/clang/bin?

Mozilla's macOS builders explicitly point CC and CXX to mozbuild's clang, so it's probably best for Firefox developers on macOS to also use mozbuild's clang like:

https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/build/macosx/local-mozconfig.common#9-10

    export CC=$MOZ_FETCHES_DIR/clang/bin/clang
    export CXX=$MOZ_FETCHES_DIR/clang/bin/clang++

(In reply to :dmajor from comment #34)

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

I knew that Apple's clang had a different version number than mozbuild's and Homebrew's clang builds, but I had hoped Firefox developers on macOS were either using mozbuild's clang or an Xcode with a new enough clang. :) That's apparently not a safe assumption, so I can add a macOS patch to either skip -ftrivial-auto-var-init or check for Xcode clang >= 10.0. I believe -ftrivial-auto-var-init first appeared in Xcode's clang 10.0. I have Xcode 10.3 with clang 10.0.1 on my macOS 10.14.6 machine.

Comment 38

[Thomas Nguyen (:tnguyen)]

Sorry, I don't remember exactly why I changed to Apple's clang. I guess I had to fix some build issue, but it has been worked for a long time.
Anw, it is always worth to make some note in our document the minimum version of clang (and recommendation one). I will try to change to mozbuild's clang and let you know. Thanks

Comment 39

[Thomas Nguyen (:tnguyen)]

(In reply to :dmajor from comment #34)

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

I knew that Apple's clang had a different version number than mozbuild's and Homebrew's clang builds, but I had hoped Firefox developers on macOS were either using mozbuild's clang or an Xcode with a new enough clang. :) That's apparently not a safe assumption, so I can add a macOS patch to either skip -ftrivial-auto-var-init or check for Xcode clang >= 10.0. I believe -ftrivial-auto-var-init first appeared in Xcode's clang 10.0. I have Xcode 10.3 with clang 10.0.1 on my macOS 10.14.6 machine.

10.0 does not work. I have upgraded to 10.0.0 and it is still failed

Comment 40

[(Away)]

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

Comment 41

[Thomas Nguyen (:tnguyen)]

(In reply to :dmajor from comment #40)

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

I did not try, but I did another upgrading, the first version works for me is clang 10.0.1.

Comment 42

[Chris Peterson [:cpeterson]]

(In reply to :dmajor from comment #40)

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

(In reply to Thomas Nguyen from comment #41)

I did not try, but I did another upgrading, the first version works for me is clang 10.0.1.

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

Comment 43

[Chris Peterson [:cpeterson]]

(In reply to Chris Peterson [:cpeterson] from comment #42)

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

I'm testing a patch that uses check_and_add_flags to detect -ftrivial-auto-var-init support, but it introduces a new complications. For one, check_and_add_flags is a no-op for clang-cl, AFAICT:

https://searchfox.org/mozilla-central/rev/dafa68a84f01897c8cfb4bbaf6829cae8df4eb57/build/moz.configure/compile-checks.configure#156-159,174-177

And even if it check_and_add_flags checked clang-cl flags, -Xclang is needed to pass -ftrivial-auto-var-init to clang-cl and a -Xclang -ftrivial-auto-var-init check wouldn't work because passing an unknown argument to -Xclang fails silently so the configure check will never fail.

At this time, I don't think changing the current -ftrivial-auto-var-init check is worth the trouble. tnguyen has been the only person to report (in this bug) problems with the check and updating to the latest Xcode version fixed the problem. The current -ftrivial-auto-var-init check works correctly with mozbuild's blessed clang and clang-cl, clang ToT, and Apple clang (if you have a Xcode >= 10.2 from March 2019). If a developer wants to build with Apple clang instead of mozbuild's blessed clang, I think updating to an Xcode version from 2019 is a reasonable requirement. (The previous Xcode version 10.1 was released in October 2018.)

Comment 44

[Nathan Froyd [:froydnj]]

(In reply to Chris Peterson [:cpeterson] from comment #43)

(In reply to Chris Peterson [:cpeterson] from comment #42)

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

I'm testing a patch that uses check_and_add_flags to detect -ftrivial-auto-var-init support, but it introduces a new complications. For one, check_and_add_flags is a no-op for clang-cl, AFAICT:

https://searchfox.org/mozilla-central/rev/dafa68a84f01897c8cfb4bbaf6829cae8df4eb57/build/moz.configure/compile-checks.configure#156-159,174-177

And even if it check_and_add_flags checked clang-cl flags, -Xclang is needed to pass -ftrivial-auto-var-init to clang-cl and a -Xclang -ftrivial-auto-var-init check wouldn't work because passing an unknown argument to -Xclang fails silently so the configure check will never fail.

tjr has been working to make check_and_add_flags work for clang-cl. I can't remember whether -ftrivial-auto-var-init exists in clang 8.0, which is our minimum version on Windows, but if it does, you could unconditionally pass it.

Comment 45

[Tom Ritter [:tjr]]

That's over in Bug 1581018 - but it's blocked on actually fixing the warnings, which is not something I'm dedicating foreground time to right now...

Comment 46

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

fyi, I am using clang from mozbuild on linux, however I met the failure.

0:01.55 checking for host system type... x86_64-pc-linux-gnu
0:01.55 checking for target system type... x86_64-pc-linux-gnu
...
0:01.94 checking for the target C compiler... /home/allstars/.mozbuild/clang/bin/clang
0:01.98 checking whether the target C compiler can be used... yes
0:01.98 checking the target C compiler version... 8.0.1
0:09.23 checking the target C compiler works... yes
0:09.23 checking for the target C++ compiler... /home/allstars/.mozbuild/clang/bin/clang++
0:09.25 checking whether the target C++ compiler can be used... yes
0:09.25 checking the target C++ compiler version... 8.0.1
0:09.29 checking the target C++ compiler works... yes
0:09.29 checking for the host C compiler... /home/allstars/.mozbuild/clang/bin/clang
0:09.33 checking whether the host C compiler can be used... yes
0:09.33 checking the host C compiler version... 8.0.1
0:09.37 checking the host C compiler works... yes
0:09.37 checking for the host C++ compiler... /home/allstars/.mozbuild/clang/bin/clang++
0:09.39 checking whether the host C++ compiler can be used... yes
0:09.39 checking the host C++ compiler version... 8.0.1
....
1:11.32 Can't read /proc/cpuinfo: No such file or directory
1:11.32 clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

I have to do ac_add_options --disable-hardening to bypass this. (from comment 35)

Comment 47

[Chris Peterson [:cpeterson]]

(In reply to Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh] from comment #46)

fyi, I am using clang from mozbuild on linux, however I met the failure.

0:09.39 checking whether the host C++ compiler can be used... yes
0:09.39 checking the host C++ compiler version... 8.0.1
....
1:11.32 Can't read /proc/cpuinfo: No such file or directory
1:11.32 clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

Hi Yoshi, sorry about that.

This is surprising because the Linux build machines are also using mozbuild's clang 8.0.1 (as shown by > checking the target C++ compiler version... 8.0.1 in the configure log from a random mozilla-central Linux64 build. That build's clang version check worked because it does not attempt to use -ftrivial-auto-var-init=pattern.

Is this a recent regression on your machine? Have you run mach bootstrap lately? Can you please share you build log file in this bug (as a file attachment because it might be big)?

Comment 48

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Would it be possible that it's because of icecc? and the failure is from another machine?
I have another machine at home and using clang 8 on linux and it is all fine,
https://pastebin.com/kBvfuEET

Comment 49

[Chris Peterson [:cpeterson]]

(In reply to Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh] from comment #48)

Would it be possible that it's because of icecc? and the failure is from another machine?
I have another machine at home and using clang 8 on linux and it is all fine,

That could be. Perhaps the remote machine has an older clang version and the toolchain.configure check for -ftrivial-auto-var-init support is only running locally? Maybe I can add a check for icecc. The proper fix would be to actually feature detect -ftrivial-auto-var-init support using check_and_add_flags, but that has its own problem (described in comment 43).

https://searchfox.org/mozilla-central/rev/7cc0f0e89cb40e43bf5c96906f13d44705401042/build/moz.configure/toolchain.configure#1217-1226

btw, I see there is another error in your log:

2:13.07 DEBUG: configure: error: Can't find header linux/joystick.h, needed for gamepad support. Please install Linux kernel headers.`