Closed Bug 1514965 Opened 2 years ago Closed 1 year ago

Enable clang -ftrivial-auto-var-init=pattern 0xAA in debug builds

Categories

(Firefox Build System :: Toolchains, enhancement, P2)

65 Branch
enhancement

Tracking

(firefox-esr60 wontfix, firefox-esr68 wontfix, firefox66 wontfix, firefox67 wontfix, firefox68 wontfix, firefox69 wontfix, firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox-esr60 --- wontfix
firefox-esr68 --- wontfix
firefox66 --- wontfix
firefox67 --- wontfix
firefox68 --- wontfix
firefox69 --- wontfix
firefox70 --- fixed

People

(Reporter: cpeterson, Assigned: cpeterson)

References

(Blocks 2 open bugs)

Details

Attachments

(2 files, 2 obsolete files)

clang trunk now has a -ftrivial-auto-var-init option to initialize automatic variables with either a pattern or with zeroes. The expected runtime overhead is 3-5%.

https://reviews.llvm.org/rL349442

Excerpt from the option's commit message:

Pattern initialization's goal is to initialize automatic variables with values which will likely transform logic bugs into crashes down the line, are easily recognizable in a crash dump, without being values which programmers can rely on for useful program semantics.

- Integers are initialized with repeated 0xAA bytes (infinite scream).

- Vectors of integers are also initialized with infinite scream.

- Pointers are initialized with infinite scream on 64-bit platforms because it's an unmappable pointer value on architectures I'm aware of. Pointers are initialize to 0x000000AA (small scream) on 32-bit platforms because 32-bit platforms don't consistently offer unmappable pages. When they do it's usually the zero page.

- Floating point values and vectors are initialized with a negative quiet NaN with repeated 0xFF payload (e.g. 0xffffffff and 0xffffffffffffffff).

- Arrays are initialized to their homogeneous elements' initialization value, repeated.

- Structs are initialized to their heterogeneous element's initialization values.
I think this would be an interesting option to enable.  We'd have to see what the performance impacts were, though.
Priority: -- → P2

The -ftrivial-auto-var-init flag is now supported in:

  • Apple Xcode 10.2's clang
  • Homebrew's clang 8.0
  • mozilla-central's clang version for Android and macOS (but not Linux or Windows??)

This patch proposes to enable -ftrivial-auto-var-init in:

  • Debug builds on Android and macOS: local, Nightly, Dev Edition, Beta, and Release
  • opt/PGO builds on Android and macOS: local, Nightly, and Dev Edition

btw, Windows builds use clang-cl, but moz.configure's add_gcc_flag() (and thus not check_and_add_gcc_flag() or check_and_add_gcc_warning()) seem to affect clang-cl's CXXFLAGS. Is that intentional?

Assignee: nobody → cpeterson

I find performance comparisons of try vs. mozilla-central unreliable; the better way to compare is to push to try with your chosen baseline revision (and --rebuild 10 or whatever) and then push to try with your changes (again, --rebuild 10), and compare those pushes. Could you please do that instead?

Flags: needinfo?(cpeterson)

Here is a try-vs-try --rebuild 10 comparison with just the -ftrivial-auto-var-init change. The results don't look as bad.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=ff8e223faef88e2a1e650ced462db465077e36e9&newProject=try&newRevision=18ca6d5e43b08b2c3315a32d8a5154068172f9fb&framework=10

Do we have a threshold for performance regressions from diagnostic changes in Nightly or local opt builds? Who would decide whether the trade-off is worthwhile? I can change this patch to only enable -ftrivial-auto-var-init in debug and fuzzing builds or on macOS Nightly but not Android. It would be interesting to see how the change fares with Nightly user testing.

Flags: needinfo?(cpeterson)

macOS regressions:

-8.41% WebAudio
-2.62% MotionMark Animometer
-1.00% MotionMark HTML Suite
-1.93% Stylebench
-1.20% Speedometer
-0.78% wasm-godot
-0.71% SunSpider

Average tp6 regression = +2.65% but Yahoo! News got 5% faster?!

+6.86% tp6: Reddit
+5.98% tp6: IMDb
+5.54% tp6: Google Sheets
+5.39% tp6: PayPal
+5.07% tp6: Yandex
+5.03% tp6: Apple
+4.85% tp6: Tumblr
+4.69% tp6: Wikipedia
+3.90% tp6: Google Docs
+2.98% tp6: Wikia
+2.89% tp6: Microsoft
+2.89% tp6: eBay
+2.75% tp6: Pinterest
+2.52% tp6: Bing
+2.40% tp6: Imgur
+1.59% tp6: Gmail
+1.57% tp6: Twitter
+1.54% tp6: Amazon
+1.45% tp6: Google Search
+1.17% tp6: YouTube
+1.05% tp6: Instagram BinAST
+0.95% tp6: Instragram
-0.04% tp6: Google Slides
-0.84% tp6: Facebook
-5.86% tp6: Yahoo! News

Android ARMv7 on Moto G5 regressions:

-2.86% Speedometer
-0.57% Unity WebGL

Average tp6m regression = +4.27% but eBay search got 6% faster?!

+13.50% tp6m: web.de
+12.57% tp6m: Amazon search
+7.26% tp6m: A-Frame
+6.81% tp6m: Google search
+6.78% tp6m: Amazon home page
+6.43% tp6m: Google Maps
+5.72% tp6m: YouTube home page
+4.81% tp6m: eBay home page
+4.76% tp6m: ESPN
+4.48% tp6m: Stack Overflow
+4.35% tp6m: Facebook user page
+4.05% tp6m: Instagram
+3.84% tp6m: IMDb
+3.23% tp6m: Jianshu
+2.97% tp6m: Booking
+2.71% tp6m: Facebook home page
+2.51% tp6m: Bing home page
+2.12% tp6m: Bing restaurants search
+2.05% tp6m: YouTube watch page
+1.73% tp6m: All Recipes
+1.22% tp6m: Reddit
+0.98% tp6m: Wikipedia
-6.65% tp6m: eBay search

Android ARM64 on Pixel 2 regressions:

-2.55% Speedometer

Average tp6m regression = +1.36% but eBay search got 26% faster?!!

+10.35% tp6m: web.de
+7.67% tp6m: Amazon home page
+5.82% tp6m: Wikipedia
+5.25% tp6m: A-Frame
+5.23% tp6m: Amazon search
+4.11% tp6m: Stack Overflow
+3.53% tp6m: IMDb
+2.79% tp6m: Facebook user page
+2.75% tp6m: Facebook home page
+2.66% tp6m: eBay home page
+2.52% tp6m: ESPN
+2.06% tp6m: Google search
+1.99% tp6m: Jianshu
+1.93% tp6m: Booking
+1.69% tp6m: Instagram
+1.57% tp6m: Bing home page
+1.55% tp6m: YouTube watch page
+0.80% tp6m: Google Maps
+0.19% tp6m: Bing restaurants search
-1.11% tp6m: YouTube home page
-1.14% tp6m: Reddit
-4.66% tp6m: All Recipes
-26.16% tp6m: eBay search

Attachment #9056364 - Attachment description: Bug 1514965 - Enable clang -ftrivial-auto-var-init 0xAA fill pattern in debug and Nightly builds. r?froydnj → Bug 1514965 - Part 1: Enable clang -ftrivial-auto-var-init in debug and Nightly builds. r?froydnj

Neither clang-cl nor mingw-clang support -ftrivial-auto-var-init yet.

Depends on D26437

Chris, if I understand correctly, the only platforms that won't have the flag are beta and release, right? I'm somewhat worried about the initialization masking bugs that only get unmasked at beta when we don't have much time left. I understand that the scream pattern is designed to reveal bugs earlier, but I still don't feel great about doing a large behavior change when a train reaches beta. I think I'd feel more comfortable if more channels had a mix of both initializing and non-initializing builds.

Another concern is that this would make it harder for developers profiling on Nightly to understand the perf-behavior of their code when it ships to users.

(In reply to David Major [:dmajor] (low availability) from comment #13)

Chris, if I understand correctly, the only platforms that won't have the flag are beta and release, right?

Yes, that's what I've currently proposed.

I still don't feel great about doing a large behavior change when a train reaches beta. I think I'd feel more comfortable if more channels had a mix of both initializing and non-initializing builds.

Another concern is that this would make it harder for developers profiling on Nightly to understand the perf-behavior of their code when it ships to users.

I see what you mean. Some other options to increase the channel mix and reduce the impact on developer profiling:

  • Enable in official Nightly builds but not in local opt builds.
  • Enable in official Dev Edition builds but not in Nightly or local opt builds.
  • Enable in local opt builds but not in official channel builds.
  • Enable in fuzzing opt builds but not in any official channel or local opt builds.

These days, Dev Edition is built from mozilla-beta with some different flags. I don't know if QA monitors Dev Edition crashes separately from Beta crashes. Any 0xAA... crashes in Dev Edition might be drowned out by the 5x more Beta users on the same Firefox version. Neither QA's Mission Control stability dashboard or Socorro's top crash list separate Dev Edition from Beta users.

I removed my review requests because I have a new patch that adds support for clang-cl and only enables -ftrivial-auto-var-init in debug builds (the least controversial option).

I'm able to build Firefox for Windows with clang-cl.exe -clang:-ftrivial-auto-var-init, but all Try tests fail with marionette.py timeouts because firefox.exe was terminated with exit code 3221225477 (0xC0000005 aka access violation):

07:59:55     INFO - REFTEST INFO | Application command: Z:\task_1555572909\build\application\firefox\firefox.exe -marionette --wait-for-browser 
-profile c:\users\task_1555572909\appdata\local\temp\tmpl5zqae.mozrunner
08:02:55    ERROR - TEST-UNEXPECTED-FAIL | None | application terminated with exit code 3221225477

But I can download the x64 Try build and run it locally without crashing. Interestingly, the windows10-aarch64 opt build passes the tests fine. These test failures only affect the x86 and x64 builds, opt or debug.

When enabling -clang:-ftrivial-auto-var-init for debug builds, the debug tests fail but the opt tests pass:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdf22b20feeb3493dc6d1eddcd3d4a5541383592

When enabling -clang:-ftrivial-auto-var-init just for opt builds, the opt tests fail and the debug tests pass:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=96b42c960293bb656913cdaedb0821e9ecfea899

I don't have a Windows build environment set up, so that's probably the next step.

I get a crash in patched_NtMapViewOfSection in the "main" process (the thing created by the launcher stub) because the compiler has inserted calls to memset, and we're so super-early in this newborn process that the loader hasn't yet resolved the import of memset from vcruntime140.dll.

At a minimum we'd want to disable var-init on that function (or file), but if that's not enough we might have to disable var-init in the sandbox for similar reasons (see also bug 1437452 comment 41).

(In reply to David Major [:dmajor] (low availability) from comment #18)

I get a crash in patched_NtMapViewOfSection in the "main" process (the thing created by the launcher stub) because the compiler has inserted calls to memset, and we're so super-early in this newborn process that the loader hasn't yet resolved the import of memset from vcruntime140.dll.

Thanks! I saw debug breaks in TargetNtMapViewOfSection when I ran my -ftrivial-auto-var-init firefox.exe build in WinDbg, but I saw the same debug breaks with the official Nightly firefox.exe build so I assumed they were expected.

Disabling -ftrivial-auto-var-init for DllBLocklistWin.cpp appears to fix the startup crash. clang allows -ftrivial-auto-var-init to be disable for individual variable definitions, but not for whole functions.

Here is a Try test run with -ftrivial-auto-var-init disabled for DllBLocklistWin.cpp. There are some oranges, but none of them appear to be related to -ftrivial-auto-var-init.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=85c9c667a7b725745978774eef6786b31fe20a97&selectedJob=241739783

Attachment #9057973 - Attachment description: Bug 1514965 - Part 3: Reuse mingw_clang flag. r?froydnj → Bug 1514965 - Part 1: Refactor mingw_clang checks for reuse. r?froydnj
Attachment #9056364 - Attachment description: Bug 1514965 - Part 1: Enable clang -ftrivial-auto-var-init in debug and Nightly builds. r?froydnj → Bug 1514965 - Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r?froydnj
Attachment #9057972 - Attachment is obsolete: true
Pushed by cpeterson@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b36a9dda426
Part 1: Refactor mingw_clang checks for reuse. r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/7a6e07feb572
Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj

My sense here is "yes", but we might want to talk to the Linux distro people that are active on Bugzilla (SuSE and RedHat, at least; I guess @glandium would speak for Debian?)

I think that enabling should be a separate bug.

I filed bug 1546873 if anyone wants --enable-hardening to also enable clang -ftrivial-auto-var-init.

Blocks: 1546873
Summary: Consider enabling clang -ftrivial-auto-var-init=pattern 0xAA in debug builds (or even Nightly and Dev Edition) → Enable clang -ftrivial-auto-var-init=pattern 0xAA in debug builds

Backed out 2 changesets (Bug 1514965) for linux build bustage

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=242510983&revision=7a6e07feb5722ba601a04efeea2e96811429450e

Backout link: https://hg.mozilla.org/integration/mozilla-inbound/rev/81efffda1929913443e758020a7d627b7af4497c

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242510983&repo=mozilla-inbound&lineNumber=39121

[task 2019-04-25T07:00:11.549Z] 07:00:11 INFO - ../python/mozbuild/mozpack/test/test_files.py::TestMercurialNativeRevisionFinder::test_recognize_repo_paths SKIPPED
[task 2019-04-25T07:00:11.549Z] 07:00:11 INFO - ===================== 48 passed, 3 skipped in 4.79 seconds =====================
[task 2019-04-25T07:00:15.856Z] 07:00:15 INFO - /builds/worker/workspace/build/src/testing/xpcshell/selftest.py
[task 2019-04-25T07:00:15.856Z] 07:00:15 INFO - ============================= test session starts ==============================
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-04-25T07:00:15.857Z] 07:00:15 INFO - collecting ... collected 55 items
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskRunNextTest PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkip TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkipAll TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskStackTrace TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestFailureInside PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestMultiple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestRejected PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestSingle TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestFailing PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestSimple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejection PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejectionJSM PASSED
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAssertStack TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAsyncCleanup TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChild TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildFail TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildHang TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildMozinfo TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseExplicit TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseInManifest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseNotExplicit TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportForeignObject TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportNonSyntaxError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportRefError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.861Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportSyntaxError TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowForeignObject TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowString TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testHangingTimeout <- ../../../../../usr/lib/python2.7/unittest/case.py SKIPPED
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testKnownFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testLogCorrectFileName TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMissingHeadFile PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMozinfo TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskMultiple TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestFail PASSED
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestEmptyTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.862Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPassFail TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testRandomExecution TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.863Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testReturnNonzero PASSED
[task 2019-04-25T07:00:15.863Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkip PASSED
[task 2019-04-25T07:00:15.863Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSyntaxError PASSED
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejection TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejectionJSM TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.864Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnexpectedPass TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.865Z] 07:00:15 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnicodeInAssertMethods TEST-UNEXPECTED-FAIL
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - =================================== FAILURES ===================================
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - ______________________ XPCShellTestsTests.testAddTaskSkip ______________________
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self = <selftest.XPCShellTestsTests testMethod=testAddTaskSkip>
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - def testAddTaskSkip(self):
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self.writeFile("test_tasks_skip.js", ADD_TASK_SKIP)
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - self.writeManifest(["test_tasks_skip.js"])
[task 2019-04-25T07:00:15.865Z] 07:00:15 INFO - > self.assertTestResult(True)
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py:1068:
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - ../testing/xpcshell/selftest.py:528: in assertTestResult
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - """ % ("passed" if expected else "failed", self.log.getvalue()))
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E AssertionError: Tests should have passed, log:
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E ========
[task 2019-04-25T07:00:15.866Z] 07:00:15 INFO - E Found node at /builds/worker/workspace/build/src/node/bin/node
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E Found moz-http2 at /builds/worker/workspace/build/src/testing/xpcshell/moz-http2/moz-http2.js
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E Running tests sequentially.
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E SUITE-START | Running 1 tests
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E profile dir is /tmp/xpcshell/xpcshellprofile
[task 2019-04-25T07:00:15.867Z] 07:00:15 INFO - E TEST-START | test_tasks_skip.js
[task 2019-04-25T07:00:15.867Z] 07:00:15 WARNING - E TEST-UNEXPECTED-FAIL | test_tasks_skip.js | xpcshell return code: -11

Flags: needinfo?(cpeterson)

Nathan, my clang -ftrivial-auto-var-init patches got backed out for xpcshell/selftest.py failures on linux32. The same tests pass on linux64. Other xpcshell tests, and all the regular Firefox tests, also pass on linux32. It's only xpcshell/selftest.py on linux32 that fails (with xpcshell return code -11).

Do you know why xpcshell/selftest.py might behave differently on linux32 and linux64 with clang -ftrivial-auto-var-init? If these failures were similar to the Windows problem dmajor described above about early use of memset during process startup, I would imagine both linux32 and linux64 would be affected.

Flags: needinfo?(cpeterson) → needinfo?(nfroyd)

(In reply to Chris Peterson [:cpeterson] from comment #23)

Nathan, my clang -ftrivial-auto-var-init patches got backed out for xpcshell/selftest.py failures on linux32. The same tests pass on linux64. Other xpcshell tests, and all the regular Firefox tests, also pass on linux32. It's only xpcshell/selftest.py on linux32 that fails (with xpcshell return code -11).

Do you know why xpcshell/selftest.py might behave differently on linux32 and linux64 with clang -ftrivial-auto-var-init? If these failures were similar to the Windows problem dmajor described above about early use of memset during process startup, I would imagine both linux32 and linux64 would be affected.

I don't have any good ideas. It certainly sounds like we're taking some kind of different path for 32-bit vs. 64-bit, but there could be a number of places that could be happening. Sure would be nice if selftest.py reported intelligent errors...

Flags: needinfo?(nfroyd)

Whenever I break selftest.py, the only way I can get a meaningful diagnosis is to run the test locally and see what my debugger complains about. Want to give it a try, Chris?

If I am honest with myself, I know I'll never get around to setting up a Linux32 dev environment to debug my patch's Linux32-only xpcshell test failure. And AFAICT the only Linux32 tests we bother running on mozilla-central are xpcshell and wpt. I'd like to resubmit my -ftrivial-auto-var-init patch but disable it on Linux32. There's little reason to not let other platforms benefit from clang's -ftrivial-auto-var-init.

Since I last submitted my patch, we are now using a version of mingw-clang that understands -ftrivial-auto-var-init=pattern. That removes one special condition.

Disable -ftrivial-auto-var-init=pattern for DllBLocklistWin.cpp with clang-cl because the file's interceptions happen so early in the main process that the loader hasn't yet resolved the import of memset (used by -ftrivial-auto-var-init) from vcruntime140.dll.

Disable -ftrivial-auto-var-init on Linux32 because it causes some xpcshell test failures.

Attachment #9056364 - Attachment is obsolete: true
Attachment #9057973 - Attachment description: Bug 1514965 - Part 1: Refactor mingw_clang checks for reuse. r?froydnj → Bug 1514965 - Part 1: Refactor mingw_clang checks for reuse. r=froydnj
Attachment #9085981 - Attachment description: Bug 1514965 - Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r?froydnj → Bug 1514965 - Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj
Pushed by cpeterson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5ad6f4f0edeb
Part 1: Refactor mingw_clang checks for reuse. r=froydnj
Pushed by cpeterson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5babe33486a1
Part 2: Enable clang -ftrivial-auto-var-init to initialize local variables with 0xAA in debug builds. r=froydnj
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70

Hi Chris,
I got build failed only for Debug bug after rebasing new source code
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'
0:07.21 DEBUG: configure: failed program was:
0:07.21 DEBUG: #line 4529 "configure"
0:07.21 DEBUG: #include "confdefs.h"
0:07.21 DEBUG:
0:07.21 DEBUG: int main() {
0:07.21 DEBUG: return 0;
0:07.21 DEBUG: ; return 0; }
0:07.21 DEBUG: configure:5077: checking for valid debug flags
0:07.21 DEBUG: configure:5088: /usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -std=gnu99 -c -fstack-protector-strong -ftrivial-auto-var-init=pattern -fno-strict-aliasing -g -Qunused-arguments -fstack-protector-strong -ftrivial-auto-var-init=pattern conftest.c 1>&5
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'
0:07.21 DEBUG: clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

I guess that this is related to this change. Running ./mach bootstrap does not help.
I tried to checkout the last version without this commit (8a08eb947133d0be8d5eeb8a9d9a3774e99eae61) then I could build without any error.
I guess that we may update some dependencies in bootstrap. Could you please take a look?

Flags: needinfo?(cpeterson)

Which version of clang do you have?

clang --version

Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Thanks

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

Thomas, if you need to get unblocked in the meantime, you can temporarily use ac_add_options --disable-hardening.

(In reply to :dmajor from comment #35)

Thomas, if you need to get unblocked in the meantime, you can temporarily use ac_add_options --disable-hardening.
0:13.68 js/src> checking for -framework ExceptionHandling... no
0:13.71 js/src> checking for -dead_strip option to ld... no
0:13.74 js/src> checking for valid debug flags... no
0:13.74 js/src> configure: error: These compiler flags are invalid: -g
0:13.74 js/src> ERROR: old-configure failed

That does not work, the lastest version works for me is 8a08eb947133d0be8d5eeb8a9d9a3774e99eae61

(In reply to Thomas Nguyen from comment #33)

Apple LLVM version 9.1.0 (clang-902.0.39.2)
...
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Thomas, is there any specific reason you are using Xcode's clang instead of the clang installed by mach bootstrap in $HOME/.mozbuild/clang/bin?

Mozilla's macOS builders explicitly point CC and CXX to mozbuild's clang, so it's probably best for Firefox developers on macOS to also use mozbuild's clang like:

https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/build/macosx/local-mozconfig.common#9-10

    export CC=$MOZ_FETCHES_DIR/clang/bin/clang
    export CXX=$MOZ_FETCHES_DIR/clang/bin/clang++

(In reply to :dmajor from comment #34)

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

I knew that Apple's clang had a different version number than mozbuild's and Homebrew's clang builds, but I had hoped Firefox developers on macOS were either using mozbuild's clang or an Xcode with a new enough clang. :) That's apparently not a safe assumption, so I can add a macOS patch to either skip -ftrivial-auto-var-init or check for Xcode clang >= 10.0. I believe -ftrivial-auto-var-init first appeared in Xcode's clang 10.0. I have Xcode 10.3 with clang 10.0.1 on my macOS 10.14.6 machine.

Flags: needinfo?(cpeterson) → needinfo?(tnguyen)

Sorry, I don't remember exactly why I changed to Apple's clang. I guess I had to fix some build issue, but it has been worked for a long time.
Anw, it is always worth to make some note in our document the minimum version of clang (and recommendation one). I will try to change to mozbuild's clang and let you know. Thanks

Flags: needinfo?(tnguyen)

(In reply to :dmajor from comment #34)

Apple clang versioning strikes again. :( Maybe we should just leave the feature turned off for Darwin hosts?

I knew that Apple's clang had a different version number than mozbuild's and Homebrew's clang builds, but I had hoped Firefox developers on macOS were either using mozbuild's clang or an Xcode with a new enough clang. :) That's apparently not a safe assumption, so I can add a macOS patch to either skip -ftrivial-auto-var-init or check for Xcode clang >= 10.0. I believe -ftrivial-auto-var-init first appeared in Xcode's clang 10.0. I have Xcode 10.3 with clang 10.0.1 on my macOS 10.14.6 machine.

10.0 does not work. I have upgraded to 10.0.0 and it is still failed

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

(In reply to :dmajor from comment #40)

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

I did not try, but I did another upgrading, the first version works for me is clang 10.0.1.

(In reply to :dmajor from comment #40)

How about using check_and_add_flags ? Conveniently, toolchain.configure already includes the necessary compile-checks.configure.

(In reply to Thomas Nguyen from comment #41)

I did not try, but I did another upgrading, the first version works for me is clang 10.0.1.

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

(In reply to Chris Peterson [:cpeterson] from comment #42)

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

I'm testing a patch that uses check_and_add_flags to detect -ftrivial-auto-var-init support, but it introduces a new complications. For one, check_and_add_flags is a no-op for clang-cl, AFAICT:

https://searchfox.org/mozilla-central/rev/dafa68a84f01897c8cfb4bbaf6829cae8df4eb57/build/moz.configure/compile-checks.configure#156-159,174-177

And even if it check_and_add_flags checked clang-cl flags, -Xclang is needed to pass -ftrivial-auto-var-init to clang-cl and a -Xclang -ftrivial-auto-var-init check wouldn't work because passing an unknown argument to -Xclang fails silently so the configure check will never fail.

At this time, I don't think changing the current -ftrivial-auto-var-init check is worth the trouble. tnguyen has been the only person to report (in this bug) problems with the check and updating to the latest Xcode version fixed the problem. The current -ftrivial-auto-var-init check works correctly with mozbuild's blessed clang and clang-cl, clang ToT, and Apple clang (if you have a Xcode >= 10.2 from March 2019). If a developer wants to build with Apple clang instead of mozbuild's blessed clang, I think updating to an Xcode version from 2019 is a reasonable requirement. (The previous Xcode version 10.1 was released in October 2018.)

(In reply to Chris Peterson [:cpeterson] from comment #43)

(In reply to Chris Peterson [:cpeterson] from comment #42)

I will look into using check_and_add_flags or requiring clang >= 10.0.1 on macOS (though that means macOS builders, currently using mozbuild's clang 8.0, won't enable -ftrivial-auto-var-init for official debug builds used in automated tests).

I'm testing a patch that uses check_and_add_flags to detect -ftrivial-auto-var-init support, but it introduces a new complications. For one, check_and_add_flags is a no-op for clang-cl, AFAICT:

https://searchfox.org/mozilla-central/rev/dafa68a84f01897c8cfb4bbaf6829cae8df4eb57/build/moz.configure/compile-checks.configure#156-159,174-177

And even if it check_and_add_flags checked clang-cl flags, -Xclang is needed to pass -ftrivial-auto-var-init to clang-cl and a -Xclang -ftrivial-auto-var-init check wouldn't work because passing an unknown argument to -Xclang fails silently so the configure check will never fail.

tjr has been working to make check_and_add_flags work for clang-cl. I can't remember whether -ftrivial-auto-var-init exists in clang 8.0, which is our minimum version on Windows, but if it does, you could unconditionally pass it.

That's over in Bug 1581018 - but it's blocked on actually fixing the warnings, which is not something I'm dedicating foreground time to right now...

fyi, I am using clang from mozbuild on linux, however I met the failure.

0:01.55 checking for host system type... x86_64-pc-linux-gnu
0:01.55 checking for target system type... x86_64-pc-linux-gnu
...
0:01.94 checking for the target C compiler... /home/allstars/.mozbuild/clang/bin/clang
0:01.98 checking whether the target C compiler can be used... yes
0:01.98 checking the target C compiler version... 8.0.1
0:09.23 checking the target C compiler works... yes
0:09.23 checking for the target C++ compiler... /home/allstars/.mozbuild/clang/bin/clang++
0:09.25 checking whether the target C++ compiler can be used... yes
0:09.25 checking the target C++ compiler version... 8.0.1
0:09.29 checking the target C++ compiler works... yes
0:09.29 checking for the host C compiler... /home/allstars/.mozbuild/clang/bin/clang
0:09.33 checking whether the host C compiler can be used... yes
0:09.33 checking the host C compiler version... 8.0.1
0:09.37 checking the host C compiler works... yes
0:09.37 checking for the host C++ compiler... /home/allstars/.mozbuild/clang/bin/clang++
0:09.39 checking whether the host C++ compiler can be used... yes
0:09.39 checking the host C++ compiler version... 8.0.1
....
1:11.32 Can't read /proc/cpuinfo: No such file or directory
1:11.32 clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

I have to do ac_add_options --disable-hardening to bypass this. (from comment 35)

(In reply to Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh] from comment #46)

fyi, I am using clang from mozbuild on linux, however I met the failure.

0:09.39 checking whether the host C++ compiler can be used... yes
0:09.39 checking the host C++ compiler version... 8.0.1
....
1:11.32 Can't read /proc/cpuinfo: No such file or directory
1:11.32 clang: error: unknown argument: '-ftrivial-auto-var-init=pattern'

Hi Yoshi, sorry about that.

This is surprising because the Linux build machines are also using mozbuild's clang 8.0.1 (as shown by > checking the target C++ compiler version... 8.0.1 in the configure log from a random mozilla-central Linux64 build. That build's clang version check worked because it does not attempt to use -ftrivial-auto-var-init=pattern.

Is this a recent regression on your machine? Have you run mach bootstrap lately? Can you please share you build log file in this bug (as a file attachment because it might be big)?

Would it be possible that it's because of icecc? and the failure is from another machine?
I have another machine at home and using clang 8 on linux and it is all fine,
https://pastebin.com/kBvfuEET

(In reply to Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh] from comment #48)

Would it be possible that it's because of icecc? and the failure is from another machine?
I have another machine at home and using clang 8 on linux and it is all fine,

That could be. Perhaps the remote machine has an older clang version and the toolchain.configure check for -ftrivial-auto-var-init support is only running locally? Maybe I can add a check for icecc. The proper fix would be to actually feature detect -ftrivial-auto-var-init support using check_and_add_flags, but that has its own problem (described in comment 43).

https://searchfox.org/mozilla-central/rev/7cc0f0e89cb40e43bf5c96906f13d44705401042/build/moz.configure/toolchain.configure#1217-1226

btw, I see there is another error in your log:

2:13.07 DEBUG: configure: error: Can't find header linux/joystick.h, needed for gamepad support. Please install Linux kernel headers.`

You need to log in before you can comment on or make changes to this bug.