1626951 - Perma build bustage fatal error: error in backend: SEH unwind data splitting not yet implemented when Gecko 76 merges to Beta on 2020-04-06

Daniel Varga [:dvarga]

Reporter

Description

•

4 years ago

•

Edited

Central as beta simulation

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=295918728&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f024b83541b154784db846e696e9a46a2e513084

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=295918728&repo=try&lineNumber=51591

[task 2020-04-02T11:37:17.001Z] 11:37:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/obj-build/dom/bindings/test'
[task 2020-04-02T11:43:28.201Z] 11:43:28     INFO -  make[4]: Entering directory '/builds/worker/workspace/obj-build/layout/style'
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoUnified_cpp_layout_style4.obj -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DUNICODE -D_UNICODE -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -D_SECURE_ATL -DCHROMIUM_BUILD -DU_STATIC_IMPLEMENTATION -DOS_WIN=1 -DWIN32 -D_WIN32 -D_WINDOWS -DWIN32_LEAN_AND_MEAN -DCOMPILER_MSVC -DWINAPI_NO_BUNDLED_LIBRARIES -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/layout/style -I/builds/worker/workspace/obj-build/layout/style -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/checkouts/gecko/ipc/glue -I/builds/worker/checkouts/gecko/layout/base -I/builds/worker/checkouts/gecko/layout/generic -I/builds/worker/checkouts/gecko/layout/svg -I/builds/worker/checkouts/gecko/layout/xul -I/builds/worker/checkouts/gecko/dom/base -I/builds/worker/checkouts/gecko/dom/html -I/builds/worker/checkouts/gecko/dom/xul -I/builds/worker/checkouts/gecko/image -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -Xclang -fexperimental-new-pass-manager  -Xclang -MP -Xclang -dependency-file -Xclang .deps/Unified_cpp_layout_style4.obj.pp -Xclang -MT -Xclang Unified_cpp_layout_style4.obj   Unified_cpp_layout_style4.cpp
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  fatal error: error in backend: SEH unwind data splitting not yet implemented
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: error: clang frontend command failed with exit code 70 (use -v to see invocation)
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang version 9.0.1
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  Target: aarch64-unknown-windows-msvc
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  Thread model: posix
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  InstalledDir: /builds/worker/fetches/clang/bin
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: note: diagnostic msg:
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  ********************
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  Preprocessed source(s) and associated run script(s) are located at:
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_style4-cf1aa1.cpp
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_style4-cf1aa1.sh
[task 2020-04-02T11:43:28.205Z] 11:43:28     INFO -  clang-9: note: diagnostic msg:
[task 2020-04-02T11:43:28.206Z] 11:43:28     INFO -  ********************
[task 2020-04-02T11:43:28.206Z] 11:43:28     INFO -  /builds/worker/checkouts/gecko/config/rules.mk:750: recipe for target 'Unified_cpp_layout_style4.obj' failed
[task 2020-04-02T11:43:28.206Z] 11:43:28    ERROR -  make[4]: *** [Unified_cpp_layout_style4.obj] Error 70
[task 2020-04-02T11:43:28.206Z] 11:43:28     INFO -  make[4]: Leaving directory '/builds/worker/workspace/obj-build/layout/style'
[task 2020-04-02T11:43:28.206Z] 11:43:28     INFO -  /builds/worker/checkouts/gecko/config/recurse.mk:74: recipe for target 'layout/style/target-objects' failed
[task 2020-04-02T11:43:28.206Z] 11:43:28    ERROR -  make[3]: *** [layout/style/target-objects] Error 2
[task 2020-04-02T11:43:28.206Z] 11:43:28     INFO -  make[3]: *** Waiting for unfinished jobs....
[task 2020-04-02T11:43:39.454Z] 11:43:39     INFO -  make[4]: Entering directory '/builds/worker/workspace/obj-build/layout/generic'
[task 2020-04-02T11:43:39.457Z] 11:43:39     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoUnified_cpp_layout_generic3.obj -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DUNICODE -D_UNICODE -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -D_SECURE_ATL -DCHROMIUM_BUILD -DU_STATIC_IMPLEMENTATION -DOS_WIN=1 -DWIN32 -D_WIN32 -D_WINDOWS -DWIN32_LEAN_AND_MEAN -DCOMPILER_MSVC -DWINAPI_NO_BUNDLED_LIBRARIES -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/layout/generic -I/builds/worker/workspace/obj-build/layout/generic -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/checkouts/gecko/ipc/glue -I/builds/worker/checkouts/gecko/dom/plugins/base -I/builds/worker/checkouts/gecko/layout/base -I/builds/worker/checkouts/gecko/layout/forms -I/builds/worker/checkouts/gecko/layout/painting -I/builds/worker/checkouts/gecko/layout/style -I/builds/worker/checkouts/gecko/layout/svg -I/builds/worker/checkouts/gecko/layout/tables -I/builds/worker/checkouts/gecko/layout/xul -I/builds/worker/checkouts/gecko/docshell/base -I/builds/worker/checkouts/gecko/dom/base -I/builds/worker/checkouts/gecko/dom/html -I/builds/worker/checkouts/gecko/dom/xul -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -I/builds/worker/workspace/obj-build/dist/include/cairo -Xclang -fexperimental-new-pass-manager  -Xclang -MP -Xclang -dependency-file -Xclang .deps/Unified_cpp_layout_generic3.obj.pp -Xclang -MT -Xclang Unified_cpp_layout_generic3.obj   Unified_cpp_layout_generic3.cpp
[task 2020-04-02T11:43:39.457Z] 11:43:39     INFO -  fatal error: error in backend: SEH unwind data splitting not yet implemented
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: error: clang frontend command failed with exit code 70 (use -v to see invocation)
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang version 9.0.1
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  Target: aarch64-unknown-windows-msvc
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  Thread model: posix
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  InstalledDir: /builds/worker/fetches/clang/bin
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: note: diagnostic msg:
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  ********************
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  Preprocessed source(s) and associated run script(s) are located at:
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_generic3-f8dc06.cpp
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_generic3-f8dc06.sh
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  clang-9: note: diagnostic msg:
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  ********************
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  /builds/worker/checkouts/gecko/config/rules.mk:750: recipe for target 'Unified_cpp_layout_generic3.obj' failed
[task 2020-04-02T11:43:39.458Z] 11:43:39    ERROR -  make[4]: *** [Unified_cpp_layout_generic3.obj] Error 70
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  make[4]: Leaving directory '/builds/worker/workspace/obj-build/layout/generic'
[task 2020-04-02T11:43:39.458Z] 11:43:39     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2020-04-02T11:43:53.785Z] 11:43:53     INFO -  make[4]: Entering directory '/builds/worker/workspace/obj-build/layout/painting'
[task 2020-04-02T11:43:53.785Z] 11:43:53     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoUnified_cpp_layout_painting0.obj -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DUNICODE -D_UNICODE -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -D_SECURE_ATL -DCHROMIUM_BUILD -DU_STATIC_IMPLEMENTATION -DOS_WIN=1 -DWIN32 -D_WIN32 -D_WINDOWS -DWIN32_LEAN_AND_MEAN -DCOMPILER_MSVC -DWINAPI_NO_BUNDLED_LIBRARIES -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/layout/painting -I/builds/worker/workspace/obj-build/layout/painting -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/checkouts/gecko/ipc/glue -I/builds/worker/checkouts/gecko/docshell/base -I/builds/worker/checkouts/gecko/dom/base -I/builds/worker/checkouts/gecko/gfx/2d -I/builds/worker/checkouts/gecko/layout/base -I/builds/worker/checkouts/gecko/layout/generic -I/builds/worker/checkouts/gecko/layout/style -I/builds/worker/checkouts/gecko/layout/svg -I/builds/worker/checkouts/gecko/layout/tables -I/builds/worker/checkouts/gecko/layout/xul -I/builds/worker/checkouts/gecko/gfx/skia -I/builds/worker/checkouts/gecko/gfx/skia/skia -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -I/builds/worker/workspace/obj-build/dist/include/cairo -Xclang -fexperimental-new-pass-manager  -Xclang -MP -Xclang -dependency-file -Xclang .deps/Unified_cpp_layout_painting0.obj.pp -Xclang -MT -Xclang Unified_cpp_layout_painting0.obj   Unified_cpp_layout_painting0.cpp
[task 2020-04-02T11:43:53.785Z] 11:43:53     INFO -  fatal error: error in backend: SEH unwind data splitting not yet implemented
[task 2020-04-02T11:43:53.785Z] 11:43:53     INFO -  clang-9: error: clang frontend command failed with exit code 70 (use -v to see invocation)
[task 2020-04-02T11:43:53.785Z] 11:43:53     INFO -  clang version 9.0.1
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  Target: aarch64-unknown-windows-msvc
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  Thread model: posix
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  InstalledDir: /builds/worker/fetches/clang/bin
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  clang-9: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  clang-9: note: diagnostic msg:
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  ********************
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  Preprocessed source(s) and associated run script(s) are located at:
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_painting0-14ee3b.cpp
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  clang-9: note: diagnostic msg: /builds/worker/artifacts/Unified_cpp_layout_painting0-14ee3b.sh
[task 2020-04-02T11:43:53.786Z] 11:43:53     INFO -  clang-9: note: diagnostic msg:

Flags: needinfo?(dmajor)

Daniel Varga [:dvarga]

Reporter

Comment 1

•

4 years ago

[Tracking Requested - why for this release]:

status-firefox74: --- → unaffected

status-firefox75: --- → unaffected

status-firefox76: --- → affected

status-firefox-esr68: --- → unaffected

tracking-firefox76: --- → ?

(Away)

Comment 2

•

4 years ago

(In reply to Daniel Varga [:dvarga] from comment #0)

Did we do this build on previous beta simulations? When did it start failing?

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f024b83541b154784db846e696e9a46a2e513084&selectedJob=295929713

This points to a bunch of mochitests. Is the link correct?

Flags: needinfo?(dmajor) → needinfo?(dvarga)

Daniel Varga [:dvarga]

Reporter

Comment 3

•

4 years ago

Changed the link to point to the build bustage.
On yesterday's run it was green: https://treeherder.mozilla.org/#/jobs?repo=try&revision=04be6ee26bb3182a09f340d814d19b5540688f81&searchStr=windows%2C2012%2Caarch64%2Copt%2Cbuild-win64-aarch64%2Fopt%2C%28b%29

Flags: needinfo?(dvarga)

(Away)

Comment 4

•

4 years ago

OK thanks. I'm looking.

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 5

•

4 years ago

Regression range: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=205d1b4e8f6946fa8ef35c86febdd338c1159715&tochange=95ddb3213aecaf87c0a4f2d3559bd17e0194148c

(Away)

Comment 6

•

4 years ago

I expect that it was started with bug 1607552, but it's not that bug's fault. More likely this is the same underlying issue as bug 1622168.

Mike Hommey [:glandium]

Assignee

Comment 7

•

4 years ago

So, like, disabling LTO triggered that? But why not on central?

(Away)

Comment 8

•

4 years ago

Likely the same (unknown) reason that bug 1622168 didn't happen on central.

Mike Hommey [:glandium]

Assignee

Comment 9

•

4 years ago

I manually retriggered the task with MOZ_LTO=cross set at the taskcluster level, and that turned in green. So at least that part is confirmed.

Mike Hommey [:glandium]

Assignee

Comment 10

•

4 years ago

Corollary: maybe the cause of bug 1622168 is not PGO, but LTO.

(Away)

Comment 11

•

4 years ago

Entirely possible. It's also possible that either one of them might suffice on its own, given that other modifications like debug/asan/ccov also hide the bug.

Mike Hommey [:glandium]

Assignee

Comment 12

•

4 years ago

Attached file Bug 1626951 - Disable new pass manager on aarch64-windows builds without LTO. — Details

Phabricator Automation

Updated

•

4 years ago

Assignee: nobody → mh+mozilla

Status: NEW → ASSIGNED

Pulsebot

Comment 13

•

4 years ago

Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/a889aea10860
Disable new pass manager on aarch64-windows builds without LTO. r=dmajor

Bogdan Tara[:bogdan_tara | bogdant]

Comment 14

•

4 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/a889aea10860

Status: ASSIGNED → RESOLVED

Closed: 4 years ago

status-firefox76: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla76

Cosmin Sabou [:CosminS]

Comment 15

•

4 years ago

verified in today's beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&author=csabou%40mozilla.com&fromchange=2afa785eb76435da609c25e3d6c52f19968af921&searchStr=Windows%2C2012%2CAArch64%2Copt%2Cbuild-win64-aarch64%2Fopt%2C%28B%29&selectedJob=296262699

Status: RESOLVED → VERIFIED

status-firefox76: fixed → verified

BugBot [:suhaib / :marco/ :calixte]

Updated

•

4 years ago

Keywords: regression

Ryan VanderMeulen [:RyanVM]

Updated

•

4 years ago

tracking-firefox76: ? → blocking

(Away)

Updated

•

4 years ago

Blocks: 1677839

BMO Automation

Updated

•

4 years ago

Has Regression Range: --- → yes