Intermittent make[4]: *** [Unified_cpp_dom_base2.o] Error 4
Categories
(Firefox Build System :: Task Configuration, defect, P5)
Tracking
(firefox-esr68 fixed, firefox73 fixed, firefox74 fixed, firefox75 fixed)
People
(Reporter: intermittent-bug-filer, Assigned: wcosta)
References
(Regression)
Details
(Keywords: regression)
Attachments
(3 files, 1 obsolete file)
Filed by: dluca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=285110322&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZBp5A99QR_edb4U3-aIiBA/runs/0/artifacts/public/logs/live_backing.log
WARNING - {standard input}:121041: Warning: end of file not at end of a line; newline inserted
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - {standard input}:121197: Error: unknown pseudo-op: `.lv'
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - g++: internal compiler error: Killed (program cc1plus)
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - Please submit a full bug report,
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - with preprocessed source if appropriate.
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - See <https://gcc.gnu.org/bugs/> for instructions.
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - /builds/worker/workspace/build/src/config/rules.mk:744: recipe for target 'Unified_cpp_dom_base2.o' failed
[task 2020-01-15T23:16:17.483Z] 23:16:17 ERROR - make[4]: *** [Unified_cpp_dom_base2.o] Error 4
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/base'
[task 2020-01-15T23:16:17.483Z] 23:16:17 INFO - make[4]: *** Waiting for unfinished jobs....
[task 2020-01-15T23:16:17.485Z] 23:16:17 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.485Z] 23:16:17 INFO - gfx/angle/targets/translator/FindFunction.o
[task 2020-01-15T23:16:17.485Z] 23:16:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.702Z] 23:16:17 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.704Z] 23:16:17 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o WrapSwitchStatementsInBlocks.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2 -MD -MP -MF .deps/WrapSwitchStatementsInBlocks.o.pp /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_ops/WrapSwitchStatementsInBlocks.cpp
[task 2020-01-15T23:16:17.704Z] 23:16:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.745Z] 23:16:17 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.746Z] 23:16:17 INFO - mkdir -p '.deps/'
[task 2020-01-15T23:16:17.746Z] 23:16:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17 INFO - dom/filesystem/compat/Unified_cpp_filesystem_compat0.o
[task 2020-01-15T23:16:17.762Z] 23:16:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:20.267Z] 23:16:20 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.276Z] 23:16:20 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o FindFunction.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2 -MD -MP -MF .deps/FindFunction.o.pp /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_util/FindFunction.cpp
[task 2020-01-15T23:16:20.276Z] 23:16:20 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20 INFO - gfx/angle/targets/translator/FindMain.o
[task 2020-01-15T23:16:20.281Z] 23:16:20 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.573Z] 23:16:20 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.576Z] 23:16:20 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o Unified_cpp_dom_canvas4.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/canvas -I/builds/worker/workspace/build/src/obj-firefox/dom/canvas -I/builds/worker/workspace/build/src/js/xpconnect/wrappers -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/html -I/builds/worker/workspace/build/src/dom/svg -I/builds/worker/workspace/build/src/dom/workers -I/builds/worker/workspace/build/src/dom/xul -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/gl -I/builds/worker/workspace/build/src/image -I/builds/worker/workspace/build/src/js/xpconnect/src -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/style -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/media/libyuv/libyuv/include -I/builds/worker/workspace/build/src/gfx/skia -I/builds/worker/workspace/build/src/gfx/skia/skia -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/widget/gtk/compat-gtk3 -pthread -I/usr/include/gtk-3.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/gtk-3.0/unix-print -Wno-error=shadow -Wno-missing-braces -Wno-error=unused-result -MD -MP -MF .deps/Unified_cpp_dom_canvas4.o.pp Unified_cpp_dom_canvas4.cpp
[task 2020-01-15T23:16:20.577Z] 23:16:20 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.593Z] 23:16:20 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.593Z] 23:16:20 INFO - gfx/angle/targets/translator/FindSymbolNode.o
Comment 1•5 years ago
|
||
Can any of you guys please take a look at this bug?
Comment 2•5 years ago
|
||
We switched from AWS workers to GCP workers for these builds, which is probably relevant, but it seems like it may also have something to do with the toolchains.
Updated•5 years ago
|
Comment 3•5 years ago
|
||
Quoting myself from slack: "if it's intermittent, I'd shrug it off"
Updated•5 years ago
|
Comment 4•5 years ago
|
||
Seems memory is too tight for the number of cores on those workers. Filed bug 1609595 to adjust at the build system level and bug 1609596 to get a better picture.
Comment hidden (Intermittent Failures Robot) |
Comment 6•5 years ago
|
||
I pretty much confirmed comment 4 in bug 1609595. Since only some gcc-based builds are affected, I suggest we set MOZ_PARALLEL_BUILD=24 on the affected builds. But we should avoid doing it while we're still on AWS...
Assignee | ||
Comment 7•5 years ago
•
|
||
I believe it is just a matter of increasing the RAM size in GCP. I will work on it today.
Update: I now see it happens only for toolchain builds, am I right? So maybe it is better to adjust the builds as :glandium suggested instead of making the instances more expensive.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 10•5 years ago
|
||
This is to fix OOM in GCP build instances.
Updated•5 years ago
|
Comment 11•5 years ago
|
||
Depends on D62103
Comment 12•5 years ago
|
||
Comment 13•5 years ago
|
||
bugherder |
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 14•5 years ago
|
||
In GCP we have the double the number of core compared to AWS
counterparts, but we use the same amount of memory. Request the builds
to be less parallel to avoid OOM.
Comment 15•5 years ago
|
||
Comment 16•5 years ago
|
||
These kind of failures are still happening even after the latest patch: https://treeherder.mozilla.org/logviewer.html#?job_id=288470457&repo=autoland
Th link: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=288470457&searchStr=Linux%2Cx64%2Cdebug%2Cbuild-linux64-fuzzing%2Fdebug%2C%28Bf%29&revision=174d5be06caa95bdc01153cdd8cff5152f406b6f
Assignee | ||
Comment 17•5 years ago
|
||
We use 16 cores in EC2 with 32GB, but in GCE we have 32 cores. We increase
the available memory to get the same proportion of memory/core.
Assignee | ||
Comment 18•5 years ago
|
||
This patch increases the available RAM in GCE instances.
Comment 19•5 years ago
|
||
bugherder |
Comment 20•5 years ago
|
||
bugherder uplift |
Comment 21•5 years ago
|
||
bugherder uplift |
Comment 22•5 years ago
|
||
bugherder uplift |
Comment hidden (Intermittent Failures Robot) |
Updated•5 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Description
•