Closed Bug 1609568 Opened 3 months ago Closed 2 months ago

Intermittent make[4]: *** [Unified_cpp_dom_base2.o] Error 4

Categories

(Firefox Build System :: Task Configuration, defect, P5, blocker)

defect

Tracking

(firefox-esr68 fixed, firefox73 fixed, firefox74 fixed, firefox75 fixed)

RESOLVED FIXED
mozilla75
Tracking Status
firefox-esr68 --- fixed
firefox73 --- fixed
firefox74 --- fixed
firefox75 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: wcosta)

References

(Blocks 1 open bug, Regression)

Details

Attachments

(3 files, 1 obsolete file)

Filed by: dluca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=285110322&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZBp5A99QR_edb4U3-aIiBA/runs/0/artifacts/public/logs/live_backing.log


 WARNING -  {standard input}:121041: Warning: end of file not at end of a line; newline inserted
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  {standard input}:121197: Error: unknown pseudo-op: `.lv'
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  g++: internal compiler error: Killed (program cc1plus)
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  Please submit a full bug report,
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  with preprocessed source if appropriate.
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  See <https://gcc.gnu.org/bugs/> for instructions.
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  /builds/worker/workspace/build/src/config/rules.mk:744: recipe for target 'Unified_cpp_dom_base2.o' failed
[task 2020-01-15T23:16:17.483Z] 23:16:17    ERROR -  make[4]: *** [Unified_cpp_dom_base2.o] Error 4
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/base'
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  gfx/angle/targets/translator/FindFunction.o
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.702Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.704Z] 23:16:17     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o WrapSwitchStatementsInBlocks.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2  -MD -MP -MF .deps/WrapSwitchStatementsInBlocks.o.pp   /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_ops/WrapSwitchStatementsInBlocks.cpp
[task 2020-01-15T23:16:17.704Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.745Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.746Z] 23:16:17     INFO -  mkdir -p '.deps/'
[task 2020-01-15T23:16:17.746Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  dom/filesystem/compat/Unified_cpp_filesystem_compat0.o
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:20.267Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.276Z] 23:16:20     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o FindFunction.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2  -MD -MP -MF .deps/FindFunction.o.pp   /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_util/FindFunction.cpp
[task 2020-01-15T23:16:20.276Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20     INFO -  gfx/angle/targets/translator/FindMain.o
[task 2020-01-15T23:16:20.281Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.573Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.576Z] 23:16:20     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o Unified_cpp_dom_canvas4.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/canvas -I/builds/worker/workspace/build/src/obj-firefox/dom/canvas -I/builds/worker/workspace/build/src/js/xpconnect/wrappers -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/html -I/builds/worker/workspace/build/src/dom/svg -I/builds/worker/workspace/build/src/dom/workers -I/builds/worker/workspace/build/src/dom/xul -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/gl -I/builds/worker/workspace/build/src/image -I/builds/worker/workspace/build/src/js/xpconnect/src -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/style -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/media/libyuv/libyuv/include -I/builds/worker/workspace/build/src/gfx/skia -I/builds/worker/workspace/build/src/gfx/skia/skia -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/widget/gtk/compat-gtk3 -pthread -I/usr/include/gtk-3.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/gtk-3.0/unix-print -Wno-error=shadow -Wno-missing-braces -Wno-error=unused-result  -MD -MP -MF .deps/Unified_cpp_dom_canvas4.o.pp   Unified_cpp_dom_canvas4.cpp
[task 2020-01-15T23:16:20.577Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.593Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.593Z] 23:16:20     INFO -  gfx/angle/targets/translator/FindSymbolNode.o

Can any of you guys please take a look at this bug?

Flags: needinfo?(wcosta)
Flags: needinfo?(mh+mozilla)

We switched from AWS workers to GCP workers for these builds, which is probably relevant, but it seems like it may also have something to do with the toolchains.

Severity: normal → blocker

Quoting myself from slack: "if it's intermittent, I'd shrug it off"

Flags: needinfo?(mh+mozilla)
Regressed by: 1547111

Seems memory is too tight for the number of cores on those workers. Filed bug 1609595 to adjust at the build system level and bug 1609596 to get a better picture.

I pretty much confirmed comment 4 in bug 1609595. Since only some gcc-based builds are affected, I suggest we set MOZ_PARALLEL_BUILD=24 on the affected builds. But we should avoid doing it while we're still on AWS...

Component: General → Task Configuration
Product: Taskcluster → Firefox Build System

I believe it is just a matter of increasing the RAM size in GCP. I will work on it today.

Update: I now see it happens only for toolchain builds, am I right? So maybe it is better to adjust the builds as :glandium suggested instead of making the instances more expensive.

Flags: needinfo?(wcosta)

This is to fix OOM in GCP build instances.

Assignee: nobody → wcosta
Status: NEW → ASSIGNED
Blocks: 1547111
Depends on: 1609595
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2ad798238dcc
Request less parallelism for gcc base-toolchain builds. r=froydnj
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla75
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

In GCP we have the double the number of core compared to AWS
counterparts, but we use the same amount of memory. Request the builds
to be less parallel to avoid OOM.

Pushed by aciure@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/aa01bf29eae7
request less parallelism for all gcc linux build r=#build

We use 16 cores in EC2 with 32GB, but in GCE we have 32 cores. We increase
the available memory to get the same proportion of memory/core.

This patch increases the available RAM in GCE instances.

Flags: needinfo?(wcosta)
Status: REOPENED → RESOLVED
Closed: 2 months ago2 months ago
Resolution: --- → FIXED
Attachment #9125971 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.