1094653 - "collect2: error: ld terminated with signal 9 [Killed]" building 33.0.3 build2

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Both linux-i686 and linux-x86_64 have this problem, mac and windows still compiling.

Logs:
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0.3-candidates/build2/logs/release-mozilla-release-linux64_build-bm74-build1-build0.txt.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/33.0.3-candidates/build2/logs/release-mozilla-release-linux_build-bm74-build1-build1.txt.gz

Snippet:
rm -f libxul.so
/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/_virtualenv/bin/python /builds/slave/rel-m-rel-l64_bld-000000000000/build/config/expandlibs_exec.py --depend .deps/libxul.so.pp --target libxul.so --uselist --  /builds/slave/rel-m-rel-l64_bld-000000000000/build/gcc/bin/g++  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Werror=int-to-pointer-cast -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -Wno-error=uninitialized -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=coverage-mismatch -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -fprofile-generate -O3 -fomit-frame-pointer -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libxul.so     -lpthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id   -fprofile-generate  -Wl,-rpath-link,/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib   ../../../toolkit/library/libxul.a -Wl,-version-script,symverscript /builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/lib/libjs_static.a -L/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3   -L/usr/lib64 -lXrender -lXext -lXdamage -lXfixes -lXcomposite  ../../../dist/lib/libmozsqlite3.a  /builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/modules/zlib/src/libmozz.a ../../../layout/media/libgkmedias.a -lasound   -lrt -L/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/lib -lnspr4 -lplc4 -lplds4 ../../../dist/lib/libmozalloc.a -L/lib64 -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0   -lgtk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lfreetype -lfontconfig -lgdk-x11-2.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0    -L/usr/lib64 -lX11  -lXext  -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0   -lXt -lgthread-2.0 -lfreetype   -lfontconfig   ../../../toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a /builds/slave/rel-m-rel-l64_bld-000000000000/build/toolkit/library/StaticXULComponents.ld  -ldl  -lrt  ../../../build/unix/stdc++compat/libstdc++compat.a   
Executing: ../../../../gcc/bin/g++ -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Werror=int-to-pointer-cast -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -Wno-error=uninitialized -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=coverage-mismatch -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe -DNDEBUG -DTRIMMED -g -fprofile-generate -O3 -fomit-frame-pointer -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libxul.so -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -fprofile-generate -Wl,-rpath-link,/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib /builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/toolkit/library/build/tmp7ElH1R.list ../../../xpcom/typelib/xpt/libxpt.a ../../../intl/unicharutil/util/internal/libintl_unicharutil_util_internal.a -Wl,-version-script,symverscript ../../../dist/lib/libjs_static.a -L/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3 -L/usr/lib64 -lXrender -lXext -lXdamage -lXfixes -lXcomposite ../../../dist/lib/libmozsqlite3.so -lasound -lrt -L/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/dist/lib -lnspr4 -lplc4 -lplds4 ../../../dist/lib/libmozalloc.so -L/lib64 -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0 -lgtk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lfreetype -lfontconfig -lgdk-x11-2.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0 -L/usr/lib64 -lX11 -lXext -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0 -lXt -lgthread-2.0 -lfreetype -lfontconfig ../../../../toolkit/library/StaticXULComponents.ld -ldl -lrt ../../../build/unix/stdc++compat/libstdc++compat.a
/builds/slave/rel-m-rel-l64_bld-000000000000/build/obj-firefox/toolkit/library/build/tmp7ElH1R.list:
...
    INPUT("../../../gfx/skia/SkPictureUtils.i_o")
    INPUT("../../../gfx/skia/SkProxyCanvas.i_o")
    INPUT("../../../gfx/skia/SkRTConf.i_o")
    INPUT("../../../gfx/skia/SkThreadUtils_pthread.i_o")
    INPUT("../../../gfx/skia/SkUnitMappers.i_o")
    INPUT("../StaticXULComponentsEnd/StaticXULComponentsEnd.i_o")

collect2: error: ld terminated with signal 9 [Killed]

This is on https://hg.mozilla.org/releases/mozilla-release/rev/983a710b51c4 - there no TBPL builds because this is on the relbranch rather than default (and that's how buildbot rolls).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like bug 1068209 actually.

Comment 3

[Mike Hommey [:glandium]]

There's a big difference between "ld returned 1" and "ld terminated with signal 9 [Killed]". The former is ld failing in some way, the latter is ld being killed by buildbot for some reason. Presumably for taking too long.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

There's no output from buildbot to confirm that though, eg the normal '2400 seconds without output', and we seem to be well below the the other timeouts. Maybe oomkiller though. We got ec2 slaves for both on the first attempt, and hardware for the rebuilds.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

It was oomkiller on both, eg the 2nd log:

Nov  6 00:04:00 bld-linux64-ec2-318 kernel: ld invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Nov  6 00:04:00 bld-linux64-ec2-318 kernel: Killed process 27754, UID 500, (ld) total-vm:7360352kB, anon-rss:7350300kB, file-rss:48kB

The AWS machines have 7.3G of RAM & no swap, hardware has 7.6 and 4G. Really hoping that the hardware is OK, but have the graphics changes tickled a bug here ?

Comment 6

[Sylvestre Ledru [:Sylvestre]]

(In reply to Nick Thomas [:nthomas] from comment #5)
> but have the graphics changes tickled a bug here ?
That is unlikely... Especially since most of the changes are Windows only

Comment 7

[Mike Hommey [:glandium]]

anon-rss:7350300kB

That's a lot of rss, especially for an AWS machine with... about that amount of RAM, and as you say no swap...

Unfortunately, I don't see much we can do besides enabling swap or increasing RAM.

We /could/ try some memory reducing LDFLAGS:
-Wl,--reduce-memory-overheads
-Wl,--no-keep-memory

Comment 8

[(not currently active) Ted Mielczarek]

If we have to hit swap when linking we're gonna have a bad time. We should either try to reduce memory usage of the linker or give our builders more ram.

Comment 9

[Chris AtLee [:catlee]]

Fixed I think?