877959 - Dep builds not rebuilding

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Phil Ringnalda (:philor)]

Consider https://hg.mozilla.org/integration/mozilla-inbound/rev/fe6ca42a5b6e, which added a test and changed one number. On that push and the next, every mochitest-1 run passed. Two pushes later, debug Mac (and thus three sets of tests on one build, 10.6, 10.7, 10.8) failed that test, failing to get the exception that should have come from going over the new limit. Retriggering them, they all failed again. Not intermittent, a bad build. The push after that, opt Linux64 had the same repeated failure. Two pushes after that, both Linux32 and Linux64 opt had it. Then we clobbered, because clobbering is what we do.

Earlier than that, in the push which relanded the stuff that disappeared because pushlog was broken and things landed in inbound's repo without going into pushlog, so we restored inbound from a backup, was https://hg.mozilla.org/integration/mozilla-inbound/rev/db53707fb4d7. Mac debug mochitest-3 failed the test, but after a clobber and a rebuild on the same push, it passed the test. Seven pushes later, Linux32 failed the same test.

One possibility is that this is actually bustage from yesterday, because the patch for bug 790765 was causing incomprehensible failures in postflight for Mac universal builds only on dep builds, and my description in bug 844288 comment 55 now sounds to me rather like I'm describing this same thing, not knowing that the failure I was seeing was the only thing that had landed after whatever caused this bustage, which could managed to visibly fail (since seeing it requires that you do something like add a test which gets built, and make a code change that the test will reflect if the code change isn't built).

Because I suspect this has been merged around, whether it's from today or from yesterday, I'm closing mozilla-inbound, mozilla-central, birch, and fx-team.

Comment 1

[Phil Ringnalda (:philor)]

One of the other charming possibilities is that yesterday's thing was separate, and that today's thing is not actually clobber versus dep, it's "has a busted shared repo on the slave from having pulled during the time of the pushlog bustage, so that it has a repo which went into the future, then back, then forward again, and is totally hosed."

Comment 2

[Phil Ringnalda (:philor)]

Third example, probably too late to make the corrupted repo theory make sense:

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=27a0f1913914 managed to cause a lot of json xpcshell failures, on Linux32 and Linux64 opt and debug, then three pushes later on Linux32 opt and Linux64 debug and Mac debug, then on and on one or two builds every two or three pushes.

Comment 3

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

gps or ted need to look into this.  glandium is moving to Japan and I'm coming back from Taipei so we're both going to be offline.

Comment 4

[Mike Hommey [:glandium]]

gps is on PTO.

Comment 5

[[e-mail] (:bhearsum)]

I just looked at the build directory for the Linux 64 build on https://hg.mozilla.org/integration/mozilla-inbound/rev/b920312f99e3. "hg status" shows it clean, which I think further shoots down the corrupted repository theory:
[cltbld@bld-linux64-ec2-666 build]$ hg status
? properties.json
[cltbld@bld-linux64-ec2-666 build]$

Comment 6

[John Hopkins (:jhopkins)]

'hg verify' of /builds/hg-shared/integration/mozilla-inbound on bld-linux64-ec2-668 was clean (no errors, only warnings, which are known)

Comment 7

[(not currently active) Ted Mielczarek]

So we looked at this push:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=b2bf68201b50

Displaying the mochitest-1 failure from Ehsan's push. bhearsum grabbed me AudioContext.o, which I disassembled and looks correct. However, looking at the build log we noted that libxul.so didn't get relinked. That seems like our smoking gun here.

Backing out bug 844288 seems like a plausible thing to do.

Comment 8

[(not currently active) Ted Mielczarek]

I think the failure mode here is something like "libxul doesn't get re-linked because the gtest libxul screws up its dependencies somehow". The log shows that libxul isn't being re-linked, but libxul-gtest is.

From the log ( https://tbpl.mozilla.org/php/getParsedLog.php?id=23615738&tree=Mozilla-Inbound&full=1 ), for posterity:

make -C toolkit/library libs
make[5]: Entering directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/config/nsinstall -R -m 644 "libxul.so" "../../dist/bin"
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/config/nsinstall -R -m 644 "libxul.so" "../../dist/sdk/lib"
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/_virtualenv/bin/python /builds/slave/m-in-l64-000000000000000000000/build/toolkit/library/dependentlibs.py libxul.so -L ../../dist/bin  > ../../dist/bin/dependentlibs.list
make[5]: Leaving directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'

However, later in the log:
make -C testing/gtest check
make[1]: Entering directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/testing/gtest'
make -C ../../toolkit/library gtestxul
make[2]: Entering directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'
sed -e "s|libxul.so|gtest/libxul.so|" ../../dist/bin/dependentlibs.list > ../../dist/bin/dependentlibs.list.gtest
mkdir -p libgtest
make libs SHARED_LIBRARY_NAME=gtest/libxul FINAL_TARGET=../../dist/bin/gtest SDK_LIBRARY= IMPORT_LIB_DEST=../../dist/lib/gtest LINK_GTEST=true
libxul.so
make[3]: Entering directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'
rm -f libgtest/libxul.so
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/_virtualenv/bin/python /builds/slave/m-in-l64-000000000000000000000/build/config/expandlibs_exec.py --depend .deps/libxul.so.pp --target libgtest/libxul.so --uselist --  /usr/bin/ccache /tools/gcc-4.7.2-0moz1/bin/g++  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks  -fno-omit-frame-pointer  -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libgtest/libxul.so  nsUnicharUtils.o nsBidiUtils.o nsSpecialCasingData.o nsUnicodeProperties.o nsRDFResource.o nsStaticXULComponents.o    -lpthread  -Wl,-z,noexecstack -Wl,--build-id   -Wl,-rpath-link,/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib   ../../media/kiss_fft/libkiss_fft.a ../../toolkit/components/osfile/libosfile_s.a ../../toolkit/xre/libxulapp_s.a  ../../accessible/src/base/libaccessibility_base_s.a ../../accessible/src/generic/libaccessibility_generic_s.a ../../accessible/src/html/libaccessibility_html_s.a ../../accessible/src/xpcom/libaccessibility_xpcom_s.a  ../../accessible/src/libaccessibility_toolkit_s.a  ../../accessible/src/xul/libaccessibility_xul_s.a ../../staticlib/components/libnecko.a ../../staticlib/components/libuconv.a ../../staticlib/components/libi18n.a ../../staticlib/components/libchardet.a ../../staticlib/components/libjar50.a ../../staticlib/components/libstartupcache.a ../../staticlib/components/libpref.a ../../staticlib/components/libhtmlpars.a ../../staticlib/components/libidentity.a ../../staticlib/components/libimglib2.a ../../staticlib/components/libmediasniffer.a ../../staticlib/components/libgkgfx.a ../../staticlib/components/libgklayout.a ../../staticlib/components/libdocshell.a ../../staticlib/components/libembedcomponents.a ../../staticlib/components/libwebbrwsr.a ../../staticlib/components/libnsappshell.a ../../staticlib/components/libtxmgr.a ../../staticlib/components/libcommandlines.a ../../staticlib/components/libtoolkitcomps.a ../../staticlib/components/libpipboot.a ../../staticlib/components/libpipnss.a ../../staticlib/components/libappcomps.a ../../staticlib/components/libjsreflect.a ../../staticlib/components/libcomposer.a ../../staticlib/components/libtelemetry.a ../../staticlib/components/libjsinspector.a ../../staticlib/components/libjsdebugger.a ../../staticlib/components/libstoragecomps.a ../../staticlib/components/librdf.a ../../staticlib/components/libwindowds.a ../../staticlib/components/libdiskspacewatcher.a ../../staticlib/components/libjsctypes.a ../../staticlib/components/libjsperf.a ../../staticlib/components/libgkplugin.a ../../staticlib/components/libunixproxy.a ../../staticlib/components/libjsd.a ../../staticlib/components/libautoconfig.a ../../staticlib/components/libauth.a ../../staticlib/components/libcookie.a ../../staticlib/components/libpermissions.a ../../staticlib/components/libuniversalchardet.a ../../staticlib/components/libfileview.a ../../staticlib/components/libplaces.a ../../staticlib/components/libtkautocomplete.a ../../staticlib/components/libsatchel.a ../../staticlib/components/libpippki.a ../../staticlib/components/libwidget_gtk2.a ../../staticlib/components/libimgicon.a ../../staticlib/components/libprofiler.a ../../staticlib/components/libremoteservice.a ../../staticlib/components/libspellchecker.a ../../staticlib/components/libzipwriter.a ../../staticlib/components/libservices-crypto.a ../../staticlib/components/libnkgio.a ../../staticlib/components/libpeerconnection.a ../../staticlib/components/libgtest.a ../../staticlib/components/libgfxtest.a ../../staticlib/libjsipc_s.a ../../staticlib/libdomipc_s.a ../../staticlib/libdomplugins_s.a ../../staticlib/libmozipc_s.a ../../staticlib/libmozipdlgen_s.a ../../staticlib/libipcshell_s.a ../../staticlib/libgfxipc_s.a ../../staticlib/libhal_s.a ../../staticlib/libdombindings_s.a ../../staticlib/libxpcom_core.a ../../staticlib/libucvutil_s.a ../../staticlib/libchromium_s.a ../../staticlib/libsnappy_s.a ../../staticlib/libgtkxtbin.a ../../staticlib/libthebes.a ../../staticlib/libgl.a ../../staticlib/libycbcr.a  -Wl,-version-script,symverscript -L../../dist/bin -L../../dist/lib /builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/dist/lib/libjs_static.a -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3   -L/usr/lib64 -lXrender  ../../dist/lib/libmozsqlite3.a  /builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/modules/zlib/src/libmozz.a ../../dist/lib/libgkmedias.a ../../media/mtransport/build/libmtransport.a ../../media/webrtc/signaling/signaling_ecc/libecc.a ../../media/webrtc/signaling/signaling_sipcc/libsipcc.a  -lasound   -lrt -L../../dist/bin -L../../dist/lib  -L/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/dist/lib -lnspr4 -lplc4 -lplds4 ../../dist/lib/libmozalloc.a -L/lib64 -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0   -L/usr/lib64 -lX11  -lXext  -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0   -lfontconfig   -lgtk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lfreetype -lfontconfig -lgdk-x11-2.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0   -lXt -lgthread-2.0 -lfreetype    -ldl  -lrt  ../../build/unix/stdc++compat/libstdc++compat.a  
chmod +x libgtest/libxul.so
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/config/nsinstall -R -m 644 "libgtest/libxul.so" "../../dist/bin/gtest"
make[3]: Leaving directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'
make[2]: Leaving directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/toolkit/library'
make[1]: Leaving directory `/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/testing/gtest'

Comment 9

[Ed Morley [:emorley]]

Thank you for tracking this down Ted - fingers crossed! :-)

Comment 10

[(not currently active) Ted Mielczarek]

Of note, from that wall of text above:
rm -f libgtest/libxul.so
/builds/slave/m-in-l64-000000000000000000000/build/obj-firefox/_virtualenv/bin/python /builds/slave/m-in-l64-000000000000000000000/build/config/expandlibs_exec.py
--depend .deps/libxul.so.pp
--target libgtest/libxul.so

The --depend line is wrong, that's probably the bug.

Comment 11

[Phil Ringnalda (:philor)]

Fixed by the backout in https://hg.mozilla.org/mozilla-central/rev/4755d50e2402