Closed Bug 886079 Opened 11 years ago Closed 11 years ago

Frequent "command timed out: 300 seconds without output, attempting to kill" running expandlibs_exec.py in libgtest

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: philor, Assigned: emorley)

References

()

Details

(Keywords: intermittent-failure)

https://tbpl.mozilla.org/php/getParsedLog.php?id=24468278&tree=Mozilla-Inbound

rm -f libgtest/libxul.so
/builds/slave/m-in-lx-d-00000000000000000000/build/obj-firefox/_virtualenv/bin/python /builds/slave/m-in-lx-d-00000000000000000000/build/config/expandlibs_exec.py --depend .deps/libgtest//libxul.so.pp --target libgtest/libxul.so --uselist --  /usr/bin/ccache /tools/gcc-4.7.2-0moz1/bin/g++ -m32 -march=pentiumpro  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -pipe  -DDEBUG -D_DEBUG -DTRACING -g -Os -freorder-blocks  -fno-omit-frame-pointer  -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libgtest/libxul.so  nsUnicharUtils.o nsBidiUtils.o nsSpecialCasingData.o nsUnicodeProperties.o nsRDFResource.o nsStaticXULComponents.o    -lpthread  -Wl,-z,noexecstack -Wl,--build-id   -Wl,-rpath-link,/builds/slave/m-in-lx-d-00000000000000000000/build/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib   ../../media/kiss_fft/libkiss_fft.a ../../toolkit/components/osfile/libosfile_s.a ../../toolkit/xre/libxulapp_s.a  ../../accessible/src/base/libaccessibility_base_s.a ../../accessible/src/generic/libaccessibility_generic_s.a ../../accessible/src/html/libaccessibility_html_s.a ../../accessible/src/xpcom/libaccessibility_xpcom_s.a  ../../accessible/src/libaccessibility_toolkit_s.a  ../../accessible/src/xul/libaccessibility_xul_s.a ../../staticlib/components/libnecko.a ../../staticlib/components/libuconv.a ../../staticlib/components/libi18n.a ../../staticlib/components/libchardet.a ../../staticlib/components/libjar50.a ../../staticlib/components/libstartupcache.a ../../staticlib/components/libpref.a ../../staticlib/components/libhtmlpars.a ../../staticlib/components/libidentity.a ../../staticlib/components/libimglib2.a ../../staticlib/components/libmediasniffer.a ../../staticlib/components/libgkgfx.a ../../staticlib/components/libgklayout.a ../../staticlib/components/libdocshell.a ../../staticlib/components/libembedcomponents.a ../../staticlib/components/libwebbrwsr.a ../../staticlib/components/libnsappshell.a ../../staticlib/components/libtxmgr.a ../../staticlib/components/libcommandlines.a ../../staticlib/components/libtoolkitcomps.a ../../staticlib/components/libpipboot.a ../../staticlib/components/libpipnss.a ../../staticlib/components/libappcomps.a ../../staticlib/components/libjsreflect.a ../../staticlib/components/libcomposer.a ../../staticlib/components/libtelemetry.a ../../staticlib/components/libjsinspector.a ../../staticlib/components/libjsdebugger.a ../../staticlib/components/libstoragecomps.a ../../staticlib/components/librdf.a ../../staticlib/components/libwindowds.a ../../staticlib/components/libdiskspacewatcher.a ../../staticlib/components/libjsctypes.a ../../staticlib/components/libjsperf.a ../../staticlib/components/libgkplugin.a ../../staticlib/components/libunixproxy.a ../../staticlib/components/libjsd.a ../../staticlib/components/libautoconfig.a ../../staticlib/components/libauth.a ../../staticlib/components/libcookie.a ../../staticlib/components/libpermissions.a ../../staticlib/components/libuniversalchardet.a ../../staticlib/components/libfileview.a ../../staticlib/components/libplaces.a ../../staticlib/components/libtkautocomplete.a ../../staticlib/components/libsatchel.a ../../staticlib/components/libpippki.a ../../staticlib/components/libwidget_gtk2.a ../../staticlib/components/libimgicon.a ../../staticlib/components/libprofiler.a ../../staticlib/components/libremoteservice.a ../../staticlib/components/libspellchecker.a ../../staticlib/components/libzipwriter.a ../../staticlib/components/libservices-crypto.a ../../staticlib/components/libgkdebug.a ../../staticlib/components/libnkgio.a ../../staticlib/components/libpeerconnection.a ../../staticlib/components/libgtest.a ../../staticlib/components/libgfxtest.a ../../staticlib/libjsipc_s.a ../../staticlib/libdomipc_s.a ../../staticlib/libdomplugins_s.a ../../staticlib/libmozipc_s.a ../../staticlib/libmozipdlgen_s.a ../../staticlib/libipcshell_s.a ../../staticlib/libgfxipc_s.a ../../staticlib/libhal_s.a ../../staticlib/libdombindings_s.a ../../staticlib/libxpcom_core.a ../../staticlib/libucvutil_s.a ../../staticlib/libchromium_s.a ../../staticlib/libsnappy_s.a ../../staticlib/libgtkxtbin.a ../../staticlib/libthebes.a ../../staticlib/libgl.a ../../staticlib/libycbcr.a  -Wl,-version-script,symverscript -L../../dist/bin -L../../dist/lib /builds/slave/m-in-lx-d-00000000000000000000/build/obj-firefox/dist/lib/libjs_static.a -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3   -L/usr/lib -lXrender  ../../dist/lib/libmozsqlite3.a  /builds/slave/m-in-lx-d-00000000000000000000/build/obj-firefox/modules/zlib/src/libmozz.a ../../dist/lib/libgkmedias.a ../../media/mtransport/build/libmtransport.a ../../media/webrtc/signaling/signaling_ecc/libecc.a ../../media/webrtc/signaling/signaling_sipcc/libsipcc.a  -lasound   -lrt -L../../dist/bin -L../../dist/lib -L/builds/slave/m-in-lx-d-00000000000000000000/build/obj-firefox/dist/lib -lnspr4 -lplc4 -lplds4 ../../dist/lib/libmozalloc.a -L/lib -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0   -lgtk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lfreetype -lfontconfig -lgdk-x11-2.0 -lgdk_pixbuf-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0    -L/usr/lib -lX11  -lXext  -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lglib-2.0   -lfontconfig   -lXt -lgthread-2.0 -lfreetype    -ldl  -lrt  ../../build/unix/stdc++compat/libstdc++compat.a  

command timed out: 300 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=24478450&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=24478476&tree=Mozilla-Central
This is just "taking a long time to link libxul", it sounds like. Any idea how long we spend linking libxul normally on these builds?
Less than 300s.

Oh, do you mean how long we spend linking libxul *when we're freakin' building, not when we're running make check*? Less than 7200s, that being the timeout for the compile step, the one where we expect to link libxul.
Blocks: 844288
Okay, so the problem here is that we snuck a slow link into the check step and made our automation unhappy. BenWa: can we just make it link during the normal build phase when we have --enable-gtest? It's not the best thing, but it's better than this.
Not impossible to increase the make check step's timeout, but that makes every single timeout in any test (and jit-tests are far too prone to fail by timing out) take longer-than-the-longest-link-libxul-time.

https://tbpl.mozilla.org/php/getParsedLog.php?id=24495435&tree=Mozilla-Central
Can we just add a step between build and make check that just link gtest libxul?
Any update on increasing the timeout duration for this build step?
Flags: needinfo?(kmoir)
I don't know how to fix this, it would be better to find someone with more expertise in this area.
Flags: needinfo?(kmoir)
Depends on: 890349
The disable-fairy is going to be visiting soon, which would be suboptimal for all parties involved... don't suppose you could drive this forwards? :-)
Flags: needinfo?(bgirard)
(In reply to Ed Morley [:edmorley UTC+1] from comment #45)
> The disable-fairy is going to be visiting soon, which would be suboptimal
> for all parties involved... don't suppose you could drive this forwards? :-)

Can the disable-fairy direct its energy to bug 890349. I was hoping to get a response sooner.
Flags: needinfo?(bgirard)
Sorry missed that dependant bug - it seemed like comment 28 was the last response on this. I'll needinfo someone from releng on the other bug.
This should go away now that bug 890349 is fixed :-)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Assignee: nobody → emorley
Component: Build Config → Release Automation
Product: Core → Release Engineering
QA Contact: bhearsum
Version: Trunk → other
Looks like 600s might not be enough (or more likely, raising the timeout isn't helping):
https://tbpl.mozilla.org/php/getParsedLog.php?id=26875528&tree=Mozilla-Inbound#error0
Blocks: 916765
You need to log in before you can comment on or make changes to this bug.