Static build runs out of disk space on the Try server when tests are enabled



Release Engineering
9 years ago
5 years ago


(Reporter: joelr, Assigned: joduinn)


Firefox Tracking Flags

(Not tracked)





9 years ago
My Linux build clocks in at 18Gb when tests are enabled. Works on my laptop but not on the Try server. 

A static build speeds up startup by > 10% and I think it ought to be tested.
For posterity, here's a log:

Which ends up failing part way through with out-of-space:
/builds/slave/sendchange-linux-hg/build/xpcom/tests/TestTArray.cpp:356:   instantiated from here
/builds/slave/sendchange-linux-hg/build/xpcom/tests/TestTArray.cpp:95: warning: comparison between signed and unsigned integer expressions
/tools/gcc-4.3.3/installed/bin/g++   -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Wno-long-long -pedantic -gstabs+ -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DNDEBUG -DTRIMMED -Os -freorder-blocks -fno-reorder-functions  -o TestTArray TestTArray.o   -lpthread   -Wl,-rpath-link,/builds/slave/sendchange-linux-hg/build/objdir/dist/bin -Wl,-rpath-link,/usr/local/lib  -rdynamic -L../../dist/bin -L../../dist/lib -L/builds/slave/sendchange-linux-hg/build/objdir/dist/lib -lplds4 -lplc4 -lnspr4 -lpthread -ldl nsStaticComponents.o ../../staticlib/components/libxpctest.a ../../staticlib/components/libpref.a ../../staticlib/components/libuconv.a ../../staticlib/components/libi18n.a ../../staticlib/components/libnecko.a ../../staticlib/components/libauth.a ../../staticlib/components/libxpconnect.a ../../staticlib/components/libjsctypes.a ../../staticlib/components/libchardet.a ../../staticlib/components/libzipwriter.a ../../staticlib/components/libjar50.a ../../staticlib/components/libcookie.a ../../staticlib/components/libpermissions.a ../../staticlib/components/libstoragecomps.a ../../staticlib/components/librdf.a ../../staticlib/components/libjsd.a ../../staticlib/components/libcaps.a ../../staticlib/components/libhtmlpars.a ../../staticlib/components/libgkgfxthebes.a ../../staticlib/components/libimgicon.a ../../staticlib/components/libimglib2.a ../../staticlib/components/libwidget_gtk2.a ../../staticlib/components/libtxmgr.a ../../staticlib/components/libcomposer.a ../../staticlib/components/libgklayout.a ../../staticlib/components/libdocshell.a ../../staticlib/components/libembedcomponents.a ../../staticlib/components/libwebbrwsr.a ../../staticlib/components/libnsappshell.a ../../staticlib/components/libuniversalchardet.a ../../staticlib/components/libaccessibility.a ../../staticlib/components/libchrome.a ../../staticlib/components/libmozfind.a ../../staticlib/components/libintlapp.a ../../staticlib/components/libwindowds.a ../../staticlib/components/libappcomps.a ../../staticlib/components/libremoteservice.a ../../staticlib/components/libcommandlines.a ../../staticlib/components/libfileview.a ../../staticlib/components/libplaces.a ../../staticlib/components/libtkautocomplete.a ../../staticlib/components/libsatchel.a ../../staticlib/components/libtoolkitcomps.a ../../staticlib/components/libspellchecker.a ../../staticlib/components/libunixproxy.a ../../staticlib/components/libxpinstall.a ../../staticlib/components/libpipboot.a ../../staticlib/components/libpipnss.a ../../staticlib/components/libpippki.a ../../staticlib/components/libautoconfig.a ../../staticlib/components/libsystem-pref.a ../../staticlib/components/libmozgnome.a ../../staticlib/components/libdbusservice.a ../../staticlib/libmozreg_s.a ../../staticlib/libunicharutil_s.a ../../staticlib/libucvutil_s.a ../../staticlib/libgtkxtbin.a ../../staticlib/libmorkreader_s.a ../../staticlib/libthebes.a ../../staticlib/libgfxpsshar.a ../../staticlib/libgkgfx.a  -L../../modules/libimg/png -lmozpng -L../../jpeg -lmozjpeg -L../../modules/zlib/src -lmozz  -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3  ../../gfx/cairo/cairo/src/libmozcairo.a ../../gfx/cairo/libpixman/src/libmozlibpixman.a   -lXrender -lfreetype -lfontconfig ../../gfx/qcms/libmozqcms.a  -lXt -lgthread-2.0 -L/lib -lpangoft2-1.0 -lpango-1.0 -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0   -L../../dist/lib -lmozsqlite3 -lasound -lm -ldl -lpthread   -L/usr/local/lib -ljs_static  -L/lib -lgtk-x11-2.0 -latk-1.0 -lgdk-x11-2.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0    -lX11  -ljs_static  -L/builds/slave/sendchange-linux-hg/build/objdir/dist/lib -lplds4 -lplc4 -lnspr4 -lpthread -ldl -ldl -lm  ../../dist/lib/libxpcom_core.a  
/usr/bin/ld: final link failed: No space left on device
collect2: ld returned 1 exit status
make[5]: *** [TestTArray] Error 1

Follow-up question for you: Are we looking at doing static builds for all Linux builds at some point? If that's the case, we're looking at all Linux builds, including Production, having comparable build directories - right?

For the try server, we could work around the issue by ensuring that all builds clean up after themselves before finishing, which should leave us with 25-30gb free at the start of any given job.

Handing to John for prioritization.
Assignee: nobody → joduinn
The problem is that all that the tests in xpcom/tests are statically linked too, and take up ~ 490MB each. The disk space requirements would be much more "normal" if this wasn't to happen.

Comment 3

9 years ago
This is a static build that we are testing and I don't know of a way to minimize disk space for it. Any suggestions? Can disk space be upped?
The linux try slaves can provide a maximum of 24 G for a single build, provided all old builds have been cleaned up. We could try to co-ordinate a time where you push to try and we jump in to clean up other builds on the slave that gets the job.

Out of interest, how big does firefox-bin end up after linking statically  ? Surely it's smaller than 490MB and there is something wrong if all the xpcom/tests binaries are that size.
Note that we don't currently build everything in xpcom/tests in our libxul builds. Is every binary in there huge, or just some of them?
Is this still an issue with the latest incarnations of the patches ?

Comment 7

8 years ago
Should not be an issue as tests are not built anymore.
I see that browser/confvars.h flips ENABLE_TESTS to undefined (in attachment 422208 [details] [diff] [review]). Is that a bandaid to solve the disk space problem found here ? If so it won't actually help the build slaves, because we'll be required to ensure --enable-tests is set in all the opt and debug mozconfigs (to continue packaging tests and handing separate suites off to slaves).

Comment 9

8 years ago
Ben, why did you want the tests disabled again?
(In reply to comment #9)
> Ben, why did you want the tests disabled again?

I may have suggested disabling the tests as a workaround for running out of disk space on the try server, but certainly not as a long term fix. I don't have any distinct memory of this though -- sorry if I've caused any confusion.

Comment 11

8 years ago
My bad, I was addressing Benjamin Smedberg :-).

Benjamin, why do you want tests disabled?

Comment 12

8 years ago
Each binary test has to statically link all of mozilla. That seems like a pretty obvious problem to me, in terms of link time for developers and the size of the packages involved.

Comment 13

8 years ago
The problem with this, though, is that it appears to be impossible to disable tests on the Try Server (Maemo) and in Places. 

It appears that the right thing to do is to -build- tests but let developers disable them, exactly as things work now. 

Would you agree?
(In reply to comment #12)
> Each binary test has to statically link all of mozilla. That seems like a
> pretty obvious problem to me, in terms of link time for developers and the size
> of the packages involved.

Can you clarify? I do not understand why "linking all of Mozilla" cause 490mb binaries for each test.

Comment 15

8 years ago
I don't know about 490MB, but it should be "the size of libxul" for each test, which is 12MB in opt builds.

You can't --disable-tests. What you need to do (and what the code already did for the most part) is stop building binary tests in this configuration using makefile tests.

Comment 16

8 years ago
The code builds binary tests all over the place, both as CPP_UNIT_TESTS and SIMPLE_PROGRAMS. Simply disabling the CPP_UNIT_TESTS [1] doesn't work so individual make files need to be edited, e.g. xpcom/tests/external/ or xpcom/tests/

Wouldn't it be more straightforward to investigate why each test is 490Mb, fix that and then give developers the ability to disable tests if they want to?

I for one -do- want to build and run tests in the static configuration since that's how I know I did not screw up. I caught and fixed several issues this way and think it's wrong to just skip binary tests in a static configuration.

talked on IRC. fixing the 490mb problem and being able to disable binary tests from building are two separate issues.

P1 is fixing the 490mb problem, so we can get the patch landed.

being able to disable binary tests is a secondary issue, to reduce build times for developers.
Last Resolved: 8 years ago
Resolution: --- → INCOMPLETE
Product: → Release Engineering
You need to log in before you can comment on or make changes to this bug.