775514 - Intermittent win32 build failure: "LINK : fatal error LNK1102: out of memory"

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Ed Morley [:emorley]]

Hmmmm...

WINNT 5.2 mozilla-beta leak test build on 2012-07-19 04:44:02 PDT for push b2487714085b

slave: w64-ix-slave18

https://tbpl.mozilla.org/php/getParsedLog.php?id=13669740&tree=Mozilla-Beta

{
make -C toolkit/library libs
make[5]: Entering directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/toolkit/library'
make[5]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/toolkit/library'
make[5]: Entering directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/toolkit/library'
dlldeps-xul.cpp
/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe -O /e/builds/moz2_slave/m-beta-w32-dbg/build/build/cl.py cl -Fodlldeps-xul.obj -c -D_HAS_EXCEPTIONS=0 -I../../dist/stl_wrappers  -DZLIB_DLL=1 -DMOZ_JSDEBUGGER -DMOZ_PREF_EXTENSIONS -DMOZ_AUTH_EXTENSION -DMOZ_PERMISSIONS -DMOZ_UNIVERSALCHARDET -DICON_DECODER -DMOZ_SPELLCHECK -DMOZ_ZIPWRITER -DIMPL_XREAPI -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -D_IMPL_NS_STRINGAPI -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET  -DENABLE_LAYOUTDEBUG -I/e/builds/moz2_slave/m-beta-w32-dbg/build/widget/windows -I/e/builds/moz2_slave/m-beta-w32-dbg/build/xpcom/base -I/e/builds/moz2_slave/m-beta-w32-dbg/build/intl/unicharutil/util -I/e/builds/moz2_slave/m-beta-w32-dbg/build/intl/unicharutil/src -I/e/builds/moz2_slave/m-beta-w32-dbg/build/config -I/e/builds/moz2_slave/m-beta-w32-dbg/build/widget/windows  -I/e/builds/moz2_slave/m-beta-w32-dbg/build/toolkit/library -I. -I../../dist/include -I../../dist/include/nsprpub  -Ie:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/include/nspr -Ie:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553  -DDEBUG -D_DEBUG -DTRACING -Zi -O1 -Oy- -MDd           -FI ../../dist/include/mozilla-config.h -DMOZILLA_CLIENT /e/builds/moz2_slave/m-beta-w32-dbg/build/toolkit/library/dlldeps-xul.cpp
dlldeps-xul.cpp

/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe /e/builds/moz2_slave/m-beta-w32-dbg/build/config/pythonpath.py -I../../config /e/builds/moz2_slave/m-beta-w32-dbg/build/config/expandlibs_exec.py --uselist -- link -NOLOGO -DLL -OUT:xul.dll -PDB:xul.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86  dlldeps-xul.obj nsStaticXULComponents.obj nsDllMain.obj nsGFXDeps.obj dlldeps-zlib.obj nsUnicharUtils.obj nsBidiUtils.obj nsSpecialCasingData.obj nsUnicodeProperties.obj nsRDFResource.obj   ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV    ../../toolkit/xre/xulapp_s.lib  ../../staticlib/components/necko.lib ../../staticlib/components/uconv.lib ../../staticlib/components/i18n.lib ../../staticlib/components/chardet.lib ../../staticlib/components/jar50.lib ../../staticlib/components/startupcache.lib ../../staticlib/components/pref.lib ../../staticlib/components/htmlpars.lib ../../staticlib/components/imglib2.lib ../../staticlib/components/gkgfx.lib ../../staticlib/components/gklayout.lib ../../staticlib/components/docshell.lib ../../staticlib/components/embedcomponents.lib ../../staticlib/components/webbrwsr.lib ../../staticlib/components/nsappshell.lib ../../staticlib/components/txmgr.lib ../../staticlib/components/commandlines.lib ../../staticlib/components/toolkitcomps.lib ../../staticlib/components/pipboot.lib ../../staticlib/components/pipnss.lib ../../staticlib/components/appcomps.lib ../../staticlib/components/jsreflect.lib ../../staticlib/components/composer.lib ../../staticlib/components/telemetry.lib ../../staticlib/components/jsinspector.lib ../../staticlib/components/jsdebugger.lib ../../staticlib/components/storagecomps.lib ../../staticlib/components/rdf.lib ../../staticlib/components/windowds.lib ../../staticlib/components/jsctypes.lib ../../staticlib/components/jsperf.lib ../../staticlib/components/gkplugin.lib ../../staticlib/components/windowsproxy.lib ../../staticlib/components/jsd.lib ../../staticlib/components/autoconfig.lib ../../staticlib/components/auth.lib ../../staticlib/components/cookie.lib ../../staticlib/components/permissions.lib ../../staticlib/components/universalchardet.lib ../../staticlib/components/places.lib ../../staticlib/components/tkautocomplete.lib ../../staticlib/components/satchel.lib ../../staticlib/components/pippki.lib ../../staticlib/components/imgicon.lib ../../staticlib/components/profiler.lib ../../staticlib/components/widget_windows.lib ../../staticlib/components/accessibility.lib ../../staticlib/components/spellchecker.lib ../../staticlib/components/zipwriter.lib ../../staticlib/components/services-crypto.lib ../../staticlib/components/gkdebug.lib ../../staticlib/jsipc_s.lib ../../staticlib/domipc_s.lib ../../staticlib/domplugins_s.lib ../../staticlib/mozipc_s.lib ../../staticlib/mozipdlgen_s.lib ../../staticlib/ipcshell_s.lib ../../staticlib/gfxipc_s.lib ../../staticlib/hal_s.lib ../../staticlib/dombindings_s.lib ../../staticlib/xpcom_core.lib ../../staticlib/ucvutil_s.lib ../../staticlib/chromium_s.lib ../../staticlib/snappy_s.lib ../../staticlib/thebes.lib ../../staticlib/gl.lib ../../staticlib/ycbcr.lib   e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/mozjs.lib e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/crmf.lib         e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/smime3.lib         e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/ssl3.lib         e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/nss3.lib         e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/nssutil3.lib   ../../dist/lib/mozsqlite3.lib  ../../modules/zlib/src/mozz.lib ../../dist/lib/gkmedias.lib    e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/nspr4.lib e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/plc4.lib e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/plds4.lib  ../../dist/lib/mozalloc.lib -DELAYLOAD:psapi.dll -DELAYLOAD:dbghelp.dll -DELAYLOAD:rasapi32.dll -DELAYLOAD:rasdlg.dll -DELAYLOAD:comdlg32.dll -DELAYLOAD:winspool.drv -DELAYLOAD:secur32.dll  -DELAYLOAD:oleacc.dll e:/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/dist/lib/mozglue.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib shell32.lib ole32.lib version.lib winspool.lib comdlg32.lib imm32.lib msimg32.lib shlwapi.lib psapi.lib ws2_32.lib dbghelp.lib rasapi32.lib rasdlg.lib iphlpapi.lib uxtheme.lib setupapi.lib secur32.lib sensorsapi.lib portabledeviceguids.lib windowscodecs.lib wininet.lib oleacc.lib delayimp.lib  usp10.lib oleaut32.lib   
LINK : fatal error LNK1102: out of memory
make[5]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox/toolkit/library'
make[4]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox'
make[3]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox'
make[5]: *** [xul.dll] Error 78
make[4]: *** [libs_tier_platform] Error 2
make[3]: *** [tier_platform] Error 2
make[2]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build/obj-firefox'
make[2]: *** [default] Error 2
make[1]: Leaving directory `/e/builds/moz2_slave/m-beta-w32-dbg/build'
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
program finished with exit code 2
}

Comment 1

[Mike Hommey [:glandium]]

It's not even a PGO build. That can't be address space starvation.

Comment 2

[(not currently active) Ted Mielczarek]

Could simply be a compiler error with some new code.

Comment 3

[Ed Morley [:emorley]]

It's on mozilla-beta, so I presume we would have seen more of this, if it were something critical.

Comment 4

[Eric Promislow]

With gcc 4.3.2 on Linux (Fedora Release 10) I get this output for

% find . -name '*.a' | xargs ls -l | fgrep -v -- '->' | sort -g -k5 -r

run from my obj dir:

-rw-rw-r-- 1 ericp ericp 185072190 2012-09-25 14:06 ./js/src/libjs_static.a
-rw-rw-r-- 1 ericp ericp   3516534 2012-09-25 13:09 ./nsprpub/pr/src/libnspr4.a
-rw-rw-r-- 1 ericp ericp   3492262 2012-09-25 14:51 ./toolkit/crashreporter/google-breakpad/src/common/libhost_breakpad_common_s.a
...

Clearly there's been a template explosion in the JS compiler code

Total size of the moz .a files:

% find . -name '*.a' | xargs ls -l | fgrep -v -- '->' | awk '{s += $5; } END { printf("total sum: %d\n", s);}'
total sum: 222266100

About      180000000 of the sum is due to template bloating.

Comment 5

[Ed Morley [:emorley]]

Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.

Comment 6

[Mark Straver]

I'm running into this build error on VS2012, using -GL (WPO) but not PGO when trying to build mozilla-beta (v24). Code base v20 builds fine without issues every time.

I'm having a hard time trying to figure out what triggers it, since the entire build system was changed around between the two versions. Looking at the info stated above, I had a look in the lib directory to find that js_static.lib is a staggering 900MB large. That can't be right, .libs should never grow to that size...

Keeping an eye on ram usage while building, it seems to do fine up to a certain point, where the linker suddenly starts eating memory like sweet rolls.

FYI: this is on a w7 x64 OS, trying to build a w32 binary - I should have max address space available. System has 16GB of RAM and plenty of swap available.