Closed Bug 1340395 Opened 4 years ago Closed 4 years ago
Linker busting between fatal errors "LNK1000: Internal error during LIB::Search" and "LNK1102: Out of Memory"
Got this with a c-c win32 build: ..\..\modules\zlib\src\inflate.obj ..\..\modules\zlib\src\inftrees.obj ..\..\modules\zlib\src\trees.obj ..\..\modules\zlib\src\uncompr.obj ..\..\modules\zlib\src\zutil.obj StaticXULComponentsEnd\StaticXULComponentsEnd.obj LINK : fatal error LNK1000: Internal error during LIB::Search Version 14.00.24213.1 ExceptionCode = C0000005 ExceptionFlags = 00000000 ExceptionAddress = 000007FEF217C387 (000007FEF2170000) "c:\builds\slave\c-c\build\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT\VCRUNTIME140.dll" NumberParameters = 00000002 ExceptionInformation[ 0] = 0000000000000000 ExceptionInformation[ 1] = 0000000405D82000 CONTEXT: Rax = 0000000011738150 R8 = 00000000031CD270 Rbx = 0000000011738110 R9 = 0000000000000000 Rcx = 0000000000000298 R10 = 0000000405D81CD0 Rdx = 00000003F4649B80 R11 = 0000000000000608 Rsp = 000000000019D1A8 R12 = 0000000000000D79 Rbp = 00000000000005C8 E13 = 00000000031CD270 Rsi = 0000000405D82000 R14 = 0000000405D81CD0 Rdi = 0000000011738480 R15 = 0000000000000000 Rip = 000007FEF217C387 EFlags = 0000000000010207 SegCs = 0000000000000033 SegDs = 000000000000002B SegSs = 000000000000002B SegEs = 000000000000002B SegFs = 0000000000000053 SegGs = 000000000000002B Dr0 = 0000000000000000 Dr3 = 0000000000000000 Dr1 = 0000000000000000 Dr6 = 0000000000000000 Dr2 = 0000000000000000 Dr7 = 0000000000000000 c:/builds/slave/c-c/build/mozilla/config/rules.mk:787: recipe for target 'xul.dll' failed mozmake.exe: *** [xul.dll] Error 1000 mozmake.exe: *** Deleting file 'xul.dll' somewhat akin to bug 436279 but that's been resolved (if only because it hasn't cropped up in a while). Just filing this just in case something needs to be done.
Summary: LINK : fatal error LNK1000: Internal error during LIB::Search → Linker busting between fatal errors "LNK1000: Internal error during LIB::Search" and "LNK1102: Out of Memory"
This is on the loaner. Win32 build. Both nightly and normal build fails with either OOM or LNK1000
OS: Unspecified → Windows
Hardware: Unspecified → x86
Last good build: https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299 First bad build: https://hg.mozilla.org/mozilla-central/rev/8c8b54b13be7
Improvement on the last good, first bad: Last good: https://hg.mozilla.org/mozilla-central/rev/e783bdf2cb50 First Bad: https://hg.mozilla.org/mozilla-central/rev/e9b926463f9e
(In reply to Edmund Wong (:ewong) from comment #4) > Improvement on the last good, first bad: > > Last good: > https://hg.mozilla.org/mozilla-central/rev/e783bdf2cb50 > Actually.. the last good is still https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299
> First Bad: > https://hg.mozilla.org/mozilla-central/rev/e9b926463f9e https://hg.mozilla.org/mozilla-central/rev/2737f66ad6ac gives a StaticXULComponentsEnd\StaticXULComponentsEnd.obj js_static.lib(Unified_cpp_js_src25.obj) : fatal error LNK1318: Unexpected PDB er ror; OK (0) '' c:/builds/slave/c-c/build/mozilla/config/rules.mk:787: recipe for target 'xul.dl l' failed mozmake.exe: *** [xul.dll] Error 1318 mozmake.exe: Leaving directory 'c:/builds/slave/c-c/build/objdir/toolkit/libr ary' c:/builds/slave/c-c/build/mozilla/config/recurse.mk:71: recipe for target 'toolk it/library/target' failed mozmake.exe: *** [toolkit/library/target] Error 2 so not sure if it's related.
How much memory has the loaner got? How much free memory has it got? Has it been rebooted recently?
(In reply to Ian Neal from comment #7) > How much memory has the loaner got? How much free memory has it got? Has it > been rebooted recently? 15GB and yes. It has been rebooted. w/ m-c = https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299, it builds. w/ any cset post a9ec72f82299, it gives me either errors. Now, with tip on c-c and m-c, it's taken more than 20 minutes to link, which isn't right since I think it's taken more than 1/2 hr. This is with a clobber build.
(In reply to Edmund Wong (:ewong) from comment #9) > Created attachment 8839043 [details] > 2nd type of error by 2nd type of error, I meant: ..\..\modules\zlib\src\gzwrite.obj ..\..\modules\zlib\src\infback.obj ..\..\modules\zlib\src\inffast.obj ..\..\modules\zlib\src\inflate.obj ..\..\modules\zlib\src\inftrees.obj ..\..\modules\zlib\src\trees.obj ..\..\modules\zlib\src\uncompr.obj ..\..\modules\zlib\src\zutil.obj StaticXULComponentsEnd\StaticXULComponentsEnd.obj js_static.lib(Unified_cpp_js_src15.obj) : fatal error LNK1318: Unexpected PDB error; OK (0) '' c:/builds/slave/c-c-ntly/build/mozilla/config/rules.mk:787: recipe for target 'xul.dll' failed mozmake.exe: *** [xul.dll] Error 1318 mozmake.exe: Leaving directory 'c:/builds/slave/c-c-ntly/build/objdir/toolkit/library'
:glandium, might you have any ideas? mozilla-central post-https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299 fails to build.
Having spoken with IanN, he suggested I backout https://hg.mozilla.org/mozilla-central/rev/b7a2f7ff5e87, which I did and it builds.
1) pull both c-c and m-c tip 2) python mozilla\mach clobber 3) build - busts with '2nd type of error' log 4) python mozilla\mach clobber 5) cd mozilla 6) hg backout -r b7a2f7ff5e87 7) cd .. 8) <build> - completes build
Now with m-c's tip, you need to backout both: https://hg.mozilla.org/mozilla-central/rev/e7c397118fd2 and https://hg.mozilla.org/mozilla-central/rev/b7a2f7ff5e87
My last two clobber builds of TB on 2017-02-18 and today failed with the same error. I only have 8GB RAM and 800MB are dedicated to internal graphics. Before I had no trouble building (and BTW, I have three development machines with 8GB). At first I followed my own advice given here: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Thunderbird_build: If on Windows you get link errors like "LNK1102: out of memory" or "LNK1318: Unexpected PDB error; OK (0)", try deleting the largest .PDB files before rushing out the door to buy more RAM. Clobbering (see below) will also remove those files. That didn't help. So then I deleted all 3587 PDB files I saw following http://stackoverflow.com/questions/174013/linker-out-of-memory-lnk1102 and the linker went through, albeit with a lot of warnings about missing PDB files. Since according to comment #8 even 16GB are not enough, I'd like to see this rectified. BTW, only few people build TB/SM in debug mode. Many TB developers are on Linux, Richard and FRG build non-debug. And Kent James will wake up when he tries to build next time.
This is a workaround that I found to work. I haven't tried the debug build yet, but I have pushed to try: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c3a7bcf4de70599f240111d5bfef6a0566ce3347
Thanks, I'll compile with the patch next time I do a full debug build.
So is this a Visual Studio compiler bug?
I don't think so. You tell the compiler to generate 3587 PDB files and then the linker chokes on those. This might work for small programs, but not for TB or SM; I haven't tried compiling FF lately.
I added CFLAGS="$CFLAGS -FS" CXXFLAGS="$CXXFLAGS -FS" mk_add_options "export COMPILE_PDB_FLAG=-Fdgenerated.pdb" to my local config and now I can build again. I get 722 PDB files (instead of 3587), mostly called generated.pdb. Question: Do we need the patch if the builds work on the servers? How much RAM do they have?
(In reply to Jorg K (GMT+1) from comment #21) > I added > CFLAGS="$CFLAGS -FS" > CXXFLAGS="$CXXFLAGS -FS" > mk_add_options "export COMPILE_PDB_FLAG=-Fdgenerated.pdb" > to my local config and now I can build again. > > I get 722 PDB files (instead of 3587), mostly called generated.pdb. > They should be all called generated.pdb (iiuc). > Question: Do we need the patch if the builds work on the servers? How much > RAM do they have? I'm not too sure how much they have but if they are similar to the loaner I use, should be around 15GB. That said, I'm wondering if 'how' they are built makes a difference. Looking at the tree and not seeing everyone complaining, it has got to be how it's built.
Comment on attachment 8840274 [details] [diff] [review] [mozconfigs] proposed workaround patch I'm undecided here. The proposed solution works locally, so r+. I still doesn't fix anyone compiling locally, see bug 1342617. I'm not sure why you don't make more noise in bug 1322703. I'm also not sure why build bot compiles don't fail.
Attachment #8840274 - Flags: review?(jorgk) → review+
(In reply to Jorg K (GMT+1) from comment #24) > Comment on attachment 8840274 [details] [diff] [review] > [mozconfigs] proposed workaround patch > > I'm undecided here. The proposed solution works locally, so r+. I still > doesn't fix anyone compiling locally, see bug 1342617. To be honest, I'm also unsure here also as to whether this fixes anything or just covers an issue with local builds vs. automation builds. > > I'm not sure why you don't make more noise in bug 1322703. I'm also not sure > why build bot compiles don't fail. well, I've ni glandium.. waiting for a response.
https://hg.mozilla.org/comm-central/rev/67c398c134dfc7b092b73b8abf2c35f40eb37ef4 Bug 1340395 - Append -FS to CFLAGS and CXXFLAGS and use generated.pdb for all the linking. r=jorgk
Comment on attachment 8840274 [details] [diff] [review] [mozconfigs] proposed workaround patch I don't think I quite have the understanding for this one. Sorry!
Patrick, never mind. It landed already. Note that in bug 1322703 they're discussing whether they will back that out, in which case this bug here can be backed out as well.
Backout: https://hg.mozilla.org/comm-central/rev/c26182d4b4d9134ad0448178bf98b10d76654795 I backed this out since bug 1322703 also got backed out.
Status: NEW → RESOLVED
Closed: 4 years ago
Product: SeaMonkey → Thunderbird
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 54.0
Version: unspecified → Trunk
You need to log in before you can comment on or make changes to this bug.