Last Comment Bug 1340395 - Linker busting between fatal errors "LNK1000: Internal error during LIB::Search" and "LNK1102: Out of Memory"
: Linker busting between fatal errors "LNK1000: Internal error during LIB::Sea...
Status: NEW
:
Product: SeaMonkey
Classification: Client Software
Component: Build Config (show other bugs)
: unspecified
: x86 Windows
-- normal with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
:
Mentors:
: 1342617 (view as bug list)
Depends on:
Blocks: 1322703
  Show dependency treegraph
 
Reported: 2017-02-16 20:01 PST by Edmund Wong (:ewong)
Modified: 2017-02-25 09:11 PST (History)
8 users (show)
ewong: needinfo? (mh+mozilla)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
log for nightly build. (6.32 MB, text/x-log)
2017-02-17 01:53 PST, Edmund Wong (:ewong)
no flags Details
2nd type of error (6.04 MB, text/x-log)
2017-02-19 21:22 PST, Edmund Wong (:ewong)
no flags Details
[mozconfigs] proposed workaround patch (7.24 KB, patch)
2017-02-22 19:34 PST, Edmund Wong (:ewong)
jorgk: review+
ewong: review? (philip.chee)
ewong: review? (clokep)
Details | Diff | Splinter Review

Description User image Edmund Wong (:ewong) 2017-02-16 20:01:05 PST
Got this with a c-c win32 build:

    ..\..\modules\zlib\src\inflate.obj
    ..\..\modules\zlib\src\inftrees.obj
    ..\..\modules\zlib\src\trees.obj
    ..\..\modules\zlib\src\uncompr.obj
    ..\..\modules\zlib\src\zutil.obj
    StaticXULComponentsEnd\StaticXULComponentsEnd.obj


LINK : fatal error LNK1000: Internal error during LIB::Search

  Version 14.00.24213.1

  ExceptionCode            = C0000005
  ExceptionFlags           = 00000000
  ExceptionAddress         = 000007FEF217C387 (000007FEF2170000) "c:\builds\slave\c-c\build\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT\VCRUNTIME140.dll"
  NumberParameters         = 00000002
  ExceptionInformation[ 0] = 0000000000000000
  ExceptionInformation[ 1] = 0000000405D82000

CONTEXT:
  Rax    = 0000000011738150  R8     = 00000000031CD270
  Rbx    = 0000000011738110  R9     = 0000000000000000
  Rcx    = 0000000000000298  R10    = 0000000405D81CD0
  Rdx    = 00000003F4649B80  R11    = 0000000000000608
  Rsp    = 000000000019D1A8  R12    = 0000000000000D79
  Rbp    = 00000000000005C8  E13    = 00000000031CD270
  Rsi    = 0000000405D82000  R14    = 0000000405D81CD0
  Rdi    = 0000000011738480  R15    = 0000000000000000
  Rip    = 000007FEF217C387  EFlags = 0000000000010207
  SegCs  = 0000000000000033  SegDs  = 000000000000002B
  SegSs  = 000000000000002B  SegEs  = 000000000000002B
  SegFs  = 0000000000000053  SegGs  = 000000000000002B
  Dr0    = 0000000000000000  Dr3    = 0000000000000000
  Dr1    = 0000000000000000  Dr6    = 0000000000000000
  Dr2    = 0000000000000000  Dr7    = 0000000000000000
c:/builds/slave/c-c/build/mozilla/config/rules.mk:787: recipe for target 'xul.dll' failed
mozmake.exe[4]: *** [xul.dll] Error 1000
mozmake.exe[4]: *** Deleting file 'xul.dll'

somewhat akin to bug 436279 but that's been resolved (if only because
it hasn't cropped up in a while).

Just filing this just in case something needs to be done.
Comment 1 User image Edmund Wong (:ewong) 2017-02-17 01:50:54 PST
This is on the loaner.  Win32 build.

Both nightly and normal build fails with either OOM or LNK1000
Comment 2 User image Edmund Wong (:ewong) 2017-02-17 01:53:09 PST
Created attachment 8838458 [details]
log for nightly build.
Comment 3 User image Edmund Wong (:ewong) 2017-02-17 07:41:44 PST
Last good build:
https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299

First bad build:
https://hg.mozilla.org/mozilla-central/rev/8c8b54b13be7
Comment 4 User image Edmund Wong (:ewong) 2017-02-18 16:57:31 PST
Improvement on the last good, first bad:

Last good:
https://hg.mozilla.org/mozilla-central/rev/e783bdf2cb50

First Bad:
https://hg.mozilla.org/mozilla-central/rev/e9b926463f9e
Comment 5 User image Edmund Wong (:ewong) 2017-02-18 17:17:35 PST
(In reply to Edmund Wong (:ewong) from comment #4)
> Improvement on the last good, first bad:
> 
> Last good:
> https://hg.mozilla.org/mozilla-central/rev/e783bdf2cb50
> 

Actually.. the last good is still https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299
Comment 6 User image Edmund Wong (:ewong) 2017-02-18 20:13:03 PST
> First Bad:
> https://hg.mozilla.org/mozilla-central/rev/e9b926463f9e

https://hg.mozilla.org/mozilla-central/rev/2737f66ad6ac gives a 


    StaticXULComponentsEnd\StaticXULComponentsEnd.obj

js_static.lib(Unified_cpp_js_src25.obj) : fatal error LNK1318: Unexpected PDB er
ror; OK (0) ''
c:/builds/slave/c-c/build/mozilla/config/rules.mk:787: recipe for target 'xul.dl
l' failed
mozmake.exe[4]: *** [xul.dll] Error 1318
mozmake.exe[4]: Leaving directory 'c:/builds/slave/c-c/build/objdir/toolkit/libr
ary'
c:/builds/slave/c-c/build/mozilla/config/recurse.mk:71: recipe for target 'toolk
it/library/target' failed
mozmake.exe[3]: *** [toolkit/library/target] Error 2

so not sure if it's related.
Comment 7 User image Ian Neal 2017-02-19 02:20:01 PST
How much memory has the loaner got? How much free memory has it got? Has it been rebooted recently?
Comment 8 User image Edmund Wong (:ewong) 2017-02-19 16:58:04 PST
(In reply to Ian Neal from comment #7)
> How much memory has the loaner got? How much free memory has it got? Has it
> been rebooted recently?

15GB and yes.  It has been rebooted.  

w/ m-c = https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299,

it builds.

w/ any cset post a9ec72f82299, it gives me either errors.  

Now, with tip on c-c and m-c,  it's taken more than 20 minutes to
link, which isn't right since I think it's taken more than 1/2 hr.
This is with a clobber build.
Comment 9 User image Edmund Wong (:ewong) 2017-02-19 21:22:36 PST
Created attachment 8839043 [details]
2nd type of error
Comment 10 User image Edmund Wong (:ewong) 2017-02-20 01:53:53 PST
(In reply to Edmund Wong (:ewong) from comment #9)
> Created attachment 8839043 [details]
> 2nd type of error

by 2nd type of error, I meant:

    ..\..\modules\zlib\src\gzwrite.obj
    ..\..\modules\zlib\src\infback.obj
    ..\..\modules\zlib\src\inffast.obj
    ..\..\modules\zlib\src\inflate.obj
    ..\..\modules\zlib\src\inftrees.obj
    ..\..\modules\zlib\src\trees.obj
    ..\..\modules\zlib\src\uncompr.obj
    ..\..\modules\zlib\src\zutil.obj
    StaticXULComponentsEnd\StaticXULComponentsEnd.obj

js_static.lib(Unified_cpp_js_src15.obj) : fatal error LNK1318: Unexpected PDB error; OK (0) ''

c:/builds/slave/c-c-ntly/build/mozilla/config/rules.mk:787: recipe for target 'xul.dll' failed
mozmake.exe[4]: *** [xul.dll] Error 1318
mozmake.exe[4]: Leaving directory 'c:/builds/slave/c-c-ntly/build/objdir/toolkit/library'
Comment 11 User image Edmund Wong (:ewong) 2017-02-20 18:03:57 PST
:glandium,  might you have any ideas? 

mozilla-central post-https://hg.mozilla.org/mozilla-central/rev/a9ec72f82299 
fails to build.
Comment 12 User image Edmund Wong (:ewong) 2017-02-20 20:53:25 PST
Having spoken with IanN, he suggested I backout https://hg.mozilla.org/mozilla-central/rev/b7a2f7ff5e87, 
which I did and it builds.
Comment 13 User image Edmund Wong (:ewong) 2017-02-20 23:22:34 PST
1) pull both c-c and m-c tip
2) python mozilla\mach clobber
3) build
   - busts with '2nd type of error' log

4) python mozilla\mach clobber
5) cd mozilla
6) hg backout -r b7a2f7ff5e87
7) cd ..
8) <build>
   - completes build
Comment 14 User image Edmund Wong (:ewong) 2017-02-21 23:55:54 PST
Now with m-c's tip, you need to backout both:

https://hg.mozilla.org/mozilla-central/rev/e7c397118fd2

and

https://hg.mozilla.org/mozilla-central/rev/b7a2f7ff5e87
Comment 15 User image Jorg K (GMT+1) 2017-02-22 02:12:29 PST
My last two clobber builds of TB on 2017-02-18 and today failed with the same error. I only have 8GB RAM and 800MB are dedicated to internal graphics. Before I had no trouble building (and BTW, I have three development machines with 8GB).

At first I followed my own advice given here:
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Thunderbird_build:

If on Windows you get link errors like "LNK1102: out of memory" or "LNK1318: Unexpected PDB error; OK (0)", try deleting the largest .PDB files before rushing out the door to buy more RAM. Clobbering (see below) will also remove those files.

That didn't help. So then I deleted all 3587 PDB files I saw following
http://stackoverflow.com/questions/174013/linker-out-of-memory-lnk1102
and the linker went through, albeit with a lot of warnings about missing PDB files.

Since according to comment #8 even 16GB are not enough, I'd like to see this rectified. BTW, only few people build TB/SM in debug mode. Many TB developers are on Linux, Richard and FRG build non-debug. And Kent James will wake up when he tries to build next time.
Comment 17 User image Edmund Wong (:ewong) 2017-02-22 19:34:06 PST
Created attachment 8840274 [details] [diff] [review]
[mozconfigs] proposed workaround patch

This is a workaround that I found to work.  I haven't tried the debug build yet, but I have pushed to try: 
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c3a7bcf4de70599f240111d5bfef6a0566ce3347
Comment 18 User image Jorg K (GMT+1) 2017-02-22 22:03:25 PST
Thanks, I'll compile with the patch next time I do a full debug build.
Comment 19 User image :aceman 2017-02-23 02:46:16 PST
So is this a Visual Studio compiler bug?
Comment 20 User image Jorg K (GMT+1) 2017-02-23 02:52:03 PST
I don't think so. You tell the compiler to generate 3587 PDB files and then the linker chokes on those. This might work for small programs, but not for TB or SM; I haven't tried compiling FF lately.
Comment 21 User image Jorg K (GMT+1) 2017-02-23 14:01:24 PST
I added
  CFLAGS="$CFLAGS -FS"
  CXXFLAGS="$CXXFLAGS -FS"
  mk_add_options "export COMPILE_PDB_FLAG=-Fdgenerated.pdb"
to my local config and now I can build again.

I get 722 PDB files (instead of 3587), mostly called generated.pdb.

Question: Do we need the patch if the builds work on the servers? How much RAM do they have?
Comment 22 User image Edmund Wong (:ewong) 2017-02-23 16:58:13 PST
(In reply to Jorg K (GMT+1) from comment #21)
> I added
>   CFLAGS="$CFLAGS -FS"
>   CXXFLAGS="$CXXFLAGS -FS"
>   mk_add_options "export COMPILE_PDB_FLAG=-Fdgenerated.pdb"
> to my local config and now I can build again.
> 
> I get 722 PDB files (instead of 3587), mostly called generated.pdb.
>

They should be all called generated.pdb (iiuc).
 
> Question: Do we need the patch if the builds work on the servers? How much
> RAM do they have?

I'm not too sure how much they have but if they are similar to the loaner
I use, should be around 15GB.  

That said, I'm wondering if 'how' they are built makes a difference.
Looking at the tree and not seeing everyone complaining, it has got to
be how it's built.
Comment 23 User image Alice0775 White 2017-02-25 08:57:36 PST
*** Bug 1342617 has been marked as a duplicate of this bug. ***
Comment 24 User image Jorg K (GMT+1) 2017-02-25 09:11:46 PST
Comment on attachment 8840274 [details] [diff] [review]
[mozconfigs] proposed workaround patch

I'm undecided here. The proposed solution works locally, so r+. I still doesn't fix anyone compiling locally, see bug 1342617.

I'm not sure why you don't make more noise in bug 1322703. I'm also not sure why build bot compiles don't fail.

Note You need to log in before you can comment on or make changes to this bug.