982310 - VS2013 unified build: unresolved external symbol std::_Debug_message in trace-malloc

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(Away)]

VS2013 Update 2 CTP 2, with --enable-debug --enable-trace-malloc. Does not happen if I --disable-unified-compilation.

17:57.95 spacetrace.exe
17:57.96 d:/src/vs2013/objxchk/_virtualenv/Scripts/python.exe d:/src/vs2013/config/expandlibs_exec.py --depend .deps/spacetrace.exe.pp --target spacetrace.exe --uselist -- link -NOLOGO -OUT:spacetrace.exe -PDB:spacetrace.pdb  -LARGEADDRESSAWARE -NXCOMPAT -RELEASE -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV     d:/src/vs2013/objxchk/dist/lib/mozglue.lib spacetrace.obj spacecategory.obj formdata.obj ./module.res kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib  tmreader.obj adreader.obj d:/src/vs2013/objxchk/dist/lib/xpcomglue_s.lib d:/src/vs2013/objxchk/dist/lib/xul.lib d:/src/vs2013/objxchk/dist/lib/mozalloc.lib d:/src/vs2013/objxchk/dist/lib/nspr4.lib d:/src/vs2013/objxchk/dist/lib/plc4.lib d:/src/vs2013/objxchk/dist/lib/plds4.lib    kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib
17:58.00 Executing: link -nologo -out:leakstats.exe -pdb:leakstats.pdb @d:\src\vs2013\objxchk\tools\trace-malloc\tmpf5sfpw.list -LARGEADDRESSAWARE -NXCOMPAT -RELEASE -DYNAMICBASE -SAFESEH -DEBUG -DEBUGTYPE:CV ..\..\dist\lib\mozglue.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib ..\..\dist\lib\xpcomglue_s.lib ..\..\dist\lib\xul.lib ..\..\dist\lib\mozalloc.lib ..\..\dist\lib\nspr4.lib ..\..\dist\lib\plc4.lib ..\..\dist\lib\plds4.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib
17:58.00 d:\src\vs2013\objxchk\tools\trace-malloc\tmpf5sfpw.list:
17:58.00     leakstats.obj
17:58.00     tmreader.obj
17:58.00     adreader.obj
17:58.00 
17:58.00    Creating library leakstats.lib and object leakstats.exp
17:58.00 
17:58.00 xpcomglue_s.lib(Unified_cpp_xpcom_glue0.obj) : error LNK2019: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z) referenced in function "unsigned long __cdecl std::_Atomic_fetch_add_4(unsigned long volatile *,unsigned long,enum std::memory_order)" (?_Atomic_fetch_add_4@std@@YAKPCKKW4memory_order@1@@Z)
17:58.00 
17:58.00 xpcomglue_s.lib(Unified_cpp_xpcom_glue1.obj) : error LNK2001: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z)
17:58.00 
17:58.00 leakstats.exe : fatal error LNK1120: 1 unresolved externals
17:58.00 
17:58.00 Executing: link -nologo -out:tmstats.exe -pdb:tmstats.pdb @d:\src\vs2013\objxchk\tools\trace-malloc\tmpgazaty.list -LARGEADDRESSAWARE -NXCOMPAT -RELEASE -DYNAMICBASE -SAFESEH -DEBUG -DEBUGTYPE:CV ..\..\dist\lib\mozglue.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib ..\..\dist\lib\xpcomglue_s.lib ..\..\dist\lib\xul.lib ..\..\dist\lib\mozalloc.lib ..\..\dist\lib\nspr4.lib ..\..\dist\lib\plc4.lib ..\..\dist\lib\plds4.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib
17:58.00 d:\src\vs2013\objxchk\tools\trace-malloc\tmpgazaty.list:
17:58.00     tmstats.obj
17:58.00     tmreader.obj
17:58.00     adreader.obj
17:58.00 
17:58.00    Creating library tmstats.lib and object tmstats.exp
17:58.00 
17:58.00 xpcomglue_s.lib(Unified_cpp_xpcom_glue0.obj) : error LNK2019: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z) referenced in function "unsigned long __cdecl std::_Atomic_fetch_add_4(unsigned long volatile *,unsigned long,enum std::memory_order)" (?_Atomic_fetch_add_4@std@@YAKPCKKW4memory_order@1@@Z)
17:58.00 
17:58.00 xpcomglue_s.lib(Unified_cpp_xpcom_glue1.obj) : error LNK2001: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z)
17:58.00 
17:58.00 tmstats.exe : fatal error LNK1120: 1 unresolved externals
17:58.00 
17:58.01 d:/src/vs2013/config/rules.mk:803: recipe for target 'leakstats.exe' failed
17:58.01 mozmake.EXE[5]: *** [leakstats.exe] Error 1120
17:58.01 mozmake.EXE[5]: *** Waiting for unfinished jobs....
17:58.01 d:/src/vs2013/config/rules.mk:803: recipe for target 'tmstats.exe' failed
17:58.01 mozmake.EXE[5]: *** [tmstats.exe] Error 1120
17:58.09    Creating library leaksoup.lib and object leaksoup.exp
17:58.09 
17:58.09 Executing: link -NOLOGO -OUT:spacetrace.exe -PDB:spacetrace.pdb -LARGEADDRESSAWARE -NXCOMPAT -RELEASE -DYNAMICBASE -SAFESEH -DEBUG -DEBUGTYPE:CV ..\..\dist\lib\mozglue.lib @d:\src\vs2013\objxchk\tools\trace-malloc\tmpn9kdd1.list module.res kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib ..\..\dist\lib\xpcomglue_s.lib ..\..\dist\lib\xul.lib ..\..\dist\lib\mozalloc.lib ..\..\dist\lib\nspr4.lib ..\..\dist\lib\plc4.lib ..\..\dist\lib\plds4.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib
17:58.09 d:\src\vs2013\objxchk\tools\trace-malloc\tmpn9kdd1.list:
17:58.09     spacetrace.obj
17:58.09     spacecategory.obj
17:58.10     formdata.obj
17:58.10     tmreader.obj
17:58.10     adreader.obj
17:58.10 
17:58.10    Creating library spacetrace.lib and object spacetrace.exp
17:58.10 
17:58.10 xpcomglue_s.lib(Unified_cpp_xpcom_glue1.obj) : error LNK2019: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z) referenced in function "unsigned long __cdecl std::_Atomic_fetch_add_4(unsigned long volatile *,unsigned long,enum std::memory_order)" (?_Atomic_fetch_add_4@std@@YAKPCKKW4memory_order@1@@Z)
17:58.10 
17:58.10 xpcomglue_s.lib(Unified_cpp_xpcom_glue0.obj) : error LNK2001: unresolved external symbol "__declspec(dllimport) void __cdecl std::_Debug_message(wchar_t const *,wchar_t const *,unsigned int)" (__imp_?_Debug_message@std@@YAXPB_W0I@Z)
17:58.10 
17:58.10 spacetrace.exe : fatal error LNK1120: 1 unresolved externals
17:58.10 
17:58.10 d:/src/vs2013/config/rules.mk:733: recipe for target 'spacetrace.exe' failed
17:58.10 mozmake.EXE[5]: *** [spacetrace.exe] Error 1120
17:58.11 mozmake.EXE[5]: Leaving directory 'd:/src/vs2013/objxchk/tools/trace-malloc'
17:58.11 d:/src/vs2013/config/recurse.mk:100: recipe for target 'tools/trace-malloc/libs' failed
17:58.11 mozmake.EXE[4]: *** [tools/trace-malloc/libs] Error 2
17:58.11 mozmake.EXE[4]: Leaving directory 'd:/src/vs2013/objxchk'
17:58.11 d:/src/vs2013/config/recurse.mk:39: recipe for target 'libs' failed
17:58.11 mozmake.EXE[3]: *** [libs] Error 2
17:58.11 mozmake.EXE[3]: Leaving directory 'd:/src/vs2013/objxchk'
17:58.12 d:/src/vs2013/config/rules.mk:593: recipe for target 'default' failed
17:58.12 mozmake.EXE[2]: *** [default] Error 2
17:58.12 mozmake.EXE[2]: Leaving directory 'd:/src/vs2013/objxchk'
17:58.12 d:/src/vs2013/client.mk:398: recipe for target 'realbuild' failed
17:58.12 mozmake.EXE[1]: *** [realbuild] Error 2
17:58.12 mozmake.EXE[1]: Leaving directory 'd:/src/vs2013'
17:58.12 client.mk:185: recipe for target 'build' failed
17:58.12 mozmake.EXE: *** [build] Error 2
17:58.13 715 compiler warnings present.

Comment 1

[(Away)]

These are the files in xpcom/glue/moz.build that I had to de-unify in order to build:

xpcom_glue_src_lcppsrcs_deunif = [
    'nsThreadUtils.cpp',
]

xpcom_gluens_src_lcppsrcs_deunif = [
    'GenericFactory.cpp',
    'nsProxyRelease.cpp',
]

SOURCES += [
    'GenericModule.cpp',
]

Comment 2

[(Away)]

Alternatively, I can remove #  define MOZ_HAVE_CXX11_ATOMICS from Atomics.h:

https://hg.mozilla.org/mozilla-central/file/23005b395ae8/mfbt/Atomics.h#l43

Comment 3

[(Away)]

Or, I can keep everything unified, keep MOZ_HAVE_CXX11_ATOMICS, but #define _INVALID_MEMORY_ORDER as nothing, in order to keep the atomics code from using _Debug_message.

All of these are hacks, not real fixes!

Comment 4

[(no longer active)]

(In reply to David Major [:dmajor] from comment #2)
> Alternatively, I can remove #  define MOZ_HAVE_CXX11_ATOMICS from Atomics.h:
> 
> https://hg.mozilla.org/mozilla-central/file/23005b395ae8/mfbt/Atomics.h#l43

Nathan, what is that used for?

FWIW, getting those files out of the unified sources is fine, but this is probably a sign of Atomics.h screwing us over, so I'd like to understand the underlying cost, and then add protections like http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/backend/recursivemake.py#663 against it so that we know about future occurrences of this problem at build time.

Comment 5

[Mike Hommey [:glandium]]

The real problem as it seems is that we rely on xpcomglue_s.lib being a library, which comes with some perks: any file it contains which symbols are not used is just dropped. So without unified sources, the files including Atomics.h are probably dropped when linking those executables because they contain no symbol they directly use. With unified sources, that's a different story, since the executables use symbols that end up in files unified with files including Atomics.h. The same problem would most likely happen if xpcomglue_s was a fake library, or if the files it contains were linked to the executables directly. To double check my theory is right, you can try this:
- Do a full build without unification
- Remove objdir/dist/lib/xpcomglue_s.lib.
- Make sure there is an objdir/dist/lib/xpcomglue_s.lib.desc file. If not, copy it from xpcom/glue.
- mozmake -C objdir/tools/trace-malloc

That should fail with the same error.
Now, I suspect this missing symbol is in the CRT, considering it's in the std namespace. Which suggest we're might "just" have a CRT discrepancy problem (like, xpcomglue_s built for the debug crt and the executables linked with the release crt, which may not contain that debug_message symbol)

Comment 6

[Mike Hommey [:glandium]]

So, please attach a complete log of mach build -v.

Comment 7

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #4)
> (In reply to David Major [:dmajor] from comment #2)
> > Alternatively, I can remove #  define MOZ_HAVE_CXX11_ATOMICS from Atomics.h:
> > 
> > https://hg.mozilla.org/mozilla-central/file/23005b395ae8/mfbt/Atomics.h#l43
> 
> Nathan, what is that used for?

It's used for determining whether we should use <atomic>.  Since MSVC 2013 is the first version that actually triggers that preprocessing path, it's not surprising that we're having issues.  We've been using the non-<atomic> codepaths in Atomics.h for a while now; I don't think it'd be problematic to bump the _MSC_VER requirement for that conditional arm or remove/comment out entirely, depending.

But it'd be much nicer to be able to use <atomic> on MSVC, so glandium's theory is probably worth checking out first.

Comment 8

[(Away)]

Attachment: mach build -v

Comment 9

[(Away)]

(In reply to Mike Hommey [:glandium] from comment #5)
> linked to the executables directly. To double check my theory is right, you
> can try this:
> - Do a full build without unification
> - Remove objdir/dist/lib/xpcomglue_s.lib.
> - Make sure there is an objdir/dist/lib/xpcomglue_s.lib.desc file. If not,
> copy it from xpcom/glue.
> - mozmake -C objdir/tools/trace-malloc
> 
> That should fail with the same error.

Yes, it does.

Comment 10

[(Away)]

(In reply to Nathan Froyd (:froydnj) from comment #7)
> It's used for determining whether we should use <atomic>.  Since MSVC 2013
> is the first version that actually triggers that preprocessing path, it's
> not surprising that we're having issues.

Actually _MSC_VER == 1700 is VS 2012, and I just confirmed that it has the same problem.

Comment 11

[(no longer active)]

Great analysis, thanks everyone!  So does this mean we should look for "just" that CRT discrepancy issue?  :-)

Comment 12

[(Away)]

Do comment 8 and 9 provide any clues?

Comment 13

[Mike Hommey [:glandium]]

(In reply to David Major [:dmajor] (Away March 22 to April 2) from comment #12)
> Do comment 8 and 9 provide any clues?

I don't see anything obvious in the build log, but comment 9 at least confirms the theory. You have to find why it's not linking to the debug CRT.

Comment 14

[(Away)]

I'm pretty sure it's linking to the debug CRT, all the |-MDd|'s look okay. It seems the question is *which* debug CRT.

std::_Debug_message is exported from msvcp120d.dll (C++ debug runtime library) but not msvcr120d.dll (C debug runtime library). 

The failing trace-malloc tools are .c files. I tried renaming everything to .cpp, and that fixed spacetrace.exe, but not tmstats.exe or leakstats.exe. Maybe it's because they pull in some NSPR C files?

Comment 15

[(Away)]

Attachment: Hack to avoid std::_Debug_message

I really don't like working around problems without understanding them, but if for some reason this needs to move forward while I am PTO, I offer this hack and a donation to the penalty jar.

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8394991 [details] [diff] [review]
Hack to avoid std::_Debug_message

I /think/ Ehsan has review power here. If not, please reassign.

Comment 17

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/626e32186d38

Comment 18

[(no longer active)]

Greg, why did you land this?  David provided this patch in case it's needed while he's away (see comment 15).  I think we should back this out and wait for a proper fix unless if there is a reason why we need to take this right now.

Comment 19

[Gregory Szorc [:gps]]

I misunderstood dmajor's intentions.

Is the patch doing any harm? If so, let's back it out.

If not, I like having it landed because it unblocks the VS2013 evaluation effort (I think). We can always unhack it later, right?

Comment 20

[(no longer active)]

I guess it's not doing any harm, and we can definitely unhack it.  But let's at least make sure the bug is not closed when this gets merged to central! :-)

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/626e32186d38

Comment 22

[(Away)]

Tinderbox builds no longer use trace-malloc as of Bug 1013014. The hack can be undone if we want (though it's not really doing any harm).

Comment 23

[Mike Hommey [:glandium]]

I'm ready to bet some people are still building locally with --enable-trace-malloc. Until that's actually removed, it should probably stay.

Comment 24

[(Away)]

I'm going to resolve this because VC12 is no longer blocked by the build failure.

I'll open a separate bug to remove the hack with a dependency on Bug 1014341 (remove-trace-malloc).