Closed Bug 795204 Opened 7 years ago Closed 7 years ago

Windows needing clobber too often

Categories

(Firefox Build System :: General, defect)

x86
Windows 7
defect
Not set

Tracking

(firefox18 fixed)

RESOLVED FIXED
mozilla19
Tracking Status
firefox18 --- fixed

People

(Reporter: philor, Assigned: glandium)

References

(Blocks 1 open bug)

Details

(Whiteboard: [capacity])

Attachments

(1 file, 2 obsolete files)

Of course, since I didn't file on any of the last two or three, I'll never find what they were again, but https://hg.mozilla.org/integration/mozilla-inbound/rev/b24a73b2c617 failed in https://tbpl.mozilla.org/php/getParsedLog.php?id=15612554&tree=Mozilla-Inbound with

e:\builds\moz2_slave\m-in-w32-dbg\build\config\rules.mk:877:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/m-in-w32-dbg/build/config/pythonpath.py -I../../config e:/builds/moz2_slave/m-in-w32-dbg/build/config/expandlibs_exec.py --depend .deps/xul.dll.pp --target xul.dll --uselist -- link -NOLOGO -DLL -OUT:xul.dll -PDB:xul.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86  dlldeps-xul.obj nsStaticXULComponents.obj nsDllMain.obj nsGFXDeps.obj dlldeps-zlib.obj nsUnicharUtils.obj nsBidiUtils.obj nsSpecialCasingData.obj nsUnicodeProperties.obj nsRDFResource.obj   ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV    ../../toolkit/components/osfile/osfile_s.lib ../../toolkit/xre/xulapp_s.lib  ../../staticlib/components/necko.lib ../../staticlib/components/uconv.lib ../../staticlib/components/i18n.lib ../../staticlib/components/chardet.lib ../../staticlib/components/jar50.lib ../../staticlib/components/startupcache.lib ../../staticlib/components/pref.lib ../../staticlib/components/htmlpars.lib ../../staticlib/components/identity.lib ../../staticlib/components/imglib2.lib ../../staticlib/components/mediasniffer.lib ../../staticlib/components/gkgfx.lib ../../staticlib/components/gklayout.lib ../../staticlib/components/docshell.lib ../../staticlib/components/embedcomponents.lib ../../staticlib/components/webbrwsr.lib ../../staticlib/components/nsappshell.lib ../../staticlib/components/txmgr.lib ../../staticlib/components/commandlines.lib ../../staticlib/components/toolkitcomps.lib ../../staticlib/components/pipboot.lib ../../staticlib/components/pipnss.lib ../../staticlib/components/appcomps.lib ../../staticlib/components/jsreflect.lib ../../staticlib/components/composer.lib ../../staticlib/components/telemetry.lib ../../staticlib/components/jsinspector.lib ../../staticlib/components/jsdebugger.lib ../../staticlib/components/storagecomps.lib ../../staticlib/components/rdf.lib ../../staticlib/components/windowds.lib ../../staticlib/components/jsctypes.lib ../../staticlib/components/jsperf.lib ../../staticlib/components/gkplugin.lib ../../staticlib/components/windowsproxy.lib ../../staticlib/components/jsd.lib ../../staticlib/components/autoconfig.lib ../../staticlib/components/auth.lib ../../staticlib/components/cookie.lib ../../staticlib/components/permissions.lib ../../staticlib/components/universalchardet.lib ../../staticlib/components/places.lib ../../staticlib/components/tkautocomplete.lib ../../staticlib/components/satchel.lib ../../staticlib/components/pippki.lib ../../staticlib/components/imgicon.lib ../../staticlib/components/profiler.lib ../../staticlib/components/widget_windows.lib ../../staticlib/components/accessibility.lib ../../staticlib/components/spellchecker.lib ../../staticlib/components/zipwriter.lib ../../staticlib/components/services-crypto.lib ../../staticlib/components/gkdebug.lib ../../staticlib/jsipc_s.lib ../../staticlib/domipc_s.lib ../../staticlib/domplugins_s.lib ../../staticlib/mozipc_s.lib ../../staticlib/mozipdlgen_s.lib ../../staticlib/ipcshell_s.lib ../../staticlib/gfxipc_s.lib ../../staticlib/hal_s.lib ../../staticlib/dombindings_s.lib ../../staticlib/xpcom_core.lib ../../staticlib/ucvutil_s.lib ../../staticlib/chromium_s.lib ../../staticlib/snappy_s.lib ../../staticlib/thebes.lib ../../staticlib/gl.lib ../../staticlib/ycbcr.lib   e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/mozjs.lib e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/crmf.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/smime3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/ssl3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nss3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nssutil3.lib   ../../dist/lib/mozsqlite3.lib  ../../dist/lib/gkmedias.lib   e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nspr4.lib e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/plc4.lib e:/builds/moz2_slave/m-in-w32-dbg/bu   Creating library xul.lib and object xul.exp
LINK : warning LNK4098: defaultlib 'LIBCMTD' conflicts with use of other libs; use /NODEFAULTLIB:library
nsGlobalWindow.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: bool __thiscall JSAutoStructuredCloneBuffer::read(struct JSContext *,class JS::Value *,struct JSStructuredCloneCallbacks const *,void *)" (__imp_?read@JSAutoStructuredCloneBuffer@@QAE_NPAUJSContext@@PAVValue@JS@@PBUJSStructuredCloneCallbacks@@PAX@Z) referenced in function "public: virtual enum tag_nsresult __stdcall PostMessageEvent::Run(void)" (?Run@PostMessageEvent@@UAG?AW4tag_nsresult@@XZ)
Events.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: bool __thiscall JSAutoStructuredCloneBuffer::read(struct JSContext *,class JS::Value *,struct JSStructuredCloneCallbacks const *,void *)" (__imp_?read@JSAutoStructuredCloneBuffer@@QAE_NPAUJSContext@@PAVValue@JS@@PBUJSStructuredCloneCallbacks@@PAX@Z)
XMLHttpRequest.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: bool __thiscall JSAutoStructuredCloneBuffer::read(struct JSContext *,class JS::Value *,struct JSStructuredCloneCallbacks const *,void *)" (__imp_?read@JSAutoStructuredCloneBuffer@@QAE_NPAUJSContext@@PAVValue@JS@@PBUJSStructuredCloneCallbacks@@PAX@Z)
IDBObjectStore.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: bool __thiscall JSAutoStructuredCloneBuffer::read(struct JSContext *,class JS::Value *,struct JSStructuredCloneCallbacks const *,void *)" (__imp_?read@JSAutoStructuredCloneBuffer@@QAE_NPAUJSContext@@PAVValue@JS@@PBUJSStructuredCloneCallbacks@@PAX@Z)
nsStructuredCloneContainer.obj : error LNK2019: unresolved external symbol __imp__JS_ClearStructuredClone referenced in function "public: virtual enum tag_nsresult __stdcall nsStructuredCloneContainer::InitFromVariant(class nsIVariant *,struct JSContext *)" (?InitFromVariant@nsStructuredCloneContainer@@UAG?AW4tag_nsresult@@PAVnsIVariant@@PAUJSContext@@@Z)
nsStructuredCloneContainer.obj : error LNK2019: unresolved external symbol __imp__JS_StructuredCloneHasTransferables referenced in function "public: virtual enum tag_nsresult __stdcall nsStructuredCloneContainer::DeserializeToVariant(struct JSContext *,class nsIVariant * *)" (?DeserializeToVariant@nsStructuredCloneContainer@@UAG?AW4tag_nsresult@@PAUJSContext@@PAPAVnsIVariant@@@Z)
WorkerPrivate.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: bool __thiscall JSAutoStructuredCloneBuffer::write(struct JSContext *,class JS::Value,class JS::Value,struct JSStructuredCloneCallbacks const *,void *)" (__imp_?write@JSAutoStructuredCloneBuffer@@QAE_NPAUJSContext@@VValue@JS@@1PBUJSStructuredCloneCallbacks@@PAX@Z) referenced in function "public: bool __thiscall mozilla::dom::workers::WorkerPrivate::PostMessageToParent(struct JSContext *,class JS::Value,class JS::Value)" (?PostMessageToParent@WorkerPrivate@workers@dom@mozilla@@QAE_NPAUJSContext@@VValue@JS@@1@Z)
xul.dll : fatal error LNK1120: 4 unresolved externals
ild/obj-firefox/dist/lib/plds4.lib  ../../dist/lib/mozalloc.lib -DELAYLOAD:psapi.dll -DELAYLOAD:dbghelp.dll -DELAYLOAD:rasapi32.dll -DELAYLOAD:rasdlg.dll -DELAYLOAD:comdlg32.dll -DELAYLOAD:winspool.drv -DELAYLOAD:secur32.dll -DELAYLOAD:wininet.dll  -DELAYLOAD:oleacc.dll e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/mozglue.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib shell32.lib ole32.lib version.lib winspool.lib comdlg32.lib imm32.lib msimg32.lib shlwapi.lib psapi.lib ws2_32.lib dbghelp.lib rasapi32.lib rasdlg.lib iphlpapi.lib uxtheme.lib setupapi.lib secur32.lib sensorsapi.lib portabledeviceguids.lib windowscodecs.lib wininet.lib oleacc.lib delayimp.lib  usp10.lib oleaut32.lib   
e:\builds\moz2_slave\m-in-w32-dbg\build\config\rules.mk:877:0: command 'e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/m-in-w32-dbg/build/config/pythonpath.py -I../../config e:/builds/moz2_slave/m-in-w32-dbg/build/config/expandlibs_exec.py --depend .deps/xul.dll.pp --target xul.dll --uselist -- link -NOLOGO -DLL -OUT:xul.dll -PDB:xul.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86  dlldeps-xul.obj nsStaticXULComponents.obj nsDllMain.obj nsGFXDeps.obj dlldeps-zlib.obj nsUnicharUtils.obj nsBidiUtils.obj nsSpecialCasingData.obj nsUnicodeProperties.obj nsRDFResource.obj   ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV    ../../toolkit/components/osfile/osfile_s.lib ../../toolkit/xre/xulapp_s.lib  ../../staticlib/components/necko.lib ../../staticlib/components/uconv.lib ../../staticlib/components/i18n.lib ../../staticlib/components/chardet.lib ../../staticlib/components/jar50.lib ../../staticlib/components/startupcache.lib ../../staticlib/components/pref.lib ../../staticlib/components/htmlpars.lib ../../staticlib/components/identity.lib ../../staticlib/components/imglib2.lib ../../staticlib/components/mediasniffer.lib ../../staticlib/components/gkgfx.lib ../../staticlib/components/gklayout.lib ../../staticlib/components/docshell.lib ../../staticlib/components/embedcomponents.lib ../../staticlib/components/webbrwsr.lib ../../staticlib/components/nsappshell.lib ../../staticlib/components/txmgr.lib ../../staticlib/components/commandlines.lib ../../staticlib/components/toolkitcomps.lib ../../staticlib/components/pipboot.lib ../../staticlib/components/pipnss.lib ../../staticlib/components/appcomps.lib ../../staticlib/components/jsreflect.lib ../../staticlib/components/composer.lib ../../staticlib/components/telemetry.lib ../../staticlib/components/jsinspector.lib ../../staticlib/components/jsdebugger.lib ../../staticlib/components/storagecomps.lib ../../staticlib/components/rdf.lib ../../staticlib/components/windowds.lib ../../staticlib/components/jsctypes.lib ../../staticlib/components/jsperf.lib ../../staticlib/components/gkplugin.lib ../../staticlib/components/windowsproxy.lib ../../staticlib/components/jsd.lib ../../staticlib/components/autoconfig.lib ../../staticlib/components/auth.lib ../../staticlib/components/cookie.lib ../../staticlib/components/permissions.lib ../../staticlib/components/universalchardet.lib ../../staticlib/components/places.lib ../../staticlib/components/tkautocomplete.lib ../../staticlib/components/satchel.lib ../../staticlib/components/pippki.lib ../../staticlib/components/imgicon.lib ../../staticlib/components/profiler.lib ../../staticlib/components/widget_windows.lib ../../staticlib/components/accessibility.lib ../../staticlib/components/spellchecker.lib ../../staticlib/components/zipwriter.lib ../../staticlib/components/services-crypto.lib ../../staticlib/components/gkdebug.lib ../../staticlib/jsipc_s.lib ../../staticlib/domipc_s.lib ../../staticlib/domplugins_s.lib ../../staticlib/mozipc_s.lib ../../staticlib/mozipdlgen_s.lib ../../staticlib/ipcshell_s.lib ../../staticlib/gfxipc_s.lib ../../staticlib/hal_s.lib ../../staticlib/dombindings_s.lib ../../staticlib/xpcom_core.lib ../../staticlib/ucvutil_s.lib ../../staticlib/chromium_s.lib ../../staticlib/snappy_s.lib ../../staticlib/thebes.lib ../../staticlib/gl.lib ../../staticlib/ycbcr.lib   e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/mozjs.lib e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/crmf.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/smime3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/ssl3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nss3.lib         e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nssutil3.lib   ../../dist/lib/mozsqlite3.lib  ../../dist/lib/gkmedias.lib   e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/nspr4.lib e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/plc4.lib e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/plds4.lib  ../../dist/lib/mozalloc.lib -DELAYLOAD:psapi.dll -DELAYLOAD:dbghelp.dll -DELAYLOAD:rasapi32.dll -DELAYLOAD:rasdlg.dll -DELAYLOAD:comdlg32.dll -DELAYLOAD:winspool.drv -DELAYLOAD:secur32.dll -DELAYLOAD:wininet.dll  -DELAYLOAD:oleacc.dll e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/dist/lib/mozglue.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib shell32.lib ole32.lib version.lib winspool.lib comdlg32.lib imm32.lib msimg32.lib shlwapi.lib psapi.lib ws2_32.lib dbghelp.lib rasapi32.lib rasdlg.lib iphlpapi.lib uxtheme.lib setupapi.lib secur32.lib sensorsapi.lib portabledeviceguids.lib windowscodecs.lib wininet.lib oleacc.lib delayimp.lib  usp10.lib oleaut32.lib   ' failed, return code 1120
<libs>: Found error
<xul.lib>: Found error
<../../dist/bin/xul.dll>: Found error
<libs>: Found error
<../../dist/lib/xul.lib>: Found error
<../../dist/sdk/lib/xul.lib>: Found error
<libs>: Found error
<libs>: Found error
<libs>: Found error
e:\builds\moz2_slave\m-in-w32-dbg\build\config\makefiles\target_libs.mk:18:0: command 'C:/mozilla-build/buildbotve/scripts/python.exe e:/builds/moz2_slave/m-in-w32-dbg/build/build/pymake/pymake/../make.py -C toolkit/library libs' failed, return code 2

which looks a great deal like the last few times we've just said "oh, well, Windows needs a clobber" when we shouldn't have just blown it off.
I've seen some import-library related weirdness myself in the last 24 hours, and I remember an irc conversation about it recently too.
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=aaf9e3020132

https://tbpl.mozilla.org/php/getParsedLog.php?id=15675302&tree=Mozilla-Inbound

LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
RasterImage.obj : error LNK2019: unresolved external symbol "bool __cdecl mozilla::gfx::Scale(unsigned char *,int,int,int,unsigned char *,int,int,int,enum mozilla::gfx::SurfaceFormat)" (?Scale@gfx@mozilla@@YA_NPEAEHHH0HHHW4SurfaceFormat@12@@Z) referenced in function "bool __cdecl ScaleFrameImage(class imgFrame *,class imgFrame *,struct gfxSize const &)" (?ScaleFrameImage@@YA_NPEAVimgFrame@@0AEBUgfxSize@@@Z)
xul.dll : fatal error LNK1120: 1 unresolved externals
https://tbpl.mozilla.org/php/getParsedLog.php?id=15681612&tree=Firefox

RasterImage.obj : error LNK2001: unresolved external symbol "bool __cdecl mozilla::gfx::Scale(unsigned char *,int,int,int,unsigned char *,int,int,int,enum mozilla::gfx::SurfaceFormat)" (?Scale@gfx@mozilla@@YA_NPAEHHH0HHHW4SurfaceFormat@12@@Z)

when that merged to m-c.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15689519&tree=Services-Central

RasterImage.obj : error LNK2019: unresolved external symbol "bool __cdecl mozilla::gfx::Scale(unsigned char *,int,int,int,unsigned char *,int,int,int,enum mozilla::gfx::SurfaceFormat)" (?Scale@gfx@mozilla@@YA_NPAEHHH0HHHW4SurfaceFormat@12@@Z) referenced in function "bool __cdecl ScaleFrameImage(class imgFrame *,class imgFrame *,struct gfxSize const &)" (?ScaleFrameImage@@YA_NPAVimgFrame@@0ABUgfxSize@@@Z)
(In reply to Phil Ringnalda (:philor) from comment #6)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=15681612&tree=Firefox
> 
> RasterImage.obj : error LNK2001: unresolved external symbol "bool __cdecl
> mozilla::gfx::Scale(unsigned char *,int,int,int,unsigned char
> *,int,int,int,enum mozilla::gfx::SurfaceFormat)"
> (?Scale@gfx@mozilla@@YA_NPAEHHH0HHHW4SurfaceFormat@12@@Z)
> 
> when that merged to m-c.

I tried to reproduce this, and blatantly failed. The core problem is that the .lib (gkmedias.lib for this one, mozjs.lib for comment 0) is not copied in dist/lib when it should be. Except that no matter what i do locally, the .lib is always copied. In fact, it's even copied when it shouldn't...

The only way i can think of that would lead to this happening is if the .lib created by msvc is somehow older than the one already in dist/lib, which would suggest some clock problem or something alike.

When that happens again, it would be nice to either isolate the builder or preserve the objdir somehow (move it to some temporary place) so that one can look at what exactly is going on with these .libs.
Blocks: 787184
https://tbpl.mozilla.org/php/getParsedLog.php?id=15713801&tree=Mozilla-Inbound from the same patch landing again after being backed out, but thanks to slave affinity meaning it's the slave most likely to be picked for the next inbound win32 debug build and it being after 5 Eastern, I probably won't get the objdir preserved for you. Slightly higher odds when that merges to m-c, since there's less likely to be another of the same job right after it finishes.
Depends on: 797458
All of Windows, not just Win64 - it's pure luck (and the fact that a lot of objdirs get removed to make space, so we do a lot of clobber builds) that you made it that far.
Thanks, I've clobbered all of Windows, but not before I got this one again:
https://tbpl.mozilla.org/php/getParsedLog.php?id=15802688&tree=Fx-Team
Duplicate of this bug: 798811
I think I have a fix for this problem, but I need to test it thoroughly, because this is a race condition. Also, chances are a clobber is actually *not* needed when that happens. The next time the same builder will be used, chances are very high that the build will succeed without a clobber.
Attachment #669072 - Flags: review?(ted.mielczarek)
Assignee: nobody → mh+mozilla
Essentially, I think the problem can be expressed by this small test makefile:
libs:: a b c

a:
        touch a
        touch b

c: b
        cp b c

In a typical case, a is gkmedias.dll, b is gkmedias.lib and c is $(DIST)/lib/gkmedias.lib.
More often than not, if you run rm a; make -f test.mk -j4 with the above, you will see:
  touch a
  touch b
and that's all. That's because by the time make tries to create c, depending on b, b hasn't changed from running the rule for a.

rules.mk does have an extra rule, though, which, in this simplified test case can be expressed as:
b: a

But it turns out it's not enough for make/pymake to be happy.

However, adding an empty rule seems to do the trick (originally, i had made it a $(TOUCH) $@, but it turns out a rule that does nothing at all works as well)
Attachment #669075 - Flags: review?(ted.mielczarek)
Attachment #669072 - Attachment is obsolete: true
Attachment #669072 - Flags: review?(ted.mielczarek)
Attachment #669075 - Attachment is obsolete: true
Attachment #669075 - Flags: review?(ted.mielczarek)
Comment on attachment 669076 [details] [diff] [review]
Trick make/pymake into considering newer import libraries when installing them in DIST/lib

Review of attachment 669076 [details] [diff] [review]:
-----------------------------------------------------------------

::: config/rules.mk
@@ +826,1 @@
>  endif

I think the more common say to express this is just:
$(IMPORT_LIBRARY): $(SHARED_LIBRARY) ;

but either way this deserves a comment expressing its purpose.
Attachment #669076 - Flags: review?(ted.mielczarek) → review+
https://hg.mozilla.org/mozilla-central/rev/b992224cdd55
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Target Milestone: --- → mozilla19
Comment on attachment 669076 [details] [diff] [review]
Trick make/pymake into considering newer import libraries when installing them in DIST/lib

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 787184
User impact if declined: Occasional build failures on Windows. See for example, bug 803684 comment 28.
Testing completed (on m-c, etc.): Landed on m-c a couple weeks ago.
Risk to taking this patch (and alternatives if risky): No risk. Note the landed patch was different, following comment 27.
String or UUID changes made by this patch: None
Attachment #669076 - Flags: approval-mozilla-aurora?
Comment on attachment 669076 [details] [diff] [review]
Trick make/pymake into considering newer import libraries when installing them in DIST/lib

[Triage Comment]
Low/no risk, let's prevent intermittent build failures.
Attachment #669076 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
I'll land this along with bug 803684 in a sec.
Blocks: clobber
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.