1381163 - [meta] Support code coverage builds on Windows

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Marco Castelluccio [:marco]]

If we could build Firefox on Windows with Clang, it would be easier to support a Windows code coverage build.

Comment 1

[Marco Castelluccio [:marco]]

The other option is to use MinGW, which is probably already supported (IIRC Tor is building Firefox with MinGW).

Comment 2

[(Away)]

> If we could build Firefox on Windows with Clang

In general this is possible already today with clang-cl. Is there something specific that isn't working?

Comment 3

[Marco Castelluccio [:marco]]

(In reply to David Major [:dmajor] from comment #2)
> > If we could build Firefox on Windows with Clang
> 
> In general this is possible already today with clang-cl. Is there something
> specific that isn't working?

I haven't actually tried, just assumed it wasn't supported yet. Glad to hear it is.

Comment 4

[Marco Castelluccio [:marco]]

This is the mozconfig I used to build a linux code coverage build with Clang:
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm

> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --disable-elf-hack
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --with-ccache
> ac_add_options --enable-coverage
>
> export CC=clang-4.0
> export CXX=clang++-4.0
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="--coverage -L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/"

Comment 5

[Marco Castelluccio [:marco]]

I've tested with the following mozconfig and clang 5.0.0 (with a patch to make clang-cl accept the "coverage" option):
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm
> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --enable-optimize
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --enable-coverage
> export CC="clang-cl.exe -v"
> export CXX="clang-cl.exe -v"
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="C:\FD\llvm-objdir\lib\clang\5.0.0\lib\windows\clang_rt.profile-i386.lib"

Clang crashes:
> 0:27.32 Wrote crash dump file "C:\Users\marco\AppData\Local\Temp\clang-cl.exe-76ccdc.dmp"
> 0:27.32 #0 0x0199dbc1 llvm::Metadata::getMetadataID(void)const  c:\fd\llvm\include\llvm\ir\metadata.h:98:0
> 0:27.32 #1 0x01cf64e3 llvm::MDNode::classof(class llvm::Metadata const *) c:\fd\llvm\include\llvm\ir\metadata.h:1077:0
> 0:27.32 #2 0x01cf673c llvm::isa_impl<class llvm::MDNode,class llvm::Metadata,void>::doit(class llvm::Metadata const &) c:\fd\llvm\include\llvm\support\casting.h:59:0
> 0:27.32 #3 0x024e740a llvm::isa_impl_cl<class llvm::MDNode,class llvm::Metadata const *>::doit(class llvm::Metadata const *) c:\fd\llvm\include\llvm\support\casting.h:107:0
> 0:27.32 #4 0x024e89de llvm::isa_impl_wrap<class llvm::MDNode,class llvm::Metadata const *,class llvm::Metadata const *>::doit(class llvm::Metadata const * const &) c:\fd\llvm\include\llvm\support\casting.h:133:0
> 0:27.32 #5 0x024e8a13 llvm::isa_impl_wrap<class llvm::MDNode,class llvm::Metadata * const,class llvm::Metadata const *>::doit(class llvm::Metadata * const &) c:\fd\llvm\include\llvm\support\casting.h:123:0
> 0:27.32 #6 0x024da3fc llvm::isa<class llvm::MDNode,class llvm::Metadata *>(class llvm::Metadata * const &) c:\fd\llvm\include\llvm\support\casting.h:143:0
> 0:27.32 #7 0x024d8fa4 llvm::dyn_cast<class llvm::MDNode,class llvm::Metadata>(class llvm::Metadata *) c:\fd\llvm\include\llvm\support\casting.h:334:0
> 0:27.32 #8 0x024f23e0 llvm::ReplaceableMetadataImpl::getOrCreate(class llvm::Metadata &) c:\fd\llvm\lib\ir\metadata.cpp:315:0
> 0:27.32 #9 0x024f1591 llvm::MetadataTracking::track(void *,class llvm::Metadata &,class llvm::PointerUnion<class llvm::MetadataAsValue *,class llvm::Metadata *>) c:\fd\llvm\lib\ir\metadata.cpp:158:0
> 0:27.32 #10 0x01758174 llvm::MetadataTracking::track(class llvm::Metadata * &) c:\fd\llvm\include\llvm\ir\metadata.h:220:0
> 0:27.32 #11 0x017581bf llvm::TrackingMDRef::track(void) c:\fd\llvm\include\llvm\ir\trackingmdref.h:86:0
> 0:27.32 #12 0x01745ee0 llvm::TrackingMDRef::TrackingMDRef(class llvm::TrackingMDRef const &) c:\fd\llvm\include\llvm\ir\trackingmdref.h:34:0
> 0:27.32 #13 0x0174518a llvm::TypedTrackingMDRef<class llvm::MDNode>::TypedTrackingMDRef<class llvm::MDNode>(class llvm::TypedTrackingMDRef<class llvm::MDNode> const &) c:\fd\llvm\include\llvm\ir\trackingmdref.h:115:0
> 0:27.32 #14 0x0174577a llvm::DebugLoc::DebugLoc(class llvm::DebugLoc const &) c:\fd\llvm\include\llvm\codegen\selectiondagnodes.h:1028:0
> 0:27.32 #15 0x017e0631 llvm::IRBuilderBase::SetInsertPoint(class llvm::Instruction *) c:\fd\llvm\include\llvm\ir\irbuilder.h:139:0
> 0:27.32 #16 0x017d95b7 llvm::IRBuilder<class llvm::ConstantFolder,class llvm::IRBuilderDefaultInserter>::IRBuilder<class llvm::ConstantFolder,class llvm::IRBuilderDefaultInserter>(class llvm::Instruction *,class llvm::MDNode *,class llvm::ArrayRef<class llvm::OperandBundleDefT<class llvm::Value *> >) c:\fd\llvm\include\llvm\ir\irbuilder.h:694:0
> 0:27.32 #17 0x028c9350 `anonymous namespace'::GCOVProfiler::emitProfileArcs c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:635:0
> 0:27.32 #18 0x028c86fa `anonymous namespace'::GCOVProfiler::runOnModule c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:468:0
> 0:27.32 #19 0x028d9ffd `anonymous namespace'::GCOVProfilerLegacyPass::runOnModule c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:147:0
> 0:27.32 #20 0x0243d95e `anonymous namespace'::MPPassManager::runOnModule c:\fd\llvm\lib\ir\legacypassmanager.cpp:1591:0
> 0:27.32 #21 0x0243e03a llvm::legacy::PassManagerImpl::run(class llvm::Module &) c:\fd\llvm\lib\ir\legacypassmanager.cpp:1694:0
> 0:27.32 #22 0x024386fd llvm::legacy::PassManager::run(class llvm::Module &) c:\fd\llvm\lib\ir\legacypassmanager.cpp:1726:0
> 0:27.32 #23 0x033374ca `anonymous namespace'::EmitAssemblyHelper::EmitAssembly c:\fd\llvm\tools\clang\lib\codegen\backendutil.cpp:786:0
> 0:29.27 #24 0x033354e1 clang::EmitBackendOutput(class clang::DiagnosticsEngine &,class clang::HeaderSearchOptions const &,class clang::CodeGenOptions const &,class clang::TargetOptions const &,class clang::LangOptions const &,class llvm::DataLayout const &,class llvm::Module *,enum clang::BackendAction,class std::unique_ptr<class llvm::raw_pwrite_stream,struct std::default_delete<class llvm::raw_pwrite_stream> >) c:\fd\llvm\tools\clang\lib\codegen\backendutil.cpp:1138:0
> 0:29.27 #25 0x06dc85cb clang::BackendConsumer::HandleTranslationUnit(class clang::ASTContext &) c:\fd\llvm\tools\clang\lib\codegen\codegenaction.cpp:261:0
> 0:29.27 #26 0x04d20e26 clang::ParseAST(class clang::Sema &,bool,bool) c:\fd\llvm\tools\clang\lib\parse\parseast.cpp:159:0
> 0:29.27 #27 0x03977d41 clang::ASTFrontendAction::ExecuteAction(void) c:\fd\llvm\tools\clang\lib\frontend\frontendaction.cpp:1003:0
> 0:29.27 #28 0x06dbfc5a clang::CodeGenAction::ExecuteAction(void) c:\fd\llvm\tools\clang\lib\codegen\codegenaction.cpp:993:0
> 0:29.27 #29 0x039778d4 clang::FrontendAction::Execute(void) c:\fd\llvm\tools\clang\lib\frontend\frontendaction.cpp:902:0
> 0:29.27 #30 0x03903fa4 clang::CompilerInstance::ExecuteAction(class clang::FrontendAction &) c:\fd\llvm\tools\clang\lib\frontend\compilerinstance.cpp:981:0
> 0:29.27 #31 0x03acd110 clang::ExecuteCompilerInvocation(class clang::CompilerInstance *) c:\fd\llvm\tools\clang\lib\frontendtool\executecompilerinvocation.cpp:251:0
> 0:29.27 #32 0x016e2eba cc1_main(class llvm::ArrayRef<char const *>,char const *,void *) c:\fd\llvm\tools\clang\tools\driver\cc1_main.cpp:221:0
> 0:29.27 #33 0x016ccca8 ExecuteCC1Tool c:\fd\llvm\tools\clang\tools\driver\driver.cpp:306:0
> 0:29.27 #34 0x016cd31d main c:\fd\llvm\tools\clang\tools\driver\driver.cpp:387:0
> 0:29.27 #35 0x06ac67de invoke_main f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:64:0
> 0:29.27 #36 0x06ac6660 _scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253:0
> 0:29.27 #37 0x06ac64fd _scrt_common_main f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:296:0
> 0:29.27 #38 0x06ac67f8 mainCRTStartup f:\dd\vctools\crt\vcstartup\src\startup\exe_main.cpp:17:0
> 0:29.27 #39 0x769e62c4 (C:\WINDOWS\System32\KERNEL32.DLL+0x162c4)
> 0:29.27 #40 0x77750f69 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x60f69)
> 0:29.27 #41 0x77750f34 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x60f34)
> 0:29.27 clang-cl.exe: error: clang frontend command failed due to signal (use -v to see invocation)
> 0:29.27 clang version 5.0.0
> 0:29.27 Target: i686-pc-windows-msvc
> 0:29.27 Thread model: posix
> 0:29.27 InstalledDir: c:\FD\llvm-objdir\bin
> 0:29.27 clang-cl.exe: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
> 0:29.27 clang-cl.exe: note: diagnostic msg:
> 0:29.27 ********************
> 0:29.27 
> 0:29.27 PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
> 0:29.27 Preprocessed source(s) and associated run script(s) are located at:
> 0:29.27 clang-cl.exe: note: diagnostic msg: C:\Users\marco\AppData\Local\Temp\w95thred-a3bd27.c
> 0:29.27 clang-cl.exe: note: diagnostic msg: C:\Users\marco\AppData\Local\Temp\w95thred-a3bd27.sh
> 0:29.27 clang-cl.exe: note: diagnostic msg:
> 0:29.27 
> 0:29.27 ********************
> 0:29.28 c:/FD/mozilla-central/config/rules.mk:795: recipe for target 'w95thred.obj' failed
> 0:29.28 mozmake.EXE[5]: *** [w95thred.obj] Error 1
> 0:29.29 c:/FD/mozilla-central/config/recurse.mk:73: recipe for target 'config/external/nspr/pr/target' failed
> 0:29.29 mozmake.EXE[4]: *** [config/external/nspr/pr/target] Error 2
> 0:29.29 mozmake.EXE[4]: *** Waiting for unfinished jobs....

Comment 6

[Sylvestre Ledru [:Sylvestre]]

You should report the bug upstream:
https://llvm.org/bugs/
I will ping a few people there.

Comment 7

[Marco Castelluccio [:marco]]

Attachment: crash.zip

Comment 8

[Marco Castelluccio [:marco]]

It looks like the problem comes from the try except, I've found a reduced test case and attached it on the llvm bug.

Comment 9

[Marco Castelluccio [:marco]]

https://reviews.llvm.org/D38221 - This is the patch to make the "-coverage" option available when using clang-cl.

Comment 10

[Marco Castelluccio [:marco]]

I've submitted a tentative patch to fix the first bug (https://reviews.llvm.org/D38223).
Once that bug is fixed, there's another one (https://bugs.llvm.org/show_bug.cgi?id=34714). I've submitted another tentative patch to fix that too (https://reviews.llvm.org/D38224).

Now, it looks like the build is failing when building something related to Rust (target "force-cargo-library-build").

Comment 11

[Marco Castelluccio [:marco]]

Attachment: Excerpt from the build log

Comment 12

[(Away)]

Looks like a Windows vs Unix path confusion? I vaguely recall having to do a bunch of trial and error on how to format my --with-libclang-path some time ago.

Comment 13

[Marco Castelluccio [:marco]]

I think the issue is that the Rust binary that is trying to load the library is 64-bit, while the library itself is 32-bit. I guess I have to select the 32-bit Rust toolchain to build Firefox.

Comment 14

[Marco Castelluccio [:marco]]

The build is now successful, but Firefox crashes at startup (the assertion at [1] fails).

[1]: https://dxr.mozilla.org/mozilla-central/source/js/src/jit/RegisterSets.h#801

Comment 15

[Marco Castelluccio [:marco]]

I've found a small test case that crashes too: https://bugs.llvm.org/show_bug.cgi?id=34833.
Not sure it's the same underlying cause, but it could be.

Comment 16

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #15)
> I've found a small test case that crashes too:
> https://bugs.llvm.org/show_bug.cgi?id=34833.
> Not sure it's the same underlying cause, but it could be.

Yes, I can run Firefox now with the patch from https://reviews.llvm.org/D38223 (fixing that bug) applied.

There are still at least a couple of issues though:
1) Gcov data is invalid when templates are used (https://bugs.llvm.org/show_bug.cgi?id=34922);
2) The flock implementation in compiler-rt is a no-op on Windows, which means multiple processes can write to the same gcda files at the same time, which corrupts the gcda files (https://bugs.llvm.org/show_bug.cgi?id=34923).

Comment 17

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #16)
> (In reply to Marco Castelluccio [:marco] from comment #15)
> > I've found a small test case that crashes too:
> > https://bugs.llvm.org/show_bug.cgi?id=34833.
> > Not sure it's the same underlying cause, but it could be.
> 
> Yes, I can run Firefox now with the patch from
> https://reviews.llvm.org/D38223 (fixing that bug) applied.
> 
> There are still at least a couple of issues though:
> 1) Gcov data is invalid when templates are used
> (https://bugs.llvm.org/show_bug.cgi?id=34922);
> 2) The flock implementation in compiler-rt is a no-op on Windows, which
> means multiple processes can write to the same gcda files at the same time,
> which corrupts the gcda files (https://bugs.llvm.org/show_bug.cgi?id=34923).

With these two issues fixed, all the gcda files are correctly generated and I can also parse them.

I'm now trying to get this working on automation. There's an issue while packaging tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cbc107a6a2cc46cec66ccbd749c47779732acbc8&selectedJob=140642521
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e0ba9f7be7a43e19ead8475dcdeff4765996183&selectedJob=140527378 (without packaging tests the build is green, but obviously the tests can't run)

Comment 18

[Marco Castelluccio [:marco]]

With bug 1413015 fixed, packaging tests is working, but there's a mozinstall error on test machines: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d42fe1917bf50913621793a96762fb7a98ff361a.

Comment 19

[Intermittent Failures Robot]

2 failures in 889 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows10-64-ccov: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1381163&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 20

[Marco Castelluccio [:marco]]

We have been running Windows coverage builds for a while now, so I think we can call this bug fixed.