[meta] Support code coverage builds on Windows

RESOLVED FIXED

Status

RESOLVED FIXED
a year ago
3 months ago

People

(Reporter: marco, Assigned: marco)

Tracking

(Depends on: 5 bugs, Blocks: 2 bugs, {meta})

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(2 attachments)

(Assignee)

Description

a year ago
If we could build Firefox on Windows with Clang, it would be easier to support a Windows code coverage build.
(Assignee)

Comment 1

a year ago
The other option is to use MinGW, which is probably already supported (IIRC Tor is building Firefox with MinGW).
> If we could build Firefox on Windows with Clang

In general this is possible already today with clang-cl. Is there something specific that isn't working?
(Assignee)

Comment 3

a year ago
(In reply to David Major [:dmajor] from comment #2)
> > If we could build Firefox on Windows with Clang
> 
> In general this is possible already today with clang-cl. Is there something
> specific that isn't working?

I haven't actually tried, just assumed it wasn't supported yet. Glad to hear it is.
(Assignee)

Comment 4

a year ago
This is the mozconfig I used to build a linux code coverage build with Clang:
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm

> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --disable-elf-hack
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --with-ccache
> ac_add_options --enable-coverage
>
> export CC=clang-4.0
> export CXX=clang++-4.0
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="--coverage -L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/"
(Assignee)

Comment 5

a year ago
I've tested with the following mozconfig and clang 5.0.0 (with a patch to make clang-cl accept the "coverage" option):
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm
> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --enable-optimize
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --enable-coverage
> export CC="clang-cl.exe -v"
> export CXX="clang-cl.exe -v"
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="C:\FD\llvm-objdir\lib\clang\5.0.0\lib\windows\clang_rt.profile-i386.lib"

Clang crashes:
> 0:27.32 Wrote crash dump file "C:\Users\marco\AppData\Local\Temp\clang-cl.exe-76ccdc.dmp"
> 0:27.32 #0 0x0199dbc1 llvm::Metadata::getMetadataID(void)const  c:\fd\llvm\include\llvm\ir\metadata.h:98:0
> 0:27.32 #1 0x01cf64e3 llvm::MDNode::classof(class llvm::Metadata const *) c:\fd\llvm\include\llvm\ir\metadata.h:1077:0
> 0:27.32 #2 0x01cf673c llvm::isa_impl<class llvm::MDNode,class llvm::Metadata,void>::doit(class llvm::Metadata const &) c:\fd\llvm\include\llvm\support\casting.h:59:0
> 0:27.32 #3 0x024e740a llvm::isa_impl_cl<class llvm::MDNode,class llvm::Metadata const *>::doit(class llvm::Metadata const *) c:\fd\llvm\include\llvm\support\casting.h:107:0
> 0:27.32 #4 0x024e89de llvm::isa_impl_wrap<class llvm::MDNode,class llvm::Metadata const *,class llvm::Metadata const *>::doit(class llvm::Metadata const * const &) c:\fd\llvm\include\llvm\support\casting.h:133:0
> 0:27.32 #5 0x024e8a13 llvm::isa_impl_wrap<class llvm::MDNode,class llvm::Metadata * const,class llvm::Metadata const *>::doit(class llvm::Metadata * const &) c:\fd\llvm\include\llvm\support\casting.h:123:0
> 0:27.32 #6 0x024da3fc llvm::isa<class llvm::MDNode,class llvm::Metadata *>(class llvm::Metadata * const &) c:\fd\llvm\include\llvm\support\casting.h:143:0
> 0:27.32 #7 0x024d8fa4 llvm::dyn_cast<class llvm::MDNode,class llvm::Metadata>(class llvm::Metadata *) c:\fd\llvm\include\llvm\support\casting.h:334:0
> 0:27.32 #8 0x024f23e0 llvm::ReplaceableMetadataImpl::getOrCreate(class llvm::Metadata &) c:\fd\llvm\lib\ir\metadata.cpp:315:0
> 0:27.32 #9 0x024f1591 llvm::MetadataTracking::track(void *,class llvm::Metadata &,class llvm::PointerUnion<class llvm::MetadataAsValue *,class llvm::Metadata *>) c:\fd\llvm\lib\ir\metadata.cpp:158:0
> 0:27.32 #10 0x01758174 llvm::MetadataTracking::track(class llvm::Metadata * &) c:\fd\llvm\include\llvm\ir\metadata.h:220:0
> 0:27.32 #11 0x017581bf llvm::TrackingMDRef::track(void) c:\fd\llvm\include\llvm\ir\trackingmdref.h:86:0
> 0:27.32 #12 0x01745ee0 llvm::TrackingMDRef::TrackingMDRef(class llvm::TrackingMDRef const &) c:\fd\llvm\include\llvm\ir\trackingmdref.h:34:0
> 0:27.32 #13 0x0174518a llvm::TypedTrackingMDRef<class llvm::MDNode>::TypedTrackingMDRef<class llvm::MDNode>(class llvm::TypedTrackingMDRef<class llvm::MDNode> const &) c:\fd\llvm\include\llvm\ir\trackingmdref.h:115:0
> 0:27.32 #14 0x0174577a llvm::DebugLoc::DebugLoc(class llvm::DebugLoc const &) c:\fd\llvm\include\llvm\codegen\selectiondagnodes.h:1028:0
> 0:27.32 #15 0x017e0631 llvm::IRBuilderBase::SetInsertPoint(class llvm::Instruction *) c:\fd\llvm\include\llvm\ir\irbuilder.h:139:0
> 0:27.32 #16 0x017d95b7 llvm::IRBuilder<class llvm::ConstantFolder,class llvm::IRBuilderDefaultInserter>::IRBuilder<class llvm::ConstantFolder,class llvm::IRBuilderDefaultInserter>(class llvm::Instruction *,class llvm::MDNode *,class llvm::ArrayRef<class llvm::OperandBundleDefT<class llvm::Value *> >) c:\fd\llvm\include\llvm\ir\irbuilder.h:694:0
> 0:27.32 #17 0x028c9350 `anonymous namespace'::GCOVProfiler::emitProfileArcs c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:635:0
> 0:27.32 #18 0x028c86fa `anonymous namespace'::GCOVProfiler::runOnModule c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:468:0
> 0:27.32 #19 0x028d9ffd `anonymous namespace'::GCOVProfilerLegacyPass::runOnModule c:\fd\llvm\lib\transforms\instrumentation\gcovprofiling.cpp:147:0
> 0:27.32 #20 0x0243d95e `anonymous namespace'::MPPassManager::runOnModule c:\fd\llvm\lib\ir\legacypassmanager.cpp:1591:0
> 0:27.32 #21 0x0243e03a llvm::legacy::PassManagerImpl::run(class llvm::Module &) c:\fd\llvm\lib\ir\legacypassmanager.cpp:1694:0
> 0:27.32 #22 0x024386fd llvm::legacy::PassManager::run(class llvm::Module &) c:\fd\llvm\lib\ir\legacypassmanager.cpp:1726:0
> 0:27.32 #23 0x033374ca `anonymous namespace'::EmitAssemblyHelper::EmitAssembly c:\fd\llvm\tools\clang\lib\codegen\backendutil.cpp:786:0
> 0:29.27 #24 0x033354e1 clang::EmitBackendOutput(class clang::DiagnosticsEngine &,class clang::HeaderSearchOptions const &,class clang::CodeGenOptions const &,class clang::TargetOptions const &,class clang::LangOptions const &,class llvm::DataLayout const &,class llvm::Module *,enum clang::BackendAction,class std::unique_ptr<class llvm::raw_pwrite_stream,struct std::default_delete<class llvm::raw_pwrite_stream> >) c:\fd\llvm\tools\clang\lib\codegen\backendutil.cpp:1138:0
> 0:29.27 #25 0x06dc85cb clang::BackendConsumer::HandleTranslationUnit(class clang::ASTContext &) c:\fd\llvm\tools\clang\lib\codegen\codegenaction.cpp:261:0
> 0:29.27 #26 0x04d20e26 clang::ParseAST(class clang::Sema &,bool,bool) c:\fd\llvm\tools\clang\lib\parse\parseast.cpp:159:0
> 0:29.27 #27 0x03977d41 clang::ASTFrontendAction::ExecuteAction(void) c:\fd\llvm\tools\clang\lib\frontend\frontendaction.cpp:1003:0
> 0:29.27 #28 0x06dbfc5a clang::CodeGenAction::ExecuteAction(void) c:\fd\llvm\tools\clang\lib\codegen\codegenaction.cpp:993:0
> 0:29.27 #29 0x039778d4 clang::FrontendAction::Execute(void) c:\fd\llvm\tools\clang\lib\frontend\frontendaction.cpp:902:0
> 0:29.27 #30 0x03903fa4 clang::CompilerInstance::ExecuteAction(class clang::FrontendAction &) c:\fd\llvm\tools\clang\lib\frontend\compilerinstance.cpp:981:0
> 0:29.27 #31 0x03acd110 clang::ExecuteCompilerInvocation(class clang::CompilerInstance *) c:\fd\llvm\tools\clang\lib\frontendtool\executecompilerinvocation.cpp:251:0
> 0:29.27 #32 0x016e2eba cc1_main(class llvm::ArrayRef<char const *>,char const *,void *) c:\fd\llvm\tools\clang\tools\driver\cc1_main.cpp:221:0
> 0:29.27 #33 0x016ccca8 ExecuteCC1Tool c:\fd\llvm\tools\clang\tools\driver\driver.cpp:306:0
> 0:29.27 #34 0x016cd31d main c:\fd\llvm\tools\clang\tools\driver\driver.cpp:387:0
> 0:29.27 #35 0x06ac67de invoke_main f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:64:0
> 0:29.27 #36 0x06ac6660 _scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253:0
> 0:29.27 #37 0x06ac64fd _scrt_common_main f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:296:0
> 0:29.27 #38 0x06ac67f8 mainCRTStartup f:\dd\vctools\crt\vcstartup\src\startup\exe_main.cpp:17:0
> 0:29.27 #39 0x769e62c4 (C:\WINDOWS\System32\KERNEL32.DLL+0x162c4)
> 0:29.27 #40 0x77750f69 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x60f69)
> 0:29.27 #41 0x77750f34 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x60f34)
> 0:29.27 clang-cl.exe: error: clang frontend command failed due to signal (use -v to see invocation)
> 0:29.27 clang version 5.0.0
> 0:29.27 Target: i686-pc-windows-msvc
> 0:29.27 Thread model: posix
> 0:29.27 InstalledDir: c:\FD\llvm-objdir\bin
> 0:29.27 clang-cl.exe: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
> 0:29.27 clang-cl.exe: note: diagnostic msg:
> 0:29.27 ********************
> 0:29.27 
> 0:29.27 PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
> 0:29.27 Preprocessed source(s) and associated run script(s) are located at:
> 0:29.27 clang-cl.exe: note: diagnostic msg: C:\Users\marco\AppData\Local\Temp\w95thred-a3bd27.c
> 0:29.27 clang-cl.exe: note: diagnostic msg: C:\Users\marco\AppData\Local\Temp\w95thred-a3bd27.sh
> 0:29.27 clang-cl.exe: note: diagnostic msg:
> 0:29.27 
> 0:29.27 ********************
> 0:29.28 c:/FD/mozilla-central/config/rules.mk:795: recipe for target 'w95thred.obj' failed
> 0:29.28 mozmake.EXE[5]: *** [w95thred.obj] Error 1
> 0:29.29 c:/FD/mozilla-central/config/recurse.mk:73: recipe for target 'config/external/nspr/pr/target' failed
> 0:29.29 mozmake.EXE[4]: *** [config/external/nspr/pr/target] Error 2
> 0:29.29 mozmake.EXE[4]: *** Waiting for unfinished jobs....
You should report the bug upstream:
https://llvm.org/bugs/
I will ping a few people there.
(Assignee)

Comment 7

a year ago
Created attachment 8909423 [details]
crash.zip
(Assignee)

Comment 8

a year ago
It looks like the problem comes from the try except, I've found a reduced test case and attached it on the llvm bug.
(Assignee)

Comment 9

a year ago
https://reviews.llvm.org/D38221 - This is the patch to make the "-coverage" option available when using clang-cl.
(Assignee)

Comment 10

a year ago
I've submitted a tentative patch to fix the first bug (https://reviews.llvm.org/D38223).
Once that bug is fixed, there's another one (https://bugs.llvm.org/show_bug.cgi?id=34714). I've submitted another tentative patch to fix that too (https://reviews.llvm.org/D38224).

Now, it looks like the build is failing when building something related to Rust (target "force-cargo-library-build").
(Assignee)

Comment 11

a year ago
Created attachment 8911590 [details]
Excerpt from the build log
Looks like a Windows vs Unix path confusion? I vaguely recall having to do a bunch of trial and error on how to format my --with-libclang-path some time ago.
(Assignee)

Comment 13

a year ago
I think the issue is that the Rust binary that is trying to load the library is 64-bit, while the library itself is 32-bit. I guess I have to select the 32-bit Rust toolchain to build Firefox.
(Assignee)

Comment 14

a year ago
The build is now successful, but Firefox crashes at startup (the assertion at [1] fails).

[1]: https://dxr.mozilla.org/mozilla-central/source/js/src/jit/RegisterSets.h#801
(Assignee)

Updated

a year ago
Depends on: 1403143
(Assignee)

Comment 15

a year ago
I've found a small test case that crashes too: https://bugs.llvm.org/show_bug.cgi?id=34833.
Not sure it's the same underlying cause, but it could be.
(Assignee)

Comment 16

a year ago
(In reply to Marco Castelluccio [:marco] from comment #15)
> I've found a small test case that crashes too:
> https://bugs.llvm.org/show_bug.cgi?id=34833.
> Not sure it's the same underlying cause, but it could be.

Yes, I can run Firefox now with the patch from https://reviews.llvm.org/D38223 (fixing that bug) applied.

There are still at least a couple of issues though:
1) Gcov data is invalid when templates are used (https://bugs.llvm.org/show_bug.cgi?id=34922);
2) The flock implementation in compiler-rt is a no-op on Windows, which means multiple processes can write to the same gcda files at the same time, which corrupts the gcda files (https://bugs.llvm.org/show_bug.cgi?id=34923).
(Assignee)

Comment 17

a year ago
(In reply to Marco Castelluccio [:marco] from comment #16)
> (In reply to Marco Castelluccio [:marco] from comment #15)
> > I've found a small test case that crashes too:
> > https://bugs.llvm.org/show_bug.cgi?id=34833.
> > Not sure it's the same underlying cause, but it could be.
> 
> Yes, I can run Firefox now with the patch from
> https://reviews.llvm.org/D38223 (fixing that bug) applied.
> 
> There are still at least a couple of issues though:
> 1) Gcov data is invalid when templates are used
> (https://bugs.llvm.org/show_bug.cgi?id=34922);
> 2) The flock implementation in compiler-rt is a no-op on Windows, which
> means multiple processes can write to the same gcda files at the same time,
> which corrupts the gcda files (https://bugs.llvm.org/show_bug.cgi?id=34923).

With these two issues fixed, all the gcda files are correctly generated and I can also parse them.

I'm now trying to get this working on automation. There's an issue while packaging tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cbc107a6a2cc46cec66ccbd749c47779732acbc8&selectedJob=140642521
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e0ba9f7be7a43e19ead8475dcdeff4765996183&selectedJob=140527378 (without packaging tests the build is green, but obviously the tests can't run)
(Assignee)

Updated

a year ago
Depends on: 1413015
(Assignee)

Updated

a year ago
Depends on: 1414962
(Assignee)

Comment 18

a year ago
With bug 1413015 fixed, packaging tests is working, but there's a mozinstall error on test machines: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d42fe1917bf50913621793a96762fb7a98ff361a.
(Assignee)

Updated

a year ago
Depends on: 1415163
(Assignee)

Updated

a year ago
Depends on: 1415568
(Assignee)

Updated

a year ago
Depends on: 1415699
(Assignee)

Updated

a year ago
Depends on: 1416008
(Assignee)

Updated

a year ago
Depends on: 1416199
(Assignee)

Updated

11 months ago
Depends on: 1416235
(Assignee)

Updated

11 months ago
Depends on: 1416239
(Assignee)

Updated

11 months ago
Depends on: 1416743
(Assignee)

Updated

11 months ago
Depends on: 1417289
(Assignee)

Updated

11 months ago
Depends on: 1417366
(Assignee)

Updated

11 months ago
Depends on: 1417436
(Assignee)

Updated

11 months ago
Depends on: 1417447
(Assignee)

Updated

11 months ago
Depends on: 1417496
(Assignee)

Updated

11 months ago
Depends on: 1417497
(Assignee)

Updated

11 months ago
Keywords: meta
Summary: Support code coverage builds on Windows → [meta] Support code coverage builds on Windows
Version: Version 3 → unspecified
(Assignee)

Updated

11 months ago
Depends on: 1418252
(Assignee)

Updated

11 months ago
Depends on: 1418253
(Assignee)

Updated

11 months ago
Depends on: 1418255
(Assignee)

Updated

11 months ago
Depends on: 1418635
(Assignee)

Updated

11 months ago
Depends on: 1393740
(Assignee)

Updated

11 months ago
Depends on: 1419475
(Assignee)

Updated

11 months ago
Depends on: 1409741
(Assignee)

Updated

11 months ago
Depends on: 1418283
(Assignee)

Updated

11 months ago
Depends on: 1419485
(Assignee)

Updated

11 months ago
Depends on: 1419486
(Assignee)

Updated

11 months ago
Depends on: 1419490
(Assignee)

Updated

11 months ago
Depends on: 1419491
(Assignee)

Updated

11 months ago
Depends on: 1419493
(Assignee)

Updated

11 months ago
Depends on: 1419494
(Assignee)

Updated

11 months ago
No longer depends on: 1419491
(Assignee)

Updated

11 months ago
Depends on: 1420947
(Assignee)

Updated

11 months ago
Depends on: 1421692
(Assignee)

Updated

11 months ago
Depends on: 1421715
(Assignee)

Updated

11 months ago
No longer depends on: 1393740
(Assignee)

Updated

11 months ago
Depends on: 1421724
(Assignee)

Updated

11 months ago
Depends on: 1401652
(Assignee)

Updated

11 months ago
No longer depends on: 1419493
(Assignee)

Updated

11 months ago
No longer depends on: 1409741
(Assignee)

Updated

11 months ago
Depends on: 1423218
Depends on: 1423667
2 failures in 889 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows10-64-ccov: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1381163&startday=2017-12-04&endday=2017-12-10&tree=all
(Assignee)

Updated

9 months ago
Blocks: 1429455
(Assignee)

Updated

9 months ago
Depends on: 1434782
(Assignee)

Comment 20

8 months ago
We have been running Windows coverage builds for a while now, so I think we can call this bug fixed.
(Assignee)

Updated

8 months ago
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
(Assignee)

Updated

8 months ago
Assignee: nobody → mcastelluccio
(Assignee)

Updated

7 months ago
Depends on: 1447727
(Assignee)

Updated

5 months ago
Depends on: 1465738
(Assignee)

Updated

4 months ago
Depends on: 1473531
(Assignee)

Updated

4 months ago
Depends on: 1473610
(Assignee)

Updated

3 months ago
Depends on: 1474254
(Assignee)

Updated

3 months ago
No longer depends on: 1473531
You need to log in before you can comment on or make changes to this bug.