Closed Bug 1335518 Opened 7 years ago Closed 5 years ago

[meta] tracking work to collect code coverage for Rust code

Categories

(Testing :: Code Coverage, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jmaher, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: meta)

      No description provided.
ideally here we will track all the bugs related to getting code coverage stood up for the Rust related code in Firefox.
as a note, initial work to get standard coverage in rustc:
https://github.com/rust-lang/rust/pull/38608

This will need to get into a version of rustc that we use in automation, but for coverage needs we could run with a custom version of rustc as needed.
(In reply to Marco Castelluccio [:marco] from comment #3)
> We could also test kcov
> (https://users.rust-lang.org/t/tutorial-how-to-collect-test-coverages-for-
> rust-project/650).

None of my initial attempts to run Firefox under kcov have proven effective -- kcov either crashes or hangs when I try it on linux.
Depends on: 1357096
Depends on: 1359941
Depends on: 1359947
Depends on: 1331022
Depends on: 1365295
Depends on: 1322803
Adding the following to the mozconfig should enable Rust code coverage, when bug 1322803 is fixed:
export LDFLAGS="--coverage -L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/ -lclang_rt.profile-x86_64"
export RUSTFLAGS="-Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads"

The brittle "-L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/ -lclang_rt.profile-x86_64" is only needed until https://github.com/rust-lang/rust/pull/42433 gets merged.
(In reply to Joel Maher ( :jmaher) from comment #2)
> as a note, initial work to get standard coverage in rustc:
> https://github.com/rust-lang/rust/pull/38608
> 
> This will need to get into a version of rustc that we use in automation, but
> for coverage needs we could run with a custom version of rustc as needed.

The rustc profiler fix has moved to a new PR:

https://github.com/rust-lang/rust/pull/42433
I've managed to get Rust code coverage working with the following mozconfig (and the patch from bug 1322803):
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm
> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --disable-elf-hack
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --with-ccache
> MOZ_CODE_COVERAGE=1
> export CC=clang-4.0
> export CXX=clang++-4.0
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="--coverage"
> export RUSTFLAGS="-Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads"
I have a patch to enable Rust coverage, the build is green (with a small hack to avoid a signing failure): https://treeherder.mozilla.org/#/jobs?repo=try&revision=48862e021d1b30845adb6df533e50b1d0472750c
The tests didn't run for some reason, I will push it to try again and hope they will run. I'm expecting some tests to fail because of timeouts, as the LLVM build looks slower than the GCC one, but we can just increase them.

Here are the logs of the build with the signing failure: https://treeherder.mozilla.org/logviewer.html#?job_id=131284360&repo=try&lineNumber=27813. Any idea what could be causing this? In theory I can skip it if code coverage is enabled, so I suppose it isn't a blocker.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e30d8c3c59a9737f1825e81e6901774f562ede00

Most tests are failing because of the same issue "TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed" (https://treeherder.mozilla.org/logviewer.html#?job_id=131854289&repo=try&lineNumber=2243) or an error with certutil (https://treeherder.mozilla.org/logviewer.html#?job_id=131854296&repo=try&lineNumber=1553, I guess they are actually the same error).
This could be related to the failure in signing with "shlibsign" that I mentioned in comment 8.

So, I'm going to need some help to figure out why certutil and shlibsign are failing and either fix the issue or ignore it if possible.
Flags: needinfo?(martin.thomson)
Flags: needinfo?(kaie)
Flags: needinfo?(franziskuskiefer)
Hm, is ccov messing with the library after it's build? I'd suggest disabling signing. You really don't need it in a coverage build.
Signing of NSS libraries in Firefox builds isn't done by the usual NSS routine [1] but while packaging [2]. Can you try what happens when you disable signing?

[1] https://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/security/moz.build#65
[2] https://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/toolkit/mozapps/installer/packager.py#306
Flags: needinfo?(franziskuskiefer)
(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #10)
> Hm, is ccov messing with the library after it's build? I'd suggest disabling
> signing. You really don't need it in a coverage build.
> Signing of NSS libraries in Firefox builds isn't done by the usual NSS
> routine [1] but while packaging [2]. Can you try what happens when you
> disable signing?
> 
> [1]
> https://searchfox.org/mozilla-central/rev/
> 1c13d5cf85f904afb8976c02a80daa252b893fca/security/moz.build#65
> [2]
> https://searchfox.org/mozilla-central/rev/
> 1c13d5cf85f904afb8976c02a80daa252b893fca/toolkit/mozapps/installer/packager.
> py#306

If I comment that code out (I did it in the build from comment 8), `shlibsign` isn't called anymore and the build succeeds.
The certutil failure during tests still stands though.
Depends on: 1401230
> The certutil failure during tests still stands though. 

I see, well there's not really a lot of information in there. The command looks fine. Can you add debug infos so that we can see what happens when certutil gets called (output/status code).
(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #12)
> > The certutil failure during tests still stands though. 
> 
> I see, well there's not really a lot of information in there. The command
> looks fine. Can you add debug infos so that we can see what happens when
> certutil gets called (output/status code).

The output is in the logs (from https://treeherder.mozilla.org/logviewer.html#?job_id=131284360&repo=try&lineNumber=27784). I've added some additional logging to shlibsign and pushed to try again.
Anyway, I will file another bug for this issue blocking this meta bug.
Flags: needinfo?(martin.thomson)
Flags: needinfo?(kaie)
Depends on: 1401742
Depends on: 1404433
Depends on: 1469804
Depends on: 1471147
Depends on: 1471339
No longer depends on: 1404433
Depends on: 1473067
Depends on: 1473069
No longer depends on: 1365295
Depends on: 1504250
The last dependency has been fixed!
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Depends on: 1540749
Depends on: 1518119
You need to log in before you can comment on or make changes to this bug.