1335518 - [meta] tracking work to collect code coverage for Rust code

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect)

Comment 1

[Joel Maher ( :jmaher ) (UTC -0800)]

ideally here we will track all the bugs related to getting code coverage stood up for the Rust related code in Firefox.

Comment 2

[Joel Maher ( :jmaher ) (UTC -0800)]

as a note, initial work to get standard coverage in rustc:
https://github.com/rust-lang/rust/pull/38608

This will need to get into a version of rustc that we use in automation, but for coverage needs we could run with a custom version of rustc as needed.

Comment 3

[Marco Castelluccio [:marco]]

We could also test kcov (https://users.rust-lang.org/t/tutorial-how-to-collect-test-coverages-for-rust-project/650).

Comment 4

[Chris Manchester (limited bugmail, email directly)]

(In reply to Marco Castelluccio [:marco] from comment #3)
> We could also test kcov
> (https://users.rust-lang.org/t/tutorial-how-to-collect-test-coverages-for-
> rust-project/650).

None of my initial attempts to run Firefox under kcov have proven effective -- kcov either crashes or hangs when I try it on linux.

Comment 5

[Marco Castelluccio [:marco]]

Adding the following to the mozconfig should enable Rust code coverage, when bug 1322803 is fixed:
export LDFLAGS="--coverage -L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/ -lclang_rt.profile-x86_64"
export RUSTFLAGS="-Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads"

The brittle "-L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/ -lclang_rt.profile-x86_64" is only needed until https://github.com/rust-lang/rust/pull/42433 gets merged.

Comment 6

[Chris Peterson [:cpeterson]]

(In reply to Joel Maher ( :jmaher) from comment #2)
> as a note, initial work to get standard coverage in rustc:
> https://github.com/rust-lang/rust/pull/38608
> 
> This will need to get into a version of rustc that we use in automation, but
> for coverage needs we could run with a custom version of rustc as needed.

The rustc profiler fix has moved to a new PR:

https://github.com/rust-lang/rust/pull/42433

Comment 7

[Marco Castelluccio [:marco]]

I've managed to get Rust code coverage working with the following mozconfig (and the patch from bug 1322803):
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-cov-llvm
> ac_add_options --disable-install-strip
> ac_add_options --disable-jemalloc
> ac_add_options --disable-crashreporter
> ac_add_options --disable-elf-hack
> ac_add_options --enable-debug
> ac_add_options --enable-debug-symbols
> ac_add_options --disable-sandbox
> ac_add_options --with-ccache
> MOZ_CODE_COVERAGE=1
> export CC=clang-4.0
> export CXX=clang++-4.0
> export CFLAGS="--coverage"
> export CXXFLAGS="--coverage"
> export LDFLAGS="--coverage"
> export RUSTFLAGS="-Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads"

Comment 8

[Marco Castelluccio [:marco]]

I have a patch to enable Rust coverage, the build is green (with a small hack to avoid a signing failure): https://treeherder.mozilla.org/#/jobs?repo=try&revision=48862e021d1b30845adb6df533e50b1d0472750c
The tests didn't run for some reason, I will push it to try again and hope they will run. I'm expecting some tests to fail because of timeouts, as the LLVM build looks slower than the GCC one, but we can just increase them.

Here are the logs of the build with the signing failure: https://treeherder.mozilla.org/logviewer.html#?job_id=131284360&repo=try&lineNumber=27813. Any idea what could be causing this? In theory I can skip it if code coverage is enabled, so I suppose it isn't a blocker.

Comment 9

[Marco Castelluccio [:marco]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e30d8c3c59a9737f1825e81e6901774f562ede00

Most tests are failing because of the same issue "TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed" (https://treeherder.mozilla.org/logviewer.html#?job_id=131854289&repo=try&lineNumber=2243) or an error with certutil (https://treeherder.mozilla.org/logviewer.html#?job_id=131854296&repo=try&lineNumber=1553, I guess they are actually the same error).
This could be related to the failure in signing with "shlibsign" that I mentioned in comment 8.

So, I'm going to need some help to figure out why certutil and shlibsign are failing and either fix the issue or ignore it if possible.

Comment 10

[Franziskus Kiefer [:franziskus]]

Hm, is ccov messing with the library after it's build? I'd suggest disabling signing. You really don't need it in a coverage build.
Signing of NSS libraries in Firefox builds isn't done by the usual NSS routine [1] but while packaging [2]. Can you try what happens when you disable signing?

[1] https://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/security/moz.build#65
[2] https://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/toolkit/mozapps/installer/packager.py#306

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #10)
> Hm, is ccov messing with the library after it's build? I'd suggest disabling
> signing. You really don't need it in a coverage build.
> Signing of NSS libraries in Firefox builds isn't done by the usual NSS
> routine [1] but while packaging [2]. Can you try what happens when you
> disable signing?
> 
> [1]
> https://searchfox.org/mozilla-central/rev/
> 1c13d5cf85f904afb8976c02a80daa252b893fca/security/moz.build#65
> [2]
> https://searchfox.org/mozilla-central/rev/
> 1c13d5cf85f904afb8976c02a80daa252b893fca/toolkit/mozapps/installer/packager.
> py#306

If I comment that code out (I did it in the build from comment 8), `shlibsign` isn't called anymore and the build succeeds.
The certutil failure during tests still stands though.

Comment 12

[Franziskus Kiefer [:franziskus]]

> The certutil failure during tests still stands though. 

I see, well there's not really a lot of information in there. The command looks fine. Can you add debug infos so that we can see what happens when certutil gets called (output/status code).

Comment 13

[Marco Castelluccio [:marco]]

(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #12)
> > The certutil failure during tests still stands though. 
> 
> I see, well there's not really a lot of information in there. The command
> looks fine. Can you add debug infos so that we can see what happens when
> certutil gets called (output/status code).

The output is in the logs (from https://treeherder.mozilla.org/logviewer.html#?job_id=131284360&repo=try&lineNumber=27784). I've added some additional logging to shlibsign and pushed to try again.
Anyway, I will file another bug for this issue blocking this meta bug.

Comment 15

[Marco Castelluccio [:marco]]

The last dependency has been fixed!