1621342 - Code coverage generation is broken on most folders

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect, P1)

Description

[Bastien Abadie [:bastien]]

As you can see on the code coverage frontend, the overall coverage has dropped from 62% to 18% over the past few days (since March 6th).

I've investigated this issue today, and it looks like the breaking change comes from mozilla-central's side on push 37188 (not the bot side, as the last release has been working on March 4th & 5th.)

Last good revision with coverage is f848103e932160dcac744a1c4b349e9a5b5f3477 on Fri, 06 Mar 2020 21:42:01

The next one is 8ab81c8e93adff2d4187aabf1ddcfcae8bfc4e16 on Fri, 06 Mar 2020 22:09:55 and has bad coverage data.

I compared the raw grcov data from a small subset (on gtest):

• f848103e
• 8ab81c8e

The diff shows a lot of change in objdir folders which may impact code coverage generation like in this place. covdir generation (our json format) on those files also show a drop on files (for example dom/base/nsINode.cpp went from 20% coverage to 0% for gtest only).

Bug 1527313 looks like the breaking change here as it moved objdir.

Comment 1

[Bastien Abadie [:bastien]]

Side note: I've disabled the low code coverage report from code review bot in order to avoid spamming developers (like here)

Comment 2

[Tom Prince [:tomprince]]

Yeah, I expect Bug 1527313 is a significant part of this. I also think that Bug 1620165 may have had an impact.

I guess https://searchfox.org/mozilla-central/rev/2fd8ffcf087bc59a8e5c962965bbb7bf230bcd28/testing/mozharness/mozharness/mozilla/testing/codecoverage.py#110-120 probably needs to be updated.

The linux path should be /builds/worker/checkouts/gecko; I'm not sure what windows should be, in light of windows-cross compiles. I'm also not sure if there are other places that need updating, or if there will need to be handling of generated files that are in the objdir, which is no longer underneath the source directory.

Comment 3

[Marco Castelluccio [:marco]]

Generated files should be good, thanks to https://searchfox.org/mozilla-central/rev/2fd8ffcf087bc59a8e5c962965bbb7bf230bcd28/python/mozbuild/mozbuild/codecoverage/packager.py#54.

Comment 4

[Bastien Abadie [:bastien]]

I've updated the linux path, and made a try push with 2 different coverage jobs: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=292803720&revision=bfaf3d779c05c2f80ac5fdf85d88a090fd8cff4b

No luck, the coverage data is still at 0 in the produced artifacts.

Tom, can you help us a bit more on this regression ? Who can i contact to get more informations on the Windows prefix ? What exactly did change in Bug 1527313 that could impact the code coverage ?

Comment 5

[Bastien Abadie [:bastien]]

I've made a try push with the prefix update + a zip dump of raw coverage files: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=293045112&revision=709bda0bfc1fcdc75f2cd06d2da88c98c4ae92e6

The raw coverage artifacts are named code-coverage-raw-gcov.zip

Comment 6

[Tom Prince [:tomprince]]

Downloading the raw coverage file, it looks like all the paths in there are missing the first component. (I see Hal.gcda instead of hal/Hal.gcda for example). I'm not sure why there is the difference in behavior between before and after Bug 1527313 regarding paths, and the number of components to strip. I vaguely wonder if prefix.count("/") was over-counting by one, but the paths were relative to the object directory, instead of the source directory, so the errors cancelled out.

If that is the case, the current path gives GCOV_PREFIX_STRIP a value of 5, and the current object directory has 4 components (/builds/worker/workspace/obj-build), which would explain stripping an extra directory off.

It looks like the windows ccov builds are not yet cross compiled. I'm not sure if there is a bug for them, but Bug 1620166 is the dependency tree for moving windows builds to be cross builds (which will switch the built paths to match linux).`

The current windows paths (before we switch to cross compiles) are:
source: z:/build/build/src
object: z:/build/workspace/obj-build

Comment 7

[Bastien Abadie [:bastien]]

Attachment: Bug 1621342 - Update linux prefix for code coverage generation, r=marco,tomprince

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d76906282445
Update linux prefix for code coverage generation, r=marco

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/d76906282445

Comment 10

[Marco Castelluccio [:marco]]

Still need to fix Windows.

Comment 11

[Bastien Abadie [:bastien]]

Attachment: Bug 1621342 - Update windows prefix for code coverage generation, r=marco,tomprince

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86149c5f9a5e
Update windows prefix for code coverage generation, r=marco

Comment 13

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/86149c5f9a5e

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Do we need to uplift these patches to Beta?

Comment 15

[Bastien Abadie [:bastien]]

Hey Ryan, uplifting is not needed here as that bug would only affect our own web platform showcasing code coverage.
As you can see https://coverage.moz.tools/ , we're back to normal now !

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:bastien, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.