Closed Bug 1621342 Opened 6 months ago Closed 6 months ago

Code coverage generation is broken on most folders

Categories

(Testing :: Code Coverage, defect, P1)

defect

Tracking

(firefox-esr68 unaffected, firefox74 unaffected, firefox75 wontfix, firefox76 fixed)

RESOLVED FIXED
mozilla76
Tracking Status
firefox-esr68 --- unaffected
firefox74 --- unaffected
firefox75 --- wontfix
firefox76 --- fixed

People

(Reporter: bastien, Assigned: bastien)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files)

As you can see on the code coverage frontend, the overall coverage has dropped from 62% to 18% over the past few days (since March 6th).

I've investigated this issue today, and it looks like the breaking change comes from mozilla-central's side on push 37188 (not the bot side, as the last release has been working on March 4th & 5th.)

Last good revision with coverage is f848103e932160dcac744a1c4b349e9a5b5f3477 on Fri, 06 Mar 2020 21:42:01

The next one is 8ab81c8e93adff2d4187aabf1ddcfcae8bfc4e16 on Fri, 06 Mar 2020 22:09:55 and has bad coverage data.

I compared the raw grcov data from a small subset (on gtest):

The diff shows a lot of change in objdir folders which may impact code coverage generation like in this place. covdir generation (our json format) on those files also show a drop on files (for example dom/base/nsINode.cpp went from 20% coverage to 0% for gtest only).

Bug 1527313 looks like the breaking change here as it moved objdir.

Side note: I've disabled the low code coverage report from code review bot in order to avoid spamming developers (like here)

Yeah, I expect Bug 1527313 is a significant part of this. I also think that Bug 1620165 may have had an impact.

I guess https://searchfox.org/mozilla-central/rev/2fd8ffcf087bc59a8e5c962965bbb7bf230bcd28/testing/mozharness/mozharness/mozilla/testing/codecoverage.py#110-120 probably needs to be updated.

The linux path should be /builds/worker/checkouts/gecko; I'm not sure what windows should be, in light of windows-cross compiles. I'm also not sure if there are other places that need updating, or if there will need to be handling of generated files that are in the objdir, which is no longer underneath the source directory.

I've updated the linux path, and made a try push with 2 different coverage jobs: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=292803720&revision=bfaf3d779c05c2f80ac5fdf85d88a090fd8cff4b

No luck, the coverage data is still at 0 in the produced artifacts.

Tom, can you help us a bit more on this regression ? Who can i contact to get more informations on the Windows prefix ? What exactly did change in Bug 1527313 that could impact the code coverage ?

Flags: needinfo?(mozilla)

I've made a try push with the prefix update + a zip dump of raw coverage files: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=293045112&revision=709bda0bfc1fcdc75f2cd06d2da88c98c4ae92e6

The raw coverage artifacts are named code-coverage-raw-gcov.zip

Downloading the raw coverage file, it looks like all the paths in there are missing the first component. (I see Hal.gcda instead of hal/Hal.gcda for example). I'm not sure why there is the difference in behavior between before and after Bug 1527313 regarding paths, and the number of components to strip. I vaguely wonder if prefix.count("/") was over-counting by one, but the paths were relative to the object directory, instead of the source directory, so the errors cancelled out.

If that is the case, the current path gives GCOV_PREFIX_STRIP a value of 5, and the current object directory has 4 components (/builds/worker/workspace/obj-build), which would explain stripping an extra directory off.

It looks like the windows ccov builds are not yet cross compiled. I'm not sure if there is a bug for them, but Bug 1620166 is the dependency tree for moving windows builds to be cross builds (which will switch the built paths to match linux).`

The current windows paths (before we switch to cross compiles) are:
source: z:/build/build/src
object: z:/build/workspace/obj-build

Flags: needinfo?(mozilla)
Version: Version 3 → unspecified
Assignee: nobody → bastien
Status: NEW → ASSIGNED
Pushed by babadie@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d76906282445
Update linux prefix for code coverage generation, r=marco
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla76

Still need to fix Windows.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by babadie@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/86149c5f9a5e
Update windows prefix for code coverage generation, r=marco
Status: REOPENED → RESOLVED
Closed: 6 months ago6 months ago
Resolution: --- → FIXED

Do we need to uplift these patches to Beta?

Flags: needinfo?(bastien)

Hey Ryan, uplifting is not needed here as that bug would only affect our own web platform showcasing code coverage.
As you can see https://coverage.moz.tools/ , we're back to normal now !

Flags: needinfo?(bastien)

The patch landed in nightly and beta is affected.
:bastien, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(bastien)
Flags: needinfo?(bastien)
You need to log in before you can comment on or make changes to this bug.