1455400 - Upload per-test coverage reports as artifacts

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Marco Castelluccio [:marco]]

With bug 1454629 we have the infrastructure to generate them, but we are not uploading them yet.

Comment 1

[Marco Castelluccio [:marco]]

Attachment: Patch

For now, the output format is the coveralls one. I think it will be faster to handle than jsdcov, as we don't need to do any translation (we already have the coveralls format as it's the output of grcov).

I could add jsdcov support to grcov, but I think that the coveralls format will also make it easier to diff with the baseline (also because the coveralls format contains hit counts, the jsdcov format doesn't).

This will need some changes on ActiveData (namely, adding a new reader), but it should be pretty easy as it's just JSON.

Comment 2

[Greg Mierzwinski [:sparky]]

Comment on attachment 8970384 [details] [diff] [review]
Patch

Review of attachment 8970384 [details] [diff] [review]:
-----------------------------------------------------------------

Good work! Just some questions/comments that need to be addressed and changes made accordingly if needed.

::: testing/mozharness/mozharness/mozilla/testing/codecoverage.py
@@ +260,5 @@
> +        with open(grcov_file, 'r') as f:
> +            report = json.load(f)
> +        os.remove(grcov_file)
> +
> +        # TODO: Diff this coverage report with the baseline one

I think we should do the diffing in the final phase - during the post-action listener. Otherwise, we will have to enforce that the baseline test is always run first so that we can diff it with all the next tests. And we might also have different suites running in a single chunk that would require a specific baseline to be subtracted.

@@ +265,5 @@
> +
> +        dest = os.path.join('per-test-coverage-reports', str(uuid.uuid4()) + '.json')
> +
> +        with open(dest, 'w') as f:
> +            json.dump({'test': test, 'report': report}, f)

Is it possible to add the flavor (suite/sub-suite) of the test to this JSON? At least as an intermediate data point before the diffing. I think it'll make it much easier to find the correct baseline's for each test when multiple suites are run in the same TC chunk.

@@ +291,5 @@
>  
>          if not self.code_coverage_enabled:
>              return
>  
>          if self.per_test_coverage:

In here is where I think we should diff the baseline's with the reports.

::: testing/mozharness/scripts/desktop_unittest.py
@@ +853,5 @@
>                  # full coverage of the test and the baseline coverage and only get
>                  # the coverage data specific to the test.
>                  if self.per_test_coverage:
>                      gcov_dir, jsvm_dir = self.set_coverage_env(env)
>                      # TODO: Run basic startup/shutdown test to collect baseline coverage.

If we remove the baseline later in the post-action listener, we can remove this TODO as well. Same for 347-349 in web_platform_tests.py

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Patch

Comment 4

[Greg Mierzwinski [:sparky]]

With these changes test-coverage should now output some per-test JSONs, even though they are not diffed. The patch looks great to me, but can you test it out on try before we land it?

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Greg Mierzwinski [:sparky] from comment #4)
> With these changes test-coverage should now output some per-test JSONs, even
> though they are not diffed. The patch looks great to me, but can you test it
> out on try before we land it?

Here's the (green) try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7994e6c880c28515fa989cf113d9859293d692e0.
It's only on Linux, but I wouldn't expect the Windows build to be different.

Comment 6

[Greg Mierzwinski [:sparky]]

This looks great to me! Two tests are run in that try push in test-coverage (TC) and two JSONs are uploaded in the artifact as well.

Comment 7

[Marco Castelluccio [:marco]]

Landed with the wrong bug number...

https://hg.mozilla.org/integration/mozilla-inbound/rev/bf368c4e5c0a

https://hg.mozilla.org/mozilla-central/rev/bf368c4e5c0a