1301978 - Enable collection of reftest code coverage with GCOV.

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect)

Description

[Greg Mierzwinski [:sparky]]

We'd like to be able to collect code coverage from reftest with GCOV. This should be similar to mochitest-browser-chrome for linux64-ccov. The "--code-coverage" flag may work already and if not, it'll have to be implemented.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

a working patch is here:
https://hg.mozilla.org/try/rev/59f825e1d7d78e14ca8e086fab2f2cb47f59bc03

a lot has changed in task configuration since then, but this is a good starting place.

Comment 2

[Greg Mierzwinski [:sparky]]

Wow has that taskcluster folder ever changed in the last bit! I've found a bug in the tests.yml file also that will be fixed with this patch.

So this patch will add a transform for code coverage jobs and also implement code coverage for reftest, crashtest, and jsreftest all at once.

Here's a link to a test run using the new patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=de0f9113824af9334dbf47019d4e3bc221394b3d .

Reftest definitely works based on a previous job so this one is testing all the tests and making sure we haven't lost or broken anything in the process. It also tests crashtest and jsreftest to see if those work.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I like the transform- the simpler we can get our tests.yml file, I think the better off we will be- this patch simplifies it a lot :)

Comment 4

[Greg Mierzwinski [:sparky]]

Can you review this patch or should I have someone else review it? It's all set, and I've also disabled e10s for ref/jsref/crash-test because I'm not sure if the ETL can handle those differences yet.

I'm going to create another bug to add jsdcov into the transform and make the appropriate changes to tests.yml for the run-on-projects flag.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I will review this!

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

the patch on the try push is looking good and the try results are great, if you want to r=me on it and we can get it landed that would be ideal!

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

and a note, there are some flake8 errors with this patch

Comment 8

[Greg Mierzwinski [:sparky]]

I'm not sure why we're getting those flake8 errors because my changes didn't affect those parts. Also, it doesn't make sense to put an extra line where it says it needs one. What do you think?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

for flake8, it could have been code from another revision that landed and then got fixed up- maybe refreshing your base revision?

for flake8 and other linters, they like to ensure there are 2 blank lines between methods/functions.  This offers a clear break.

Comment 10

[Greg Mierzwinski [:sparky]]

Ok, I've fixed the flaxe8 error. I was looking in the wrong 'tests' file. Here's a test run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf323b25dd5e75ef87c9d60fbb9c07c3f1e0ffa9 .

I'll have a patch up for review later today.

Comment 11

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1301978 - Enable reftest, crashtest, & jsreftest and use a transform on linux64-ccov tasks.

This bug makes it simpler to add new test suites to linux64-ccov with the help of a new transform and also cleans up the test suite definitions. It also enables reftest, crashtest and jsreftest at the same time.

Review commit: https://reviewboard.mozilla.org/r/107028/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107028/

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8830126 [details]
Bug 1301978 - Enable reftest, crashtest, & jsreftest and use a transform on linux64-ccov tasks.

https://reviewboard.mozilla.org/r/107028/#review108222

thank you for putting this together!

Comment 13

[Greg Mierzwinski [:sparky]]

Thanks for the quick review! Would it be possible to have these changes landed soon?

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d0b05a7251a7
Enable reftest, crashtest, & jsreftest and use a transform on linux64-ccov tasks. r=jmaher

Comment 15

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d0b05a7251a7