1368115 - Enable remaining test-suites for linux64-ccov.

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Greg Mierzwinski [:sparky]]

There are still a few test-suites that need to be enabled on linux64-ccov.

Here is a list:
  Mochitests on Valgrind (all broken)

  AWSY (one test failing)

  Mochitest-browser-screenshots (no failures, ready)

  Marionette-Headless (no failures, ready)

  Web-platform-tests-reftests (no failures, ready)

  Web-platform-tests-wdspec (no failures, ready)
  

And here's a try push for reference to the bugs, (note that mochitests on valgrind is disabled because of it's error) : https://treeherder.mozilla.org/#/jobs?repo=try&revision=fdeec7c79770235149dfb3236ce8f6d8cba4e213

Comment 1

[Greg Mierzwinski [:sparky]]

This is a try push that shows the valgrind error: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2bf5de2d35ddba1b07eb4d42c7a6e038f2c827a5

Comment 2

[Greg Mierzwinski [:sparky]]

Here's a test run with all the suites (except for SM, Talos, and Valgrind): https://treeherder.mozilla.org/#/jobs?repo=try&revision=e224ef569ab89ec350cf15248a0d8a1309592ad8

The Wd failures are known and the frequency has not changed. The AWSY failure is just me forgetting to apply a patch, and the bc failure is nothing of concern and already known as well.

The talos suite needs a bit more work: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bca2d12abe12d9cd54ba53afd9fdb56cda18a561

The failures are coming from the profiler which is profiling every single gcda that was/could've been generated and this gives a large amount of output.

Joel, do we want to leave profiling on when we run the talos tests? There is a flag we can use to disable it and I am testing it out right now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=73bee6dc011888669509dc71fcab2a84ee913eba

Also, should we leave it riding along the linux64 name or get it into the linux64-ccov category?

Comment 3

[Greg Mierzwinski [:sparky]]

Actually, we might have to move it to linux64-ccov to get code coverage artifacts uploaded.

Comment 4

[Greg Mierzwinski [:sparky]]

So, I found that we will have to add linux64-ccov to the list of build bot names at [1] to be able to get code coverage, or change the 'cc' to match 'ccov'.

The reason for this is because linux64 does not get --code-coverage and I think that just making the change in buildbot will be easier than trying to hack in the command on linux64 when we're using it for coverage on talos.

I'm running a test at [2] to see what will happen when 'cc' is matched.

[1]: https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py
[2]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3cc9b702e5a040ad0c81cff9c9c50cd5f465ade0

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I don't see a need for having profiling enabled whilst running Talos.  We really need any Talos results to show up as linux64-ccov, the problem I think we will hit is that this is on hardware, not on a VM.  This means that it goes through buildbot and might be problematic.  In this case, I don't think we need hardware since the timing is not of concern and we could hack talos to run on the VM :)

If we get everything but Talos done now, then we can sort out the remaining talos hacks.

Comment 6

[Greg Mierzwinski [:sparky]]

Ok, I'll make a new bug for talos and submit a patch here for enabling everything else but that.

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1368115 - Enable most of the remaining test suites for linux64-ccov.

This enables all test suites on linux64-ccov except for the talos test suite (more work needs to be done), spidermonkey tests (more work needs to be done), and the mochitest-valgrind test suite (not going to run linux64-ccov).

Review commit: https://reviewboard.mozilla.org/r/148204/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/148204/

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8876875 [details]
Bug 1368115 - Enable most of the remaining test suites for linux64-ccov.

https://reviewboard.mozilla.org/r/148204/#review152556

::: taskcluster/ci/test/test-platforms.yml:122
(Diff revision 1)
>          - qr-tests
>  
>  linux64-ccov/opt:
>      build-platform: linux64-ccov/opt
>      test-sets:
> -        - ccov-code-coverage-tests
> +        - common-tests

can we remove ccov-code-coverage-tests from the test-sets.yml file as well?

Comment 9

[Greg Mierzwinski [:sparky]]

Comment on attachment 8876875 [details]
Bug 1368115 - Enable most of the remaining test suites for linux64-ccov.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/148204/diff/1-2/

Comment 10

[Greg Mierzwinski [:sparky]]

Comment on attachment 8876875 [details]
Bug 1368115 - Enable most of the remaining test suites for linux64-ccov.

https://reviewboard.mozilla.org/r/148204/#review152556

> can we remove ccov-code-coverage-tests from the test-sets.yml file as well?

Yes we can, thanks for catching that!

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8876875 [details]
Bug 1368115 - Enable most of the remaining test suites for linux64-ccov.

https://reviewboard.mozilla.org/r/148204/#review152564

thanks for the updated change

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/922b31170819
Enable most of the remaining test suites for linux64-ccov. r=jmaher

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/922b31170819