1434729 - Always schedule all tests for code coverage builds

Status: VERIFIED FIXED

(Taskcluster :: General, enhancement)

Description

[Marco Castelluccio [:marco]]

For code coverage builds, we should ignore SCHEDULES and always run all tests.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

the specific concern is that we skip jsreftests often (good for saving resources) and end up with variable code coverage when analyzing a full run.

I assume we can fix this with a transform?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Are these the cron-driven coverage builds?  If so, they should just include all of the relevant tests in the target task set, rather than trying to adjust the optimization settings.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

these are regular test sets scheduled on mozilla-central only (i.e. every m-c build, not a cron task):
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#129
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#185

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, so the idea would be to optimize these normally on all branches where they run, but on mozilla-central to never optimize them?  Which will mean always performing the ccov build on each platform, then running the jsreftest tasks?  I'm just trying to get an idea of the cost and E2E time..

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Hm, so the idea would be to optimize these normally on all branches where
> they run, but on mozilla-central to never optimize them?  Which will mean
> always performing the ccov build on each platform, then running the
> jsreftest tasks?  I'm just trying to get an idea of the cost and E2E time..

I think we would need to never optimize them on mozilla-central only for the ccov builds.
The other builds should be unaffected.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

OK, then I think implementing that by setting optimization with a `by: branch: ..` would be OK.

Comment 7

[Marco Castelluccio [:marco]]

We have two options:
1) define the optimization in the yaml file, disabling them for ccov builds;
2) modify the transform we have for ccov builds and set the optimizations there.

The advantage of the first solution is that it is cleaner (even though it requires modifying many yaml files) and less likely to regress.
The advantage of the second solution is that we don't have to remember to add the `optimization` to any new yaml file.

Comment 8

[Marco Castelluccio [:marco]]

Attachment: Make it possible to overwrite 'optimization' in tests transforms

Comment 9

[Marco Castelluccio [:marco]]

Attachment: Never optimize test suites for coverage builds

Comment 10

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

Comment on attachment 8952485 [details]
Never optimize test suites for coverage builds

as far as I can tell this looks good.  My knowledge of tasks has become less useful with the schedules work, it seems to have overflowed what I can process and understand!

I would have thought that just |test['optimization'] = None| would have solved this.  Maybe there is transform logic that requires the removal of 'schedules-component' and 'when' clauses as well.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8952484 [details] [diff] [review]
Make it possible to overwrite 'optimization' in tests transforms

Review of attachment 8952484 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great.  Sorry for the delay in review!!

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4cbab279837a
Make it possible to overwrite 'optimization' in tests transforms. r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/eeb50080d6de
Never optimize test suites for coverage builds. r=jmaher

Comment 13

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4cbab279837a
https://hg.mozilla.org/mozilla-central/rev/eeb50080d6de

Comment 14

[Marco Castelluccio [:marco]]

Needinfoing me so I remember to verify everything is working as it should.