Closed Bug 1434729 Opened 4 years ago Closed 4 years ago

Always schedule all tests for code coverage builds

Categories

(Taskcluster :: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED
mozilla60

People

(Reporter: marco, Assigned: marco)

References

Details

Attachments

(2 files)

For code coverage builds, we should ignore SCHEDULES and always run all tests.
the specific concern is that we skip jsreftests often (good for saving resources) and end up with variable code coverage when analyzing a full run.

I assume we can fix this with a transform?
Are these the cron-driven coverage builds?  If so, they should just include all of the relevant tests in the target task set, rather than trying to adjust the optimization settings.
Hm, so the idea would be to optimize these normally on all branches where they run, but on mozilla-central to never optimize them?  Which will mean always performing the ccov build on each platform, then running the jsreftest tasks?  I'm just trying to get an idea of the cost and E2E time..
(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Hm, so the idea would be to optimize these normally on all branches where
> they run, but on mozilla-central to never optimize them?  Which will mean
> always performing the ccov build on each platform, then running the
> jsreftest tasks?  I'm just trying to get an idea of the cost and E2E time..

I think we would need to never optimize them on mozilla-central only for the ccov builds.
The other builds should be unaffected.
See Also: → 1362010
OK, then I think implementing that by setting optimization with a `by: branch: ..` would be OK.
We have two options:
1) define the optimization in the yaml file, disabling them for ccov builds;
2) modify the transform we have for ccov builds and set the optimizations there.

The advantage of the first solution is that it is cleaner (even though it requires modifying many yaml files) and less likely to regress.
The advantage of the second solution is that we don't have to remember to add the `optimization` to any new yaml file.
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #8952484 - Flags: review?(dustin)
Comment on attachment 8952485 [details]
Never optimize test suites for coverage builds

as far as I can tell this looks good.  My knowledge of tasks has become less useful with the schedules work, it seems to have overflowed what I can process and understand!

I would have thought that just |test['optimization'] = None| would have solved this.  Maybe there is transform logic that requires the removal of 'schedules-component' and 'when' clauses as well.
Attachment #8952485 - Flags: review?(jmaher) → review+
Comment on attachment 8952484 [details] [diff] [review]
Make it possible to overwrite 'optimization' in tests transforms

Review of attachment 8952484 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great.  Sorry for the delay in review!!
Attachment #8952484 - Flags: review?(dustin) → review+
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4cbab279837a
Make it possible to overwrite 'optimization' in tests transforms. r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/eeb50080d6de
Never optimize test suites for coverage builds. r=jmaher
https://hg.mozilla.org/mozilla-central/rev/4cbab279837a
https://hg.mozilla.org/mozilla-central/rev/eeb50080d6de
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Needinfoing me so I remember to verify everything is working as it should.
Flags: needinfo?(mcastelluccio)
Status: RESOLVED → VERIFIED
Flags: needinfo?(mcastelluccio)
You need to log in before you can comment on or make changes to this bug.