Closed Bug 1640902 Opened 6 months ago Closed 5 months ago

Create integration tests for ci + taskgraph

Categories

(Firefox Build System :: Task Configuration, task, P2)

task

Tracking

(firefox80 fixed)

RESOLVED FIXED
mozilla80
Tracking Status
firefox80 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(3 files, 1 obsolete file)

We have pretty good unit tests for the taskgraph module itself, but no integration tests that cover what actually gets generated with the ci directory as input. In otherwords, tests that generate the actual graphs and make assertions on the results.

In general I don't want to get too carried away here, as we don't want to end up in a situation where each change to the configs needs a corresponding change to test expectations.

But there are certain conditions we want to make sure don't regress. A couple examples:

  1. Bug 1639812 fixed an exception that only showed up when running with the parameters set by |mach try auto|.
  2. Bug 1640890 fixed a regression where fuzzing-builds were accidentally removed from central. It's important that these always run on central.

These tests will be very expensive as we'll likely need to generate the taskgraph many times with varying parameters. So maybe they should be part of a new "suite" (i.e, not bundled in with the taskgraph unittests).

Tom, curious if you've put thought into this before and whether you had any suggestions.

Flags: needinfo?(mozilla)

(In reply to Andrew Halberstadt [:ahal] from comment #0)

These tests will be very expensive as we'll likely need to generate the taskgraph many times with varying parameters. So maybe they should be part of a new "suite" (i.e, not bundled in with the taskgraph unittests).

Maybe we could instead invest some effort into making taskgraph generation faster?

When hacking on taskgraph, we find it useful to run taskgraph-gen and taskgraph-diff. These build various release graphs and on-push graphs on various branches. By generating these graphs against both a clean and patched revision, we can diff to pinpoint what we're changing.

A while back, I wondered if we could generate these graphs whenever the taskgraph changes. We could then diff against the previous indexed taskgraph generation task's artifacts. These diffs would likely be more useful to humans than automation.

(Edit: We could, however, add some checks here to make sure that we don't see unexpected changes in the release graphs or on other branches like Try.)

(In reply to Nathan Froyd [:froydnj] from comment #2)

Maybe we could instead invest some effort into making taskgraph generation faster?

Not mutually exclusive, by stand up a new "suite" I meant put the tests in a different location from the taskgraph tests and use a different subsuite value. So maybe ~10 minutes of extra effort.

Also even if we get taskgraph generation back down to 20s somehow, these would still be very expensive tests compared to the others that are usually measured in milliseconds.

See also bug 1617598.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #3)

When hacking on taskgraph, we find it useful to run taskgraph-gen and taskgraph-diff. These build various release graphs and on-push graphs on various branches. By generating these graphs against both a clean and patched revision, we can diff to pinpoint what we're changing.

A while back, I wondered if we could generate these graphs whenever the taskgraph changes. We could then diff against the previous indexed taskgraph generation task's artifacts. These diffs would likely be more useful to humans than automation.

This would be awesome. See also bug 1409733.

Assignee: nobody → ahal
Severity: -- → S3
Status: NEW → ASSIGNED
Priority: -- → P2
Flags: needinfo?(mozilla)

This creates a general pytest plugin under "config/mozunit" which implements
the ability to mark tests as "slow". Slow tests will be skipped by default
unless "--run-slow" is passed in.

Initially this suite will only include a test for |mach try auto| pushes.

Depends on D81402

Adds some additional assertions against what would be ran with |mach try auto|.
This also fixes a legit bug that the 'test_important_manifests_only' test
caught.

Depends on D81403

See Also: → 1650067

Comment on attachment 9160158 [details]
Bug 1640902 - [taskgraph] Fix some Python 3 compatibility issues in the 'optimize' directory, r?tomprince

Revision D81577 was moved to bug 1650067. Setting attachment 9160158 [details] to obsolete.

Attachment #9160158 - Attachment is obsolete: true
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3601782567e4
[python-test] Add ability to mark tests as "slow" and an arugment to run them, r=raphael
https://hg.mozilla.org/integration/autoland/rev/ff858fa5508b
[ci] Add integration tests for the 'taskcluster' directory, r=tomprince
https://hg.mozilla.org/integration/autoland/rev/2d6c930db96c
[taskcluster.test] Add some additional checks to test_mach_try_auto.py, r=marco
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla80
You need to log in before you can comment on or make changes to this bug.