1640902 - Create integration tests for ci + taskgraph

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P2)

Description

[Andrew Halberstadt [:ahal]]

We have pretty good unit tests for the taskgraph module itself, but no integration tests that cover what actually gets generated with the ci directory as input. In otherwords, tests that generate the actual graphs and make assertions on the results.

In general I don't want to get too carried away here, as we don't want to end up in a situation where each change to the configs needs a corresponding change to test expectations.

But there are certain conditions we want to make sure don't regress. A couple examples:

1. Bug 1639812 fixed an exception that only showed up when running with the parameters set by |mach try auto|.
2. Bug 1640890 fixed a regression where fuzzing-builds were accidentally removed from central. It's important that these always run on central.

These tests will be very expensive as we'll likely need to generate the taskgraph many times with varying parameters. So maybe they should be part of a new "suite" (i.e, not bundled in with the taskgraph unittests).

Comment 1

[Andrew Halberstadt [:ahal]]

Tom, curious if you've put thought into this before and whether you had any suggestions.

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to Andrew Halberstadt [:ahal] from comment #0)

These tests will be very expensive as we'll likely need to generate the taskgraph many times with varying parameters. So maybe they should be part of a new "suite" (i.e, not bundled in with the taskgraph unittests).

Maybe we could instead invest some effort into making taskgraph generation faster?

Comment 3

[Aki Sasaki (not active)]

When hacking on taskgraph, we find it useful to run taskgraph-gen and taskgraph-diff. These build various release graphs and on-push graphs on various branches. By generating these graphs against both a clean and patched revision, we can diff to pinpoint what we're changing.

A while back, I wondered if we could generate these graphs whenever the taskgraph changes. We could then diff against the previous indexed taskgraph generation task's artifacts. These diffs would likely be more useful to humans than automation.

(Edit: We could, however, add some checks here to make sure that we don't see unexpected changes in the release graphs or on other branches like Try.)

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to Nathan Froyd [:froydnj] from comment #2)

Maybe we could instead invest some effort into making taskgraph generation faster?

Not mutually exclusive, by stand up a new "suite" I meant put the tests in a different location from the taskgraph tests and use a different subsuite value. So maybe ~10 minutes of extra effort.

Also even if we get taskgraph generation back down to 20s somehow, these would still be very expensive tests compared to the others that are usually measured in milliseconds.

See also bug 1617598.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #3)

When hacking on taskgraph, we find it useful to run taskgraph-gen and taskgraph-diff. These build various release graphs and on-push graphs on various branches. By generating these graphs against both a clean and patched revision, we can diff to pinpoint what we're changing.

A while back, I wondered if we could generate these graphs whenever the taskgraph changes. We could then diff against the previous indexed taskgraph generation task's artifacts. These diffs would likely be more useful to humans than automation.

This would be awesome. See also bug 1409733.

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1640902 - [python-test] Add ability to mark tests as "slow" and an arugment to run them, r?raphael

This creates a general pytest plugin under "config/mozunit" which implements
the ability to mark tests as "slow". Slow tests will be skipped by default
unless "--run-slow" is passed in.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1640902 - [ci] Add integration tests for the 'taskcluster' directory, r?tomprince

Initially this suite will only include a test for |mach try auto| pushes.

Depends on D81402

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1640902 - [taskcluster.test] Add some additional checks to test_mach_try_auto.py, r?marco

Adds some additional assertions against what would be ran with |mach try auto|.
This also fixes a legit bug that the 'test_important_manifests_only' test
caught.

Depends on D81403

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1640902 - [taskgraph] Fix some Python 3 compatibility issues in the 'optimize' directory, r?tomprince

Comment 9

[Phabricator Automation]

Comment on attachment 9160158 [details]
Bug 1640902 - [taskgraph] Fix some Python 3 compatibility issues in the 'optimize' directory, r?tomprince

Revision D81577 was moved to bug 1650067. Setting attachment 9160158 [details] to obsolete.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3601782567e4
[python-test] Add ability to mark tests as "slow" and an arugment to run them, r=raphael
https://hg.mozilla.org/integration/autoland/rev/ff858fa5508b
[ci] Add integration tests for the 'taskcluster' directory, r=tomprince
https://hg.mozilla.org/integration/autoland/rev/2d6c930db96c
[taskcluster.test] Add some additional checks to test_mach_try_auto.py, r=marco

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/3601782567e4
https://hg.mozilla.org/mozilla-central/rev/ff858fa5508b
https://hg.mozilla.org/mozilla-central/rev/2d6c930db96c