Investigate removing `fission-run-on-projects` and `webrender-run-on-projects`
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(firefox89 fixed)
Tracking | Status | |
---|---|---|
firefox89 | --- | fixed |
People
(Reporter: ahal, Assigned: ahal)
References
Details
Attachments
(18 files, 1 obsolete file)
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
416.73 KB,
patch
|
Details | Diff | Splinter Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review |
The fission-run-on-projects
key was a hack I implemented to allow us to rapidly meet the fission team's needs when fission was still nascent. Since the tasks it supported were very sporadic, we ended up with a ton of by-variant
statements all over the test configs. Duplicating the run-on-projects
key for fission ended up being less complicated at the time.
Now that fission runs pretty much everywhere, I think the key is causing more trouble than its solving. We should attempt to remove it and go back to using by-variant
statements when fission needs to deviate from non-fission.
At some point, someone added webrender-run-on-projects
as well. I'm not sure if it was actually needed, or if it was a cargo cult. We should check this assumption as well.
Comment 1•4 years ago
|
||
I think this should be higher priority- we have added 2 new variants in the last 2 months, so I would prefer we have less footguns for people to add.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 2•4 years ago
|
||
This will allow us to use 'by-variant' in the run-on-projects key.
Assignee | ||
Comment 3•4 years ago
|
||
This means that any task that doesn't set 'fission-run-on-projects' has it
default to []. Instead I'd like tasks that omit this key to default to using
'run-on-projects' (which means we can solve this bug piece by piece rather than
try to land it all at once).
The full taskgraph of this revision is identical to the previous revision.
Depends on D107962
Assignee | ||
Comment 4•4 years ago
|
||
The full taskgraph from this revision is identical to the previous revision.
Depends on D107963
Assignee | ||
Comment 5•4 years ago
|
||
Depends on D107964
Assignee | ||
Comment 6•4 years ago
|
||
The full taskgraph on this revision is identical to the previous revision.
Depends on D107965
Assignee | ||
Comment 7•4 years ago
|
||
The full taskgraph on this revision is identical to the previous revision.
Depends on D107966
Assignee | ||
Comment 8•4 years ago
|
||
This is a diff of the entire stack. It's functionally equivalent to the baseline. The only difference is some command line flag re-ordering caused by moving the ccov transform a little down in the ordering.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 9•4 years ago
|
||
Fixed colour formatting.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 10•4 years ago
|
||
This changes the taskgraph slightly for
'test-macosx1014-64-devedition-qr/opt-test-verify-fis-e10s' by removing
'integration' from its 'run-on-projects' key.
The reason is that the 'test-verify' tasks has 'built-projects-only: true'.
This key ensures that the task can only ever run when the build is already
running (so it doesn't cause builds to get pulled in).
Turns out the 'built-projects-only' feature was erroneously ignoring
'fission-run-on-projects', so this small change in behaviour is actually
desired.
Depends on D107967
Assignee | ||
Comment 11•4 years ago
|
||
Depends on D108075
Comment 12•4 years ago
|
||
Comment 13•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/268d317bac31
https://hg.mozilla.org/mozilla-central/rev/ac54e90586f7
https://hg.mozilla.org/mozilla-central/rev/0f5eaa4e72c1
https://hg.mozilla.org/mozilla-central/rev/87bd024d9028
https://hg.mozilla.org/mozilla-central/rev/50e961f7a2c0
https://hg.mozilla.org/mozilla-central/rev/68501658a1e6
https://hg.mozilla.org/mozilla-central/rev/5b3ed85eaf66
Assignee | ||
Comment 14•4 years ago
|
||
These tasks all vary in their 'fission-run-on-projects'. So while this looks
like more complexity in isolation, it will greatly simplify the configs (and
the diff) later on in the stack.
Assignee | ||
Comment 15•4 years ago
|
||
Depends on D108681
Comment 16•4 years ago
|
||
Comment 17•4 years ago
|
||
bugherder |
Assignee | ||
Comment 18•4 years ago
|
||
Comment 19•4 years ago
|
||
Comment 20•4 years ago
|
||
bugherder |
Assignee | ||
Comment 21•4 years ago
|
||
We are deprecating the 'fission-run-on-projects' and 'fission-tier' keys in
favour of more familiar 'by-variant' keys. While the diff here might look
confusing, it's designed to produce an identical taskgraph as before. So
shouldn't cause any changes in CI.
Assignee | ||
Comment 22•4 years ago
|
||
Some tasks need to use the 'by-variant' key here, so anything that requires a 'tier' needs
to come after we split the variants.
The full taskgraph is identical before / after this patch.
Comment 23•4 years ago
|
||
Assignee | ||
Comment 24•4 years ago
|
||
This change causes a slight change in the taskgraph (see phabricator for diff).
Essentially it changes the optimization for 'browsertime-tp6-firefox' tasks
from 'skip-unless-expanded' to 'skip-unless-backstop'. This change happened
because the 'apply_raptor_tier_optimization' transform did not take
'fission-tier' into account. Now that 'fission-tier' has merged with 'tier',
these tasks are being modified by this transform. In my view this change is a
desired outcome.
Assignee | ||
Comment 25•4 years ago
|
||
The full taskgraph before / after this change is identical.
Depends on D109662
Assignee | ||
Comment 26•4 years ago
|
||
This change causes a slight change in the taskgraph. Essentially it changes the
optimization for some fission raptor tasks from 'skip-unless-expanded' to
'skip-unless-backstop'. This change happened because the
'apply_raptor_tier_optimization' transform did not take 'fission-tier' into
account. Now that 'fission-tier' has merged with 'tier', these tasks are being
modified by this transform. In my view this change is a desired outcome.
Depends on D109663
Assignee | ||
Comment 27•4 years ago
|
||
Depends on D109664
Comment 28•4 years ago
|
||
bugherder |
Assignee | ||
Updated•4 years ago
|
Comment 29•4 years ago
|
||
Comment 30•4 years ago
|
||
Backed out for causing gecko decision task failures
backout: https://hg.mozilla.org/integration/autoland/rev/2d6f2d37409866e55cbbf11dc3181a0b2676016a
push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=a0b814900c3676410d090d20a2cf131f3d394f86
failure log: https://treeherder.mozilla.org/logviewer?job_id=334586106&repo=autoland&lineNumber=1599
[task 2021-03-26T18:43:03.750Z] Error loading tasks for kind test:
[task 2021-03-26T18:43:03.751Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "bugbug_push_schedules_time", "value": 0.08206899700007853, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}, {"name": "bugbug_push_schedules_retries", "value": 0, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[task 2021-03-26T18:43:03.751Z] Traceback (most recent call last):
[task 2021-03-26T18:43:03.751Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 27, in validate_schema
[task 2021-03-26T18:43:03.751Z] schema(obj)
[task 2021-03-26T18:43:03.751Z] File "/builds/worker/checkouts/gecko/third_party/python/voluptuous/voluptuous/schema_builder.py", line 267, in __call__
[task 2021-03-26T18:43:03.751Z] return self._compiled([], data)
[task 2021-03-26T18:43:03.751Z] File "/builds/worker/checkouts/gecko/third_party/python/voluptuous/voluptuous/schema_builder.py", line 589, in validate_dict
[task 2021-03-26T18:43:03.751Z] return base_validate(path, iteritems(data), out)
[task 2021-03-26T18:43:03.751Z] File "/builds/worker/checkouts/gecko/third_party/python/voluptuous/voluptuous/schema_builder.py", line 427, in validate_mapping
[task 2021-03-26T18:43:03.751Z] raise er.MultipleInvalid(errors)
[task 2021-03-26T18:43:03.751Z] voluptuous.error.MultipleInvalid: expected a list @ data['run-on-projects']['by-variant']['default']
[task 2021-03-26T18:43:03.751Z]
[..]
[task 2021-03-26T18:43:03.752Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 32, in validate_schema
[task 2021-03-26T18:43:03.752Z] raise Exception("\n".join(msg) + "\n" + pprint.pformat(obj))
[task 2021-03-26T18:43:03.752Z] Exception: In unknown task:
[task 2021-03-26T18:43:03.752Z] expected a list @ data['run-on-projects']['by-variant']['default']
[task 2021-03-26T18:43:03.752Z] {'allow-software-gl-layers': False,
[task 2021-03-26T18:43:03.752Z] 'app': 'firefox',
[task 2021-03-26T18:43:03.752Z] 'attributes': {'retrigger': True},
[task 2021-03-26T18:43:03.752Z] 'build-attributes': {'accepted-mar-channel-ids': 'firefox-mozilla-central',
[task 2021-03-26T18:43:03.752Z] 'always_target': False,
[task 2021-03-26T18:43:03.752Z] 'build_platform': 'linux64',
[task 2021-03-26T18:43:03.752Z] 'build_type': 'opt',
[task 2021-03-26T18:43:03.752Z] 'enable-build-signing': True,
[task 2021-03-26T18:43:03.752Z] 'kind': 'build',
[task 2021-03-26T18:43:03.752Z] 'mar-channel-id': 'firefox-mozilla-central',
[task 2021-03-26T18:43:03.752Z] 'run_on_projects': ['integration'],
[task 2021-03-26T18:43:03.752Z] 'shipping_phase': None,
[task 2021-03-26T18:43:03.752Z] 'shipping_product': None},
[task 2021-03-26T18:43:03.752Z] 'build-label': 'build-linux64/opt',
[task 2021-03-26T18:43:03.752Z] 'build-platform': 'linux64/opt',
[task 2021-03-26T18:43:03.752Z] 'mar-channel-id': 'firefox-mozilla-central',
[task 2021-03-26T18:43:03.752Z] 'run_on_projects': ['integration'],
[task 2021-03-26T18:43:03.752Z] 'shipping_phase': None,
[task 2021-03-26T18:43:03.752Z] 'shipping_product': None},
[task 2021-03-26T18:43:03.752Z] 'build-label': 'build-linux64/opt',
[task 2021-03-26T18:43:03.752Z] 'build-platform': 'linux64/opt',
[task 2021-03-26T18:43:03.752Z] 'build-signing-label': 'build-signing-linux64/opt',
[task 2021-03-26T18:43:03.752Z] 'built-projects-only': False,
[task 2021-03-26T18:43:03.752Z] 'checkout': False,
[task 2021-03-26T18:43:03.752Z] 'chunks': 1,
Comment 31•4 years ago
|
||
Comment 32•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/45fd21a641f4
https://hg.mozilla.org/mozilla-central/rev/13db75ac9ace
https://hg.mozilla.org/mozilla-central/rev/d226c414af12
https://hg.mozilla.org/mozilla-central/rev/f4789d603967
Assignee | ||
Updated•4 years ago
|
Description
•