1653050 - Allow optimizing by replacement existing (pending/unscheduled) cached tasks

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Justin Wood (:Callek)]

We should allow depending on tasks from a prior push, specifically this solves the following situation a bit:

User A pushes a change to an early toolchain --> This triggers many other toolchains and docker image builds

10 minutes later User B pushes any other change, but is now waiting on that entire path from User A, and since nothing is done the taskgraph recreates and runs the entire set of toolchains/docker image builds that User A also triggered. Which adds more time that User B has to wait for results and adds extra cost.

---

We haven't pushed toolchain/docker image changes at that scale too often, so this has not been a priority, however over in Bug 1628981 we want to be able to depend on an already produced build in progress from a prior push in order to avoid an extra build on a test-only push.

This has the downside on UX for sheriffs in terms of being able to identify the situation, I've touched base with treeherder team and will be following up with sheriffs as well (I'll file that as a new bug)

Comment 1

[Justin Wood (:Callek)]

Attachment: Bug 1653050 - Allow optimizing by replacement existing (pending/unscheduled) cached tasks. r=tomprince

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c2746ff5e6b4
Allow optimizing by replacement existing (pending/unscheduled) cached tasks. r=tomprince

Comment 3

[Natalia Csoregi [:nataliaCs]]

Backed out for failing gecko decision task on attributes.rst

backout: https://hg.mozilla.org/integration/autoland/rev/1baff6edcc55d2d4c7d3ef3a32a282c4e74c1b62

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=fjzHAwJrTYOR9C318PPDkQ.0&revision=c2746ff5e6b47a52c90eddc1cf7a37c97ded89e4&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2Cd

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=311467103&repo=autoland&lineNumber=3813

[task 2020-07-30T02:07:28.229Z] PERFHERDER_DATA: {"suites": [{"value": 0.30357109800024773, "name": "bugbug_push_schedules_time", "shouldAlert": false, "subtests": [], "lowerIsBetter": true}, {"value": 0, "name": "bugbug_push_schedules_retries", "shouldAlert": false, "subtests": [], "lowerIsBetter": true}], "framework": {"name": "build_metrics"}}
[task 2020-07-30T02:07:28.230Z] Traceback (most recent call last):
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 205, in taskgraph_decision
[task 2020-07-30T02:07:28.230Z] return taskgraph.decision.taskgraph_decision(options)
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 251, in taskgraph_decision
[task 2020-07-30T02:07:28.230Z] full_task_json = tgg.full_task_graph.to_json()
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 163, in full_task_graph
[task 2020-07-30T02:07:28.230Z] return self._run_until('full_task_graph')
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 375, in _run_until
[task 2020-07-30T02:07:28.230Z] k, v = next(self._run)
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 290, in _run
[task 2020-07-30T02:07:28.230Z] self.verify_attributes(all_tasks)
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 406, in verify_attributes
[task 2020-07-30T02:07:28.230Z] appearing_as="heading"
[task 2020-07-30T02:07:28.230Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/verify.py", line 135, in verify_docs
[task 2020-07-30T02:07:28.230Z] .format(appearing_as, identifier, filename)
[task 2020-07-30T02:07:28.230Z] Exception: heading: eager_index missing from doc file: attributes.rst
[taskcluster 2020-07-30 02:07:29.704Z] === Task Finished ===
[taskcluster 2020-07-30 02:07:35.129Z] Unsuccessful task run with exit code: 1 completed in 254.501 seconds

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/299d719022da
Allow optimizing by replacement existing (pending/unscheduled) cached tasks. r=tomprince

Comment 5

[Justin Wood (:Callek)]

Attachment: Bug 1653050 followup, import copy. CLOSED TREE r=CosminS

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/893df5ae4443
followup, import copy. CLOSED TREE r=CosminS

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/299d719022da
https://hg.mozilla.org/mozilla-central/rev/893df5ae4443

Comment 8

[Aki Sasaki (not active)]

This is causing issues when the toolchain tasks fail on my try staging testing. I wonder if we can recreate the cached task if the existing indexed task has an exception or failure state.