1665312 - Intermittent Exception: New label (test-linux1804-64-qr/debug-mochitest-media-fis-e10s-1) was not found in the task-graph

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=315814327&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FyQfh7dFSz-U4Bjm0WU38w/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-09-16T09:03:30.043Z] Running in Taskcluster instance https://firefox-ci-tc.services.mozilla.com with taskcluster-proxy
[task 2020-09-16T09:03:30.050Z] Starting new HTTPS connection (1): firefox-ci-tc.services.mozilla.com
[task 2020-09-16T09:03:30.167Z] "GET /api/queue/v1/task/StDUiIVPQfSetG7NEcwbNA/artifacts/public/parameters.yml HTTP/1.1" 303 116
[task 2020-09-16T09:03:30.169Z] Starting new HTTPS connection (1): firefoxci.taskcluster-artifacts.net
[task 2020-09-16T09:03:30.212Z] "GET /StDUiIVPQfSetG7NEcwbNA/0/public/parameters.yml HTTP/1.1" 200 1109
[task 2020-09-16T09:03:30.217Z] loading config from `taskcluster/ci/config.yml`
[task 2020-09-16T09:03:30.304Z] validating the graph config.
[task 2020-09-16T09:03:30.417Z] Retreving the full task graph and labels.
[task 2020-09-16T09:03:35.560Z] Load taskgraph from JSON.
[task 2020-09-16T09:03:36.132Z] Fetching additional tasks from action and cron tasks.
[task 2020-09-16T09:03:36.289Z] Traceback (most recent call last):
[task 2020-09-16T09:03:36.289Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 267, in action_callback
[task 2020-09-16T09:03:36.289Z]     test=False)
[task 2020-09-16T09:03:36.289Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/registry.py", line 317, in trigger_action_callback
[task 2020-09-16T09:03:36.289Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2020-09-16T09:03:36.289Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/backfill.py", line 247, in add_task_with_original_manifests
[task 2020-09-16T09:03:36.289Z]     label = new_label(label, full_task_graph.tasks)
[task 2020-09-16T09:03:36.289Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/backfill.py", line 181, in new_label
[task 2020-09-16T09:03:36.289Z]     raise Exception('New label ({}) was not found in the task-graph'.format(label))
[task 2020-09-16T09:03:36.289Z] Exception: New label (test-linux1804-64-qr/debug-mochitest-media-fis-e10s-1) was not found in the task-graph
[taskcluster 2020-09-16 09:03:37.065Z] === Task Finished ===
[taskcluster 2020-09-16 09:03:37.217Z] Artifact "public/docker-contexts" not found at "/builds/worker/checkouts/gecko/docker-contexts"
[taskcluster 2020-09-16 09:03:37.321Z] Unsuccessful task run with exit code: 1 completed in 21.361 seconds```

Comment 1

[Andrew Halberstadt [:ahal]]

It looks like the push in question didn't have any "mochitest-media" tasks in its full-task-graph, I guess because bugbug didn't tell it to select any test manifests. The backfill action depends on at least an unchunked or chunk-1 task existing in the full taskgraph.

I think the solution here will be to ensure that we always have at least one task even if there are no manifests to run in the test transforms. Then to filter out any tasks that don't have test manifests at the target tasks phase.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1665312 - Ensure a test task is never completely removed from full_task_graph due to test_manifest_loader, r?#taskgraph-reviewers

This causes them to get filtered out at the target tasks phase instead. We need
to do this since backfilling depends on the task existing in the full task
graph.

Comment 3

[Andrew Halberstadt [:ahal]]

I believe this patch should fix the backfill issue, but I don't know for sure as I'm not sure how to test it on try.

I was able to confirm that without this patch, mochitest-media does not exist in the full taskgraph, and with this patch it does. I also confirmed that the backfill action is using the full-task-graph to find the task definition, and then adjusting the test-manifests key. So I'm pretty sure this will work.

Comment 4

[Intermittent Failures Robot]

8 failures in 3760 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform breakdown:

• gecko-decision: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1665312&startday=2020-09-14&endday=2020-09-20&tree=all

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/38124b70ea29
Ensure a test task is never completely removed from full_task_graph due to test_manifest_loader, r=taskgraph-reviewers,aki

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/38124b70ea29