1713582 - Permanent busted decision and cron tasks: Exception: Build job build-macosx64-ccov/opt has no tests, but specifies MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment. Unset MOZ_AUTOMATION_PACKAGE_TESTS in the task definition to fix.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=341260460&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RaJueGH5RZ2jwm9lB32xCg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-05-31T09:58:27.102Z] Generating full task graph
[task 2021-05-31T09:58:27.206Z] Full task graph contains 24331 tasks and 106966 dependencies
[task 2021-05-31T09:58:37.070Z] Traceback (most recent call last):
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 354, in taskgraph_decision
[task 2021-05-31T09:58:37.070Z]     ret = taskgraph.decision.taskgraph_decision(options)
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 238, in taskgraph_decision
[task 2021-05-31T09:58:37.070Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 174, in full_task_graph
[task 2021-05-31T09:58:37.070Z]     return self._run_until("full_task_graph")
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 435, in _run_until
[task 2021-05-31T09:58:37.070Z]     k, v = next(self._run)
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 341, in _run
[task 2021-05-31T09:58:37.070Z]     "full_task_graph", full_task_graph, graph_config, parameters
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/verify.py", line 61, in __call__
[task 2021-05-31T09:58:37.070Z]     parameters=parameters,
[task 2021-05-31T09:58:37.070Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/verify.py", line 409, in verify_test_packaging
[task 2021-05-31T09:58:37.070Z]     raise Exception("\n".join(exceptions))
[task 2021-05-31T09:58:37.070Z] Exception: Build job build-macosx64-ccov/opt has no tests, but specifies MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment. Unset MOZ_AUTOMATION_PACKAGE_TESTS in the task definition to fix.

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The decision task and cron tasks like coverity are broken due to this. This prevents further merges both from autoland to mozilla-central and from mozilla-central to mozilla-beta (today is merge day).

The changes in the latest merge look unrelated and the mentioned macOS code coverage task has test artifacts.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/fafcc4a3b16a
Skip verification of update packaging for macOS code coverage (ccov) builds. a=decision-task-fix

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: WIP: Bug 1713582 - Skip verification of update packaging for macOS code coverage (ccov) builds. a=decision-task-fix

The decision task started to complain about the macOS ccov build setting up the
environment to export tests but not doing so. Previous tasks exported tests and
there are no recent changes altering this.

Comment 4

[Andrew Halberstadt [:ahal]]

Looks like the only tests that are scheduled to depend on that build are test-verify tasks. Those tasks can not exist in the full taskgraph depending on which files are modified. So whenever this situation arises this build will hit that exception. In light of this, I think the landed patch is actually the correct solution and not a hack at all.

There are two issues however:

1. Marco thought that the test tasks for macos ccov existed but weren't being run. This does not appear to be the case, they don't even exist. Maybe we could fix these to exist but not run anywhere.
2. TV tasks should ideally always be present in the full taskgraph and if the "perfile" logic says not to run them, we should filter them out at the optimization phase.

But both of these issues warrant separate bugs and are tangential to this one.

Comment 5

[Andrew Halberstadt [:ahal]]

Actually, jmaher has been migrating OSX tests from 10.14 -> 10.15, so this might have regressed 1) and we didn't notice since they weren't running anywhere anyway.

Comment 6

[Andrew Halberstadt [:ahal]]

Technically this was regressed by bug 1713329 (so adding for posterity), but in reality there was a confluence of factors that made this possible. It would have been very difficult to catch this edge case ahead of time. Some more detailed explanation here.

Comment 7

[Intermittent Failures Robot]

1 failures in 3065 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1713582&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 8

[Julien Cristau [:jcristau] (back on 2026-01-06)]

I guess we can call this fixed by comment 2?