1713497 - Perma bustage - Gecko decision task for cron jobs - raise ManifestError

Reporter

Description

•

3 years ago

[task 2021-05-30T10:02:23.757Z] Error loading tasks for kind test:
[task 2021-05-30T10:02:23.758Z] Traceback (most recent call last):
[task 2021-05-30T10:02:23.758Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 205, in taskgraph_decision
[task 2021-05-30T10:02:23.758Z] return taskgraph.decision.taskgraph_decision(options)
[task 2021-05-30T10:02:23.758Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 223, in taskgraph_decision
[task 2021-05-30T10:02:23.758Z] full_task_json = tgg.full_task_graph.to_json()
[task 2021-05-30T10:02:23.758Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 153, in full_task_graph
[task 2021-05-30T10:02:23.758Z] return self._run_until('full_task_graph')
[task 2021-05-30T10:02:23.758Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 359, in _run_until
[task 2021-05-30T10:02:23.759Z] k, v = next(self._run)
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 267, in _run
[task 2021-05-30T10:02:23.759Z] new_tasks = kind.load_tasks(parameters, list(all_tasks.values()))
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 79, in load_tasks
[task 2021-05-30T10:02:23.759Z] for task_dict in transforms(trans_config, inputs)]
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2017, in check_run_task_caches
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1935, in check_task_dependencies
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1923, in check_task_identifiers
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1906, in chain_of_trust
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1745, in build_task
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1736, in try_task_config_routes
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1726, in try_task_config_chemspill_prio
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1716, in try_task_config_env
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1681, in add_index_routes
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1486, in validate
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1463, in task_name_from_label
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1423, in set_defaults
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1402, in set_implementation
[task 2021-05-30T10:02:23.759Z] for task in tasks:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/init.py", line 313, in make_task_description
[task 2021-05-30T10:02:23.759Z] for job in jobs:
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/init.py", line 201, in use_fetches
[task 2021-05-30T10:02:23.759Z] for job in order_tasks(config, jobs):
[task 2021-05-30T10:02:23.759Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
[task 2021-05-30T10:02:23.760Z] pending = deque(tasks)
[task 2021-05-30T10:02:23.760Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/init.py", line 156, in set_label
[task 2021-05-30T10:02:23.760Z] for job in jobs:
[task 2021-05-30T10:02:23.760Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/init.py", line 140, in set_implementation
[task 2021-05-30T10:02:23.760Z] for job in jobs:
[task 2021-05-30T10:02:23.760Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/init.py", line 120, in rewrite_when_to_optimization
[task 2021-05-30T10:02:23.760Z] for job in jobs:
[task 2021-05-30T10:02:23.761Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/base.py", line 76, in call
[task 2021-05-30T10:02:23.761Z] for task in tasks:
[task 2021-05-30T10:02:23.761Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1678, in make_job_description
[task 2021-05-30T10:02:23.762Z] for task in tasks:
[task 2021-05-30T10:02:23.762Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1644, in set_schedules_components
[task 2021-05-30T10:02:23.762Z] for task in tasks:
[task 2021-05-30T10:02:23.762Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1589, in set_worker_type
[task 2021-05-30T10:02:23.763Z] for task in tasks:
[task 2021-05-30T10:02:23.763Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1579, in set_test_type
[task 2021-05-30T10:02:23.763Z] for task in tasks:
[task 2021-05-30T10:02:23.763Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1570, in set_tag
[task 2021-05-30T10:02:23.763Z] for task in tasks:
[task 2021-05-30T10:02:23.764Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1558, in set_profile
[task 2021-05-30T10:02:23.764Z] for task in tasks:
[task 2021-05-30T10:02:23.764Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1548, in set_retry_exit_status
[task 2021-05-30T10:02:23.764Z] for task in tasks:
[task 2021-05-30T10:02:23.764Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1537, in set_schedules_for_webrender_android
[task 2021-05-30T10:02:23.764Z] for task in tasks:
[task 2021-05-30T10:02:23.764Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1523, in enable_webrender
[task 2021-05-30T10:02:23.764Z] for task in tasks:
[task 2021-05-30T10:02:23.764Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1508, in allow_software_gl_layers
[task 2021-05-30T10:02:23.765Z] for task in tasks:
[task 2021-05-30T10:02:23.765Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1460, in split_chunks
[task 2021-05-30T10:02:23.765Z] for task in tasks:
[task 2021-05-30T10:02:23.765Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1427, in resolve_dynamic_chunks
[task 2021-05-30T10:02:23.765Z] for task in tasks:
[task 2021-05-30T10:02:23.765Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/tests.py", line 1412, in set_test_manifests
[task 2021-05-30T10:02:23.765Z] frozenset(mozinfo.items()),
[task 2021-05-30T10:02:23.765Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 1018, in method_call
[task 2021-05-30T10:02:23.765Z] cache[args] = self.func(instance, *args)
[task 2021-05-30T10:02:23.765Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/chunking.py", line 214, in get_manifests
[task 2021-05-30T10:02:23.766Z] tests = self.get_tests(flavor, subsuite)
[task 2021-05-30T10:02:23.766Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 1018, in method_call
[task 2021-05-30T10:02:23.766Z] cache[args] = self.func(instance, *args)
[task 2021-05-30T10:02:23.766Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/chunking.py", line 208, in get_tests
[task 2021-05-30T10:02:23.766Z] return list(resolver.resolve_tests(flavor=flavor, subsuite=subsuite))
[task 2021-05-30T10:02:23.766Z] File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 833, in resolve_tests
[task 2021-05-30T10:02:23.766Z] for test in self._resolve(**kwargs):
[task 2021-05-30T10:02:23.766Z] File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 605, in _resolve
[task 2021-05-30T10:02:23.767Z] self.add_wpt_manifest_data()
[task 2021-05-30T10:02:23.767Z] File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 732, in add_wpt_manifest_data
[task 2021-05-30T10:02:23.767Z] update=True, logger=logger)
[task 2021-05-30T10:02:23.767Z] File "/builds/worker/checkouts/gecko/testing/web-platform/manifestupdate.py", line 123, in run
[task 2021-05-30T10:02:23.767Z] cache_root=kwargs["cache_root"])
[task 2021-05-30T10:02:23.767Z] File "/builds/worker/checkouts/gecko/testing/web-platform/manifestupdate.py", line 192, in load_and_update
[task 2021-05-30T10:02:23.767Z] cache_root=this_cache_root)
[task 2021-05-30T10:02:23.767Z] File "/builds/worker/checkouts/gecko/testing/web-platform/tests/tools/manifest/manifest.py", line 376, in load_and_update
[task 2021-05-30T10:02:23.768Z] allow_cached=allow_cached)
[task 2021-05-30T10:02:23.768Z] File "/builds/worker/checkouts/gecko/testing/web-platform/tests/tools/manifest/manifest.py", line 336, in _load
[task 2021-05-30T10:02:23.768Z] callee_owns_obj=True)
[task 2021-05-30T10:02:23.768Z] File "/builds/worker/checkouts/gecko/testing/web-platform/tests/tools/manifest/manifest.py", line 290, in from_json
[task 2021-05-30T10:02:23.768Z] raise ManifestError
[task 2021-05-30T10:02:23.768Z] ManifestError

Flags: needinfo?(james)

Natalia Csoregi [:nataliaCs]

Updated

•

3 years ago

Summary: Perma bustage - Gecko descision task for cron job searchfox -index → Perma bustage - Gecko decision task for cron jobs - raise ManifestError

Comment hidden (Intermittent Failures Robot)

Julien Cristau [:jcristau]

Assignee

Comment 2

•

3 years ago

Looks like this will block shipping 78.11.

status-firefox-esr78: --- → affected

tracking-firefox-esr78: --- → 89+

Julien Cristau [:jcristau]

Assignee

Comment 3

•

3 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-esr78/rev/01d6500ecb25

status-firefox-esr78: affected → fixed

Julien Cristau [:jcristau]

Assignee

Comment 4

•

3 years ago

Comment 3 ought to unblock 78.11esr. There's still a question of is it ok for branches to use a manifest from m-c, and how come this started breaking yesterday.

Andrew Halberstadt [:ahal]

Comment 5

•

3 years ago

This will likely happen again until we fix it properly. I'd assume we should be pulling that file from the repo where the task is running and not always from central, though hopefully James can clarify.

James Graham [:jgraham]

Comment 6

•

3 years ago

So it's supposed to use a recent version of the manifest. Presumably in this case the format changed and instead of doing something sensible like rebuilding from scratch (which is pretty slow) it just failed. So that's one thing we could fix. It would also make sense to use the head of the same repo, but I don't know how one is supposed to know that (in general or on CI in particular). Do we have existing code for that, or is there an environment variable that's set in CI, or something?

Flags: needinfo?(james)

Aki Sasaki (not active)

Comment 7

•

3 years ago

We have "GECKO_HEAD_REPOSITORY": "https://hg.mozilla.org/releases/mozilla-esr78" in the environment; is that sufficient?

Comment hidden (Intermittent Failures Robot)

Ryan VanderMeulen [:RyanVM]

Updated

•

3 years ago

Comment 9

•

2 years ago

Should we do this for ESR91 too?

Flags: needinfo?(jcristau)

Julien Cristau [:jcristau]

Assignee

Comment 10

•

2 years ago

Attached file Bug 1713497 - look for wpt manifests in the esr91 index as well. r?jgraham — Details

Phabricator Automation

Updated

•

2 years ago

Assignee: nobody → jcristau

Status: NEW → ASSIGNED

Julien Cristau [:jcristau]

Assignee

Updated

•

2 years ago

Assignee: jcristau → nobody

Status: ASSIGNED → NEW

Flags: needinfo?(jcristau)

Keywords: leave-open

Phabricator Automation

Updated

•

2 years ago

Assignee: nobody → jcristau

Status: NEW → ASSIGNED

BugBot [:suhaib / :marco/ :calixte]

Comment 11

•

2 years ago

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:jcristau, could you have a look please?
For more information, please visit auto_nag documentation.

Flags: needinfo?(jcristau)

Flags: needinfo?(james)

Pulsebot

Comment 12

•

2 years ago

Pushed by jcristau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3d3287b010d1
look for wpt manifests in the esr91 index as well. r=jgraham

Julien Cristau [:jcristau]

Assignee

Comment 13

•

2 years ago

•

Edited

Comment on attachment 9253910 [details]
Bug 1713497 - look for wpt manifests in the esr91 index as well. r?jgraham

ESR Uplift Approval Request

If this is not a sec:{high,crit} bug, please state case for ESR consideration: latent esr91 test bustage when the wpt manifest on central gets updated
User impact if declined: none
Fix Landed on Version: 97
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): test-only change, hoping to avoid a situation down the road where an update on central breaks tests on esr91

Flags: needinfo?(jcristau)

Attachment #9253910 - Flags: approval-mozilla-esr91?

Julien Cristau [:jcristau]

Assignee

Updated

•

2 years ago

Flags: needinfo?(james)

Sandor Molnar[:smolnar]

Comment 14

•

2 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/3d3287b010d1

Julien Cristau [:jcristau]

Assignee

Updated

•

2 years ago

Status: ASSIGNED → RESOLVED

Closed: 2 years ago

status-firefox97: --- → fixed

Keywords: leave-open

Resolution: --- → FIXED

Target Milestone: --- → 97 Branch

Ryan VanderMeulen [:RyanVM]

Comment 15

•

2 years ago

Comment on attachment 9253910 [details]
Bug 1713497 - look for wpt manifests in the esr91 index as well. r?jgraham

Approved for ESR91, thanks.

Attachment #9253910 - Flags: approval-mozilla-esr91? → approval-mozilla-esr91+

Ryan VanderMeulen [:RyanVM]

Updated

•

2 years ago

status-firefox-esr91: --- → affected

tracking-firefox-esr91: --- → 96+

Ryan VanderMeulen [:RyanVM]

Comment 16

•

2 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-esr91/rev/047e4a66f9dc

status-firefox-esr91: affected → fixed