2018744 - Perma esr115/esr140 ci job taskcluster/test/test_generate_params.py::test_generate_graphs[autoland-cron-l10n-bumper] TEST-UNEXPECTED-FAIL

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=550252264&repo=mozilla-esr140&task=amUdr62YTuu3zqc7T1_H3w.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/amUdr62YTuu3zqc7T1_H3w/runs/0/artifacts/public/logs/live_backing.log

---

[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-ship-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[me-push-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-ship-devedition] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[autoland-cron-l10n-bumper] TEST-UNEXPECTED-FAIL
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-onpush] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-promote-firefox-partials] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[me-promote-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-push-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mc-desktop-nightly] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-push-firefox-partials] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-promote-firefox-rc] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mc-android-nightly] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-push-android] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[try-syntax] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-promote-devedition] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mc-nightly-all] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-push-android] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-onpush] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[me-ship-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-promote-devedition-partials] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-ship-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-promote-firefox] PASSED
[task 2026-02-23T17:13:39.462Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-onpush-geckoview] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[try-config] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-promote-firefox] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[autoland-onpush] TEST-UNEXPECTED-FAIL
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-ship-android] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mb-promote-android] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mc-cron-os-integration] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mr-ship-firefox-rc] PASSED
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py::test_generate_graphs[mc-cron-system-symbols] PASSED
[task 2026-02-23T17:13:39.463Z] 
[task 2026-02-23T17:13:39.463Z] =================================== FAILURES ===================================
[task 2026-02-23T17:13:39.463Z] _______________ test_generate_graphs[autoland-cron-l10n-bumper] ________________
[task 2026-02-23T17:13:39.463Z] taskcluster/test/test_generate_params.py:55: in test_generate_graphs
[task 2026-02-23T17:13:39.463Z]     pytest.fail("An exception was raised during graph generation!")
[task 2026-02-23T17:13:39.463Z] E   Failed: An exception was raised during graph generation!
[task 2026-02-23T17:13:39.463Z] ----------------------------- Captured stdout call -----------------------------
[task 2026-02-23T17:13:39.463Z] concurrent.futures.process._RemoteTraceback:
[task 2026-02-23T17:13:39.463Z] """
[task 2026-02-23T17:13:39.463Z] Traceback (most recent call last):
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 667, in send
[task 2026-02-23T17:13:39.463Z]     resp = conn.urlopen(
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 896, in urlopen
[task 2026-02-23T17:13:39.463Z]     return self.urlopen(
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 896, in urlopen
[task 2026-02-23T17:13:39.463Z]     return self.urlopen(
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 896, in urlopen
[task 2026-02-23T17:13:39.463Z]     return self.urlopen(
[task 2026-02-23T17:13:39.463Z]   [Previous line repeated 2 more times]
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 886, in urlopen
[task 2026-02-23T17:13:39.463Z]     retries = retries.increment(method, url, response=response, _pool=self)
[task 2026-02-23T17:13:39.463Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/util/retry.py", line 594, in increment
[task 2026-02-23T17:13:39.464Z]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
[task 2026-02-23T17:13:39.464Z] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='bugbug.herokuapp.com', port=443): Max retries exceeded with url: /push/autoland/00327c4fd62255eb3ab07a275d90ad09cce51ac3/schedules (Caused by ResponseError('too many 503 error responses'))
[task 2026-02-23T17:13:39.464Z] 
[task 2026-02-23T17:13:39.464Z] During handling of the above exception, another exception occurred:
[task 2026-02-23T17:13:39.464Z] 
[task 2026-02-23T17:13:39.464Z] Traceback (most recent call last):
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/fetches/python/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
[task 2026-02-23T17:13:39.464Z]     r = call_item.fn(*call_item.args, **call_item.kwargs)
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/main.py", line 149, in format_taskgraph
[task 2026-02-23T17:13:39.464Z]     tg = getattr(tgg, options["graph_attr"])
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 220, in morphed_task_graph
[task 2026-02-23T17:13:39.464Z]     return self._run_until("morphed_task_graph")
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 427, in _run_until
[task 2026-02-23T17:13:39.464Z]     k, v = next(self._run)  # type: ignore
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 311, in _run
[task 2026-02-23T17:13:39.464Z]     new_tasks = kind.load_tasks(
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in load_tasks
[task 2026-02-23T17:13:39.464Z]     tasks = [
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in <listcomp>
[task 2026-02-23T17:13:39.464Z]     tasks = [
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2584, in check_run_task_caches
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2494, in check_task_dependencies
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2480, in check_task_identifiers
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2461, in chain_of_trust
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2249, in build_task
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2185, in set_task_and_artifact_expiry
[task 2026-02-23T17:13:39.464Z]     for job in jobs:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2158, in try_task_config_routes
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2145, in try_task_config_chemspill_prio
[task 2026-02-23T17:13:39.464Z]     yield from tasks
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2125, in try_task_config_env
[task 2026-02-23T17:13:39.464Z]     yield from tasks
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2091, in add_index_routes
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 1870, in validate
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 1848, in task_name_from_label
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.464Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 1838, in setup_raptor
[task 2026-02-23T17:13:39.464Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 1787, in set_defaults
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 1744, in set_implementation
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 264, in use_fetches
[task 2026-02-23T17:13:39.465Z]     for job in order_tasks(config, jobs):
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/cached_tasks.py", line 20, in order_tasks
[task 2026-02-23T17:13:39.465Z]     pending = deque(tasks)
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 189, in use_system_python
[task 2026-02-23T17:13:39.465Z]     for job in jobs:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 152, in make_task_description
[task 2026-02-23T17:13:39.465Z]     for job in jobs:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 136, in set_label
[task 2026-02-23T17:13:39.465Z]     for job in jobs:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 118, in set_implementation
[task 2026-02-23T17:13:39.465Z]     for job in jobs:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/transforms/run/__init__.py", line 115, in rewrite_when_to_optimization
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/transforms/base.py", line 144, in __call__
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/__init__.py", line 464, in make_job_description
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/__init__.py", line 448, in define_tags
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/__init__.py", line 443, in run_remaining_transforms
[task 2026-02-23T17:13:39.465Z]     yield from xforms(config, [task])
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/chunk.py", line 254, in split_chunks
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/chunk.py", line 198, in resolve_dynamic_chunks
[task 2026-02-23T17:13:39.465Z]     for task in tasks:
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/test/chunk.py", line 107, in set_test_manifests
[task 2026-02-23T17:13:39.465Z]     task["test-manifests"] = loader.get_manifests(
[task 2026-02-23T17:13:39.465Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 931, in method_call
[task 2026-02-23T17:13:39.465Z]     cache[args] = self.func(instance, *args)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/util/chunking.py", line 341, in get_manifests
[task 2026-02-23T17:13:39.466Z]     data = push_schedules(self.params["project"], self.params["head_rev"])
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 922, in __call__
[task 2026-02-23T17:13:39.466Z]     self[args] = self.func(*args)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/util/bugbug.py", line 94, in push_schedules
[task 2026-02-23T17:13:39.466Z]     r = session.get(url)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 602, in get
[task 2026-02-23T17:13:39.466Z]     return self.request("GET", url, **kwargs)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 589, in request
[task 2026-02-23T17:13:39.466Z]     resp = self.send(prep, **send_kwargs)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 703, in send
[task 2026-02-23T17:13:39.466Z]     r = adapter.send(request, **kwargs)
[task 2026-02-23T17:13:39.466Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 691, in send
[task 2026-02-23T17:13:39.466Z]     raise RetryError(e, request=request)
[task 2026-02-23T17:13:39.466Z] requests.exceptions.RetryError: HTTPSConnectionPool(host='bugbug.herokuapp.com', port=443): Max retries exceeded with url: /push/autoland/00327c4fd62255eb3ab07a275d90ad09cce51ac3/schedules (Caused by ResponseError('too many 503 error responses'))
[task 2026-02-23T17:13:39.466Z] """
[task 2026-02-23T17:13:39.466Z] 
[task 2026-02-23T17:13:39.466Z] The above exception was the direct cause of the following exception:
[task 2026-02-23T17:13:39.466Z] 
[task 2026-02-23T17:13:39.466Z] requests.exceptions.RetryError: None: Max retries exceeded with url: /push/autoland/00327c4fd62255eb3ab07a275d90ad09cce51ac3/schedules (Caused by None)

Comment 1

[Cosmin Sabou [:CosminS]]

Started with this land https://bugzilla.mozilla.org/show_bug.cgi?id=2018654#c3 and fails also where it was green before.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

:eijebong, since you are the author of the regressor, bug 2018654, could you take a look?

For more information, please visit BugBot documentation.

Comment 3

[Julien Cristau [:jcristau]]

marco, esr140 is still pointing at the old bugbug.

Comment 4

[Marco Castelluccio [:marco]]

Can we change the URL in esr140 to avoid this problem? Or should we keep the Heroku bugbug up for the entire lifetime of 140?

Comment 5

[Julien Cristau [:jcristau]]

Up to you.

Comment 6

[Marco Castelluccio [:marco]]

Attachment: Bug 2018744 - Use new bugbug deployment for esr140. r=#taskgraph-reviewers

Comment 7

[Phabricator Automation]

firefox-esr140 Uplift Approval Request

• User impact if declined: No end user impact, only CI breakage
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: low
• Explanation of risk level: It doesn't affect end users in any way. The only risk is I missed something and CI breaks.
• String changes made/needed: N/A
• Is Android affected?: no

Comment 8

[Intermittent Failures Robot]

12 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr140: 12

Table

	no_variant
linux-x86/unknown build	12

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018744&startday=2026-02-23&endday=2026-03-01&tree=all

Comment 9

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/81b0071349fd

Comment 10

[Intermittent Failures Robot]

4 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 4

Table

	no_variant
linux-x86/unknown build	4

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018744&startday=2026-03-23&endday=2026-03-29&tree=all

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/e1c887ccd202