1999031 - Geckoprofile action task broken due to multiple failures

Status: RESOLVED FIXED

(Testing :: Performance, defect, P1)

Description

[Greg Mierzwinski [:sparky]]

The geckoprofile action task is currently broken due to multiple failures. The most recent one is this one: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&searchStr=gecko&revision=1c9634718ce17d30ba459e4bfae1997c1c039759&selectedTaskRun=N830ZW31TgCUw_G_mE8hAw.0

[task 2025-11-07T20:27:39.440Z] Traceback (most recent call last):
[task 2025-11-07T20:27:39.440Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/main.py", line 554, in action_callback
[task 2025-11-07T20:27:39.440Z]     return trigger_action_callback(
[task 2025-11-07T20:27:39.440Z]            ^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:27:39.440Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/actions/registry.py", line 342, in trigger_action_callback
[task 2025-11-07T20:27:39.440Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2025-11-07T20:27:39.440Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/actions/gecko_profile.py", line 106, in geckoprofile_action
[task 2025-11-07T20:27:39.440Z]     fetch_graph_and_labels(push_params, graph_config)
[task 2025-11-07T20:27:39.440Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/actions/util.py", line 159, in fetch_graph_and_labels
[task 2025-11-07T20:27:39.440Z]     decision_task_id = find_decision_task(parameters, graph_config)
[task 2025-11-07T20:27:39.440Z]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:27:39.440Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/util/taskgraph.py", line 19, in find_decision_task
[task 2025-11-07T20:27:39.440Z]     parameters["project"],
[task 2025-11-07T20:27:39.440Z]     ~~~~~~~~~~^^^^^^^^^^^
[task 2025-11-07T20:27:39.440Z] TypeError: '_io.BytesIO' object is not subscriptable

Before this one, there's an auth failure happening: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&searchStr=action%2Cgeckoprofile&revision=6e0dd1421c8affd4915c151970e6fe9e979c759b

[task 2025-11-07T20:29:26.492Z] http://taskcluster:80 "GET /api/index/v1/task/gecko.v2.mozilla-central.pushlog-id.43469.decision/artifacts/public%2Fparameters.yml HTTP/1.1" 401 561
[task 2025-11-07T20:29:26.492Z] Received HTTP Status:    401
[task 2025-11-07T20:29:26.492Z] Received HTTP Headers: {'Accept-Ranges': 'bytes', 'Access-Control-Allow-Headers': 'X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control', 'Access-Control-Allow-Methods': 'OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Max-Age': '900', 'Access-Control-Request-Method': '*', 'Cache-Control': 'no-store no-cache must-revalidate', 'Content-Length': '561', 'Content-Security-Policy': "report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';", 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Fri, 07 Nov 2025 20:29:26 GMT', 'Etag': 'W/"231-eWsgV2Me/ovnrCwMHhh3H1Q5kwQ"', 'Server': 'openresty', 'Strict-Transport-Security': 'max-age=31536000', 'Via': '1.1 google, 1.1 varnish, 1.1 varnish', 'Www-Authenticate': 'hawk', 'X-Cache': 'MISS, MISS', 'X-Cache-Hits': '0, 0', 'X-Content-Type-Options': 'nosniff', 'X-For-Request-Id': '2c88b2f7-becb-45d2-aea3-80f90b6ffb1b', 'X-For-Trace-Id': '2097537ddf4b43804d7b608a18a7ac22', 'X-Served-By': 'cache-chi-kigq8000061-CHI, cache-chi-kigq8000061-CHI', 'X-Taskcluster-Authorized-Scopes': '["assume:repo:hg.mozilla.org/mozilla-central:action:generic","queue:create-artifact:H-7HwVU0SgiwgjTM9IAEOg/0"]', 'X-Taskcluster-Endpoint': 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.pushlog-id.43469.decision/artifacts/public%2Fparameters.yml', 'X-Taskcluster-Proxy-Revision': '', 'X-Taskcluster-Proxy-Temp-Clientid': 'task-client/H-7HwVU0SgiwgjTM9IAEOg/0/on/us-central1-b/8665922999022774980/until/1762548456.209', 'X-Taskcluster-Proxy-Temp-Scopes': '["queue:reclaim-task:H-7HwVU0SgiwgjTM9IAEOg/0","queue:resolve-task:H-7HwVU0SgiwgjTM9IAEOg/0","queue:create-artifact:H-7HwVU0SgiwgjTM9IAEOg/0","assume:repo:hg.mozilla.org/mozilla-central:action:generic"]', 'X-Taskcluster-Proxy-Version': '38.0.5', 'X-Timer': 'S1762547366.396313,VS0,VE89'}
[task 2025-11-07T20:29:26.493Z] Traceback (most recent call last):
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster/taskcluster/client.py", line 503, in _makeHttpRequest
[task 2025-11-07T20:29:26.493Z]     response.raise_for_status()
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 1026, in raise_for_status
[task 2025-11-07T20:29:26.493Z]     raise HTTPError(http_error_msg, response=self)
[task 2025-11-07T20:29:26.493Z] requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://taskcluster/api/index/v1/task/gecko.v2.mozilla-central.pushlog-id.43469.decision/artifacts/public%2Fparameters.yml
[task 2025-11-07T20:29:26.493Z] 
[task 2025-11-07T20:29:26.493Z] During handling of the above exception, another exception occurred:
[task 2025-11-07T20:29:26.493Z] 
[task 2025-11-07T20:29:26.493Z] Traceback (most recent call last):
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/main.py", line 550, in action_callback
[task 2025-11-07T20:29:26.493Z]     return trigger_action_callback(
[task 2025-11-07T20:29:26.493Z]            ^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/actions/registry.py", line 342, in trigger_action_callback
[task 2025-11-07T20:29:26.493Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/actions/gecko_profile.py", line 102, in geckoprofile_action
[task 2025-11-07T20:29:26.493Z]     push_params = get_artifact_from_index(
[task 2025-11-07T20:29:26.493Z]                   ^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 242, in get_artifact_from_index
[task 2025-11-07T20:29:26.493Z]     response = index.findArtifactFromTask(index_path, artifact_path)
[task 2025-11-07T20:29:26.493Z]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster/taskcluster/generated/index.py", line 183, in findArtifactFromTask
[task 2025-11-07T20:29:26.493Z]     return self._makeApiCall(self.funcinfo["findArtifactFromTask"], *args, **kwargs)
[task 2025-11-07T20:29:26.493Z]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster/taskcluster/client.py", line 269, in _makeApiCall
[task 2025-11-07T20:29:26.493Z]     response = self._makeHttpRequest(entry['method'], _route, payload)
[task 2025-11-07T20:29:26.493Z]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-11-07T20:29:26.493Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster/taskcluster/client.py", line 531, in _makeHttpRequest
[task 2025-11-07T20:29:26.493Z]     raise exceptions.TaskclusterAuthFailure(
[task 2025-11-07T20:29:26.493Z] taskcluster.exceptions.TaskclusterAuthFailure: Cannot use two authentication schemes at once this request has both bewit in querystring and 'authorization' header
[task 2025-11-07T20:29:26.493Z] 
[task 2025-11-07T20:29:26.493Z] ---
[task 2025-11-07T20:29:26.493Z] 
[task 2025-11-07T20:29:26.493Z] * method:     getLatestArtifact
[task 2025-11-07T20:29:26.493Z] * errorCode:  AuthenticationFailed
[task 2025-11-07T20:29:26.493Z] * statusCode: 401
[task 2025-11-07T20:29:26.493Z] * time:       2025-11-07T20:29:26.459Z

Comment 1

[Greg Mierzwinski [:sparky]]

Retriggered a geckoprofile run on a push that was previously passing and it's failed now. Seems like there's an issue with infra at this point: https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=26a4b3bbf1f2411870927b34a12ecbd0b0f51bda&searchStr=geckoprofile&selectedTaskRun=YQAjn59LQIqBwCzw11DR_A.0

Comment 2

[Greg Mierzwinski [:sparky]]

:ahal, it looks like these failures started on your patch here: https://treeherder.mozilla.org/jobs?repo=autoland&tier=1%2C2%2C3&searchStr=geckoprofile&tochange=05a7f41cd3fced799e32a3502fe50dc664490321&fromchange=77cb0f20e63cf7b484909d82df13f7b2b72a4fbc

Could you look into this?

Comment 3

[Greg Mierzwinski [:sparky]]

Actually, maybe it's not that patch? For some reason, I'm getting passes on the mozilla-central push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&searchStr=geckoprofile&tochange=54e9a1e89e6e1c7989efda42d7d7839b561e9ea9&fromchange=426d0bb3c4514da69402bd11bac79f8bb749949f

Comment 4

[Greg Mierzwinski [:sparky]]

Oh nvm, it's pretty likely that patch. I see it was reverted in the Oct 23rd m-c push, but it stuck in the Oct 24th push.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1990567

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1990567

Comment 7

[Andrew Halberstadt [:ahal]]

Sorry, this slipped my mind.

What changed was we started using the official Taskcluster client for our Taskcluster related requests. As part of that, if TASKCLUSTER_PROXY_URL is set in the env, we just assume that the Taskcluster proxy is configured and working. I'm not sure why these tasks seemingly have TASKCLUSTER_PROXY_URL set, but then hit a 401 when they try to use it.

Could you try adding something like:

if "TASKCLUSTER_PROXY_URL" in os.environ:
    del os.environ["TASKCLUSTER_PROXY_URL"]

Before that call to get_artifact_from_index()? That should cause the request to use the regular unauthenticated url (which should work as long as the artifact is public).

Comment 8

[Andrew Halberstadt [:ahal]]

We may also want to understand why the proxy isn't working in this case, but I don't really have time to dig at the moment.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

:abhishekmadan, since you are the author of the regressor, bug 1990567, could you take a look?

For more information, please visit BugBot documentation.

Comment 10

[Abhishek Madan[:abhishekmadan]]

Yep will do !

Comment 11

[Markus Stange [:mstange]]

(Removing "TypeError: '_io.BytesIO' object is not subscriptable" from the title again, that part is bug 1999039)

Comment 12

[Abhishek Madan[:abhishekmadan]]

A patch landed for Bug 1999039. I am unable to reproduce the AuthenticationFailed issue though, is anyone able to reproduce this issue ?

Comment 13

[Greg Mierzwinski [:sparky]]

I think it's only an issue on mozilla-central/autoland kind of branches. I've been seeing passing runs on try or some reason. I see that the patch for bug 1999039 only just landed so I can try to reproduce it again on a mozilla-central task tomorrow.

Comment 14

[Greg Mierzwinski [:sparky]]

It's working now and the auth failure is gone too for some reason: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=gecko%2Cprofile&revision=66a2bb093b81686973b18f82f747f20ca020c487

Thanks for the fixes :abhisekmadan!

Comment 15

[Andrew Creskey [:acreskey]]

This is very helpful for bug 1997949 and thus bug 1994314 - thank you for the fixes.