Closed Bug 1515188 Opened 5 years ago Closed 5 years ago

404's for a taskgroup list that does exist

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1492664

People

(Reporter: dustin, Assigned: dustin)

References

Details

Attachments

(1 obsolete file)

https://tools.taskcluster.net/groups/a-o8xVkHRL68E1SrFSl5XQ/tasks/V1VVk8mhT5iiY4vLb23IEw/runs/0/logs/public%2Flogs%2Flive.log
[task 2018-12-18T21:19:01.235Z] Traceback (most recent call last):
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 255, in action_callback
[task 2018-12-18T21:19:01.235Z]     test=False)
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/registry.py", line 366, in trigger_action_callback
[task 2018-12-18T21:19:01.235Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id, task)
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/cancel_all.py", line 40, in cancel_all_action
[task 2018-12-18T21:19:01.235Z]     for t in list_task_group(task_group_id) if t != own_task_id
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 260, in list_task_group
[task 2018-12-18T21:19:01.235Z]     resp = _do_request(url, params=params).json()
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 47, in _do_request
[task 2018-12-18T21:19:01.235Z]     response.raise_for_status()
[task 2018-12-18T21:19:01.235Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2018-12-18T21:19:01.236Z]     raise HTTPError(http_error_msg, response=self)
[task 2018-12-18T21:19:01.236Z] HTTPError: 404 Client Error: Not Found for url: https://queue.taskcluster.net/v1/task-group/a-o8xVkHRL68E1SrFSl5XQ/list
[taskcluster 2018-12-18 21:19:01.498Z] === Task Finished ===

the decision task for that taskgroup ran at 20:04, so the taskgroup definitely existed at 21:19
  https://tools.taskcluster.net/groups/a-o8xVkHRL68E1SrFSl5XQ/tasks/a-o8xVkHRL68E1SrFSl5XQ/runs/0

In fact, from the queue logs, the decision task description was fetched just before that time:

Dec 18 21:18:39 taskcluster-queue heroku/router: at=info method=GET path="/v1/task/a-o8xVkHRL68E1SrFSl5XQ" host=queue.taskcluster.net request_id=5b5ff2c8-e2eb-437f-90fe-b0b16c74a94e fwd="24.141.16.253" dyno=web.18 connect=0ms service=153ms status=200 bytes=4892 protocol=https 
Dec 18 21:19:01 taskcluster-queue heroku/router: at=info method=POST path="/v1/task-group/a-o8xVkHRL68E1SrFSl5XQ/list" host=queue.taskcluster.net request_id=9a394316-49ea-4a5b-b9be-279bc3f15975 fwd="35.166.84.64" dyno=web.22 connect=1ms service=524ms status=404 bytes=728 protocol=https 

And another 404 occurred soon after:

Dec 18 21:30:02 taskcluster-queue heroku/router: at=info method=POST path="/v1/task-group/a-o8xVkHRL68E1SrFSl5XQ/list" host=queue.taskcluster.net request_id=a3a5f121-afda-4cd2-902d-956b99a829fb fwd="18.234.190.217" dyno=web.8 connect=0ms service=26ms status=404 bytes=728 protocol=https 

but one succeeded a few minutes later (with a limit):

Dec 18 21:31:32 taskcluster-queue heroku/router: at=info method=GET path="/v1/task-group/a-o8xVkHRL68E1SrFSl5XQ/list?limit=20" host=queue.taskcluster.net request_id=e09a5e82-1042-4370-a828-d2147e03e2a8 fwd="24.141.16.253" dyno=web.2 connect=0ms service=464ms status=200 bytes=114246 protocol=https 

That 404 can only have come from
  https://github.com/taskcluster/taskcluster-queue/blob/1e47a960a34cd36322900cad5bfc2834fcea7b8f/src/api.js#L250
meaning the preceding call to this.TaskGroup.load({taskGroupId}, true) failed.  Presence or absence of a limit seems unrelated.

But that row is created by ensureTaskGroup, which is called in createTask and defineTask, and thus should have occurred when the decision task was created (or any of the other tasks in the taskGroup).

As of right now, https://queue.taskcluster.net/v1/task-group/a-o8xVkHRL68E1SrFSl5XQ/list returns a big old list of tasks.

I wonder if this was some Azure error?
From bug 1515246:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217825301&repo=try&lineNumber=151
[task 2018-12-19T03:24:29.059Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2018-12-19T03:24:29.059Z]     raise HTTPError(http_error_msg, response=self)
[task 2018-12-19T03:24:29.059Z] HTTPError: 404 Client Error: Not Found for url: https://queue.taskcluster.net/v1/task-group/EXzwF3FfRMqMwfn02hmZvw/list

I wonder if this is a bug in the cancel_all action -- maybe some whitespace snuck into that URL?
Indeed, I can reproduce:

>>> import os
>>> os.environ['TASKCLUSTER_ROOT_URL'] = 'https://taskcluster.net'
>>> from taskgraph.util.taskcluster import list_task_group
>>> list(list_task_group('EXzwF3FfRMqMwfn02hmZvw'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dustin/p/m-c/taskcluster/taskgraph/util/taskcluster.py", line 260, in list_task_group
    resp = _do_request(url, params=params).json()
  File "/home/dustin/p/m-c/taskcluster/taskgraph/util/taskcluster.py", line 47, in _do_request
    response.raise_for_status()
  File "/home/dustin/p/m-c/third_party/python/requests/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://queue.taskcluster.net/v1/task-group/EXzwF3FfRMqMwfn02hmZvw/list
Component: Queue → Task Configuration
Product: Taskcluster → Firefox Build System
QA Contact: jhford
Assignee: nobody → dustin
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/20bedd28b46d
fix list_task_group to use GET r=Callek
OK, bug 1492664 got backed out in the interim.  I'll pile this patch on top of it :)
Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(dustin)
Resolution: --- → DUPLICATE
Attachment #9032333 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: