1904784 - find_task_id_batched(): Intermittent ConnectionResetError: [Errno 104] Connection reset by peer

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Johan Lorenzo [:jlorenzo]]

Attachment: Perfherder - ConnectionResetError on Gecko Decision Task

Discovered by :CosminS in bug 1658531 comment 181. When looking at the past 6 months[1], we had literally 0 ConnectionResetError: [Errno 104] Connection reset by peer between January and May on the Gecko Decision Tasks. Starting June, we got 14 errors (see screenshot). All of these errors show the same stacktrace:

[task 2024-06-25T22:26:15.453Z] Traceback (most recent call last):
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 714, in urlopen
[task 2024-06-25T22:26:15.453Z]     httplib_response = self._make_request(
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 466, in _make_request
[task 2024-06-25T22:26:15.453Z]     six.raise_from(e, None)
[task 2024-06-25T22:26:15.453Z]   File "<string>", line 3, in raise_from
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 461, in _make_request
[task 2024-06-25T22:26:15.453Z]     httplib_response = conn.getresponse()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
[task 2024-06-25T22:26:15.453Z]     response.begin()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 316, in begin
[task 2024-06-25T22:26:15.453Z]     version, status, reason = self._read_status()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
[task 2024-06-25T22:26:15.453Z]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/socket.py", line 669, in readinto
[task 2024-06-25T22:26:15.453Z]     return self._sock.recv_into(b)
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
[task 2024-06-25T22:26:15.453Z]     return self.read(nbytes, buffer)
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/ssl.py", line 1099, in read
[task 2024-06-25T22:26:15.453Z]     return self._sslobj.read(len, buffer)
[task 2024-06-25T22:26:15.453Z] ConnectionResetError: [Errno 104] Connection reset by peer
[task 2024-06-25T22:26:15.453Z] 
[task 2024-06-25T22:26:15.453Z] During handling of the above exception, another exception occurred:
[task 2024-06-25T22:26:15.453Z] 
[task 2024-06-25T22:26:15.453Z] Traceback (most recent call last):
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 486, in send
[task 2024-06-25T22:26:15.453Z]     resp = conn.urlopen(
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 798, in urlopen
[task 2024-06-25T22:26:15.453Z]     retries = retries.increment(
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/util/retry.py", line 550, in increment
[task 2024-06-25T22:26:15.453Z]     raise six.reraise(type(error), error, _stacktrace)
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/packages/six.py", line 769, in reraise
[task 2024-06-25T22:26:15.453Z]     raise value.with_traceback(tb)
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 714, in urlopen
[task 2024-06-25T22:26:15.453Z]     httplib_response = self._make_request(
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 466, in _make_request
[task 2024-06-25T22:26:15.453Z]     six.raise_from(e, None)
[task 2024-06-25T22:26:15.453Z]   File "<string>", line 3, in raise_from
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/third_party/python/urllib3/urllib3/connectionpool.py", line 461, in _make_request
[task 2024-06-25T22:26:15.453Z]     httplib_response = conn.getresponse()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
[task 2024-06-25T22:26:15.453Z]     response.begin()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 316, in begin
[task 2024-06-25T22:26:15.453Z]     version, status, reason = self._read_status()
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
[task 2024-06-25T22:26:15.453Z]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/socket.py", line 669, in readinto
[task 2024-06-25T22:26:15.453Z]     return self._sock.recv_into(b)
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
[task 2024-06-25T22:26:15.453Z]     return self.read(nbytes, buffer)
[task 2024-06-25T22:26:15.453Z]   File "/usr/lib/python3.8/ssl.py", line 1099, in read
[task 2024-06-25T22:26:15.453Z]     return self._sslobj.read(len, buffer)
[task 2024-06-25T22:26:15.453Z] urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[task 2024-06-25T22:26:15.453Z] 
[task 2024-06-25T22:26:15.453Z] During handling of the above exception, another exception occurred:
[task 2024-06-25T22:26:15.453Z] 
[task 2024-06-25T22:26:15.453Z] Traceback (most recent call last):
[task 2024-06-25T22:26:15.453Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 209, in taskgraph_decision
[task 2024-06-25T22:26:15.453Z]     ret = taskgraph_commands["decision"].func(options)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/main.py", line 703, in decision
[task 2024-06-25T22:26:15.454Z]     taskgraph_decision(options)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/decision.py", line 227, in taskgraph_decision
[task 2024-06-25T22:26:15.454Z]     write_artifact("task-graph.json", tgg.morphed_task_graph.to_json())
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 220, in morphed_task_graph
[task 2024-06-25T22:26:15.454Z]     return self._run_until("morphed_task_graph")
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 425, in _run_until
[task 2024-06-25T22:26:15.454Z]     k, v = next(self._run)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 399, in _run
[task 2024-06-25T22:26:15.454Z]     optimized_task_graph, label_to_taskid = optimize_task_graph(
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/optimize/base.py", line 91, in optimize_task_graph
[task 2024-06-25T22:26:15.454Z]     index_to_taskid = find_task_id_batched(indexes)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 214, in find_task_id_batched
[task 2024-06-25T22:26:15.454Z]     response = _do_request(
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 121, in _do_request
[task 2024-06-25T22:26:15.454Z]     response = getattr(session, method)(url, **kwargs)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 637, in post
[task 2024-06-25T22:26:15.454Z]     return self.request("POST", url, data=data, json=json, **kwargs)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 589, in request
[task 2024-06-25T22:26:15.454Z]     resp = self.send(prep, **send_kwargs)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 703, in send
[task 2024-06-25T22:26:15.454Z]     r = adapter.send(request, **kwargs)
[task 2024-06-25T22:26:15.454Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 501, in send
[task 2024-06-25T22:26:15.454Z]     raise ConnectionError(err, request=request)
[task 2024-06-25T22:26:15.454Z] requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

find_task_id_batched()[2] was added in bug 1840830. I think we should add a retry mechanism in this function. What do you think, :ahal?

[1] https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1658531&startday=2024-01-01&endday=2024-06-26&tree=all
[2] https://searchfox.org/mozilla-central/rev/2f48061aef8c8976b73749ee845e7b85751f5f2f/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py#195

Comment 1

[Julien Cristau [:jcristau]]

Bug 1903695 is another case that's affected by this new POST request not being retried.

Comment 2

[Julien Cristau [:jcristau]]

Attachment: [taskcluster/taskgraph] Bug 1904784 - retry batched taskcluster requests (#529)

Comment 3

[Andrew Halberstadt [:ahal]]

Thanks Julien!

Comment 4

[BugBot [:suhaib / :marco]]

:alphare33, since you are the author of the regressor, bug 1840830, could you take a look?

For more information, please visit BugBot documentation.

Comment 5

[Julien Cristau [:jcristau]]

Fix vendored into m-c in bug 1909260.