release-runner3 couldn't kick release off: Can't download actions.json

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
10 months ago
9 months ago

People

(Reporter: jlorenzo, Assigned: jlorenzo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

10 months ago
> Status: Failed to start release promotion. Error(s): HTTPSConnectionPool(host='taskcluster-artifacts.net', port=443): Max retries exceeded with url: /dTdpi_rFT5udi6kw3H6vLg/0/public/actions.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTP

It seems we hit a network error. A rerun fixed it.
(Assignee)

Comment 1

10 months ago
> 2018-07-04 02:36:44,155 - DEBUG - Found a positional argument: public/actions.json
> 2018-07-04 02:36:44,155 - DEBUG - After processing positional arguments, we have: {'name': 'public/actions.json', 'taskId': u'dTdpi_rFT5udi6kw3H6vLg'}
> 2018-07-04 02:36:44,155 - DEBUG - After keyword arguments, we have: {'name': 'public/actions.json', 'taskId': u'dTdpi_rFT5udi6kw3H6vLg'}
> 2018-07-04 02:36:44,156 - DEBUG - Starting new HTTPS connection (1): queue.taskcluster.net
> 2018-07-04 02:36:44,874 - DEBUG - https://queue.taskcluster.net:443 "GET /v1/task/dTdpi_rFT5udi6kw3H6vLg/artifacts/public%2Factions.json HTTP/1.1" 303 104
> 2018-07-04 02:36:44,876 - DEBUG - Starting new HTTPS connection (1): taskcluster-artifacts.net
> 2018-07-04 02:36:44,891 - INFO - mark as failed Firefox-61.0.1-build1
> 2018-07-04 02:36:44,892 - DEBUG - Resetting dropped connection: ship-it.mozilla.org
> 2018-07-04 02:36:44,921 - DEBUG - https://ship-it.mozilla.org:443 "HEAD /csrf_token HTTP/1.1" 200 0
> 2018-07-04 02:36:44,922 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-61.0.1-build1
> 2018-07-04 02:36:44,923 - DEBUG - Data sent: {'ready': False, 'status': "Failed to start release promotion. Error(s): HTTPSConnectionPool(host='taskcluster-artifacts.net', port=443): Max ret
> ries exceeded with url: /dTdpi_rFT5udi6kw3H6vLg/0/public/actions.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff612e001d0>: Failed to establis
> h a new connection: [Errno 111] Connection refused',))", 'csrf_token': '20180704100644##7a61171582b8aa76a0ee3ce54aef6263c5a9cee5'}
> 2018-07-04 02:36:44,923 - DEBUG - attempt 1/5
> 2018-07-04 02:36:44,923 - DEBUG - retry: Calling <function _req at 0x7ff612df28c0> with args: (), kwargs: {}, attempt #1
> 2018-07-04 02:36:44,924 - DEBUG - Starting new HTTPS connection (2): ship-it.mozilla.org
> 2018-07-04 02:36:44,979 - DEBUG - https://ship-it.mozilla.org:443 "POST /releases/Firefox-61.0.1-build1 HTTP/1.1" 200 0
> 2018-07-04 02:36:44,981 - ERROR - Failed to start release "Firefox-61.0.1-build1". Error(s): HTTPSConnectionPool(host='taskcluster-artifacts.net', port=443): Max retries exceeded with url: /
> dTdpi_rFT5udi6kw3H6vLg/0/public/actions.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff612e001d0>: Failed to establish a new connection: [Errn
> o 111] Connection refused',))
> Traceback (most recent call last):
>   File "release-runner3.py", line 255, in main
>     action_task_input=action_task_input,
>   File "/builds/releaserunner3/tools/lib/python/kickoff/actions.py", line 38, in generate_action_task
>     actions = fetch_actions_json(decision_task_id)
>   File "/builds/releaserunner3/tools/lib/python/kickoff/actions.py", line 23, in fetch_actions_json
>     q = requests.get(actions_url)
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/api.py", line 72, in get
>     return request('get', url, params=params, **kwargs)
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/api.py", line 58, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
>     resp = self.send(prep, **send_kwargs)
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/sessions.py", line 640, in send
>     history = [resp for resp in gen] if allow_redirects else []
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/sessions.py", line 218, in resolve_redirects
>     **adapter_kwargs
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
>     r = adapter.send(request, **kwargs)
>   File "/builds/releaserunner3/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
>     raise ConnectionError(e, request=request)
> ConnectionError: HTTPSConnectionPool(host='taskcluster-artifacts.net', port=443): Max retries exceeded with url: /dTdpi_rFT5udi6kw3H6vLg/0/public/actions.json (Caused by NewConnectionError('
> <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff612e001d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Even though the error message reads it reached the max numbers of retries, I only see 1 try. Rail, do you think all the tries were done?
Flags: needinfo?(rail)
(Assignee)

Updated

10 months ago
Summary: release-runner3 couldn't kick release off: Can't download action task → release-runner3 couldn't kick release off: Can't download actions.json

Comment 2

10 months ago
I don't think we retry in https://hg.mozilla.org/build/tools/file/default/lib/python/kickoff/actions.py#l23. should be easy to use redo there or wrap it in the caller https://hg.mozilla.org/build/tools/file/default/lib/python/kickoff/actions.py#l38
Flags: needinfo?(rail)
(Assignee)

Comment 3

10 months ago
Retry fetch {action,decision} task
(Assignee)

Updated

10 months ago
Assignee: nobody → jlorenzo
Comment on attachment 8990333 [details]
Bug 1473288 - Retry fetch {action,decision} task r=rail

Rail Aliiev [:rail] ⌚️ET has approved the revision.

https://phabricator.services.mozilla.com/D2003
Attachment #8990333 - Flags: review+
(Assignee)

Comment 5

9 months ago
Phabricator didn't close this bug. It was landed at https://hg.mozilla.org/build/tools/rev/1b273d683d4e21bf603c274716e521694af72610
Status: NEW → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.