1620262 - [scriptworker] Intermittent asyncio.exceptions.TimeoutError

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Alexandru Michis [:malexandru]]

Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/FGZhXRKsT2OAu-zIfTdAGw/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FFGZhXRKsT2OAu-zIfTdAGw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Fchain_of_trust.log#L123

Raw log: https://firefoxci.taskcluster-artifacts.net/FGZhXRKsT2OAu-zIfTdAGw/0/public/logs/chain_of_trust.log

 'workerType': 'gecko-3-signing'},
 'takenUntil': '2020-03-05T13:40:16.635Z',
 'workerGroup': 'gecko-3-signing',
 'workerId': 'gecko-3-signing-j59r6udbsguvyqfuq7uqa'}
2020-03-05T13:20:16    DEBUG -  waiting 300 seconds before reclaiming...
2020-03-05T13:20:24    ERROR - SCRIPTWORKER_UNEXPECTED_EXCEPTION task 
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker/worker.py", line 55, in do_run_task
    await run_cancellable(verify_chain_of_trust(chain))
  File "/app/lib/python3.8/site-packages/scriptworker/worker.py", line 156, in _run_cancellable
    result = await self.future
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1908, in verify_chain_of_trust
    await verify_task_types(chain)
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1668, in verify_task_types
    await valid_task_types[task_type](chain, obj)
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1592, in verify_parent_task
    await verify_parent_task_definition(chain, link)
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1463, in verify_parent_task_definition
    jsone_context, tmpl = await get_jsone_context_and_template(chain, parent_link, decision_link, tasks_for)
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1432, in get_jsone_context_and_template
    tmpl = await get_in_tree_template(decision_link)
  File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1289, in get_in_tree_template
    tmpl = await load_json_or_yaml_from_url(context, source_url, os.path.join(context.config["work_dir"], "{}_taskcluster.yml".format(link.name)), auth=auth)
  File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 735, in load_json_or_yaml_from_url
    await retry_async(download_file, args=(context, url, path), kwargs=kwargs, retry_exceptions=(DownloadError, aiohttp.ClientError))
  File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 259, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 655, in download_file
    async with session.get(url, auth=auth) as resp:
  File "/app/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/app/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/app/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 860, in start
    self._continue = None
  File "/app/lib/python3.8/site-packages/aiohttp/helpers.py", line 596, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
2020-03-05T13:20:24    DEBUG -  "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2020-03-05T13:20:24     INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Intermittent Failures Robot]

1 failures in 5483 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows2012-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1620262&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 2

[Intermittent Failures Robot]

2 failures in 4773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows2012-64-shippable: 1
• windows2012-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1620262&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

We seem to be going through a spell of higher than normal network errors, or possibly something changed behaviour in a dependency. eg CoT download errors in
https://firefoxci.taskcluster-artifacts.net/Fv9yv1CKRqCecDECQW4eUA/0/public/logs/chain_of_trust.log
https://firefoxci.taskcluster-artifacts.net/Fv9yv1CKRqCecDECQW4eUA/0/public/logs/chain_of_trust.log

It looks like we should port the fix in bug 1624794 for upload_file() to download_file too. Maybe grep over scriptworker for use of aiohttp.ClientError to retry where we don't also use asyncio.TimeoutError.

Comment 4

[Aki Sasaki (not active)]

Attachment: [scriptworker] catch TimeoutError in load_json_or_yaml_from_url

Comment 5

[Aki Sasaki (not active)]

Scriptworker 33.0.1 is in production.