1395578 - signing script workers should retry on signing server timeouts

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

From https://public-artifacts.taskcluster.net/UUAD-rsFTF244Puh2J2c7Q/0/public/logs/live_backing.log

2017-08-31 14:04:39,758 - asyncio - DEBUG - Using selector: EpollSelector
2017-08-31 14:04:39,761 - signingscript.script - INFO - validating task
2017-08-31 14:04:39,761 - signingscript.task - DEBUG - {'type': 'object', 'required': ['scopes', 'payload'], 'properties': {'payload': {'type': 'object', 'required': ['upstreamArtifacts'], 'properties': {'upstreamArtifacts': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'object', 'required': ['taskId', 'taskType', 'paths', 'formats'], 'properties': {'taskType': {'type': 'string'}, 'taskId': {'type': 'string'}, 'paths': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'formats': {'type': 'array', 'uniqueItems': True, 'items': {'type': 'string'}}}}}}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}}, 'title': 'Taskcluster signing task minimal schema'}
2017-08-31 14:04:39,764 - signingscript.utils - INFO - Loading signing server config from /builds/scriptworker/passwords.json
2017-08-31 14:04:39,765 - signingscript.utils - INFO - Signing server config loaded from /builds/scriptworker/passwords.json
2017-08-31 14:04:39,766 - signingscript.sign - INFO - Certificate types: ['project:releng:signing:cert:release-signing']
2017-08-31 14:04:39,767 - signingscript.script - INFO - getting token
2017-08-31 14:04:39,767 - signingscript.task - INFO - getting token from signing5.srv.releng.scl3.mozilla.com:9120
2017-08-31 14:04:39,768 - scriptworker.utils - DEBUG - retry_async: Calling <function request at 0x7f5987cf8ae8>, attempt 1
2017-08-31 14:04:39,769 - scriptworker.utils - DEBUG - POST https://signing5.srv.releng.scl3.mozilla.com:9120/token
Traceback (most recent call last):
  File "/builds/scriptworker/bin/signingscript", line 9, in <module>
    load_entry_point('signingscript==4.1.2', 'console_scripts', 'signingscript')()
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/script.py", line 140, in main
    loop.run_until_complete(async_main(context, conn=conn))
  File "/tools/python35/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/tools/python35/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/tools/python35/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/script.py", line 56, in async_main
    await get_token(context, os.path.join(work_dir, 'token'), cert_type, all_signing_formats)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/task.py", line 105, in get_token
    auth=auth, return_type='text')
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/utils.py", line 92, in retry_request
    args=args, kwargs=kwargs, **retry_async_kwargs)
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/utils.py", line 251, in retry_async
    return await func(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/utils.py", line 70, in request
    return resp
  File "/builds/scriptworker/lib/python3.5/site-packages/async_timeout/__init__.py", line 46, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
exit code: 1

Comment 1

[Aki Sasaki (not active)]

>>> import concurrent.futures._base
>>> import asyncio
>>> concurrent.futures._base.TimeoutError is asyncio.TimeoutError
True

Let's add asyncio.TimeoutError to the tuple of exceptions to retry for in retry_request().

Comment 2

[Aki Sasaki (not active)]

Attachment: add asyncio.TimeoutError to the tuple of retry exceptions in retry_request

Jordan, are you comfortable reviewing this?

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8915706 [details] [diff] [review]
add asyncio.TimeoutError to the tuple of retry exceptions in retry_request

Review of attachment 8915706 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm code wise.

I was worried that if we timeout, we should be conscious about retry load but it seems that this function uses calculate_sleep_time[0] so a default jitter and delay factor is in place.

[0] https://github.com/mozilla-releng/scriptworker/blob/601fcf23103c9893e8ea44678fe6e41b4ae73691/scriptworker/utils.py#L188

Comment 4

[Aki Sasaki (not active)]

Awesome. Do you think you can mark the PR approved?

Comment 5

[Aki Sasaki (not active)]

scriptworker 5.2.2 https://hg.mozilla.org/build/puppet/rev/63ec1deee923