signing script workers should retry on signing server timeouts



a year ago
10 months ago


(Reporter: catlee, Assigned: aki)


Firefox Tracking Flags

(Not tracked)



(1 attachment)



a year ago

2017-08-31 14:04:39,758 - asyncio - DEBUG - Using selector: EpollSelector
2017-08-31 14:04:39,761 - signingscript.script - INFO - validating task
2017-08-31 14:04:39,761 - signingscript.task - DEBUG - {'type': 'object', 'required': ['scopes', 'payload'], 'properties': {'payload': {'type': 'object', 'required': ['upstreamArtifacts'], 'properties': {'upstreamArtifacts': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'object', 'required': ['taskId', 'taskType', 'paths', 'formats'], 'properties': {'taskType': {'type': 'string'}, 'taskId': {'type': 'string'}, 'paths': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'formats': {'type': 'array', 'uniqueItems': True, 'items': {'type': 'string'}}}}}}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}}, 'title': 'Taskcluster signing task minimal schema'}
2017-08-31 14:04:39,764 - signingscript.utils - INFO - Loading signing server config from /builds/scriptworker/passwords.json
2017-08-31 14:04:39,765 - signingscript.utils - INFO - Signing server config loaded from /builds/scriptworker/passwords.json
2017-08-31 14:04:39,766 - signingscript.sign - INFO - Certificate types: ['project:releng:signing:cert:release-signing']
2017-08-31 14:04:39,767 - signingscript.script - INFO - getting token
2017-08-31 14:04:39,767 - signingscript.task - INFO - getting token from
2017-08-31 14:04:39,768 - scriptworker.utils - DEBUG - retry_async: Calling <function request at 0x7f5987cf8ae8>, attempt 1
2017-08-31 14:04:39,769 - scriptworker.utils - DEBUG - POST
Traceback (most recent call last):
  File "/builds/scriptworker/bin/signingscript", line 9, in <module>
    load_entry_point('signingscript==4.1.2', 'console_scripts', 'signingscript')()
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/", line 140, in main
    loop.run_until_complete(async_main(context, conn=conn))
  File "/tools/python35/lib/python3.5/asyncio/", line 387, in run_until_complete
    return future.result()
  File "/tools/python35/lib/python3.5/asyncio/", line 274, in result
    raise self._exception
  File "/tools/python35/lib/python3.5/asyncio/", line 241, in _step
    result = coro.throw(exc)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/", line 56, in async_main
    await get_token(context, os.path.join(work_dir, 'token'), cert_type, all_signing_formats)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/", line 105, in get_token
    auth=auth, return_type='text')
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/", line 92, in retry_request
    args=args, kwargs=kwargs, **retry_async_kwargs)
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/", line 251, in retry_async
    return await func(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/", line 70, in request
    return resp
  File "/builds/scriptworker/lib/python3.5/site-packages/async_timeout/", line 46, in __exit__
    raise asyncio.TimeoutError from None
exit code: 1

Comment 1

11 months ago
>>> import concurrent.futures._base
>>> import asyncio
>>> concurrent.futures._base.TimeoutError is asyncio.TimeoutError

Let's add asyncio.TimeoutError to the tuple of exceptions to retry for in retry_request().

Comment 2

11 months ago
Created attachment 8915706 [details] [diff] [review]
add asyncio.TimeoutError to the tuple of retry exceptions in retry_request

Jordan, are you comfortable reviewing this?
Assignee: nobody → aki
Attachment #8915706 - Flags: review?(jlund)

Comment 3

11 months ago
Comment on attachment 8915706 [details] [diff] [review]
add asyncio.TimeoutError to the tuple of retry exceptions in retry_request

Review of attachment 8915706 [details] [diff] [review]:

lgtm code wise.

I was worried that if we timeout, we should be conscious about retry load but it seems that this function uses calculate_sleep_time[0] so a default jitter and delay factor is in place.

Attachment #8915706 - Attachment is patch: true
Attachment #8915706 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #8915706 - Flags: review?(jlund) → review+

Comment 4

10 months ago
Awesome. Do you think you can mark the PR approved?

Comment 5

10 months ago
scriptworker 5.2.2
Last Resolved: 10 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.