Intermittent build bustage scriptworker.utils - WARNING - retry_async: call_autograph: too many retries! ends with MemoryError
Categories
(Release Engineering :: Release Automation, defect)
Tracking
(Not tracked)
People
(Reporter: bogdan_tara, Unassigned)
References
Details
2019-09-23 10:45:33,373 - signingscript.utils - INFO - mkdir /builds/scriptworker/work/ojtarqhdtt3k7
2019-09-23 10:47:00,288 - signingscript.sign - INFO - Creating tarfile /builds/scriptworker/work/public/build/target.tar.bz2...
2019-09-23 10:49:58,091 - signingscript.task - INFO - sign(): Signing /builds/scriptworker/work/public/build/target.tar.bz2 with autograph_gpg...
2019-09-23 10:50:55,720 - scriptworker.utils - WARNING - retry_async: call_autograph: too many retries!
Traceback (most recent call last):
File "/builds/scriptworker/bin/signingscript", line 11, in <module>
sys.exit(main())
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/script.py", line 138, in main
async_main, default_config=get_default_config()
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 164, in sync_main
loop.run_until_complete(_handle_asyncio_loop(async_main, context))
File "/tools/python3/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
await async_main(context)
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/script.py", line 74, in async_main
context, os.path.join(work_dir, path), path_dict["formats"]
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/task.py", line 216, in sign
output = await signing_func(context, output, fmt)
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/sign.py", line 1086, in sign_gpg_with_autograph
signature = await sign_with_autograph(s, input_bytes, fmt, "data")
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/sign.py", line 1014, in sign_with_autograph
sleeptime_kwargs={"delay_factor": 2.0},
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 261, in retry_async
return await func(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/sign.py", line 943, in call_autograph
r = session.post(url, json=request_json, auth=auth)
File "/builds/scriptworker/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/requests/sessions.py", line 519, in request
prep = self.prepare_request(req)
File "/builds/scriptworker/lib/python3.6/site-packages/requests/sessions.py", line 462, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 317, in prepare
self.prepare_auth(auth, url)
File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 548, in prepare_auth
r = auth(self)
File "/builds/scriptworker/lib/python3.6/site-packages/requests_hawk/init.py", line 79, in call
_timestamp=self._timestamp
File "/builds/scriptworker/lib/python3.6/site-packages/mohawk/sender.py", line 100, in init
self.req_resource.gen_content_hash())
File "/builds/scriptworker/lib/python3.6/site-packages/mohawk/base.py", line 208, in gen_content_hash
self.content_type)
File "/builds/scriptworker/lib/python3.6/site-packages/mohawk/util.py", line 65, in calculate_payload_hash
.format(parts=pprint.pformat(parts)))
File "/tools/python3/lib/python3.6/pprint.py", line 58, in pformat
compact=compact).pformat(object)
File "/tools/python3/lib/python3.6/pprint.py", line 144, in pformat
self._format(object, sio, 0, 0, {}, 0)
File "/tools/python3/lib/python3.6/pprint.py", line 161, in _format
rep = self._repr(object, context, level)
File "/tools/python3/lib/python3.6/pprint.py", line 393, in _repr
self._depth, level)
File "/tools/python3/lib/python3.6/pprint.py", line 405, in format
return _safe_repr(object, context, maxlevels, level)
File "/tools/python3/lib/python3.6/pprint.py", line 553, in _safe_repr
return format % ", ".join(components), readable, recursive
MemoryError
exit code: 1
|
||
Comment 2•6 years ago
•
|
||
Aki, Justin, could you look into this? Is hitting beta and esr68 at the moment.
|
||
Comment 3•6 years ago
|
||
we are currently debugging. thanks for escalating.
|
|
||
Comment 5•6 years ago
|
||
tried backing out a pyup patch that changed signingscript deps. That didn't work https://github.com/mozilla-releng/build-puppet/commit/663861a97adb1e8e4465b9ffd9506e5996438ece
we then bumped instance types from t2.medium to t2.large across all signing-linux-* workers. Didn't touch dep or mobile. this fixed the problem. we now have a green task
https://taskcluster-ui.herokuapp.com/tasks/ZMu7u0OQQtqpS_Hc1dI7gw/runs/5
Rerunning all the things
fyi - @catlee @mtabara - this may affect workers and configuration in gcp?
|
|
||
Comment 6•6 years ago
|
||
nothing in there looks like it would impact workers in gcp
|
|
||
Comment 7•6 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #6)
nothing in there looks like it would impact workers in gcp
I'll double-check the t2.large hardware specifications against our current hardware configurations for GCP signing workers. Well, we only have the signing-dev for now.
|
|
||
Comment 8•6 years ago
|
||
We ended up bumping memory in GCP counterparts too in https://github.com/mozilla-services/cloudops-infra/pull/1401.
|
|
||
Updated•6 years ago
|
| Comment hidden (Intermittent Failures Robot) |
|
||
Updated•1 year ago
|
Description
•