1501782 - Intermittent signingscript.exceptions.FailedSubprocess: Command `/builds/scriptworker/bin/signtool -v -n /builds/scriptworker/work/nonce -t /builds/scriptworker/work/token -c /host.cert -H ... firefox/updater.exe` failed

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4c162e4eda0879b7ea1761c3db4679e403273479&searchStr=windows%2C2012%2Cdebug%2Cbuild-signing-win32%2Fdebug%2C%28bs%29&selectedJob=207573564

Task details : https://tools.taskcluster.net/groups/KK1HuQaXSXyy5QFsVspJog/tasks/MmBzHDwJSHSEZm7B7BVtFQ/runs/0

Raw log: https://taskcluster-artifacts.net/MmBzHDwJSHSEZm7B7BVtFQ/0/public/logs/live_backing.log

2018-10-24 17:20:19,586 - signingscript.utils - INFO - 2018-10-24 17:20:19,586 - Starting new HTTPS connection (1): signing7.srv.releng.mdc1.mozilla.com:9110
2018-10-24 17:20:20,082 - signingscript.utils - INFO - 2018-10-24 17:20:20,082 - https://signing7.srv.releng.mdc1.mozilla.com:9110 "POST /sign/sha2signcode HTTP/1.1" 202 0
2018-10-24 17:20:21,086 - signingscript.utils - INFO - 2018-10-24 17:20:21,084 - 42b3bd0080de318bd4e8b11419c40325ae13ddcb: processing /builds/scriptworker/work/ziphmnx0m01/firefox/updater.exe on https://signing7.srv.releng.mdc1.mozilla.com:9110
2018-10-24 17:20:21,087 - signingscript.utils - INFO - 2018-10-24 17:20:21,084 - 42b3bd0080de318bd4e8b11419c40325ae13ddcb: GET https://signing7.srv.releng.mdc1.mozilla.com:9110/sign/sha2signcode/42b3bd0080de318bd4e8b11419c40325ae13ddcb
2018-10-24 17:20:21,089 - signingscript.utils - INFO - Traceback (most recent call last):
2018-10-24 17:20:21,090 - signingscript.utils - INFO -   File "/builds/scriptworker/bin/signtool", line 11, in <module>
2018-10-24 17:20:21,090 - signingscript.utils - INFO -     sys.exit(main())
2018-10-24 17:20:21,091 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signtool.py", line 278, in main
2018-10-24 17:20:21,091 - signingscript.utils - INFO -     sign(options, args)
2018-10-24 17:20:21,092 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signtool.py", line 265, in sign
2018-10-24 17:20:21,092 - signingscript.utils - INFO -     if not remote_signfile(options, urls, f, fmt, token, dest):
2018-10-24 17:20:21,093 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 85, in remote_signfile
2018-10-24 17:20:21,093 - signingscript.utils - INFO -     r = getfile(url, filehash, fmt, options.cert)
2018-10-24 17:20:21,094 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 16, in getfile
2018-10-24 17:20:21,094 - signingscript.utils - INFO -     return method(url, verify=cert)
2018-10-24 17:20:21,095 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/api.py", line 75, in get
2018-10-24 17:20:21,096 - signingscript.utils - INFO -     return request('get', url, params=params, **kwargs)
2018-10-24 17:20:21,096 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/api.py", line 60, in request
2018-10-24 17:20:21,097 - signingscript.utils - INFO -     return session.request(method=method, url=url, **kwargs)
2018-10-24 17:20:21,097 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-10-24 17:20:21,098 - signingscript.utils - INFO -     resp = self.send(prep, **send_kwargs)
2018-10-24 17:20:21,098 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-10-24 17:20:21,098 - signingscript.utils - INFO -     r = adapter.send(request, **kwargs)
2018-10-24 17:20:21,098 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/adapters.py", line 416, in send
2018-10-24 17:20:21,098 - signingscript.utils - INFO -     self.cert_verify(conn, request.url, verify, cert)
2018-10-24 17:20:21,098 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/adapters.py", line 228, in cert_verify
2018-10-24 17:20:21,098 - signingscript.utils - INFO -     "invalid path: {}".format(cert_loc))
2018-10-24 17:20:21,098 - signingscript.utils - INFO - OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /builds/scriptworker/lib/python3.6/site-packages/signingscript/data/host.cert
2018-10-24 17:20:21,110 - signingscript.utils - INFO - exitcode 1
2018-10-24 17:20:21,111 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 203, in _handle_asyncio_loop
    await async_main(context)
  File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/script.py", line 54, in async_main
  File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/task.py", line 207, in sign
  File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/sign.py", line 246, in sign_signcode
  File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/sign.py", line 151, in sign_file
  File "/builds/scriptworker/lib/python3.6/site-packages/signingscript/utils.py", line 159, in execute_subprocess
signingscript.exceptions.FailedSubprocess: Command `/builds/scriptworker/bin/signtool -v -n /builds/scriptworker/work/nonce -t /builds/scriptworker/work/token -c /builds/scriptworker/lib/python3.6/site-packages/signingscript/data/host.cert -H signing7.srv.releng.mdc1.mozilla.com:9110 -H signing8.srv.releng.mdc1.mozilla.com:9110 -H signing9.srv.releng.mdc1.mozilla.com:9110 -H signing10.srv.releng.mdc2.mozilla.com:9110 -H signing11.srv.releng.mdc2.mozilla.com:9110 -H signing12.srv.releng.mdc2.mozilla.com:9110 -f sha2signcode -o /builds/scriptworker/work/ziphmnx0m01/firefox/updater.exe /builds/scriptworker/work/ziphmnx0m01/firefox/updater.exe` failed
exit code: 5

Comment 1

[Aki Sasaki (not active)]

This sounds like fallout from a signingscript update. One happened this morning: https://github.com/mozilla-releng/build-puppet/pull/275 . A rerun (not retrigger) should work here.

Comment 2

[Aki Sasaki (not active)]

The long term fix here is docker (or other similar) non-puppet solutions... that won't be ready this quarter.

Comment 3

[Intermittent Failures Robot]

1 failures in 2584 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1501782&startday=2018-10-22&endday=2018-10-28&tree=all

Comment 4

[Aki Sasaki (not active)]

We've moved to docker, and no longer use signtool.