1624496 - Perma build signing bustages http.client_exceptions.ClientResponseError: 502, message='Bad Gateway', url=URL('https://autograph-external.stage.autograph.services.mozaws.net/sign/file')

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&fromchange=ba7de7a93a714f71a8e2fb1f0ab1584d90ad2e57&tochange=93cf130f59f97acc6ebb84c3505880cdcaca36c0&searchStr=build%2Csignin&selectedJob=294446627

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=294446627&repo=autoland&lineNumber=753

aiohttp.client_exceptions.ClientResponseError: 502, message='Bad Gateway', url=URL('https://autograph-external.stage.autograph.services.mozaws.net/sign/file')
2020-03-24 00:08:34,928 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-124' coro=<sign_omnija_with_autograph() done, defined at /app/lib/python3.8/site-packages/signingscript/sign.py:90> exception=ClientResponseError(RequestInfo(url=URL('https://autograph-external.stage.autograph.services.mozaws.net/sign/file'), method='POST', headers=<CIMultiDictProxy('Host': 'autograph-external.stage.autograph.services.mozaws.net', 'Authorization': 'Hawk mac="ffcpElKLb6H6hIKwTivcjMI0iLQBczarefevTWmQo34=", hash="zq8BqbICdUr+S6tPme+notvMrbXvihFllXP7jwwgE7M=", id="releng_systemaddon", ts="1585008514", nonce="gwPVi8"', 'Content-Type': 'application/json', 'Content-Length': '62045222', 'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.8 aiohttp/3.6.2')>, real_url=URL('https://autograph-external.stage.autograph.services.mozaws.net/sign/file')), (), status=502, message='Bad Gateway', headers=<CIMultiDictProxy('Date': 'Tue, 24 Mar 2020 00:08:34 GMT', 'Content-Type': 'text/html', 'Content-Length': '166', 'Connection': 'keep-alive')>)>
Traceback (most recent call last):
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 95, in wrapped
return await f(*args, **kwargs)
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 1230, in sign_omnija_with_autograph
await sign_file_with_autograph(context, from_, "autograph_omnija", to=signed_out, extension_id="[e-mail]")
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 95, in wrapped
return await f(*args, **kwargs)
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 994, in sign_file_with_autograph
signed_bytes = base64.b64decode(await sign_with_autograph(context.session, a, input_file, fmt, "file", extension_id=extension_id))
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 95, in wrapped
return await f(*args, **kwargs)
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 960, in sign_with_autograph
sign_resp = await retry_async(
File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 262, in retry_async
_check_number_of_attempts(attempt, attempts, func, "retry_async")
File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 259, in retry_async
return await func(*args, **kwargs)
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 95, in wrapped
return await f(*args, **kwargs)
File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 882, in call_autograph
resp.raise_for_status()
File "/app/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 941, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 502, message='Bad Gateway', url=URL('https://autograph-external.stage.autograph.services.mozaws.net/sign/file')
exit code: 1

Comment 1

[Aki Sasaki (not active)]

This is a releng bug. We shouldn't be pointing at autograph-external.stage for anything tree-closing.

Comment 2

[Aki Sasaki (not active)]

Attachment: scriptworker-scripts patch

Also landed a sops patch to change auth.

Comment 3

[Aki Sasaki (not active)]

I accidentally overwrote the widevine dep password instead of the omnija? which slowed down the fix. I think we're good now.

In the morning, let's postmortem to figure out why we were still pointing at autograph-external.stage.

Comment 4

[Intermittent Failures Robot]

19 failures in 873 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19

Platform breakdown:

• windows2012-aarch64-shippable: 9
• windows2012-64-shippable: 1
• linux64-shippable: 1
• windows2012-64: 3
• windows2012-aarch64: 3
• windows2012-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1624496&startday=2020-03-23&endday=2020-03-23&tree=all

Comment 6

[Aki Sasaki (not active)]

It looks like the docker configs come from August 13, 2019 https://github.com/mozilla-releng/signingscript/pull/123 .
We pointed puppet configs at autograph-external.prod for gpg on August 16, 2019: https://github.com/mozilla-releng/build-puppet/commit/f4ea4c5380f6031f265220c4bde16d5d93ba2c39

We may have just not noticed that we were pointing at autograph-external.stage for two formats in dep all this time.

Comment 7

[Intermittent Failures Robot]

19 failures in 4773 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 19

Platform breakdown:

• windows2012-64: 3
• windows2012-aarch64-shippable: 9
• windows2012-aarch64: 3
• windows2012-32: 2
• windows2012-64-shippable: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1624496&startday=2020-03-23&endday=2020-03-29&tree=all