Closed Bug 1761398 Opened 4 years ago Closed 4 years ago

Intermittent Balrog requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://aus4-admin.mozilla.org/api/v2/releases/Devedition-99.0b8-build1-No-WNP

Categories

(Release Engineering :: Release Automation, defect, P1)

Product:
Release Engineering
Component:
Release Automation
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: jbuck)

References

(Regression)

Details

(Keywords: intermittent-failure)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=372263272&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BZja89DLRF2Xl4dCmP5B5g/runs/0/artifacts/public/logs/live_backing.log

2022-03-24 21:59:31,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): aus4-admin.mozilla.org:443
2022-03-24 22:00:01,676 - urllib3.connectionpool - DEBUG - https://aus4-admin.mozilla.org:443 "POST /api/v2/releases/Devedition-99.0b8-build1-No-WNP HTTP/1.1" 502 332
2022-03-24 22:00:01,677 - balrogclient.api - ERROR - Caught HTTPError: b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n'
2022-03-24 22:00:01,677 - balrogclient.api - DEBUG - REQUEST STATS: {"timestamp": 1648159201.6772387, "method": "POST", "url": "https://aus4-admin.mozilla.org/api/v2/releases/Devedition-99.0b8-build1-No-WNP", "status_code": 502, "elapsed_secs": 30.034594}
2022-03-24 22:00:01,677 - redo - DEBUG - retry: Caught exception: 
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/redo/__init__.py", line 170, in retry
    return action(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/balrogscript/script.py", line 110, in <lambda>
    retry(lambda: submitter.run(**release), jitter=5, sleeptime=10, max_sleeptime=30, attempts=10)
  File "/app/lib/python3.9/site-packages/balrogscript/submitter/cli.py", line 466, in run
    balrog_request(session, "post", url, json=data)
  File "/app/lib/python3.9/site-packages/balrogclient/api.py", line 90, in balrog_request
    resp.raise_for_status()
  File "/app/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://aus4-admin.mozilla.org/api/v2/releases/Devedition-99.0b8-build1-No-WNP
2022-03-24 22:00:01,680 - redo - DEBUG - sleeping for 7.32s (attempt 1/10)

Jobs are failing on Beta: https://treeherder.mozilla.org/jobs?repo=mozilla-beta&group_state=expanded&revision=07a291b2c115530da9a34fe1075ec6579cc97e92&searchStr=balrog

Retriggers were done, but the jobs have been pending for the last 3-4 hours.
Issue is escalated on Matrix, on #Firefox-CI and on Slack #taskcluster-cloudops.

Looks like there was a bug in Balrog that wasn't surfaced until after deploy. I filed https://github.com/mozilla-releng/balrog/issues/2479 to track it on the Balrog side. JBuck is rolling back the deploy, so things should hopefully improve soon.

Regressed by: 1760606

All pending tasks have been successfully run after the rollback.

Closing as fixed by the rollback. Will track the balrog-side fix on github.

Assignee: nobody → jbuckley
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
See Also: → https://github.com/mozilla-releng/balrog/issues/2479
Severity: S4 → S1
tracking-firefox100: --- → ?
tracking-firefox99: --- → ?
Priority: P5 → P1
Component: Release Automation: Updates → Release Automation
You need to log in before you can comment on or make changes to this bug.