Closed Bug 1825582 Opened 1 year ago Closed 1 year ago

Add retries on GCS beetmover uploads

Categories

(Release Engineering :: Release Automation: Uploading, enhancement)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gabriel, Assigned: gabriel)

References

Details

Attachments

(1 file)

When there's beetmover bustage caused by Google-side service degradation or intermittent issues in their storage API we re-run tasks to retry the requests.

These busted langpack tasks on the beta's l10n bump are an example of what can happen.

These were getting intermittent 503 - Service Unavailable errors from storage.googleapis.com

The Google storage client has retry strategies [1]

Also we could use redo [2]

[1] https://cloud.google.com/storage/docs/retry-strategy
[2] https://github.com/mozilla-releng/scriptworker-scripts/blob/master/beetmoverscript/src/beetmoverscript/script.py#L15

oops I forgot to delete some dependencies on this bug. I cloned another beetmover bug as a staring point

Assignee: nobody → gabriel
Status: NEW → ASSIGNED

scriptworker-scripts/pull/702 should mitigate the intermittents failures seen on Bug 1817535 (see here) and Bug 1644408 (see here) by retrying the requests. Hopefully saving us some re-runs and making the tasks more resilient / less flaky.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: