1825582 - Add retries on GCS beetmover uploads

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Gabriel Bustamante [:gabriel]]

When there's beetmover bustage caused by Google-side service degradation or intermittent issues in their storage API we re-run tasks to retry the requests.

These busted langpack tasks on the beta's l10n bump are an example of what can happen.

These were getting intermittent 503 - Service Unavailable errors from storage.googleapis.com

The Google storage client has retry strategies [1]

Also we could use redo [2]

[1] https://cloud.google.com/storage/docs/retry-strategy
[2] https://github.com/mozilla-releng/scriptworker-scripts/blob/master/beetmoverscript/src/beetmoverscript/script.py#L15

Comment 1

[Gabriel Bustamante [:gabriel]]

oops I forgot to delete some dependencies on this bug. I cloned another beetmover bug as a staring point

Comment 2

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1825582 - Force retries on GCS operations (#702)

Comment 3

[Gabriel Bustamante [:gabriel]]

scriptworker-scripts/pull/702 should mitigate the intermittents failures seen on Bug 1817535 (see here) and Bug 1644408 (see here) by retrying the requests. Hopefully saving us some re-runs and making the tasks more resilient / less flaky.