1817535 - Intermittent google.api_core.exceptions.ServiceUnavailable: 503 POST https://storage.googleapis.com/upload/storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o?uploadType=multipart: Service Unavailable

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Iulian Moraru]

Failure summary does not appear in treeherder.

• Push where the failure occurred
• Parsed failure log
• Raw failure log

2023-02-17 23:24:05,977 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:05,978 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster beetmover task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'upload_date': {'type': 'number'}, 'build_number': {'type': 'number'}, 'locale': {'type': 'string'}, 'maxRunTime': {'type': 'number'}, 'version': {'type': 'string'}, 'next_version': {'$comment': 'ignored'}, 'appVersion': {'$comment': 'ignored'}, 'releaseProperties': {'type': 'object', 'properties': {'appName': {'type': 'string'}, 'buildid': {'type': 'string'}, 'appVersion': {'type': 'string'}, 'hashType': {'type': 'string'}, 'platform': {'type': 'string'}, 'branch': {'type': 'string'}}, 'required': ['appName', 'buildid', 'appVersion', 'hashType', 'platform', 'branch']}, 'upstreamArtifacts': {'type': 'array', 'items': {'type': 'object', 'properties': {'taskType': {'type': 'string'}, 'locale': {'type': 'string'}, 'taskId': {'type': 'string'}, 'paths': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}}, 'required': ['taskId', 'taskType', 'paths', 'locale']}, 'minItems': 1, 'uniqueItems': True}}, 'required': ['upload_date', 'upstreamArtifacts', 'releaseProperties']}}, 'required': ['payload', 'dependencies']}
2023-02-17 23:24:05,992 - beetmoverscript.task - INFO - Buckets: ['nightly']
2023-02-17 23:24:05,993 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:05,993 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:05,994 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:05,995 - google.auth._default - DEBUG - Checking /tmp/tmpzc13fvaz for explicit credentials as part of auth process...
2023-02-17 23:24:06,074 - google.auth._default - DEBUG - Checking /tmp/tmpzc13fvaz for explicit credentials as part of auth process...
2023-02-17 23:24:06,125 - urllib3.util.retry - DEBUG - Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
2023-02-17 23:24:06,128 - google.auth.transport.requests - DEBUG - Making request: POST https://oauth2.googleapis.com/token
2023-02-17 23:24:06,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): oauth2.googleapis.com:443
2023-02-17 23:24:06,168 - urllib3.connectionpool - DEBUG - https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
2023-02-17 23:24:06,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): storage.googleapis.com:443
2023-02-17 23:24:06,221 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "GET /storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery?fields=name&prettyPrint=false HTTP/1.1" 200 57
2023-02-17 23:24:06,221 - beetmoverscript.gcloud - INFO - Found GCS bucket <Bucket: moz-fx-productdelivery-pr-38b5-productdelivery> - proceeding with GCS uploads.
2023-02-17 23:24:06,222 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:06,223 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:06,224 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:06,352 - beetmoverscript.script - INFO - upload_to_s3: /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums -> s3://net-mozaws-prod-delivery-firefox/pub/firefox/nightly/2023/02/2023-02-17-16-53-16-mozilla-central-l10n/firefox-112.0a1.id.mac.checksums
2023-02-17 23:24:06,369 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:06,369 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:06,390 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "GET /storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o/pub%2Ffirefox%2Fnightly%2F2023%2F02%2F2023-02-17-16-53-16-mozilla-central-l10n%2Ffirefox-112.0a1.id.mac.checksums?fields=name&prettyPrint=false HTTP/1.1" 404 427
2023-02-17 23:24:06,391 - beetmoverscript.gcloud - INFO - upload_to_gcs: /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums -> Bucket: gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/firefox/nightly/2023/02/2023-02-17-16-53-16-mozilla-central-l10n/firefox-112.0a1.id.mac.checksums
2023-02-17 23:24:16,399 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "POST /upload/storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o?uploadType=multipart HTTP/1.1" 503 19
2023-02-17 23:24:16,400 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:16,400 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:16,410 - beetmoverscript.script - INFO - upload_to_s3: /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums -> s3://net-mozaws-prod-delivery-firefox/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-112.0a1.id.mac.checksums
2023-02-17 23:24:16,411 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2023-02-17 23:24:16,412 - beetmoverscript.task - DEBUG - Loading release_props from task's payload: {'appName': 'Firefox', 'appVersion': '112.0a1', 'branch': 'mozilla-central', 'buildid': '20230217165316', 'hashType': 'sha512', 'platform': 'macosx64'}
2023-02-17 23:24:16,434 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "GET /storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o/pub%2Ffirefox%2Fnightly%2Flatest-mozilla-central-l10n%2Ffirefox-112.0a1.id.mac.checksums?fields=name&prettyPrint=false HTTP/1.1" 200 91
2023-02-17 23:24:16,435 - beetmoverscript.gcloud - WARNING - upload_to_gcs: Overriding file: pub/firefox/nightly/latest-mozilla-central-l10n/firefox-112.0a1.id.mac.checksums
2023-02-17 23:24:16,435 - beetmoverscript.gcloud - INFO - upload_to_gcs: /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums -> Bucket: gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-112.0a1.id.mac.checksums
2023-02-17 23:24:16,735 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "POST /upload/storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o?uploadType=multipart HTTP/1.1" 200 1206
2023-02-17 23:24:17,104 - beetmoverscript.script - INFO - put /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums: 200
2023-02-17 23:24:17,104 - charset_normalizer - DEBUG - Encoding detection on empty bytes, assuming utf_8 intention.
2023-02-17 23:24:17,137 - beetmoverscript.script - INFO - put /app/workdir/cot/TE76eVCtTc-hMLLx7cmxnA/public/target.checksums: 200
2023-02-17 23:24:17,137 - charset_normalizer - DEBUG - Encoding detection on empty bytes, assuming utf_8 intention.
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2539, in upload_from_file
    created_json = self._do_upload(
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2354, in _do_upload
    response = self._do_multipart_upload(
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1889, in _do_multipart_upload
    response = upload.transmit(
  File "/app/lib/python3.9/site-packages/google/resumable_media/requests/upload.py", line 153, in transmit
    return _request_helpers.wait_and_retry(
  File "/app/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py", line 171, in wait_and_retry
    raise error
  File "/app/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py", line 148, in wait_and_retry
    response = func()
  File "/app/lib/python3.9/site-packages/google/resumable_media/requests/upload.py", line 149, in retriable_request
    self._process_response(result)
  File "/app/lib/python3.9/site-packages/google/resumable_media/_upload.py", line 114, in _process_response
    _helpers.require_status_code(response, (http.client.OK,), self._get_status_code)
  File "/app/lib/python3.9/site-packages/google/resumable_media/_helpers.py", line 108, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/bin/beetmoverscript", line 8, in <module>
    sys.exit(main())
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 683, in main
    client.sync_main(async_main, config_path=config_path, default_config=default_config, should_validate_task=False)
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 166, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 335, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 124, in push_to_nightly
    await move_beets(context, context.artifacts_to_beetmove, artifact_map=context.task["payload"]["artifactMap"])
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 426, in move_beets
    await raise_future_exceptions(beets)
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 409, in raise_future_exceptions
    succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 443, in _process_future_exceptions
    raise exc
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 444, in move_beet
    await retry_upload(context=context, destinations=destinations, path=source)
  File "/app/lib/python3.9/site-packages/beetmoverscript/script.py", line 634, in retry_upload
    await await_and_raise_uploads(cloud_uploads, context.config["clouds"], context.bucket)
  File "/app/lib/python3.9/site-packages/beetmoverscript/utils.py", line 333, in await_and_raise_uploads
    await raise_future_exceptions(cloud_uploads[cloud])
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 409, in raise_future_exceptions
    succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 443, in _process_future_exceptions
    raise exc
  File "/app/lib/python3.9/site-packages/beetmoverscript/gcloud.py", line 101, in upload_to_gcs
    return blob.upload_from_filename(path, content_type=mime_type)
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2683, in upload_from_filename
    self.upload_from_file(
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2556, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/app/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 4368, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.ServiceUnavailable: 503 POST https://storage.googleapis.com/upload/storage/v1/b/moz-fx-productdelivery-pr-38b5-productdelivery/o?uploadType=multipart: Service Unavailable: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
exit code: 1

Comment 1

[Intermittent Failures Robot]

1 failures in 3856 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1817535&startday=2023-02-13&endday=2023-02-19&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 3966 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• windows2012-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1817535&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 3

[Intermittent Failures Robot]

4 failures in 3863 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• osx-shippable: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1817535&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 4

[Intermittent Failures Robot]

21 failures in 670 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 21

Platform and build breakdown:

• windows2012-64-shippable: 10
  • opt: 10
• windows2012-32-shippable: 9
  • opt: 9
• osx-shippable: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1817535&startday=2023-04-03&endday=2023-04-03&tree=all

Comment 5

[Intermittent Failures Robot]

21 failures in 3386 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

Repository breakdown:

• mozilla-central: 21

Platform and build breakdown:

• windows2012-64-shippable: 10
  • opt: 10
• windows2012-32-shippable: 9
  • opt: 9
• osx-shippable: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1817535&startday=2023-04-03&endday=2023-04-09&tree=all