Closed Bug 1554501 Opened 9 months ago Closed 9 months ago

Make push-apk tasks idempotent

Categories

(Release Engineering :: Release Automation: Pushapk, enhancement, P5)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: jlorenzo)

References

Details

Attachments

(2 files)

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=248419120&repo=mozilla-beta
Full log: https://queue.taskcluster.net/v1/task/ZtH_l64nTBqpIap6nM_y1Q/runs/0/artifacts/public/logs/live_backing.log


2019-05-26 11:59:06,537 - mozapkpublisher.common.googleplay - INFO - Uploading "/builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk"
2019-05-26 11:59:06,538 - googleapiclient.discovery - WARNING - media_mime_type argument not specified: trying to auto-detect for /builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk
2019-05-26 11:59:06,593 - googleapiclient.discovery - INFO - URL being requested: POST https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/08285841811199333129/apks?alt=json&uploadType=media
2019-05-26 11:59:19,077 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "apkUpgradeVersionConflict"
Traceback (most recent call last):
File "/builds/scriptworker/bin/pushapkscript", line 11, in <module>
sys.exit(main())
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 95, in main
client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 164, in sync_main
loop.run_until_complete(_handle_asyncio_loop(async_main, context))
File "/tools/python3/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
await async_main(context)
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 54, in async_main
strings_file)
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/googleplay.py", line 36, in publish_to_googleplay
skip_checks_fennec=bool(product_config.get('skip_checks_fennec')),
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 111, in push_apk
rollout_percentage,
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 135, in _upload_apks
edit_service.upload_apk(path)
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 66, in _transaction_required
return method(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 86, in upload_apk
media_body=apk_path
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/08285841811199333129/apks?alt=json&uploadType=media returned "APK specifies a version code that has already been used.">
exit code: 1

This issue has been known for a few years, but has never been an annoying-enough issue because:

  • We hit occasionally on staging releases, and it was an expected error
  • mozilla-central has always had a couple of pushes so we've always had new code to deploy

As a consequence, it's not an intermittent failure, per se, and it's not a regressions.

To me, the real fix would be to let the nightly decision task detect that a previous nightly graph was successfully resolved. However, that looks like an easy way to break existing workflow. Therefore, the easiest way is to just make the task not fail when the APKs have already been pushed. That's the paradigm we usually have on Taskcluster and we've implemented something similar in bug 1491262.

I'll get this fixed this cycle, because we'll see this bug more often once Fennec Nightly is built off mozilla-esr68.

Assignee: nobody → jlorenzo
Blocks: ESR-Fennec
Type: defect → enhancement
Depends on: 1548864
See Also: → 1491262
Summary: Intermittent googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/08285841811199333129/apks?alt=json&uploadType=media returned "APK specifies → Make push-apk tasks idempotent

Deployed and tested: https://tools.taskcluster.net/groups/b-xfKZ8CRHSdf-yRym0gvA/tasks/QuFIZpdRRFmjpO4G3_PHVA/runs/1/logs/public%2Flogs%2Flive_backing.log#L211

push-apk doesn't error out anymore when an APK has already been shipped.

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Blocks: 1562465
You need to log in before you can comment on or make changes to this bug.