Closed Bug 1554501 Opened 9 months ago Closed 9 months ago

Make push-apk tasks idempotent


(Release Engineering :: Release Automation: Pushapk, enhancement, P5)



(Not tracked)



(Reporter: intermittent-bug-filer, Assigned: jlorenzo)




(2 files)

Filed by: apavel [at]
Parsed log:
Full log:

2019-05-26 11:59:06,537 - mozapkpublisher.common.googleplay - INFO - Uploading "/builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk"
2019-05-26 11:59:06,538 - googleapiclient.discovery - WARNING - media_mime_type argument not specified: trying to auto-detect for /builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk
2019-05-26 11:59:06,593 - googleapiclient.discovery - INFO - URL being requested: POST
2019-05-26 11:59:19,077 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "apkUpgradeVersionConflict"
Traceback (most recent call last):
File "/builds/scriptworker/bin/pushapkscript", line 11, in <module>
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/", line 95, in main
client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/", line 164, in sync_main
loop.run_until_complete(_handle_asyncio_loop(async_main, context))
File "/tools/python3/lib/python3.6/asyncio/", line 468, in run_until_complete
return future.result()
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/", line 204, in _handle_asyncio_loop
await async_main(context)
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/", line 54, in async_main
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/", line 36, in publish_to_googleplay
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/", line 111, in push_apk
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/", line 135, in _upload_apks
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/", line 66, in _transaction_required
return method(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/", line 86, in upload_apk
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting returned "APK specifies a version code that has already been used.">
exit code: 1

This issue has been known for a few years, but has never been an annoying-enough issue because:

  • We hit occasionally on staging releases, and it was an expected error
  • mozilla-central has always had a couple of pushes so we've always had new code to deploy

As a consequence, it's not an intermittent failure, per se, and it's not a regressions.

To me, the real fix would be to let the nightly decision task detect that a previous nightly graph was successfully resolved. However, that looks like an easy way to break existing workflow. Therefore, the easiest way is to just make the task not fail when the APKs have already been pushed. That's the paradigm we usually have on Taskcluster and we've implemented something similar in bug 1491262.

I'll get this fixed this cycle, because we'll see this bug more often once Fennec Nightly is built off mozilla-esr68.

Assignee: nobody → jlorenzo
Blocks: ESR-Fennec
Type: defect → enhancement
Depends on: 1548864
See Also: → 1491262
Summary: Intermittent googleapiclient.errors.HttpError: <HttpError 403 when requesting returned "APK specifies → Make push-apk tasks idempotent

Deployed and tested:

push-apk doesn't error out anymore when an APK has already been shipped.

Closed: 9 months ago
Resolution: --- → FIXED
Blocks: 1562465
You need to log in before you can comment on or make changes to this bug.