1899553 - Remove push-bundle from Android nightly graphs

Status: RESOLVED FIXED

(Release Engineering :: General, task)

Description

[Ryan VanderMeulen [:RyanVM]]

Right now, due to the ongoing issues with Google Play review lag, we need to be monitoring the Play Console continuously to check whether there is currently a submission in review and cancel the push-bundle task on a new Nightly graph if there is in order to avoid having the review clock reset on us. This requires people to be monitoring and taking action outside of their regular working hours which is far from ideal.

Since we have no reason to believe that this situation will improve any time soon based on the responses we've received from Google support, we should change our Nightly graphs to stop automatically submitting new builds to the Play Store so that we have safer defaults that require less manual intervention.

I propose the following:

• Remove push-bundle-fenix-nightly and push-bundle-focus-nightly from the Android Nightly graph
  • All other prerequisite jobs should remain in place. Only the final push jobs should be removed.
  • This should only apply to Nightly graphs. Beta and Release should continue to schedule their respective push-bundle jobs as usual.
• Change the cron back to scheduling nightly-all at both 10:00 and 22:00 rather than the current split between nightly-desktop and nightly-all.
• Release Management will continue to monitor the Play Console for submissions in review and use the Treeherder "Add new jobs" functionality to submit new builds to the store once ready.

If/when things return back to normal with respect to review time, we can add push-bundle back into the Nightly graphs, but for now this at least means we won't have to babysit every Nightly graph and manually cancel jobs lest we risk accidentally letting a new submission get through on us.

Comment 1

[Andrew Halberstadt [:ahal]]

Alternatively, is there a way to programatically check if a review is pending? Maybe the push-bundle tasks could bail out if they notice this. I agree your proposal is more expedient however.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I had that same thought and agree that it would also work if the Play Store API had such ability. The only concern I have is that there may be times when we want to submit a new build even if it does mean restarting the review clock, so we'd need some mechanism for overriding that behavior in those situations.

Comment 3

[Johan Lorenzo [:jlorenzo]]

Alternatively, is there a way to programatically check if a review is pending?

I had a quick look at the API[1]. I didn't see anything relevant. The review API they expose is to interact with the reviews users leave about your app. The server responses we log[2] don't show anything useful either:

2024-05-28 11:57:57,007 - googleapiclient.discovery - DEBUG - URL being requested: POST https://androidpublisher.googleapis.com/androidpublisher/v3/applications/org.mozilla.fenix/edits?alt=json
[...]
2024-05-28 11:57:58,790 - googleapiclient.discovery - DEBUG - URL being requested: POST https://androidpublisher.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.fenix/edits/07345100893184259321/bundles?alt=json&uploadType=media
2024-05-28 11:58:59,333 - mozapkpublisher.common.store - DEBUG - Upload response: {'versionCode': 2016023391, 'sha1': '98882171b4d44315cdf2c384b03d1ac4cba6e6cc', 'sha256': 'bbed83dd8e421e71cb396b346c0723bc2161e3bf1ecff79547417857f43f2f57'}
2024-05-28 11:58:59,335 - googleapiclient.discovery - DEBUG - URL being requested: PUT https://androidpublisher.googleapis.com/androidpublisher/v3/applications/org.mozilla.fenix/edits/07345100893184259321/tracks/production?alt=json
[...]
2024-05-28 11:58:59,550 - mozapkpublisher.common.store - DEBUG - Track update response: {'track': 'production', 'releases': [{'versionCodes': ['2016023391'], 'status': 'completed'}]}
2024-05-28 11:58:59,550 - googleapiclient.discovery - DEBUG - URL being requested: POST https://androidpublisher.googleapis.com/androidpublisher/v3/applications/org.mozilla.fenix/edits/07345100893184259321:commit?alt=json
[...]

[1] https://developers.google.com/android-publisher/api-ref/rest/v3/
[2] e.g.: https://firefox-ci-tc.services.mozilla.com/tasks/Pkl803OvRXuImWo3cPth_A/runs/1/logs/public/logs/live_backing.log

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1899553: create 'googleplay' action permission and grant it pushapk scopes r?#releng-reviewers!

This includes a new action permission (googleplay) and an explicit grant of the pushapk scopes to action on mozilla-central.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: Bug 1899553: grant relman access to trigger google play pushes via action hook r?#releng-reviewers!

Comment 6

[[e-mail] (:bhearsum)]

Attachment: Bug 1899553: Add google play action to allow manual triggering of google play upload r?#releng-reviewers!

Comment 7

[[e-mail] (:bhearsum)]

Attachment: Bug 1899553: switch nightly-all scheduling back to 2x day r?#releng-reviewers!

Comment 8

[[e-mail] (:bhearsum)]

Attachment: Bug 1899553: don't automatically schedule google play uploads on nightly r?#releng-reviewers!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/2b4575f02e47
create 'googleplay' action permission and grant it pushapk scopes r=releng-reviewers,jcristau
https://hg.mozilla.org/ci/ci-configuration/rev/e29e72b4fd9b
grant relman access to trigger google play pushes via action hook r=releng-reviewers,jcristau

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac6b7d5e1eb6
Add google play action to allow manual triggering of google play upload r=releng-reviewers,taskgraph-reviewers,jcristau
https://hg.mozilla.org/integration/autoland/rev/bcafa06d7a28
switch nightly-all scheduling back to 2x day r=releng-reviewers,jcristau
https://hg.mozilla.org/integration/autoland/rev/255542e0f255
don't automatically schedule google play uploads on nightly r=releng-reviewers,taskgraph-reviewers,jcristau

Comment 11

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/ac6b7d5e1eb6
https://hg.mozilla.org/mozilla-central/rev/bcafa06d7a28
https://hg.mozilla.org/mozilla-central/rev/255542e0f255