1393727 - [tracking] improving mozapkpublisher to eliminate Google Play l10n contents fetch as SPOF

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P3)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We recently discovered that pushing the apk to the Google Play store can be blocker (at least automation wise, manually is a fallback option) to the Fennec releases if the Google Play l10n contents (what's new and Google Play listings, etc) website is down. To not be confused - this is not about the localized apk it self, this is not touching the binary at all - this is about the strings that are being showed in the Google Play store for users along with serving the new apk.  

Current logic asks for those l10n contents from a website[1] that is hosted by Community IT. Recent events[2] showed that due to various reasons, the website is occasionally down. A recent postmortem happened in bug 1385245. For this reason, we need to eliminate this fetch-data call as a single point of failure (SPOF) and not block the release on it.

:jlorenzo suggested the first thing we should solve should be caching the results elsewhere, in a RelEng owned infrastructure, or in any case a back-up mirroring solution, so that we never end up in situations like this. This should cover us for at least 99% of the possible fall-outs. 

How's that working?
* use something similar to l10n bumper[4] to mirror content from [1] into a github repository or alike.
* have mozapkpublisher talk to that instead
* if Github/hg is down, default to [1]

Additional improvement after this:
* decouple the GP l10n contents from the actual push apk process so that we can either ignore it or rerun it or tweak it easier, without having it block the actual push of the release

Will file blocking bugs against this one, for all the improvements that are to follow-up here.
Also, :jlorenzo pointed out some Github issues[5][6][7][8] to match for these bugs. 

[1]: http://l10n.mozilla-community.org/
[2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1383642
[3]: https://bugzilla.mozilla.org/show_bug.cgi?id=1385245
[4]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/testing/mozharness/configs/l10n_bumper/mozilla-beta.py
[5]: https://github.com/mozilla-releng/mozapkpublisher/issues/37
[6]: https://github.com/mozilla-releng/mozapkpublisher/issues/36
[7]: https://github.com/mozilla-releng/pushapkscript/issues/25
[8]: https://github.com/mozilla-releng/mozapkpublisher/issues/39

Comment 1

[Johan Lorenzo [:jlorenzo]]

We now have the ability to ship Fennec without depending on strings thanks to bug 1385401. If l10n.mozilla-community.org was to be down again, we can:
* forget about updating these strings
* tweak the google-play-strings task to fetch data elsewhere.

It's not a single point of failure anymore, we can now close this bug! \o/