1314063 - Latest Aurora build on Google Play is in English

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Théo Chevalier]

Attachment: about:buildconfig

After this initial discussion [1], I checked the Aurora build I installed from Google Play, and it was in English (it used to be in French). I have the latest update.

Flod checked the ftp, the multilocale build seem fine, so we might be serving single locale en-US build.

From what I can see, the list of languages in Settings only contains “System language”.

[1] https://groups.google.com/d/msg/mozilla.dev.l10n/hT8d-PEaIG4/p3G8cIS8BgAJ

Comment 1

[Johan Lorenzo [:jlorenzo]]

That's very likely related to the interaction of both pushapkworker and Sylvestre's cron script. Investigating.

Comment 2

[Johan Lorenzo [:jlorenzo]]

I suspect a misbehavior between Sylvestre's cron script and the data pushed by the worker.

pushapkworker fetches artifacts from [1]. :Callek pointed out these builds are en-US builds only. We should get l10n builds from [2], which are very likely uploaded once every day. Until fennecs nighlies are all done in taskcluster, a workaround would be to either:
* listen to pulse messages linked to [2]
* or listen to builbot pulse messages (proposed by :catlee)

I looked at pushapkworker configuration and it turns out it's a little bit more than just update its configuration. I'll take care of the code changes and the configuration. Once that is done, all new apks will multilocales, and no bad behavior between the worker and the cron script should happen.


[1] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-aurora.nightly.latest.mobile/gecko.v2.mozilla-aurora.nightly.latest.mobile
[2] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-aurora.latest.mobile-l10n.android-api-15-opt/gecko.v2.mozilla-aurora.latest.mobile-l10n.android-api-15-opt.multi

Comment 3

[Sylvestre Ledru [:Sylvestre]]

I haven't changed anything about the script and I am not using pushapkworker yet.
Are the multi APK on the ftp correct?

Here is my script:
----
VERSION=$( curl -s https://product-details.mozilla.org/1.0/firefox_versions.json|grep 'FIREFOX_AURORA": '|sed -e 's|.*": "\(.*\)",|\1|g')
cd /home/sylvestre/dev/mozilla/mozilla-central.hg/testing/mozharness/scripts/
rm -rf apk-download
python get_apk.py --latest-aurora
python push_apk.py --package-name org.mozilla.fennec_aurora --service-account @developer.gserviceaccount.com --credentials googleplay.p12 --apk-x86=apk-download/fennec-$VERSION.multi.android-i386.apk --apk-armv7-v15=apk-download/fennec-$VERSION.multi.android-arm-api-15.apk --push_apk --track beta
----

Comment 4

[Théo Chevalier]

(In reply to Sylvestre Ledru [:sylvestre] from comment #3)

> Are the multi APK on the ftp correct?
This build seem correct on my phone (Got it in French) http://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-15/fennec-51.0a2.multi.android-arm.apk

When I install back the Store version, it’s in English

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: task creator (for pushapkworker) PR

:Sylvestre: I suspect the en-US build got in there when I asked you (over IRC) if we could update the beta track to a given version code. If that's the case, pushapkworker was the one which uploaded the APK. And that APK came from the nightly namespace (see comment 2).

:Callek: Here's the PR which changes the task creator for pushapkworker. Rail has been the one reviewing patch for it, but based on the discussion we had on IRC yesterday, I believe you have more context about what this PR fixes.

Once a new pulse message is emitted, the task creator verifies if APKs for every Android architecture is ready, by checking the TC index. The PR basically makes where the namespace location configurable, so we may tweak where to find builds without any code change.

The default value has also been changed, so we point to the multi-locale builds by default.

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8806292 [details] [review]
task creator (for pushapkworker) PR

First glance this looks correct. The new index namespace is correct, I'll let rail review the remaining code changes here.

Comment 7

[Johan Lorenzo [:jlorenzo]]

:Sylvestre and I found out the root cause. Here's the error returned by Sylvestre's script:
> 13:06:28    FATAL - HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload
> /androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/01409953595506263635
> /apks?uploadType=media&alt=json returned "APK specifies a version code that has already been used.">

This means if an APK is uploaded once, it can't be uploaded one more time, even if that's for another track. You can only promote an existing apk from one track to another. So even if pushapkworker pushes to the "alpha" track, the "beta" one can't be correctly updated if APK share the same version code.

I opened up the current latest en-US (nightly) build[1] and the multi-locale one[2], with apktool[3]. They both do show the same version code. That's why the multi-locale APK can't be uploaded by Sylvestre's script. Pushapkworker is too fast and has pushed en-US builds right after they're uploaded on Taskcluster.

I landed attachment 8806292 [details] [review][4], but I shut down the task creator, so pushapkworker won't publish anything. Let's keep this situation as is, until we manually push a multi-locale Aurora. It can't be done until a new Aurora is created. Like the error message shows, Google Play is very restrictive about what version codes are uploaded. Also, for security concerns, Google Play store doesn't allow to unpublish an existing APK.

Once things are back to normal, I'll restart the task creator, and manually verify if the alpha track delivers multi-locale builds. If test results are okay, I'll add a verification step in mozapkpublisher (proposed by :catlee), so we won't publish a non-multi-locale build anymore. 

[1] https://public-artifacts.taskcluster.net/LeIvdXoLShK9Y0IIekKavQ/0/public/build/fennec-51.0a2.en-US.android-i386.apk
[2] https://public-artifacts.taskcluster.net/SPbk015SS2SJGv4M108Kxw/0/public/build/fennec-51.0a2.multi.android-i386.apk
[3] https://bitbucket.org/iBotPeaches/apktool
[4] https://github.com/mozilla-releng/fennec-aurora-task-creator/commit/9d05cc64e527dfdccb578640d50df7b9579c811f

Comment 8

[Sylvestre Ledru [:Sylvestre]]

Johan disabled it scripts, mine took over and uploaded it today, this is back in French now :)

Comment 9

[Johan Lorenzo [:jlorenzo]]

My Android updated to a French build as well. Théo told me (on IRC) his phone is back too. Marking this issue as fixed.