Latest Aurora build on Google Play is in English

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: tchevalier, Assigned: jlorenzo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Created attachment 8805988 [details]
about:buildconfig

After this initial discussion [1], I checked the Aurora build I installed from Google Play, and it was in English (it used to be in French). I have the latest update.

Flod checked the ftp, the multilocale build seem fine, so we might be serving single locale en-US build.

From what I can see, the list of languages in Settings only contains “System language”.

[1] https://groups.google.com/d/msg/mozilla.dev.l10n/hT8d-PEaIG4/p3G8cIS8BgAJ
(Assignee)

Comment 1

2 years ago
That's very likely related to the interaction of both pushapkworker and Sylvestre's cron script. Investigating.
Component: Build Config & IDE Support → Release Automation
Product: Firefox for Android → Release Engineering
QA Contact: rail
(Assignee)

Comment 2

2 years ago
I suspect a misbehavior between Sylvestre's cron script and the data pushed by the worker.

pushapkworker fetches artifacts from [1]. :Callek pointed out these builds are en-US builds only. We should get l10n builds from [2], which are very likely uploaded once every day. Until fennecs nighlies are all done in taskcluster, a workaround would be to either:
* listen to pulse messages linked to [2]
* or listen to builbot pulse messages (proposed by :catlee)

I looked at pushapkworker configuration and it turns out it's a little bit more than just update its configuration. I'll take care of the code changes and the configuration. Once that is done, all new apks will multilocales, and no bad behavior between the worker and the cron script should happen.


[1] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-aurora.nightly.latest.mobile/gecko.v2.mozilla-aurora.nightly.latest.mobile
[2] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-aurora.latest.mobile-l10n.android-api-15-opt/gecko.v2.mozilla-aurora.latest.mobile-l10n.android-api-15-opt.multi
Assignee: nobody → jlorenzo
(Assignee)

Updated

2 years ago
Status: NEW → ASSIGNED
I haven't changed anything about the script and I am not using pushapkworker yet.
Are the multi APK on the ftp correct?

Here is my script:
----
VERSION=$( curl -s https://product-details.mozilla.org/1.0/firefox_versions.json|grep 'FIREFOX_AURORA": '|sed -e 's|.*": "\(.*\)",|\1|g')
cd /home/sylvestre/dev/mozilla/mozilla-central.hg/testing/mozharness/scripts/
rm -rf apk-download
python get_apk.py --latest-aurora
python push_apk.py --package-name org.mozilla.fennec_aurora --service-account @developer.gserviceaccount.com --credentials googleplay.p12 --apk-x86=apk-download/fennec-$VERSION.multi.android-i386.apk --apk-armv7-v15=apk-download/fennec-$VERSION.multi.android-arm-api-15.apk --push_apk --track beta
----
(In reply to Sylvestre Ledru [:sylvestre] from comment #3)

> Are the multi APK on the ftp correct?
This build seem correct on my phone (Got it in French) http://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-15/fennec-51.0a2.multi.android-arm.apk

When I install back the Store version, it’s in English
(Assignee)

Comment 5

2 years ago
Created attachment 8806292 [details] [review]
task creator (for pushapkworker) PR

:Sylvestre: I suspect the en-US build got in there when I asked you (over IRC) if we could update the beta track to a given version code. If that's the case, pushapkworker was the one which uploaded the APK. And that APK came from the nightly namespace (see comment 2).

:Callek: Here's the PR which changes the task creator for pushapkworker. Rail has been the one reviewing patch for it, but based on the discussion we had on IRC yesterday, I believe you have more context about what this PR fixes.

Once a new pulse message is emitted, the task creator verifies if APKs for every Android architecture is ready, by checking the TC index. The PR basically makes where the namespace location configurable, so we may tweak where to find builds without any code change.

The default value has also been changed, so we point to the multi-locale builds by default.
Attachment #8806292 - Flags: review?(bugspam.Callek)
Comment on attachment 8806292 [details] [review]
task creator (for pushapkworker) PR

First glance this looks correct. The new index namespace is correct, I'll let rail review the remaining code changes here.
Attachment #8806292 - Flags: review?(rail)
Attachment #8806292 - Flags: review?(bugspam.Callek)
Attachment #8806292 - Flags: feedback+
Attachment #8806292 - Flags: review?(rail) → review+
(Assignee)

Comment 7

2 years ago
:Sylvestre and I found out the root cause. Here's the error returned by Sylvestre's script:
> 13:06:28    FATAL - HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload
> /androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/01409953595506263635
> /apks?uploadType=media&alt=json returned "APK specifies a version code that has already been used.">

This means if an APK is uploaded once, it can't be uploaded one more time, even if that's for another track. You can only promote an existing apk from one track to another. So even if pushapkworker pushes to the "alpha" track, the "beta" one can't be correctly updated if APK share the same version code.

I opened up the current latest en-US (nightly) build[1] and the multi-locale one[2], with apktool[3]. They both do show the same version code. That's why the multi-locale APK can't be uploaded by Sylvestre's script. Pushapkworker is too fast and has pushed en-US builds right after they're uploaded on Taskcluster.

I landed attachment 8806292 [details] [review][4], but I shut down the task creator, so pushapkworker won't publish anything. Let's keep this situation as is, until we manually push a multi-locale Aurora. It can't be done until a new Aurora is created. Like the error message shows, Google Play is very restrictive about what version codes are uploaded. Also, for security concerns, Google Play store doesn't allow to unpublish an existing APK.

Once things are back to normal, I'll restart the task creator, and manually verify if the alpha track delivers multi-locale builds. If test results are okay, I'll add a verification step in mozapkpublisher (proposed by :catlee), so we won't publish a non-multi-locale build anymore. 

[1] https://public-artifacts.taskcluster.net/LeIvdXoLShK9Y0IIekKavQ/0/public/build/fennec-51.0a2.en-US.android-i386.apk
[2] https://public-artifacts.taskcluster.net/SPbk015SS2SJGv4M108Kxw/0/public/build/fennec-51.0a2.multi.android-i386.apk
[3] https://bitbucket.org/iBotPeaches/apktool
[4] https://github.com/mozilla-releng/fennec-aurora-task-creator/commit/9d05cc64e527dfdccb578640d50df7b9579c811f
(Assignee)

Updated

2 years ago
Blocks: 1311624
Johan disabled it scripts, mine took over and uploaded it today, this is back in French now :)
(Assignee)

Comment 9

2 years ago
My Android updated to a French build as well. Théo told me (on IRC) his phone is back too. Marking this issue as fixed.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Blocks: 1314712
You need to log in before you can comment on or make changes to this bug.