Closed Bug 1349039 Opened 3 years ago Closed 3 years ago

mozapkpublisher: Let push_apk.py publish new locales

Categories

(Release Engineering :: Release Automation: Other, enhancement)

enhancement
Not set

Tracking

(firefox53blocking fixed)

RESOLVED FIXED
Tracking Status
firefox53 blocking fixed

People

(Reporter: aki, Assigned: jlorenzo)

Details

Attachments

(2 files)

See https://github.com/mozilla-releng/mozapkpublisher/issues/21 .

We seem to have hit this in https://github.com/mozilla-l10n/stores_l10n/issues/105 and https://github.com/mozilla-l10n/stores_l10n/issues/43

We need some way to publish 53.0b4, or make sure we don't hit this in 53.0b5.
Marking this as a blocker for 53. Might it also affect 52.0.2 if we have that planned?  Why would it affect only beta?
Johan, is this something you can take on, so that we can start the beta 5 build for fennec? Thanks!
Flags: needinfo?(jlorenzo)
You didn't see this because the locale wasn't complete, like it happened for es-MX, and was skipped during the publish.

This is getting really annoying.
(In reply to Francesco Lodolo [:flod] from comment #3)
> You didn't see this because the locale wasn't complete, like it happened for
> es-MX, and was skipped during the publish.

I've disable nb-NO on my side, but we clearly need a long term solution here
https://github.com/mozilla-l10n/stores_l10n/pull/116

We have the problem with es-MX and nb-NO at this point.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #2)

As we've previously taken out es-MX and en-GB, I'd suggest we keep nb-NO disabled on stores_l10n. Thanks to that, 53.0b4 has correctly been uploaded[1]. That'll allow the next betas to be updated as well. If this locale needs to be updated, we're still able to manually change the strings on the web interface.

What do you all think?

[1] https://tools.taskcluster.net/task-group-inspector/#/XiDs4q2uThqW8dS42R4IIw/K3gDs-y8RXG0z0tmkN8Elg?_k=ku0mg4
Flags: needinfo?(jlorenzo)
I don't mind keeping nb-NO and es-MX disabled until we find a proper solution.

Note that, if a locale is disabled, we stop exposing content for localization all over the place.
:nalexander,

I proposed this as a possible long-term fix here [1]:

> add a flag into the l10n process + l10n-changesets.json to specify we should repack locale nb-NO as locale no-NO.
> Since the multilocale build needs the nb-NO name to know which repo to pull from, we need some build/repack magic
> to repack this under a different name, so the apk shows no-NO and we don't have to do any translating during push
> time.

That means l10n-changesets.json could look like

{
    "nb-NO": {
        "google_locale": "no-NO",
        "platforms": [
            "android", 
            "android-api-15", 
            "android-multilocale"
        ],
        "revision": "default"
    },
    ...
}

* we'd pull the nb-NO strings
* we'd somehow tell the build system to use nb-NO strings but call it no-NO in the apk
** possibly by cloning into an no-NO/ dir? or some other way

Then, during push-to-google-play time, there would be no discrepancy between the apk locale (no-NO) and the google play locale (no-NO).

Does this sound plausible or crazy?

[1] https://github.com/mozilla-releng/mozapkpublisher/issues/21#issue-215545578
Flags: needinfo?(nalexander)
(In reply to Aki Sasaki [:aki] from comment #7)
> :nalexander,
> 
> I proposed this as a possible long-term fix here [1]:
> 
> > add a flag into the l10n process + l10n-changesets.json to specify we should repack locale nb-NO as locale no-NO.
> > Since the multilocale build needs the nb-NO name to know which repo to pull from, we need some build/repack magic
> > to repack this under a different name, so the apk shows no-NO and we don't have to do any translating during push
> > time.
> 
> That means l10n-changesets.json could look like
> 
> {
>     "nb-NO": {
>         "google_locale": "no-NO",
>         "platforms": [
>             "android", 
>             "android-api-15", 
>             "android-multilocale"
>         ],
>         "revision": "default"
>     },
>     ...
> }
> 
> * we'd pull the nb-NO strings
> * we'd somehow tell the build system to use nb-NO strings but call it no-NO
> in the apk
> ** possibly by cloning into an no-NO/ dir? or some other way
> 
> Then, during push-to-google-play time, there would be no discrepancy between
> the apk locale (no-NO) and the google play locale (no-NO).
> 
> Does this sound plausible or crazy?
> 
> [1]
> https://github.com/mozilla-releng/mozapkpublisher/issues/21#issue-215545578

I'm really struggling to put together the context here, and wonder if I'm the right person to help.  Should this be Jeff Beatty, who knows more about the l10n + Android details?  Or Callek, who has done the most work on getting l10n and Task Cluster to play together?

I can speak to the small point you had about the build system handling some of these remappings.  I expect it is possible, although perhaps not easy or fun, to map `nb-NO` to `no_NO` at repack time.  We already do some small remappings, and it might "Just Work" to do a few more: see https://dxr.mozilla.org/mozilla-central/rev/05bfa2831c0ba4a26fa72328ffe6a99aba9c356a/mobile/android/base/locales/Makefile.in#9.  If I read that correctly, we currently take strings from

https://hg.mozilla.org/l10n-central/he/file/tip/mobile/android/base/android_strings.dtd

and they'll end up in

$OBJDIR/mobile/android/base/res/values-iw

as Android requires.  That seems to be what you're hoping to achieve here, although I'm quite lost trying to figure out exactly which layer is responsible for this change.
Flags: needinfo?(nalexander)
I think that's very helpful.

We could continue that pattern across all non-matching locale names [1], which could get ugly, or deal with it in another location... possibly the mozharness level, where I'm probably the person to ask =\

[1] https://l10n.mozilla-community.org/stores_l10n/api/v1/google/localesmapping/?reverse (which seems to be missing en-GB, es-MX, and nb-NO atm due to bugs like this one?)
CCing :guerojeff in case he has something to add to comment #8 :)
CCing a bunch of other people for partners and work being done on l20n+mobile.

My concerns:
* Is this going to affect the language switcher used in Firefox for Android?

* We're starting to ship other Android products (Focus), and looking into using the same automation. We don't use m-c build system there, that would mean finding ad-hoc solutions for every project to overcome this limitation.

* I believe the long term goal it to ship a build with only en-US, and download other languages as language packs. It's a long way to come, but when that happens, this becomes a much larger problem. We won't be able to update any store description in that scenario, since the only locale available at submission would be en-US (unless we ship a minimal locale).

IMO the first step should be figuring out with Google why they find it smart to block us from updating no-NO if we don't ship that localization in the apk. 

As far as I'm able to understand (no visibility on our Play Store administration), we can't update any data, not just whatsnew (proof is that es-419 was es-ES content, not es-MX). Can someone confirm this? Is this affecting every store detail, or just whatsnew?
Summary: renamed locale prevents uploading to google play → Mismatch between shipping locale codes and Play Store locale codes prevents uploading to Google Play
So, the status quo, AFAICT, is that we map he to strings-il.xml and id to strings-in.xml, and that's all we do in terms of modifying data.

I've talked to gandalf about language selection and switching yesterday, so sending a "do me a favor" request his way:

Gandalf, can you check that locale switching to and from Indonesian and Hebrew work as expected? Probably both as OS language and explicitly set Firefox language.

BrowserLocaleManager.java has nothing special for them, nor the other three places I looked (browser.js and intl's android locale code). Thus I ask.
Flags: needinfo?(gandalf)
(In reply to Francesco Lodolo [:flod] from comment #11)
> CCing a bunch of other people for partners and work being done on
> l20n+mobile.
> 
> My concerns:
> * Is this going to affect the language switcher used in Firefox for Android?
> 
> * We're starting to ship other Android products (Focus), and looking into
> using the same automation. We don't use m-c build system there, that would
> mean finding ad-hoc solutions for every project to overcome this limitation.
> 
> * I believe the long term goal it to ship a build with only en-US, and
> download other languages as language packs. It's a long way to come, but
> when that happens, this becomes a much larger problem. We won't be able to
> update any store description in that scenario, since the only locale
> available at submission would be en-US (unless we ship a minimal locale).

These are good concerns to raise.  Let's figure these out before we proceed with any multilocale build changes.
For the last point, I believe that means we'd only be able to update the en-US what's new if en-US is the only in-apk locale.

> IMO the first step should be figuring out with Google why they find it smart
> to block us from updating no-NO if we don't ship that localization in the
> apk. 
> 
> As far as I'm able to understand (no visibility on our Play Store
> administration), we can't update any data, not just whatsnew (proof is that
> es-419 was es-ES content, not es-MX). Can someone confirm this? Is this
> affecting every store detail, or just whatsnew?

I don't know the details here, but aiui Google Play has certain rules and to prevent product owners from doing the wrong thing.  These change over time, and we have no control over them.  We can raise issues like this to them, but I think if we rely on them to change their behavior, we may be waiting a long time.
For the record, "ar" just failed with Fennec 53.0b5[1]. The same error occurred.

[1] https://tools.taskcluster.net/task-inspector/#dX_L3q2ORgWDDyMACYI8Xg/0
We may have solved this by updating the play store locale list.  Fingers crossed.
Attached file mozapkpublisher PR
Assignee: nobody → jlorenzo
Status: NEW → ASSIGNED
Flags: needinfo?(gandalf)
Attachment #8851008 - Flags: review?(aki)
Yesterday, I ran [1] which published listings for new locales. This allowed to publish 53.0b5[2] *with* the ar locale. Hence, mozapkpublisher can take care or uploading new locales every time we need it. In order to test this assumption, nb-NO hasn't been pushed with the rest of the locales at [2]. We'll use it to see if attachment 8851008 [details] [review] fixes it under real life conditions. This is likely gonna happen with 53.0b7.

[1] https://github.com/mozilla-releng/mozapkpublisher/blob/6575e5468c3b7002fa32a7407437df99e58eb49e/mozapkpublisher/update_apk_description.py
[2] https://tools.taskcluster.net/task-inspector/#dX_L3q2ORgWDDyMACYI8Xg/2
Summary: Mismatch between shipping locale codes and Play Store locale codes prevents uploading to Google Play → mozapkpublisher: Let push_apk.py publish new locales
Attachment #8851008 - Flags: review?(aki) → review+
And it worked! nb-NO managed to be uploaded automatically via the worker.

https://tools.taskcluster.net/task-inspector/#RC939naoRoqPpZpO9nSGdw/
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
I *think* this should be fixed for fx53 release; is that right Johan?
(We probably need to change the status-firefox53 flag to "fixed" if so)
Flags: needinfo?(jlorenzo)
It is. The patch was on the worker itself, which is shared across every branch. Toggling flag to fixed.
Flags: needinfo?(jlorenzo)
You need to log in before you can comment on or make changes to this bug.