1424953 - Move ja-JP-mac to be a well-formed BCP47 language tag

Status: RESOLVED DUPLICATE of bug 1450656

(Core :: Internationalization, enhancement, P3)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

Currently, we have a single locale in Firefox that is not a well-formed BCP47 tag.

This locale costs us some code complexity both at build and run time, and blocks us from further tightening the code to ensure BCP47 input/normalization.

I'd like to discuss various ways in which we could attempt to mitigate the problem.

I see three ways:

1) Migrate all Firefox to Fluent

This would allow us to use `PLATFORM` selector to separate strings that are specific to MacOS in `ja-JP` build, like this:

```
key =
    { PLATFORM() ->
        [mac] Mac specific string
       *[other] Regular string
    }
```

but the problem here is that the diff is pretty massive (1.1MB!) and I'm not sure what timeline can we realistically hope for to get all the strings affected to be migrated to Fluent and also it'll require some special migration scripts to merge ja and ja-JP-mac.

2) Move the code

This is my original idea - just change the repository name, move the code in all-locales and add some nsBrowserGlue.js migration code to change all `ja-JP-mac` to `ja-JP-macos`.

3) Start a new locale, deprecate the old one

Last idea, which I think may be the easiest one to implement, is to create a new locale `ja-JP-macos`, fill it with data from `ja-JP-mac` and over a couple releases migrate users of `ja-JP-mac` to `ja-JP-macos`.
Once the number of `ja-JP-mac` users on the latest release drops to 0, we can remove it.

I'd like to keep this bug as a place to discuss those and other options and gather feedback from stakeholders.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

NI the first set of likely stakeholders for their feedback.

Comment 2

[Axel Hecht [:Pike]]

Adding marsf and dynamis. I'd be curious if you guys have looked at the current features of fluent yet to see if that has what you'd need to implement the differences between ja and ja-JP-mac. Assume that we'd find a way to give you a central file to centralize most of the logic.

Technically, we'll want to also see what we need to or can do on the data side. Georg, are you a good person to talk to about that?

Also, if we migrate users away from the update-locale ja-JP-mac, that's going to have impact on releng and updates. catlee?

I'm still not happy with the target code. I think we should for sure drop the -JP. And I'd prefer to have a valid locale code to a well-formed (i.e., "macos" isn't a registered subtag).

Last but not least, I'd add another option:


4) Keep building ja-JP-mac, but transform that into ja-macos for runtime.

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

> 4) Keep building ja-JP-mac, but transform that into ja-macos for runtime.

That sounds very tempting. Can you elaborate?

I see a couple entry points:
 - update.locale
 - multilocale.json
 - chrome registry

We'd need to filter the input from them via `ja-JP-mac -> ja-macos` right?

> And I'd prefer to have a valid locale code to a well-formed (i.e., "macos" isn't a registered subtag).

What's the value for that? The well-formed BCP47 tags gives us access to all BCP47 parsing/serializing/manipulating algorithms and libraries (including the whole ECMA402 Intl API). What's the reason for requiring it to be valid?

Comment 4

[Masahiko Imanaka [:marsf]]

Historically, this is the second time to change the locale code for mac. 


(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> 1) Migrate all Firefox to Fluent
> 
> This would allow us to use `PLATFORM` selector to separate strings that are
> specific to MacOS in `ja-JP` build, like this:
> 
> (...snip)
> 
> but the problem here is that the diff is pretty massive (1.1MB!) and I'm not
> sure what timeline can we realistically hope for to get all the strings
> affected to be migrated to Fluent and also it'll require some special
> migration scripts to merge ja and ja-JP-mac.

I've heard about "PLATFORM" selector from Stas last year. But, merging ja and ja-JP-mac is not a solution. Because the "-mac" locale is a dialect of ja, and it just has no user on Windows/Linux platform.


> 2) Move the code
> 
> This is my original idea - just change the repository name, move the code in
> all-locales and add some nsBrowserGlue.js migration code to change all
> `ja-JP-mac` to `ja-JP-macos`.

This may be same with the first time (, without migration code). I'm not sure what was happened on build side.


> 3) Start a new locale, deprecate the old one
> 
> Last idea, which I think may be the easiest one to implement, is to create a
> new locale `ja-JP-macos`, fill it with data from `ja-JP-mac` and over a
> couple releases migrate users of `ja-JP-mac` to `ja-JP-macos`.
> Once the number of `ja-JP-mac` users on the latest release drops to 0, we
> can remove it.

This will be related to releng. I have no comment on this.

Comment 5

[Masahiko Imanaka [:marsf]]

FYI: https://bugzilla.mozilla.org/show_bug.cgi?id=288244 At the first time :)

Comment 6

[Axel Hecht [:Pike]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #3)
> > 4) Keep building ja-JP-mac, but transform that into ja-macos for runtime.
> 
> That sounds very tempting. Can you elaborate?
> 
> I see a couple entry points:
>  - update.locale

This should be ja-JP-mac, to serve continuity on updates.
We'd probably need to fiddle with https://dxr.mozilla.org/mozilla-central/source/intl/locale/LocaleService.cpp#627, either by making this read multilocale.json, or by tweaking the locale code.

>  - multilocale.json

At generation time?

>  - chrome registry

Not sure? Also, do we have a bug to move langneg here away from hand-written code? Was there a catch-22?

Also, we'd need to understand the impact on metrics still.

> > And I'd prefer to have a valid locale code to a well-formed (i.e., "macos" isn't a registered subtag).
> 
> What's the value for that? The well-formed BCP47 tags gives us access to all
> BCP47 parsing/serializing/manipulating algorithms and libraries (including
> the whole ECMA402 Intl API). What's the reason for requiring it to be valid?

Well, valid means that other tools can understand what our token means. The other is future compatibility. The "macos" subtag might at some point become a registered subtag, but with a conflicting meaning.


Regardless, my preferred route is to make ja-JP-mac a thing of the past via fluent.

Comment 7

[Axel Hecht [:Pike]]

(In reply to Masahiko Imanaka [:marsf] from comment #4)
> Historically, this is the second time to change the locale code for mac. 
> 
> 
> (In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> > 1) Migrate all Firefox to Fluent
> > 
> > This would allow us to use `PLATFORM` selector to separate strings that are
> > specific to MacOS in `ja-JP` build, like this:
> > 
> > (...snip)
> > 
> > but the problem here is that the diff is pretty massive (1.1MB!) and I'm not
> > sure what timeline can we realistically hope for to get all the strings
> > affected to be migrated to Fluent and also it'll require some special
> > migration scripts to merge ja and ja-JP-mac.
> 
> I've heard about "PLATFORM" selector from Stas last year. But, merging ja
> and ja-JP-mac is not a solution. Because the "-mac" locale is a dialect of
> ja, and it just has no user on Windows/Linux platform.

Well, right now, you have a single localization that you script into two different variants. This proposal would just move that scripting from build time and your landing script into runtime.

Comment 8

[Masahiko Imanaka [:marsf]]

(In reply to Axel Hecht [:Pike] from comment #7)
> Well, right now, you have a single localization that you script into two
> different variants. This proposal would just move that scripting from build
> time and your landing script into runtime.

Is that mean, we have to replace all "word variables" to "PLATFORM selector" as committing source?

Comment 9

[Georg Fritzsche [:gfritzsche]]

(In reply to Axel Hecht [:Pike] from comment #2)
> Technically, we'll want to also see what we need to or can do on the data
> side. Georg, are you a good person to talk to about that?

Sorry, i'm not sure what the context here is or what you need.
Can you expand?

Comment 10

[Mike Kaply [:mkaply]]

I don't think I have anything to offer here. From my perspective, we just need to make sure the search engines work properly between the two.

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

Bug 1450656 brings me back to this. I think I'll try to tackle this in 62 by moving the ja-JP-mac outside of core Gecko and only into packaging/updating/resource loading.

It would be significantly better if we could switch it to `ja-JP-macos` because ICU canonicalizes it to `ja-JP-x-lvariant-mac` which doesn't roll off the tongue really and a langpack with such name wouldn't look neat.

For that reason I may actually special-case `ja-JP-mac` to be canonicalized into `ja-JP-macos` on the "input", rather than into "ja-JP-x-lvariant-mac".

Comment 12

[Zibi Braniecki [:zbraniecki][:gandalf]]

This has been done in bug 1450656. I think that's as far as we can go with moving ja-JP-mac out of the system for now.