1968951 - Add an API to BrowsingContext to override locale in JS APIs

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, task, P2)

Description

[Alexandra Borovova [:Sasha]]

In order to implement the WebDriver BiDi command to emulate browser locale, we require an API on BrowsingContext to override the locale with the provided value.

Comment 1

[Alexandra Borovova [:Sasha]]

Attachment: Bug 1968951 - Add an API to BrowsingContext to override locale in JS APIs.

Comment 2

[Henri Sivonen (:hsivonen)]

thorin, this is likely relevant to your interests. Is there anything in particular that has arisen from the spoof_english experience that should be taken into account in the design here?

Comment 3

[Thorin [:thorin]]

spoof_english has been a bit of a pain over the years and we've found a number of places where the app language is used in web content

• click [example] on https://arkenfox.github.io/TZP/tests/elementforms.html
  • forms (default labels, sizes etc), input formats
• https://arkenfox.github.io/TZP/tests/applang.html
  • MediaDocument.properties
  • validation messages
    • including input formats
  • XML errors
  • other XML - e.g. Bug 1954813
  • intl.css - Bug 1534581 (also see attachment on https://bugzilla.mozilla.org/show_bug.cgi?id=1588797#c1)
• fonts e.g. Bug 1906643
• must be some more I've missed

And spoof_english needs to set intl.accept_languages and the locale, which in FF may still be problematic (in Tor Browser we found languages got in a stuck state due to a years old decision upstream, which I know hasn't been addressed - hard to find the issues right now)

Also, at some point, we would like to think we can replace spoof_english with spoof_language - i.e separate app and content fully now we know all the places they leak - i.e have a german interface, but request french content

Also, live language reload misses a few things and requires a restart (which we enforce in Tor Browser)

I'm not super savvy on BiDi and why/how it is used - but emulation with spoof_english (which also requires RFP to be enabled) isn't going to be perfect

ni piero for any further insights, items I've missed

Comment 4

[Pier Angelo Vendrame]

(In reply to Henri Sivonen (:hsivonen) (on vacation) from comment #2)

thorin, this is likely relevant to your interests. Is there anything in particular that has arisen from the spoof_english experience that should be taken into account in the design here?

spoof_english is kinda hacky 😒.
As Thorin said, it sets intl.accepted_language, which is problematic in tests (test validation detects this change and make tests fail).
So, there's a bug about reimplementing it without this change (I don't remember the number).

As regards strings, you often need to do special cases for spoof English.
This is problematic especially for strings in .properties, since you will need to always include them in omni.ja.
I tried to look into making the chrome://*/locale/ resolver aware of spoof English, but it was kinda difficult, because context is lost at a certain point.

Fluent makes things easier for spoof English, because all English .ftl files are always available.
However, this doesn't help for other languages (which might be this case, or the hypothetical spoof_language).
I recently added a constructor that accepts specific languages, but it needs to be called as a special case.
It'd be great if Fluent instead could know that it's running in a context where the language was changed.
At that point, spoof English would tell this context to use en-US, and it would hopefully prevent any future leak (or even existing ones that haven't been found yet).
However, I didn't have time to investigate that.

Comment 5

[Alexandra Borovova [:Sasha]]

Since using the spoof_english feature doesn't seem to be necessarily a good idea. I also would like to add maybe a bit more context here.

Although in the future, the clients might want the emulation to affect more APIs, at the moment it has to change only the following things (see more details in this spec issue):

• Accept-Language HTTP Headers;
• JavaScript APIs (Date, Intl), ensure that JavaScript Date and Internationalization APIs (Intl) reflect the correct locale-based formats (e.g., date, currency, numbers);
• Navigator.language/s.

So for us, it would be acceptable (and probably even preferable) to start just with emulation of these features.

Comment 6

[Thorin [:thorin]]

So for us, it would be acceptable (and probably even preferable) to start just with emulation of these features

I agree. spoof_english + BiDi is not a supported Tor Browser mode :)

Accept-Language HTTP Headers / Navigator.language/s

note: there are/may-be differences in geckoview (we fixed them in Tor Browser Android IIRC because we enforce and limit to our smaller set of app+ content languages) - not sure how deep the rabbit hole runs - e.g. Bug 1961578

Comment 7

[Alexandra Borovova [:Sasha]]

Hey Dan, since you reviewed the changes related to setting locale to SpiderMonkey APIs, maybe you can help me with the issues/questions which I have around it. We have tests written for this API written by a developer from Google, which I'm now using to verify my implementation, and they generally work fine apart from a few cases:

• three letter languages like: ast, yue, apc, gsw are reset to en-GB in new Intl.DateTimeFormat().resolvedOptions().locale. If call JS_GetDefaultLocale(context) after setting the locale, it returns the correct value. Also, if I call new Intl.DateTimeFormat("ast").resolvedOptions() it returns ast (the same for yue, gsw, but not for apc where I get en-DE). So not sure if that's expected, or I'm doing something wrong.
• some locales values get truncated: de-DE-1996 becomes de-DE, sl-Roza-biske -> sl, ca-ES-valencia -> ca-ES, sl-1994 -> sl, zh-Latn-CN-variant1-a-extend1-u-co-pinyin-x-private-zh-Latn-CN-variant1 -> zn (expected zh-Latn-CN-variant1) but the results are consistent with passing the locale to new Intl.DateTimeFormat(locale).resolvedOptions().locale, so I guess that probably fine, and we just can not expect the same behavior across the browsers for such locales?
• some locales values get truncated only with override: th-TH-u-nu-thai -> th-TH, en-US-u-ca-gregory -> en-US-u-ca-gregory, but e.g., new Intl.DateTimeFormat("th-TH-u-nu-thai").resolvedOptions().locale returns th-TH-u-nu-thai. So looks like in this case numberingSystem is not set, not sure if we then have to override something else?

Comment 8

[Alexandra Borovova [:Sasha]]

After some discussion, we decided to scope this bug to override locale in Javascript APIs only, which is what currently in the WebDriver BiDi specification. The override for navigator.language and Accept-Language header requires more spec discussion and will be probably added later.

Comment 9

[Dan Minor [:dminor]]

That all seems related to locale canonicalization, I wonder if it's not being applied consistently. It looks like behind the scenes the implementation for JS_SetDefaultLocale doesn't do any canonicalization, it's just copying the string.

I think it would be worth trying canonicalizing the locale before passing it in, here's an example. You should be able to call toSpan() to get a locale string back out.

It seems to me that JS_SetDefaultLocale should canonicalize the locale, but before I file a bug for that, I'd like to see if it makes any difference here.
From what you said, it also seems like JS_ResetDefaultLocale is not working as expected.

Hopefully this helps, if not, let me know and we can look into this further.

Comment 10

[Alexandra Borovova [:Sasha]]

Maybe I'm doing something wrong because I actually had a hard with getting the locale string back, because toSpan() doesn't seem to be available (I've got no member named 'toSpan' in 'mozilla::intl::Locale' error), but in the end it didn't seem to change anything. I would still be able to set locales like de-DE, but not ast. Just for the reference, I tried the following:

nsAutoCString lang = NS_ConvertUTF16toUTF8(GetLanguageOverride());
mozilla::intl::Locale loc;
if (mozilla::intl::LocaleParser::TryParse(lang, loc).isOk() && loc.Canonicalize().isOk()) {
  Vector<char, 32> locale;
  intl::VectorToBufferAdaptor buffer(locale);
  loc.ToString(buffer);
  JS_SetDefaultLocale(runtime, JS_EncodeStringToASCII(context, JS_NewStringCopyN(context, locale.begin(), locale.length())).get());
}

Comment 11

[Alexandra Borovova [:Sasha]]

Hey Dan, since the patch on this bug is approved now, and I think it's good enough for us to start. I've decided to file a bug already for the remaining issues: https://bugzilla.mozilla.org/show_bug.cgi?id=1975931, so I can unblock the review on BiDi API. Please move it to the right component (if the one I picked is wrong) or add the info if I missed anything.

Comment 12

[Dan Minor [:dminor]]

Yes, that makes sense to me. The APIs you're using don't have a lot of existing use at the moment, so it's possible they are buggy. Probably the best way ahead is to step through with a debugger for the problematic locales, and see if anything pops out as being weird.

Comment 13

[Alexandra Borovova [:Sasha]]

Attachment: Bug 1968951 - [wdspec] Remove valid "es-419" test case.

Comment 14

[Phabricator Automation]

Comment on attachment 9498938 [details]
Bug 1968951 - [wdspec] Remove valid "es-419" test case.

Revision D256296 was moved to bug 1968952. Setting attachment 9498938 [details] to obsolete.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d72bf8d7856f
https://hg.mozilla.org/integration/autoland/rev/afcaec3c8856
Add an API to BrowsingContext to override locale in JS APIs. r=jfkthame,dminor

Comment 16

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/afcaec3c8856