Closed Bug 1461332 Opened 6 years ago Closed 6 years ago

Language changes to English for some non-English locales after updating in-browser to Firefox 60

Categories

(External Software Affecting Firefox :: Other, defect)

Product:
External Software Affecting Firefox
Component:
Other
Platform:
x86_64
Windows 10
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(firefox-esr60 fixed, firefox60blocking fixed, firefox61 fixed, firefox62 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr60 --- fixed
firefox60 blocking fixed
firefox61 --- fixed
firefox62 --- fixed

People

(Reporter: mklltech, Unassigned)

References

Details

(Keywords: jp-critical, Whiteboard: [AV:Kaspersky]) 

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Build ID: 20180510203301

Steps to reproduce:

Updated browser (Japanese) via the Firefox Menu >> (?) Help >> About Firefox method.



Actual results:

Upon restart, the browser menus and related items were no longer in Japanese, but instead in English.


Expected results:

The menu's should not have changed to en-US (English)

The solution right now is to run the Japanese installer from here: https://ftp.mozilla.org/pub/firefox/releases/60.0/win64/ja/
I should note, this is mainly happening on Windows - not Linux.
Component: Untriaged → General
Keywords: jp-critical
Product: Firefox → Release Engineering
QA Contact: catlee
Version: 60 Branch → unspecified
Hardware: Unspecified → x86_64
OS: Unspecified → Windows 10
Component: General → Release Automation: L10N
QA Contact: catlee → bugspam.Callek
To be clear, this is "user on Firefox 59, Japanese; updates via in-app updater and gets English Firefox (with no Japanese translations)" without using a language pack, merely a fully translated copy?
Flags: needinfo?(mklltech)
Yes, they are using the Japanese locale version of Firefox from https://www.mozilla.org/en-US/firefox/all/
Flags: needinfo?(mklltech)
To be clear, I just tried to reproduce using Windows8, Win64, 59.0.3 japanese (https://archive.mozilla.org/pub/firefox/candidates/59.0.3-candidates/build1/win64/ja/firefox-59.0.3.zip) and then updated to 60.0, and all menu items and such are still japanese after restart.
I asked for more information, and these users were using auto-updates.
Another report of this happening via Twitter: https://twitter.com/c_sis/status/996028334235582465
Even more users are reporting this is happening with automatic update now: https://twitter.com/Ajx_Resphoina/status/996038118905016325
This is only happening when users are using Automatic updates - not from manual as originally thought.
I'm still unable to reproduce, so can we try and gather:

* "specifically what version did a user have before the update"
* "were they using a language pack (or even have any installed)"

Additionally I'd love if we can get any internal QA to validate these user reports.


What I tested locally:

* Win64
* Via .zip file (since my windows is actively using Firefox and didn't want to deal with installer changes affecting system with locale stuff)
* Previous versions: 59.0, 59.0.2 and 59.0.3
* Update Types Complete from 59.0 and partial from 59.0.2 and 59.0.3
* Language "ja" (Japanese)
* Update via in-product dialog
* With a fresh profile each attempt.
CCing mkaply, too.

It'd be good to know if these people have distributions of Firefox, or if they installed themselves.

Also, bug 1450270 has QE verification, but maybe we missed something.
Is it possible these were english users with the Japanese Locale? And after update, the Japanese Locale wasn't available yet?
No, I don't believe so, as these users speak Japanese only.
> Also, bug 1450270 has QE verification, but maybe we missed something.

How is this bug related to Acer case?

Also, this is not just language negotiation. Majority of Firefox ja-JP UI which doesn't use Fluent yet does not have English resources at all, so it cannot just "switch to English". It has to switch from "Firefox ja-JP" to "Firefox en-US" or something...

Based on the description I'm wondering if it's possible that we somehow push them partial/full update to en-US?
Another report on Twitter: https://ftp.mozilla.org/pub/firefox/releases/60.0/win64/ja/
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #15)
> > Also, bug 1450270 has QE verification, but maybe we missed something.
> Based on the description I'm wondering if it's possible that we somehow push
> them partial/full update to en-US?

Based on my own testing I find it unlikely they got en-US as an update (complete update gave me japanese, and partials which would fail if existing app doesn't match what it was generated against, also gives me japanese).

(I'm open to anyone here who can reproduce telling me I'm wrong though)
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #15)
> > Also, bug 1450270 has QE verification, but maybe we missed something.
> 
> How is this bug related to Acer case?
> 
> Also, this is not just language negotiation. Majority of Firefox ja-JP UI
> which doesn't use Fluent yet does not have English resources at all, so it
> cannot just "switch to English". It has to switch from "Firefox ja-JP" to
> "Firefox en-US" or something...
> 
> Based on the description I'm wondering if it's possible that we somehow push
> them partial/full update to en-US?

I looked into this a bit when I saw the report on reddit, and I couldn't see a way this could happen. ja users update with urls like https://aus5.mozilla.org/update/3/Firefox/59.0.2/20180323154952/WINNT_x86_64-msvc/ja/release/default/default/default/update.xml?force=1

Those urls point at MARs such as http://download.mozilla.org/?product=firefox-60.0-partial-59.0.2&os=win64&lang=ja&force=1.

Unpacking that, I get files such as localization/ja/browser/preferences/containers.ftl in omni.ja.

So, that confirms that bouncer + files on archive are OK. I can't find a way to make the update server point at non-Japanese bouncer links for Japanese update URLs. If we could get update logs (app.update.log -> True, then check for updates) on a case that reproduces this, we could confirm everything on the update mechanism.
Mkll - can you or someone who can reproduce it copy&paste here the bottom section (Internationalization & Localization ) of their `about:support` page before and after the language change?

We should be able to reject some hypothesis based on that.
Flags: needinfo?(mklltech)
we were seeing numerous reports on german support channels of users whose ui got switched to english in the auto update to english as well.

here's the inconspicuous about:support log of one affected user: https://mozhelp.dynvpn.de/paste/?707bcfe9ece4ea19#z0qnJI7A6u9e21DWrp5K7j30amVLcrqqN2s4DfK5qD0=
Can you have this person check the values of:

general.useragent.locale
intl.locale.matchOS

and I need to know if they have default values or user set values.

Also the value of distribution.id
Looking at this about:support, there is definitely a problem:

Internationalization & Localization
-----------------------------------

Application Settings
Requested Locales: ["en-US"]
Available Locales: ["en-US"]
App Locales: ["en-US"]
Regional Preferences: ["en-US"]
Default Locale: "en-US"
Operating System
System Locales: ["de-DE"]
Regional Preferences: ["de-DE"]

de-DE isn't there for available locales.
We have it in about:support output:

```
Internationalization & Localization
-----------------------------------

Application Settings
Requested Locales: ["en-US"] // intl.locale.requested
Available Locales: ["en-US"]
App Locales: ["en-US"]
Regional Preferences: ["en-US"]
Default Locale: "en-US" // update.locale
Operating System
System Locales: ["de-DE"]
Regional Preferences: ["de-DE"]
```

This looks bad. I'd like to see the same entries prior to the switch so if anyone can reliably reproduce it, I'd appreciate that.

The Default Locale is the most interesting one for us since it is the locale Firefox is packaged into. If it's en-US it means that the user has "Firefox en-US" single locale build. If before that they had "de" or "ja" then it means they used to have a different build and we move them.

I think we have enough external evidence to switch to NEW and based on the severity of the experience I'm marking this as critical.

Should we consider pausing 60 uplift?
Severity: normal → critical
Status: UNCONFIRMED → NEW
Ever confirmed: true
(In reply to Mike Kaply [:mkaply] from comment #22)
> Can you have this person check the values of:
> 
> general.useragent.locale
> intl.locale.matchOS
> 
> and I need to know if they have default values or user set values.
> 
> Also the value of distribution.id

non of the preferences or language packs in about:addons were present. two affected users i could ask said they have kaspersky security software, which contains a software updater feature - perhaps while we're turning off auto updates, they take over and cause this mess?
(In reply to Mike Kaply [:mkaply] from comment #23)
> Looking at this about:support, there is definitely a problem:
> 
> Internationalization & Localization
> -----------------------------------
> 
> Application Settings
> Requested Locales: ["en-US"]
> Available Locales: ["en-US"]
> App Locales: ["en-US"]
> Regional Preferences: ["en-US"]
> Default Locale: "en-US"
> Operating System
> System Locales: ["de-DE"]
> Regional Preferences: ["de-DE"]
> 
> de-DE isn't there for available locales.

Possibly notable: Our german locale is "de", not "de-DE". I wonder if it's the same for Japan, where we use "ja"/"ja-JP-mac", not "ja-JP".
> non of the preferences or language packs in about:addons were present.

OK, that eliminates the distribution change I made. That's helpful I guess.

That seems to imply they somehow got updated to English,
Our team could not reproduce on the following:

58.0.2 ja --> to 60 ja on Win 10 x64
59.0.3 ja --> 60.0 ja on Win 10 x 64
59.0.3 ja --> 60.0 ja on Win 7 x 64
We have tested this issue with updating through Kaspersky Software Updater 2.0.0.623 on the following: 

59.0.1 ja --> 60 ja on Win 10 x86
59.0.3 ja --> 60.0 ja on Win 10 x64
59.0.3 ja --> 60.0 ja on Win 7 x64
59.0 ja   --> 60.0 ja on Win 7 x86


After an update, the browser did not start in a different locale.
Matt, can you think of anything in the installer and/or updater that might cause issues like the locale switching reported here  or the double entries in windows settings from comment 11?
Flags: needinfo?(mhowell)
Hi sorry, for the late response, but it seems you got the info you needed from me via another person.
Flags: needinfo?(mklltech)
We don't seem to have a pre-upgrade about:support for any affected user, no.  If you can reproduce or somehow get hold of one, that might be helpful.  Also, the update log as in comment 19.
Flags: needinfo?(mklltech)
Ah, understood - I'll try to get one of those.
I installed win10 fr, firefox 59 fr, updated to 60, firefox restarted in French.
However the windows control panel's "installed programs" screen shows en-US: https://cristau.org/~julien/tmp/win10-installed-programs.png
I guess comment 37 might be due to bug 1436662?
Any more information about it possibly being caused by Kaspersky?
Flags: needinfo?(mklltech)
Nevermind, got reports that don't even have antivirus!
I made a mistake installing the current version and not doing the update the way the report specified. I thought I should note, however, that the uninstaller was in English. Is that an issue?

This test is being done on a clean install of windows 10 on a VM.
See comment 38
I'm not sure what's going on here. I'd love to get some registry keys from the submitter of that Reddit post in comment 11, but it would be the exact ones that they say they've now deleted. I'll see if I can get anywhere on reproducing any of this.

Julien is right about comment 37, it's due to bug 1436662 and isn't related to the rest of what's happening here; I tried to get that fix uplifted for 60 but couldn't get it approved.
Flags: needinfo?(mhowell)
One aspect of the reports that haven't been explored is what's mentioned in comment 9 about automatic vs. manual updates.

For people trying to reproduce the issue, it'd be good to *not* trigger the update through the About Dialog, but instead reduce the update timer (app.update.interval) and wait for the browser to update itself.


The issue reported on the Reddit post appears to be the same, because according to the screenshot, the version that the user had as 59 was en-GB, and the version on 60 is en-US.
Background updates are off for 60 at the moment, so you'd only get 59.0.2 that way.
It would be possible to test background updates on the release-cdntest channel though.
Confirmed to be caused by Kaspersky.
I think I just reproduced this, with the following steps:

- on a fresh win10 fr install
- install kaspersky internet security's trial version (https://www.kaspersky.fr/internet-security)
- install firefox 58.0 win64 fr
- background update to 59.0.2 fr
- check for software updates in kaspersky
- accept the firefox 60.0 update there (https://cristau.org/~julien/tmp/kaspersky-update.png)
- now firefox is in english

Random theories that might (or might not) explain this:
- kaspersky keys off the language of our installer, and since that's not translated (bug 1436662) it picks the english version
- kaspersky keys off general.useragent.locale and since that's gone in 59 it falls back to en-US
... and since our background updates for 60 have been off for a few days, the kaspersky tool goes in and updates behind our updater's back and this might explain why we're getting reports early this week.
Do we have any estimation of the affected population (windows+kaspersky+non-en-US)?
(In reply to Julien Cristau [:jcristau] from comment #48)
> Random theories that might (or might not) explain this:
> - kaspersky keys off the language of our installer, and since that's not
> translated (bug 1436662) it picks the english version

i've tested this and they really seem to go just by the locale's suffix that's showing up in the uninstaller string of the windows control panel and that got wrongly set to en-us in bug bug 1436662 & bug 1451480
Depends on: 1436662
Can we also please ask them to use different method? They should really look into `update.locale` file.
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #52)
> Can we also please ask them to use different method? They should really look
> into `update.locale` file.

Having external tools look into our omnijar doesn't sound like a reasonable place to go either, to be honest.
They shouldn't be doing this at all. We should just ask them to stop.
Folks, tomorrow we will disable FireFox update in Kaspersky Software Updater, till the problem wont be cleared. But as written in reddit the problem is still exist even without Kaspersky product - https://www.reddit.com/r/firefox/comments/8ifibw/since_v60_i_have_two_different_entries_of_firefox/dz02le2/.
I doubt there's a single scenario where we are not pushing updates to our users and yet they should be pushed to them.

If we don't, it means it shouldn't be done. So I'm with Mike - Kaspersky should not do this at all.
We disable updates for Firefox in Kaspersky Software Updater, the problem should be gone. May be Mike and Zibi right, it is not completely correct to force update applications with auto update feature. We will check our Requirement for conflict with other software.
Sorry for inconvenience.
Component: Release Automation: L10N → Other
Product: Release Engineering → External Software Affecting Firefox
QA Contact: bugspam.Callek
Whiteboard: [AV:Kaspersky]
I'll go ahead and call the immediate issue fixed, we got the translated uninstaller back in 60.0.1 and Kaspersky stopped force-updating Firefox.
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox60: affected → fixed
status-firefox61: ? → fixed
status-firefox62: ? → fixed
status-firefox-esr60: ? → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.