1628405 - [meta] Removed app-provided Search Engines reappear on update when using language packs

Status: RESOLVED FIXED

(Firefox :: Search, defect, P2)

Description

[Mark Banner (:standard8)]

STR

• Download 74.0.1 en-US
• Create a new profile, and create a prefs.js file with:

user_pref("app.update.auto", false);
user_pref("browser.search.region", "CA");

• Start the new profile.
• Go into Preferences -> General -> Language
• From the language drop down, select "Search for more languages"
• Select to add a different language and add "French"
• Add it to the list (it should be at the top).
• Click ok
• Apply and restart.
• Go into Preferences -> Search -> One-Click Search Engines
• Remove "Qwant" and "Wikipedia (fr)" from the list.
• Restart and confirm that they are still removed.
• Firefox -> About -> Update to 75.0.0 & restart
• Tools -> Add-ons -> Languages

See the add-on pack is out of date

• Cog symbol -> Check for updates
• Restart Firefox
• Check the search engines

Actual Results

"Qwant" and "Wikipedia (fr)" have re-appeared

Expected Results

They should have stayed hidden.

Comment 1

[Mark Banner (:standard8)]

The issue here is to do with how the search service and language packs work together.

The search service uses appLocaleAsBCP47 for determining the current set of search engines. Currently for fr in the CA region, the list of app-provided search engines includes Qwant and Wikipedia (fr).

When Firefox updates, the language pack is out of date, so appLocaleAsBCP47 changes to en-US as that's the original build in use and the search service picks up on this and gets a new set of engines which no longer includes Qwant and Wikipedia (fr). So the settings (that they've been removed) for those engines are lost.

After the language pack updates, appLocaleAsBCP47 resets to fr and so Qwant and Wikipedia (fr) are now in the list of app-provided engines again. As the settings were lost, they get displayed again.

I've had various reports of this happening over the last few months.

I'm not quite sure what the solution is here. Axel tells me that "requestedLocale(s) should be considered unsafe data that users set to weird stuff in about:config", so switching to that might not be the right thing to do.

Comment 4

[Mark Banner (:standard8)]

I'd like to start working towards a fix for this - we're getting users hit it every release.

Axel/Zibi, any idea for a solution here? Please see comment 1 for in-depth details.

The fact that appLocaleAsBCP47 changes on upgrade due to the delay in the language pack being applied is breaking us. I don't think it is sensible for us to work around this in the search service, as that would require somehow storing settings longer, and then expiring them later.

This seems like more of an issue for the language pack handling to provide something consistent.

Comment 5

[Axel Hecht [:Pike]]

Romain has a proposal, AFAIK.

Comment 6

[Romain Testard [:RT]]

Shane and Mike, can you please help confirm if this is a duplicate of bug 1566084?

Comment 7

[Mark Banner (:standard8)]

Thanks for the redirect Axel.

(In reply to Romain Testard [:RT] from comment #6)

Shane and Mike, can you please help confirm if this is a duplicate of bug 1566084?

I think it is the same... as long as the solution makes sure that when the search service starts up, the language pack is applied.

However, I would like to keep the bug as a dependency for now, so that we have something direct to reference for the search description.

I'll leave the NIs open for now, to make sure Shane/Mike are aware.

Comment 8

[Zibi Braniecki [:zbraniecki][:gandalf]]

I think the LocaleService is doing the right thing here. It accurately reflects the state of the systems and reports on changes throughout the lifetime of the app.
I agree with Mark that this is not a dupe of bug 1566084 - it's a inderect consequence of it, so this bug should be blocked by that one.

Comment 9

[Mike Kaply [:mkaply]]

We'll make sure to test this with the new langpack solutions.

Comment 10

[Shane Caraveo (:mixedpuppy)]

I have nothing to add on top of the last couple comments.

Comment 11

[Mark Banner (:standard8)]

This is being worked on in bug 1566084 by other people. Dropping from my assigned list as I'm not active on it.

Comment 13

[Shane Caraveo (:mixedpuppy)]

Is there any chance you could test this on top of D81607? You may be able to base a test on how it's being tested in toolkit/mozapps/extensions/test/xpcshell/test_webextension_langpack.js

Comment 14

[Mathew Hodson]

Is this fixed now that all the bugs under bug 1566084 are fixed?

Comment 15

[Michael]

Today I have received a Firefox / Language Pack update.
All the search engines are back.

This is really annoying.

Comment 16

[Mark Banner (:standard8)]

(In reply to Mathew Hodson from comment #14)

Is this fixed now that all the bugs under bug 1566084 are fixed?

I think there still may be one case. I'm trying to find out.

(In reply to Michael from comment #15)

Today I have received a Firefox / Language Pack update.
All the search engines are back.

There are some changes in Firefox 81 (from bug 1647443 & others) that I believe should help this for the next update. Unfortunately we needed to get them into the released version before they will take effect.

Comment 17

[Mark Banner (:standard8)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #13)

Is there any chance you could test this on top of D81607? You may be able to base a test on how it's being tested in toolkit/mozapps/extensions/test/xpcshell/test_webextension_langpack.js

Filed bug 1668048.

Comment 18

[Michael]

(In reply to Mark Banner (:standard8) from comment #16)

There are some changes in Firefox 81 (from bug 1647443 & others) that I believe should help this for the next update. Unfortunately we needed to get them into the released version before they will take effect.

I don't know if I got it right. Should it be better now?

Because today all the search engines are back after an update of the UK language pack using Firefox 82.0.

Comment 19

[riccardo.robecchi]

I'd like to chime in to add that on my install the problem manifests itself even without updating. Following advice from another thread (on Reddit, I think) I tried disabling the search engine defined by the language pack, which in my case is Chambers (UK), but it always comes back after I restart Firefox. The odd thing is that in that case it has no icon, whereas after an update it does.
I'm using Firefox 84.0.2 on KDE neon (based on Ubuntu Focal).

Comment 20

[Mark Banner (:standard8)]

(In reply to riccardo.robecchi from comment #19)

I'd like to chime in to add that on my install the problem manifests itself even without updating. Following advice from another thread (on Reddit, I think) I tried disabling the search engine defined by the language pack, which in my case is Chambers (UK), but it always comes back after I restart Firefox. The odd thing is that in that case it has no icon, whereas after an update it does.
I'm using Firefox 84.0.2 on KDE neon (based on Ubuntu Focal).

This should start working properly after Firefox 85 is out - due out tomorrow - however, if you're still seeing issues, please file a bug against your distribution.

Comment 21

[riccardo.robecchi]

(In reply to Mark Banner (:standard8) from comment #20)

This should start working properly after Firefox 85 is out - due out tomorrow - however, if you're still seeing issues, please file a bug against your distribution.

The bug is still there with Firefox 85. Please note I am using the official Mozilla-maintained PPA for Ubuntu and I am not using the language pack provided by the distribution, but the one provided by Mozilla.

Comment 22

[Mark Banner (:standard8)]

(In reply to riccardo.robecchi from comment #21)

(In reply to Mark Banner (:standard8) from comment #20)

This should start working properly after Firefox 85 is out - due out tomorrow - however, if you're still seeing issues, please file a bug against your distribution.

The bug is still there with Firefox 85. Please note I am using the official Mozilla-maintained PPA for Ubuntu and I am not using the language pack provided by the distribution, but the one provided by Mozilla.

By "after Firefox 85" is out, I meant that it wouldn't apply until updates after you've updated to at least 85 and re-hidden the engines. Updates to 85 are still likely to be broken.

Comment 23

[riccardo.robecchi]

(In reply to Mark Banner (:standard8) from comment #22)

By "after Firefox 85" is out, I meant that it wouldn't apply until updates after you've updated to at least 85 and re-hidden the engines. Updates to 85 are still likely to be broken.

That was actually understood. Now that Firefox 86 is out and that I have updated to 85 one month ago, re-hidden the engines back then and done all the dance again, I can definitely say that the bug is still there. I just updated the UK language pack and the "Chambers (UK)" search engine popped up again, so the bug is not fixed.

Comment 24

[Mark Banner (:standard8)]

Please can you file a separate bug about that, I've no idea why that would be happening still, though I wonder if something is messing with the settings file (search.json.mozlz4).

Comment 25

[Mark Banner (:standard8)]

We think the main issue here has been fixed by bug 1675624 and various other work. If you are still seeing issues, please file new bugs and include as much information as possible.