Closed Bug 1691706 Opened 2 years ago Closed 9 months ago

Removed search engines and disabled search engine suggestions reappear after updating Firefox

Categories

(Firefox :: Search, defect, P3)

Firefox 85
defect

Tracking

()

RESOLVED FIXED
99 Branch
Tracking Status
firefox99 --- fixed

People

(Reporter: mjr, Assigned: standard8)

References

Details

Attachments

(14 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:85.0) Gecko/20100101 Firefox/85.0

Steps to reproduce:

I've consistently removed, not just unchecked, all of the search engines in Firefox preferences except for DuckDuckGo. I've left Bookmarks, Tabs, History, in place.

I'm not a point & click user. Throughout the day I hit Cmd+L and begin typing in a URL. I have the "Remember History" options unchecked. I only expect to see a few things when I type in the URL bar:

  1. A matching bookmark.
  2. A matching open tab.
  3. The "search with DuckDuckGo" prompt, which I can hit submit by pressing enter

I do have a handful of styles in my UserChrome.css. And I have disabled some of the URL bar features by setting flags to false. The primary issue, though, is not that the search suggestions are appearing in the URL bar—it's that they're being restored to my Search preferences.

I'm attaching the URL bar portion of my about:support dump.

Actual results:

Every time I update Firefox, all of the default search engines that I've removed appear back in my Search preferences, enabled. As a result, when I interact with the URL bar, I see "This time, try..." suggestions with all of those options.

Expected results:

My URL bar should have appeared unchanged from the last time I used it, before updating Firefox. Removed search engines should have remained removed.

Component: Untriaged → Search

Hi, thank you for filing this. Sorry for the delay in getting to it.

Just to confirm a few of things:

  • Is it only the removed search engines that are reappearing? e.g. you're not getting search suggestions re-enabled as well.
  • DuckDuckGo is your default engine and remains so even after the other engines re-appear?
  • Does this only happen on upgrade, or any time you restart the browser? How often do you normally restart the browser?

And some more questions:

  • Are you using any add-ons, if so, which ones?
  • Would you mind attaching a copy of your search.json.mozlz4 file from your profile? You can get to the profile location from about:support. This contains only the current settings for your search engines (urls, alias, hidden, order, default engines).
Flags: needinfo?(mjr)

Is it only the removed search engines that are reappearing? e.g. you're not getting search suggestions re-enabled as well.

I'm not sure I understand the distinction. What are "search suggestions?" I realize this has happened after I update Firefox because when I type in the URL bar, it says "this time try searching with..." and lists the search engines that have been re-enabled. That's the undesirable aspect. I removed the search engines in part because I do not want to use them. I also don't want to see them in my URL bar interface.

DuckDuckGo is your default engine and remains so even after the other engines re-appear?

That is correct.

Does this only happen on upgrade, or any time you restart the browser? How often do you normally restart the browser?

I shut down my mac mini daily, and I would say 80% of the time I quit Firefox before doing so. I always close all the windows before doing so. This only happens AFTER I've been prompted to download a Firefox update, when I next (re)start Firefox and the update takes effect.

Are you using any add-ons, if so, which ones?

  • Privacy Badger
  • HTTPS Everywhere
  • uBlock Origin
  • Custom Style Script
  • DuckDuckGo Privacy Essentials
  • OneTab
  • Disable JavaScript
  • Bypass Paywalls
  • Vue DevTools
  • Raindrop.io
  • React Developer Tools
  • WAVE

Would you mind attaching a copy of your search.json.mozlz4 file from your profile?

Sure thing.

Flags: needinfo?(mjr)
Attached file search.json.mozlz4

Thank you for the information, unfortunately I don't see anything out of place here. I just tried reproducing it with your search.json.mozlz4 and I wasn't able to.

I think the next steps are to try and get some logging information out, and see if that helps.

To turn logging on, enter about:config in the address bar, acknowledge the warning, then search for browser.search.log. Double-click it and it should set the value to true.

When the issue next happens, and as soon as it happens do this:

  • Go to Tools -> Web Developer -> Browser Console (not web console).
  • Right-click and select all
  • Copy with cmd-c
  • Then paste it the output into an attachment here.

The next release is due to be published on the 23rd February.

Flags: needinfo?(mjr)

The other thought here is to check that you haven't got any apps like "Cookie" or "CCleaner" trying to do clean-up in Firefox from outside.

The other thought here is to check that you haven't got any apps like "Cookie" or "CCleaner" trying to do clean-up in Firefox from outside.

Confirmed that I don't have anything running externally, targeting Firefox or otherwise.

I set a reminder to capture screenshots and enable logging before the next update drops. Thanks!

Flags: needinfo?(mjr)

I enabled logging earlier today anticipating an update.

This evening I hit check for updates and hit the button to download and update to v86.0.

I OK'd the restart after it updated.

And...After restarting I don't see the recommendations in my URL bar and I don't see the search engines restored in my preferences. :)

Mark

Thank you for testing that. I'm not quite sure what's been happening, there were some related changes in 85 that may have helped, so maybe that was it. We'll call this "works for me" for now, and if it re-occurs feel free to re-open.

Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME

It's back. Firefox 88.0.

Tried to simulate the update pathway:

  1. Created a profile in Firefox 87.
  2. Removed all search engines but ddg.
  3. Added all extensions as listed in comment 2.
  4. Downloaded update + restart to update to Firefox 88.

Unfortunately, couldn't reproduce it the easy way: no search engine was reinstanted, ddg was the only engine before and after update.

Mjr, have you managed by any chance to get a browser console log for the update ?

Flags: needinfo?(mjr)

have you managed by any chance to get a browser console log for the update

No, because after the Firefox 86 update did not reintroduce the search engines, I thought the problem was solved and disabled logging. Now the same thing happened: All of the default search engines I had removed were reintroduced, all were enabled, and all were presented to me in the URL bar.

Flags: needinfo?(mjr)

So this is happening again.

v94.0.2

Similarly with comment #10, I still cannot reproduce this with a simple update from 93 to 94 (without specified add-ons, will try shortly with a simulated profile containing those).
Mark, is there any code path in which we would restore disabled search engines after update ?

Mjr, could you please also specify the region (open about:config and find browser.search.region) and localization you are using?

Status: RESOLVED → REOPENED
Ever confirmed: true
Flags: needinfo?(standard8)
Flags: needinfo?(mjr)
Resolution: WORKSFORME → ---

Not just disabled—removed :) I've actually removed them from the list of available search engines at each pass, and they come back.

browser.search.region = US

Which flag should I be looking for for localization? Searching for that revealed one that had to be revealed, but that was just true/false.

Flags: needinfo?(mjr)

(In reply to mjr from comment #17)

Which flag should I be looking for for localization? Searching for that revealed one that had to be revealed, but that was just true/false.

About:support -> Internationalization & Localization, but I'll guess that you are en-US. Any chance that you are using a VPN service?

Attached file about:support

Would you mind re-enabling the logs as described in comment 4 and live a bit with the browser console noise, update shall follow soonish? I'm afraid that without further information we cannot really figure this one out.

Flags: needinfo?(mjr)

Obviously this shouldn't be happening, but I'm not really sure what to suggest to look at. Here's some ideas:

  • Check that you don't have any adware/malware - use something like the free version of malwarebytes - there is some adware that's known to change search settings, but I've not seen any with this exact behaviour so far.
  • Confirm if you're switching languages or not. We fixed this a while ago, and it shouldn't have the effects here, but you never know.
  • Attach copies of your search.json.mozlz4 available from your profile (in about:support), for a "good" startup, and a "bad" startup. The file only contains details about your search engines and preferences.
  • Catch up broken startup, and record the logs from it (as per comment 4).
Flags: needinfo?(standard8)

Thanks for following up on this.

Check that you don't have any adware/malware - use something like the free version of malwarebytes - there is some adware that's known to change search settings, but I've not seen any with this exact behaviour so far.

I've seen this happen on relatives' computers and I'm pretty sure this isn't the case on my mac, and am disinclined to install more stuff to verify stuff didn't get installed. I've been using privacy & ad blockers since purchasing and setting up this computer. I don't install stuff that I don't know the source of.

Confirm if you're switching languages or not. We fixed this a while ago, and it shouldn't have the effects here, but you never know.

Nope. I've never switched languages in the browser or OS.

Attach copies of your search.json.mozlz4 available from your profile (in about:support), for a "good" startup, and a "bad" startup. The file only contains details about your search engines and preferences.

Will do.

Catch up broken startup, and record the logs from it (as per comment 4).

I'm not opposed to doing this, but the last time I turned this on, subsequent updates did not cause the issue again. It had gone away. I have it back on as of...now.

Flags: needinfo?(mjr)
Attached file addonStartup.json.lz4

Just to note that you attached addonStartup.json.lz4, rather than search.json.mozlz4.

I think the logs are probably going to be the best way to get information about what is happening, though if it doesn't reproduce with those enabled, that will make it a bit more difficult, though it does point to a possible timing issue.

Severity: -- → S3
Priority: -- → P3
Attached file search.json.mozlz4

I attached the output from the browser console after seeing the search engines were back in the URL bar today.

Hello!

I uploaded my browser console output after I saw this again recently, but haven't seen any activity here.

It's back again today.

I can see in this log excerpt where these search engines are being added back in. I did not do that.

Thank you for the log - that is so useful. Unfortunately the earlier ones weren't but this one is interesting.

Shane, this looks like an issue with add-on manager. Can you take a look please?

Here's what I see happening:

  • Search Service is starting and determines the application provided engines to load from the configuration.
  • Search Services asks add-on manager for the add-on details of the application provided (built-in) engines
  • Add-on Manager tries, but fails: TypeError: AddonManagerInternal._getProviderByName(...) is undefined SearchService.jsm:608:17.
  • Search Service finishes its initialisation process, but doesn't have any engines at all, characterised by: SearchService: Could not find a replacement default engine. SearchService.jsm:2135:18
  • Add-on Manager then seems to kick in and load the add-ons associated with the application provided engines.
  • Add-on Manager tells the Search Service of these engines, and so the Search Service knows they are in the configuration and loads them.

Unfortunately because of that initial problem with not loading the engines, the settings are "lost" - although the engines get loaded later, we're not expecting to load the settings at that stage (as there typically wouldn't be). Really the issue here is due to the add-on manager not having started up enough.

I think this is also likely to be causing bug 1755740.

Blocks: 1755740
Flags: needinfo?(mixedpuppy)

Glad to hear that's helpful and there's something to diagnose! :)

I've spoken to Shane and he's pointed out this is the search service not waiting for the add-on manager initialisation to complete. Normally these are quite far apart and we get away with it, unfortunately there's obviously some cases where these can overlap and cause issues. I'm taking this to work on a fix.

Assignee: nobody → standard8
Status: REOPENED → ASSIGNED
Flags: needinfo?(mixedpuppy)

The error AddonManagerInternal._getProviderByName(...) is undefined is triggered within installBuiltinAddon, and shows that gStarted is true, which is set right before the providers are started, but that the providers have somehow not registered, despite _callProvider being a synchronous function that marks the provider as registered in the end: https://searchfox.org/mozilla-release/rev/f8a114d1d43fe6a329547882faef5c724e6c81a9/toolkit/mozapps/extensions/AddonManager.jsm#575,584-591,621-622

That could potentially happen when the addon manager is spinning an event loop AND the search service initialization is triggered during the nested event loop. I tried to look up the callers of the SearchService's init method, but it is not obvious how that could happen.

In any case, if this analysis is correct, then the proposed patch (where the search service awaits the startup of the addon manager) is a fix in the right direction.

No longer blocks: 1755740
See Also: → 1755740
Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0c511360ce13
Ensure Add-on Manager has finished initialising before the Search Service starts using it. r=daleharvey,robwu,agi
Status: ASSIGNED → RESOLVED
Closed: 2 years ago9 months ago
Resolution: --- → FIXED
Target Milestone: --- → 99 Branch

@mjr, I just wanted to say thank you for sticking with us and getting that final log, that was very helpful in diagnosing this issue.

Hopefully this will now fix the issue permanently (once it gets released around 5th April), but if it does happen again after that please file a new bug.

You need to log in before you can comment on or make changes to this bug.