Closed Bug 1578807 Opened Last month Closed 28 days ago

Upgrading to Firefox 69 removes most search engines and changes default

Categories

(Firefox :: Search, defect, P1)

69 Branch
defect
Points:
5

Tracking

()

VERIFIED FIXED
Firefox 71
Iteration:
71.1 - Sept 2 - 15
Tracking Status
firefox69 + wontfix
firefox70 + verified
firefox71 --- verified

People

(Reporter: bradleysepos, Assigned: standard8)

References

Details

Attachments

(6 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Upgraded to Firefox 69.0 for macOS from 67.0.4.

Actual results:

Nearly all search engines were removed. The default search engine was removed, thus the default was set to a different search engine.

Expected results:

No changes to installed search engines or default search engine.

Workaround:

  • Quit Firefox
  • Backup and remove search.json.mozlz4
  • Launch Firefox (will regenerate the file)

I've attached a zip file including the original search.json.mozlz4 from Firefox 67.0.4, the result after upgrading to 69.0, and the result after removing it and allowing Firefox to regenerate it, along with about:support from Firefox 69.0.

Taking to look into this.

Assignee: nobody → standard8
Component: Untriaged → Search
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Bradley, I've been trying this locally with your json files, and a built-up profile, and unfortunately I can't reproduce at the moment.

Are you able to reproduce it still? If so, would you be able to set the preference browser.search.log to true in about:config before doing the upgrade.

Once the upgrade has happened, go to Tools -> Web Developer -> Browser Console, then copy out everything there and post it here?

Flags: needinfo?(bradleysepos)

Mkaply NI in case he has something to add here.

Flags: needinfo?(mozilla)

I don't. This would have to do with the move to WebExtensions for search which I wasn't involved in.

Flags: needinfo?(mozilla)

I was just looking at the data again, as I hadn't really looked at the bad search.json.mozlz4 file.

I realised that some of the hashes in the config file rely on the profile directory name - so starting it up with a random configuration would mean they're immediately invalid, which would not be helpful.

Hence I'm now going to try checking those hashes out, and seeing if I can reproduce again.

Flags: needinfo?(bradleysepos)

More investigations:

  • I still can't reproduce the issue on upgrade at the moment.
  • The hashes all seem valid in their various files.
  • With the search.json.mozlz4.ff69-bad file I can reproduce the issue with 69 where only Wikipedia is shown.
    • The engines array only has Wikipedia within it, the other 9 engines have been dropped for some reason.
    • This has the knock-on effect that restoring default engines doesn't work, since we expect the default engines to be listed in that array as well.
    • We could get back to a partial good state by attempting to ensure any default engines missing from the array are re-added, however in this case, the user would still loose any/some engines they've added manually.

Dale, any suggestions on what's happening here?

Flags: needinfo?(dharvey)

Bradley, can you check a couple more things for me please:

  • The values of your browser.search.cohort and browser.search.region preferences - you'll need to go into about:config for these.
  • If you're able to check the 67 profile, could you look in searchplugins and check all the search engines that you added are there?
  • Also, if you still have the complete 67 profile, could you try what I suggest in comment 3 please?
Flags: needinfo?(bradleysepos)

My profile name is Bradley.default. I will attempt to gather more data per your suggestions.

Firefox 67.0.4 and 69.0:

browser.search.cohort           modified    string    nov17-2
browser.search.hiddenOneOffs    modified    string    Yahoo,Amazon.com,Food Network - Recipes,Flickr Tags,Wikipedia (en),Wikipedia (English),Bing,Twitter,DuckDuckGo,eBay
browser.search.region           modified    string    US

$ ls -l Bradley.default/searchplugins
total 80
-rw-r--r--@ 1 bradley  staff  10339 Apr  2  2013 duckduckgo-1.xml
-rw-r--r--@ 1 bradley  staff  10339 Apr  2  2013 duckduckgo.xml
-rw-r--r--@ 1 bradley  staff   2125 Jul  6  2013 flickr-tags.xml
-rw-r--r--@ 1 bradley  staff   5500 Mar 30  2010 food-network-recipes.xml
-rw-r--r--@ 1 bradley  staff   1068 Mar  4  2007 wikipedia-english.xml

Browser console after upgrade:

1567719228750	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e8fd786c-25ab-bc40-90fc-400437f3c886, overwrite: false
1567719228750	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 80a4c22a-6800-e549-8cd1-e1ff72c08d32, overwrite: false
1567719228750	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4c89fab7-8a34-ff48-97f3-1b5b4c2e6900, overwrite: false
1567719228750	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 74587c69-96b7-5a48-8db3-364eb849627f, overwrite: false
1567719228756	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 693f0831-a53a-a34b-9253-793235562a7b, overwrite: false
1567719228756	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c0005625-fbec-f04a-8878-e3e9e10e3f8f, overwrite: false
1567719228756	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 37e17341-7c1c-bb4c-b4d2-ce5ab430d0af, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 66a7105c-eeea-4c46-ab92-76fa79d2fae2, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: abc1ccd8-0fb2-8c4d-ba70-505b27ac6e33, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 26397de2-f5cb-cd49-a327-882414aaaa9c, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 16a0a3ba-7403-0743-8358-559b003bc195, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 944f5126-89a3-374a-9a8f-8121edb88bc9, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 9bdfdb73-c8f2-ed43-9704-f327c80a209c, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 6bf1b8ea-a0fb-9d4f-ad1e-72ba710eb682, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 6597f5ec-a72a-8943-9325-6afec2973c19, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c7440d62-3804-aa43-a6b1-c08f298212f1, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: bb516466-7390-a84f-9cae-f2cbc3964650, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 8a07ed41-f253-9e4f-bb87-ced09c92dde5, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: f36cc72c-9884-7942-89a6-fb95b0b95019, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 68a2fa9a-eae8-7e42-8b26-b6f39c42bb3e, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: d20a84be-3412-1142-9a4f-2626a9dcac7c, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: afb8ab39-5339-c94d-85bc-6d8e84e0ad07, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e697fff5-23ef-8c41-8bc0-a1690cbabc69, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ceda6fc0-7741-484f-8126-81f3c49395f0, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e705c51d-7889-dc4b-81d2-9540547d7ac1, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: fcd141f7-4a11-d242-9dbd-1e3f96e4f591, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 340fa5f4-c160-a044-adee-6ecae1631ae7, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 178d9c63-8d00-ca4d-9d3a-b2381b71e94e, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 41bd087f-272e-a144-863e-d85623d98d17, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 5bd9a750-5e54-a240-a563-36008bccc27f, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2a90f9d1-b0a4-4d43-8a2e-b2fefc361a63, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 27a0e66c-17ea-b248-b5e1-478ed60ea876, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7372f1d4-15d5-e54e-baa8-2fd960046c58, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: fa2ef5ab-e5fd-0d41-a9cb-aff3a5e1a6fa, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: fa627899-cf5f-6e42-8b2b-d8ae23fb62bd, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 6de0daf7-6953-eb43-ae00-09299c362f85, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: a4ef301d-0b47-e54b-ad53-a7b935fd0009, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b1362a05-43da-c14e-86a2-51e2e67ea0e1, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 0ae497b9-122e-c54d-95ed-d184edda0aa9, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b9a1b0c4-9acb-5646-87ed-e0733679d731, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c8d5b816-ed7b-014c-95ac-56403b6e82b4, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3366e1d5-255f-1845-b577-48299375b348, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3d985560-223b-e149-956e-24c2c18ae659, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ce53570b-ac19-7c4a-8529-3d922bdd7e39, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: f2ff9078-320b-b549-8c77-037807b718fb, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3827b49e-1297-9849-be99-aaebef0948d8, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: da5e23a0-2f10-c54d-9067-adf76047f4bd, overwrite: false
1567719228757	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 10a86f84-08fc-3244-b0b9-a38eb7c8ffec, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2e4e9add-0dfc-734a-a602-e292e0829f73, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ea076f69-00e6-2748-b158-1506a33a8c60, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 28640d84-1e50-0a4e-8891-2364175342bf, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: be9577b3-25ad-2345-abeb-c1b3a586bac3, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2b643e19-974d-5a4f-9860-4faa0843d73f, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 09d2ca59-b2b7-7e47-96c1-b44683ac00b7, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b4d6752f-3740-a749-b002-927f5008789c, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ed3c9dff-9abe-b14c-bb30-3e11e0bda1ac, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: fff1f995-ee0c-a045-8c97-6bc04ce892c2, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 897f07b6-0f36-8746-ade8-d51ff60a6bbc, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3344baa5-5b1b-2b44-8bb5-91d167389ef2, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c609953d-662e-1a48-a320-d7b0bc5b0e70, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7ea8b617-7a93-e147-a745-364313379021, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: cccd32f1-e4ab-0b45-9331-31e7ecd1c5ac, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4bce7701-e617-ee4c-9435-a933dd6b43c9, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 37f5fa85-74ce-6245-a6b1-8d55103882af, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 1180e788-9c61-eb4e-abf7-473b9636cbac, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: d4e35450-8d6a-3d4f-8cc8-1bf1f4cb0fc4, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e1dfcfd7-7e2f-6e45-ba66-224193f56142, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7cc746a1-5a80-ad44-bb7e-3021507ed394, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 5d71ecc4-7293-f442-a4e4-8161bd8035b2, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 652d5d11-ad36-b34c-8729-68027ff68350, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 57e1feb2-cea6-a347-ac1f-a37e81241e8c, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ca10ce89-7906-8344-8dc4-932b08287dce, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 9b4af52f-119e-004d-80f0-796b8bfcdcbc, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 69fc9af3-f534-5b4a-a747-0e12118d3c5a, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 07b4f305-2486-bd4d-b9e0-44a6ffce463c, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b16d6903-a964-7f47-819c-ff45aff17742, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3862c5ef-c8c9-1444-a7b3-022cf36911cb, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b8a25236-b574-334c-882d-5eeba7336869, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3f94d487-89e1-f348-9bde-15265077dd60, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 953d164a-6c5c-dd42-9ecc-0d6c2a09de1f, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: eb772c96-a1cc-fc45-9964-41c569c0f3d0, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3ef48447-c458-8549-bf19-a5526d9739ca, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 1b5c6e6f-7535-3949-b4e6-2c4f86e8e659, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 9ee4ac9a-fd3f-1842-9a2c-160803155eb1, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 750bb38e-1970-3545-9150-a852657779dc, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: a4c079d8-399a-ed40-a923-4e9db57aa2c2, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 97c3ec40-3b36-6e41-a6c6-a0c96da7001a, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7afc743f-b340-914c-acd7-4d526d96c87f, overwrite: false
1567719228758	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ecca6ad1-61c1-8442-ad20-f59f89bcd65c, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 6fff1731-7c70-6c4f-a735-74b93ac01ce9, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c2a1ec42-90ff-3544-bd8d-ca1b71ec1a2a, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 5400089d-beb9-c64f-bf2a-98edfd0c5adb, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 52d89212-808f-4b45-8e8f-02bd5c612291, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: ab3bd00e-a030-564d-bda5-b1d4bf3cf3d6, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 37cb325c-f78e-b44a-a613-dbd43e381542, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 57229aa0-b31b-a746-918c-11fd9838ec98, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 12150230-1f89-e646-b031-cb9f6832de8d, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 673779b3-f1b0-b44e-8a73-f2d7701be822, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: bf93992c-d628-7044-95fe-8c47c76f52cd, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4e978122-fa7a-0c4e-b16e-1f32dd32cb52, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b7a38168-236e-3940-9e66-dcbc4295d6c9, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7a9de5c8-fcaf-cb43-b3d3-04fe7752d700, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 0d369e21-4ade-1b42-bc2e-8fd438f9e086, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4715347a-13af-674d-a7b5-924fe206c6af, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: c88cb813-2b64-f543-8318-3cae52fc7d96, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 86f5b44e-3df1-7d49-999b-63c056b8e38e, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 73b55d89-dd67-0b49-9a2b-c4529a9aefbc, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4a99bce9-1739-8c47-a1e4-c2a0f77deea3, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 98b69532-f7ec-3640-8329-a55ab81e3cad, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 77d177c3-d3e8-6844-a841-f618ae8ce1ab, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 1153b60b-d838-e040-953f-b5dd1e77551b, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 0a3359fb-4299-e14a-a04c-64d0a05761f5, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e13dba09-9439-b940-aa62-ed768f563dc7, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 36ef0cf0-5fe6-1e4f-a334-1c0572a75df4, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 463a7ddb-d288-bf4d-8aec-00269d5ccf21, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 0fd15049-745c-2941-b3ee-c4793b4e2342, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: aaea6795-6591-094f-955f-c3ff6972c206, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 9e77804b-aed9-aa42-836e-e50a092a9213, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 18fd3070-e23d-d64c-a188-4c9d2b72e2df, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 33f20cce-d8a1-a44c-b69c-9497efd11b1e, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 06663053-991f-c44b-a0a6-859037270f52, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4c64380a-35f5-244e-b948-dc4b9ed419c6, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 332054a4-7fb3-2d40-be61-fbd0a18313d9, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3daff8a9-f80e-564d-8db2-83f5bef1dabc, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4d8a50fb-d9a0-514b-bc36-4941a0f18bd9, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2dc1c5e7-5050-c74f-aef6-f14c4f7eaa5f, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 1d3fe6de-77bc-fe45-9048-27e2bf70c07d, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4237a0e0-0e52-4b4c-b191-baa75e69e314, overwrite: false
1567719228759	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: b50fbb66-9f6d-df46-9fff-f038e8e32254, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: f1fc7b52-d6f4-0d4c-a586-be145dc04d63, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 264fa185-7346-8044-8f66-46174159f4e2, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 3a60f5c5-6727-1348-be79-034971af52aa, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2f6a7085-aaaf-864b-8803-bc973d41884c, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 2d919984-9e39-0a4c-870b-75ec41a8ce38, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 5fe23ea4-60e5-2a41-9f1f-24603bd872b4, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: dacf06a2-9b8f-8747-a1c4-e3949121b49d, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 7f39b4e2-28bb-fb4a-8175-9ffab8e8c020, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: e8fd786c-25ab-bc40-90fc-400437f3c886, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 80a4c22a-6800-e549-8cd1-e1ff72c08d32, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 4c89fab7-8a34-ff48-97f3-1b5b4c2e6900, overwrite: false
1567719228760	Toolkit.Telemetry	WARN	TelemetryStorage::_scanArchive - have seen this id before: 74587c69-96b7-5a48-8db3-364eb849627f, overwrite: false
stderr output from native app 2bua8c4s2c.com.agilebits.1password: 2019-09-05 17:33:48.884 1PasswordNativeMessageHost[25251:401535] 70301008 [EXT_NMH:(Secondary Thread 0x7fc683648b10):OPNMXPCConnection.swift] E connect() | Connection Established
Error: No matching message handler 89 undefined
In add-on {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}, attempting to use listener "notifications.onButtonClicked", which is unimplemented. ExtensionParent.jsm:1225:13
Password fields present on an insecure (http://) page. This is a security risk that allows user login credentials to be stolen.
login.php
Error processing windowTypes: This property is deprecated global.min.js:250
Security Error: Content at moz-extension://17c57b60-fb18-5d4e-9016-7fb98bd7ba0e/ may not load data from blob:null/e44b28b8-7d6f-444e-9b27-fa8c8256d57d.
Error processing windowTypes: This property is deprecated global.min.js:250
[Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMWindowUtils.removeSheet]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/ExtensionCommon.jsm :: runSafeSyncWithoutClone :: line 75"  data: no] ExtensionCommon.jsm:75:12
SearchService.init 3
getDefaultEngines: engineOrder: ({Google:1, Bing:2})
getVisibleEngines: getting all visible engines
SearchService.init
Failed to load strings for Activity Stream about:preferences AboutPreferences.jsm:150
Use of nsIFile in content process is deprecated. 2 NetUtil.jsm:253:8
Security Error: Content at moz-extension://17c57b60-fb18-5d4e-9016-7fb98bd7ba0e/ may not load data from blob:null/57b56559-2520-854d-bced-ae0c30315a40.
Error processing windowTypes: This property is deprecated global.min.js:250
[Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMWindowUtils.removeSheet]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/ExtensionCommon.jsm :: runSafeSyncWithoutClone :: line 75"  data: no] ExtensionCommon.jsm:75:12
OpenGL compositor Initialized Succesfully.
Version: 2.1 NVIDIA-12.0.24 355.11.10.50.10.103
Vendor: NVIDIA Corporation
Renderer: NVIDIA GeForce GTX 670 OpenGL Engine
FBO Texture Target: TEXTURE_2D

The upgrade seemingly worked flawlessly this time. Only change I am aware of is browser.search.log. I'll investigate further.

Flags: needinfo?(bradleysepos)

I cannot reproduce the issue under any known circumstance. Given there are other reports of this on the web, it would seem to be valid yet intermittent. I'm not sure what else to do here.

So yeh I have been looking into this but cant reproduce with a fresh 67 profile upgrading to 69 (or nightly)

Mark said he is going to do a patch to fill in the missing builtIn engines, this doesnt solve the root cause but will mean it falls back better, will be keeping an eye out for anything that could potentially cause this in the meantime

Flags: needinfo?(dharvey)

The attached patch would reset the cache, which would fix the issue. I'm still debating if the way I've added the check is the right solution or not, and if it could cause issues when engines are updated remotely.

Attached file searchOLD.json.mozlz4

I got the same problem yesterday, and here's the file before I reset it.

I've caught the reset of the default engines without upgrade path on Ubuntu 16.04, with a new profile:

  1. Installed Firefox 69 fr.
  2. Changed language to en-US. (about:preferences/Language)
  3. Restarted the browser to take effect.

Upon restart, all default search engines would be gone, only google remains.

Any attempt to reproduce this failed.

Here's a summary of where I am with this:

  • As far as I can tell, this is semi-random as to when it happens, but is infrequent if it does, and doesn't seem to be consistently reproducible.
  • Adrian's issue above of changing the language pack was also confirmed by someone on twitter, but again, this isn't reproducible.
  • Upgrade might also be a trigger point for this, but it is hard to know for sure.
  • The important parts of the SearchService initialisation and region handling look fine with respect to possible race conditions. There could still be a race condition in there, but if so it is hard to see by manual analysis.
  • Search modernisation (bug 1542235) is likely going to be simplifying some of the initialisation, as we won't have ABSearch to deal with. There's potential also for simplifying the cache, but we need to consider that a bit more we commit to it.
  • The broken profiles all appear to have most or all of the built-in engines missing from the cache.

Given all this, I think the best way forward at the moment is to land the patch that I posted earlier, which will at least allow users to recover from the situation. As we proceed forward with search modernisation and cleaning things up, we can keep an eye out for circumstances in which this can happen.

Iteration: --- → 71.1 - Sept 2 - 15
Points: --- → 5
Priority: -- → P1

Also, thank you to those that have provided reports and files for me to look at, they've been very helpful, even if we've not been able to figure out the root cause yet.

Can we add Telemetry for this so we have a sense for how often users are hitting this in the wild?

Flags: needinfo?(standard8)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #21)

Can we add Telemetry for this so we have a sense for how often users are hitting this in the wild?

Yes, lets.

Flags: needinfo?(standard8)
Attachment #9092158 - Flags: data-review?(bmiroglio)
Attachment #9091468 - Attachment description: Bug 1578807 - Reset the cache if the count of built in search engines is different to the number in the engines cache. → Bug 1578807 - Reset the cache if the count of built in search engines is different to the number in the engines cache. r?daleharvey!,r?chutten!
Attached image search.jpg

Following up comment 17, Mark asked yesterday about how fast I was doing it when the problem reproduced and given that this scenario has also been reported during the update, I've managed to narrow a scenario to reproduce the problem consistently:

  1. Install Firefox 69 fr. (probably works with any locale)
  2. From about:preferences/Language add another language (in my tests I used fr build with language change to en-us). - this option is available only in release channel, I don't know which preference controls this for the other channels.
  3. At this point you should have two languages and when selecting the one that you want to switch to the "Apply and Restart" is available as in the screenshot.
  4. Hit "Apply and Restart".
  5. Check the search engines.

If after the restart the search engines are ok, change again the language and hit restart. Do it a few times and 2-3 changes and restarts will likely mess up the search engine list.

Given the above scenario using the change language and the originally reported issue during major version update, I would guess that some kind of race condition happens on initialization/re-initialization of the search engines list.

**Edit: Browser console log with browser.search.log = true:
https://pastebin.com/KYJ7dbmn

See Also: → 1580737

Thank you Adrian for the extra STR and the logs. This appears to be a specific case where we start to hit shutdown whilst we're reinitialising the search service.

I've split that specific case off to bug 1580737.

It could be that these are the same bug, and I guess there's a small chance that if someone started Firefox and shut it down real quick they could potentially hit something similar. It is unclear if that's what happening in the other cases we've seen.

In any case, at the moment I think it is best to go forward with both these bugs and then we'll also have telemetry to know if bug 1580737 fixes the issues, or if there's some still lying around.

Discussing this with Mark, I think it'd be better to let any fixes bake on Beta for a bit rather than trying to jump the trains right to Release since we're still mostly in the realm of speculation about what's going on here. I'd be open to landing the Telemetry probe now, however, if we wanted to get a better sense of frequency before then.

See Also: → 1581882
Comment on attachment 9092158 [details]
EnginesCacheDataCollectionReview.md

# Data Review Form (to be filled by Data Stewards)

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? (see [here](https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md), [here](https://github.com/mozilla-mobile/focus/wiki/Install-and-event-tracking-with-the-Adjust-SDK), and [here](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/index.html) for examples).  Refer to the appendix for "documentation" if more detail about documentation standards is needed.

Yes, this will be documented in the probe dictionary: https://telemetry.mozilla.org/probe-dictionary/.

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this can be turned off in Firefox Preferences.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

This is not permanent.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1: technical data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes,

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

Yes, check-in in 6 months.

9) Does the data collection use a third-party collection tool?

No.

data-review: r+
Attachment #9092158 - Flags: data-review?(bmiroglio) → data-review+
Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7debf9ae7642
Reset the cache if the count of built in search engines is different to the number in the engines cache. r=daleharvey
Status: ASSIGNED → RESOLVED
Closed: 28 days ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 71
Regressions: 1582086
Regressions: 1582081

Comment on attachment 9091468 [details]
Bug 1578807 - Reset the cache if the count of built in search engines is different to the number in the engines cache. r?daleharvey!,r?chutten!

Beta/Release Uplift Approval Request

  • User impact if declined: If the user gets into a broken cache state, they might not be able to get out of it until the next release, without manually removing files.

This also adds telemetry so we can check how often it is occurring.

  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: Bug 1582081
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Worst case we'll be doing extra rebuilds of the cache on startup.
  • String changes made/needed: None
Attachment #9091468 - Flags: approval-mozilla-beta?

Comment on attachment 9091468 [details]
Bug 1578807 - Reset the cache if the count of built in search engines is different to the number in the engines cache. r?daleharvey!,r?chutten!

Allows us to recover from a dataloss issue and measure how often users are hitting it in the wild. Approved for 70.0b8.

Attachment #9091468 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
QA Whiteboard: [qa-triaged]
Flags: needinfo?(aflorinescu)

Using STR from https://bugzilla.mozilla.org/show_bug.cgi?id=1581882#c0, created a broken search cache profile, then proceed to verify that the proper search engines are reset accordingly with:

70.0b8 2019-09-19
71.0a1 2019-09-22

on:
Windows 10, Ubuntu 16.04, Mac 10.14.6

I'd like to also check that the telemetry is recorded properly for the reset case. Mark, could you please list what is the telemetry probe for this?

Status: RESOLVED → VERIFIED
Flags: needinfo?(aflorinescu) → needinfo?(standard8)

(In reply to Adrian Florinescu [:adrian_sv] from comment #33)

I'd like to also check that the telemetry is recorded properly for the reset case. Mark, could you please list what is the telemetry probe for this?

It is a scalar, browser.searchinit.engines_cache_corrupted. Though you might have trouble reproducing it (I did when I landed it on nightly). The issue is that you need the cache with the correct hashes and build data otherwise it'll just get rebuilt because they're invalid.

The easiest way I found to do this, was to get an instance of Firefox up to date with associated profile, then intentionally break the cache file.

To do that, shutdown the Firefox profile, then with a different profile, open the Browser Toolbox and enter the following into the console:

bytes = await OS.File.read("/Users/mark/Downloads/search.json.mozlz4", {compression: "lz4"});
json = JSON.parse(new TextDecoder().decode(bytes))
json.engines = [json.engines[0]]
bytes = new TextEncoder().encode(JSON.stringify(json))
await OS.File.writeAtomic("/Users/mark/Downloads/search.json.mozlz4", bytes, {compression: "lz4"})
Flags: needinfo?(standard8)
Duplicate of this bug: 1586614
QA Whiteboard: [qa-triaged]
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.