Search engines are gone with v78.0
Categories
(Core :: Storage: IndexedDB, defect, P2)
Tracking
()
People
(Reporter: ulli.conrad, Assigned: janv)
References
Details
Attachments
(10 files)
27.61 KB,
text/plain
|
Details | |
15.81 KB,
text/plain
|
Details | |
260 bytes,
text/plain
|
Details | |
3.94 KB,
application/zip
|
Details | |
1.56 KB,
text/plain
|
Details | |
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
RyanVM
:
approval-mozilla-esr78+
|
Details | Review |
960 bytes,
text/plain
|
Details | |
38.71 KB,
text/plain
|
Details | |
38.71 KB,
text/plain
|
Details | |
38.71 KB,
text/plain
|
Details |
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
Steps to reproduce:
Updated to v78.0
Actual results:
All search engines are gone, list of one-click search engines is empty now
Auto complete in the address bar doesn't work any longer
Search function on the start page doesn't start a search any longer
Expected results:
Search engines should disappear after an update
Updated•5 years ago
|
Comment 2•5 years ago
|
||
Hi Ulli, please can you provide us with some more information to help us diagnose the issue? Here's what would be useful:
-
Go to the three bar menu -> Help -> Troubleshooting Information.
-
Click the "Copy text to Clipboard" Button.
-
Back on this bug, just above your first comment, click "Attach new file", click in the box at the top and do ctrl-v (or right-click and select paste).
-
Give it a description and then hit submit at the bottom.
-
Then, restart Firefox, as soon as it is started, go to the three bar menu -> Web Developer -> Browser Console (not the Web Console).
-
Right-click and Select All
-
Then Right-click again and Copy Message.
-
Paste that into another new attachment as per the above instructions.
That will hopefully give us enough information, but we may need a little more. Either way, we'll hopefully be able to get you running again.
Reporter | ||
Comment 3•5 years ago
|
||
Reporter | ||
Comment 4•5 years ago
|
||
Comment 5•5 years ago
|
||
Thank you, for some reason the local database that we use to store some data is not working. This errors are particularly pointing to that:
IndexedDB UnknownErr: ActorsParent.cpp:17128
UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code.
IndexedDB UnknownErr: ActorsParent.cpp:17128
UnknownError: IndexedDB: main/anti-tracking-url-decoration getLastModified() IndexedDB: The operation failed for reasons unrelated to the database itself and not covered by any other error code. IDBHelpers.jsm:24:5
IndexedDB: main/hijack-blocklists getLastModified() IndexedDB: The operation failed for reasons unrelated to the database itself and not covered by any other error code. IDBHelpers.jsm:24
Unfortunately our search service has just switched to using that local database, and as a result isn't working properly.
I'll see if I can find some people to have a look at it.
Reporter | ||
Comment 6•5 years ago
|
||
Thank you, at least a quickt idea what's goin' wrong :-)
Comment 7•5 years ago
|
||
Jan/Baku, please could you take a look at this? The search service is now using remote settings in 78, and if indexed DB isn't working, that stops that and a lot of other things from working.
Assignee | ||
Comment 8•5 years ago
|
||
I'm looking at this, the attached browser console output is very useful.
Assignee | ||
Updated•5 years ago
|
Comment 9•5 years ago
|
||
Ulli, from your other bug, you implied that you've restored an old profile from backup (not downgraded) and you're using that with 77 now.
Do you get similar error messages on the browser console with that older profile? Obviously the search issue won't be there, but I'm wondering about the IndexedDB side.
Reporter | ||
Comment 10•5 years ago
|
||
Doesn't look so, but it is a newly created profile with all needed files (places.sqlite,prefs.js and so on) copied back from backup. All the rubbish collected over the years of course is gone now...
Reporter | ||
Comment 11•5 years ago
|
||
Comment 12•5 years ago
|
||
The line would be https://searchfox.org/mozilla-release/rev/e9fe0d92b780775234419a0651fef84f6e8311f2/dom/indexedDB/ActorsParent.cpp#17128, which is within QuotaClient::UpgradeStorageFrom1_0To2_0
, which would be consistent with this happening with an old profile.
Updated•5 years ago
|
Assignee | ||
Comment 13•5 years ago
•
|
||
The good news is that remote settings code doesn't depend on temporary storage initialization (storage for the web). The relevant IDB files live in permanent/chrome/idb/
, 3870112724rsegmnoittet-es.files and 3870112724rsegmnoittet-es.sqlite
We have a telemetry for initialization of directories like this [1]
The relevant keys for this bug are "Storage" and "UpgradeStorageFrom1_0To2_0"
"Storage" is at 99.99% for Nightly which roughly matches Release.
The failure at [2] is covered by the "UpgradeStorageFrom1_0To2_0" key and there are currently 194 failures out of 280 tries on Nightly for last month (we track it only once per FF session).
We can also provide exact data for Release using custom queries.
There are other upgrades with relevant telemetry, for example "UpgradeStorageFrom2_0To2_1", but they all look good.
So there's a high probability that fixing the problem at [2] will improve overall initialization success rate and mitigate the problem with search engines.
I'm now preparing a custom FF 78 build which can be used to identify precise directory in user's profile which causes the upgrade to fail.
We will then ask the reporter to provide full recursive listing of the directory.
Assignee | ||
Comment 14•5 years ago
•
|
||
Here is data for Release 78:
Storage key
number of all storage initializations (roughly matches number of FF sessions): 27925
number of failed storage initializations: 9
success rate: 99.97%
UpgradeFromIndexedDBDirectory key
number of all storage upgrades: 1
success rate: 100%
UpgradeFromPersistentStorageDirectory
number of all storage upgrades: 13
success rate: 100%
UpgradeStorageFrom0_0To1_0
number of all storage upgrades: 35
success rate: 100%
UpgradeStorageFrom1_0To2_0
number of all storage upgrades: 46
number of failed storage initializations: 7
success rate: 84.78%
UpgradeStorageFrom2_0To2_1
number of all storage upgrades: 46
success rate: 100%
UpgradeStorageFrom2_1To2_2
number of all storage upgrades: 83
success rate: 100%
UpgradeStorageFrom2_2To2_3
number of all storage upgrades: 116
success rate: 100%
There are only 7 failed initializations for UpgradeStorageFrom1_0To2_0, but number of total failures is 9.
It seems there can be something else which breaks the initialization and is not covered by individual keys.
Assignee | ||
Comment 15•5 years ago
|
||
Sorry, I made a mistake in the last comment, it's now fixed.
Assignee | ||
Comment 16•5 years ago
|
||
Assignee | ||
Comment 17•5 years ago
|
||
I attached a zip file (storage.zip) which can be used to simulate a profile with broken storage (leading to broken search).
Steps for creating broken profile and reproducing the search issue:
- Create a new profile
- Close FF
- Remove or rename the "storage" folder and the "storage.sqlite" database in the new profile (you can find location for it in about:profiles)
- Unzip storage.zip in the new profile (it will create the "storage" folder and the "storage.sqlite" database)
- Run FF for given profile
Let me know if it doesn't work for you.
Comment 18•5 years ago
|
||
Search engines are gone Yandex ?:C why?what for ?
Assignee | ||
Comment 19•5 years ago
|
||
Here are custom FF 78 release builds with additional debugging info (reported to browser console):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=42699d073a2a44ac1846da87266afd09ff45c1ac
Ulli, here's a custom build for x64 windows (win64):
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ic3gHdMuQuKAES04xMTjgQ/runs/0/artifacts/public/build/target.zip
Can you run your profile with the custom build ?
You should see something like this in the browser console:
IndexedDB Directory (<path_to_profile>\storage\default\<some_origin>\idb\<basename>..files) already exists!: ActorsParent.cpp:17130
Can you send me full recursive listing of the <some_origin> directory ?
Thank you!
Assignee | ||
Updated•5 years ago
|
Reporter | ||
Comment 20•5 years ago
|
||
Wow you guys are quick :-) Thank you very much indeed! No default storage message, I've send the permanent folder listing instead
Assignee | ||
Comment 21•5 years ago
|
||
Ok, thanks.
So you probably see messages like this:
IndexedDB Directory (<path_to_profile>\storage\permanent\moz-safe-about+home\idb\818200132aebmoouht.files) already exists!: ActorsParent.cpp:17130
I suspect this was caused by running an older FF (version between 36 and 48 probably) with a profile which was already upgraded by newer FF.
These days, something like that is not possible (fortunately).
Working on a fix and a test now.
Comment 22•5 years ago
|
||
The search issue is fixed for 78.0.1 via bug 1649558.
Updated•5 years ago
|
Reporter | ||
Comment 24•5 years ago
|
||
Just updated to 78.0.1
Everything seems fine now :-) Thank you very much indeed!
Assignee | ||
Comment 25•5 years ago
|
||
If an old IDB files directory exists, it no longer causes a failure during the
storage upgrade. The directory is removed instead.
Comment 26•5 years ago
|
||
Comment 27•5 years ago
|
||
bugherder |
Comment 28•5 years ago
|
||
Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.
Comment 29•5 years ago
|
||
Please nominate this for Beta and ESR78 approval when you get a chance. Also, should we throw together a test build for the reporter to try out to confirm that this works with their profile as well?
Assignee | ||
Comment 30•5 years ago
|
||
Yeah, I'll do that once telemetry data is available (should be tomorrow).
Assignee | ||
Comment 31•5 years ago
|
||
Here are custom FF 78 release builds with the fix for the upgrade function:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d19f3c2f7aa7c8ed9fee89a2809235018c15584d
Ulli, here's a custom build for x64 windows (win64):
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZWv_IPQiRCSCqwgtOEx2Kg/runs/0/artifacts/public/build/target.zip
Would you have time to run your profile with this custom build ?
You can either set the preference browser.search.modernConfig
to true to test if the new search works for you
or go to https://firefox-storage-test.glitch.me/ to see if your storage is all good.
Reporter | ||
Comment 32•5 years ago
|
||
Search engines are back and work with that build although storage seems to be broken
Assignee | ||
Comment 33•5 years ago
|
||
Ok, there must be something else broken, but the issue reported in comment 4 seems to be fixed.
I might ping you in a different bug about the other issues.
Thank you!
Reporter | ||
Comment 34•5 years ago
|
||
Thank >you<, keep up the great work ;-)
Assignee | ||
Comment 35•5 years ago
|
||
Comment on attachment 9160680 [details]
Bug 1649393 - Remove old IDB files directories during 1.0 to 2.0 storage upgrade; r=#dom-workers-and-storage-reviewers
Beta/Release Uplift Approval Request
- User impact if declined: Quota Manager storage initialization breaks for some users which may lead to browser usability problems.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): The patch has been tested on Nightly. The success rate of storage initialization significantly improved after landing.
- String changes made/needed: None
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enabling of search modernization depends on this patch.
- User impact if declined: Quota Manager storage initialization breaks for some users which may lead to browser usability problems.
- Fix Landed on Version: 80
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): The patch has been tested on Nightly. The success rate of storage initialization significantly improved after landing.
- String or UUID changes made by this patch: None
Comment 36•5 years ago
|
||
tested fix so far using comment#17 in:
Win10 and Ubuntu 18.04 64bit:
FX Nightly 80.0a1 > modernization is ON. i was able to see search engines.
FX Release 78.0.2 > with custom FF 78 release build having user.js with modernization turned on, i saw the search engines.
Mac0s 10.15
Nightly 80 80.0a1 > modernization is ON. i was able to see search engines.
Comment 37•5 years ago
|
||
Comment on attachment 9160680 [details]
Bug 1649393 - Remove old IDB files directories during 1.0 to 2.0 storage upgrade; r=#dom-workers-and-storage-reviewers
Sounds like there's still some follow-up work to do, but this still strictly sounds like an improvement over the status quo and helps avoid a totally broken browser for some users with corrupted profiles. Approved for 79.0b5 and 78.1esr.
Comment 38•5 years ago
|
||
bugherder uplift |
Comment 39•5 years ago
|
||
bugherder uplift |
Comment 40•5 years ago
•
|
||
Proceeded to verify the uplift, with the following steps:
- Created broken profiles with help of the steps on comment 17.
- forced an user.js with user_pref("browser.search.modernConfig", true);
- Reproduced the broken index db and seach initialization, outputting no search engines
- Update to versions that contained the fix.
with the following results:
79.0a1 2020-06-03(broken) updated to 80.0a1 2020-07-10(fixed)
79.0b2 2020-06-30 (broken) updated to 79.0b5 2020-07-08(fixed)
79.0b2 2020-06-30 (broken) updated to 79.0b6 2020-07-09 (fixed)
78.0.1esr 2020-06-30(broken) updated to 78.0.2esr 20200708170510(broken) - paste bin reflecting the index db is still broken: https://pastebin.com/grfak1TN
The above scenarios and results are the same across the tested platforms: Windows10, Ubuntu 20 and Mac 10.13.6 .
Jan, can you take a look over the results, since my assumption is that for esr 78.0.2 there is something missing for this fix to function as expected.
Note:
This verification tries to exclude bug 1649700, hence the upgrade to 79beta5 which does not contain it.
Comment 41•5 years ago
|
||
This patch is not in 78.0.2esr, it's only on the default branch (future 78.1esr).
Comment 42•5 years ago
•
|
||
thanks @Julien, I was a bit confused by the 78.0.2 branching; adding reminder to re-check this on 78.1esr when it is near release to reduce any risk from other uplifts regression.
Comment 43•5 years ago
|
||
This issue is verified fixed using Windows 10 64bit, macOS 10.14 and Ubuntu 20.
Updated from 78.0esr (broken) to 78.1.0esr (fixed). (using the same scenario as in comment 40)
Comment 44•4 years ago
|
||
Comment 45•4 years ago
|
||
Comment 46•4 years ago
|
||
Comment 47•4 years ago
|
||
after update to firefox 90.0 got the same issue
it seem to come from extensions.
1: To restore search engines I use:
"Stop firefox"
mkdir /tmp/EXT/
cd ~/.mozilla/firefox/your.profile/
mv extension* /tmp/EXT/
"Start firefox" all is good
2: To get the bug again:
"Stop firefox"
cd ~/.mozilla/firefox/your.profile/
mv /tmp/EXT/extension* ./
"Start firefox"
"Stop firefox"
"Start firefox" autocomplete in search bar is disabled, search engines are hidden
So to fix I use 1: then have to reinstall extensions
Description
•