Closed Bug 1649393 Opened 1 month ago Closed 1 month ago

Search engines are gone with v78.0

Categories

(Core :: Storage: IndexedDB, defect, P2)

78 Branch
defect

Tracking

()

VERIFIED FIXED
mozilla80
Tracking Status
firefox-esr68 --- wontfix
firefox-esr78 79+ verified
firefox77 --- wontfix
firefox78 - disabled
firefox79 + verified
firefox80 + verified

People

(Reporter: ulli.conrad, Assigned: janv)

References

(Blocks 1 open bug)

Details

Attachments

(7 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

Updated to v78.0

Actual results:

All search engines are gone, list of one-click search engines is empty now
Auto complete in the address bar doesn't work any longer
Search function on the start page doesn't start a search any longer

Expected results:

Search engines should disappear after an update

Component: Untriaged → Search
Duplicate of this bug: 1649389

Hi Ulli, please can you provide us with some more information to help us diagnose the issue? Here's what would be useful:

  • Go to the three bar menu -> Help -> Troubleshooting Information.

  • Click the "Copy text to Clipboard" Button.

  • Back on this bug, just above your first comment, click "Attach new file", click in the box at the top and do ctrl-v (or right-click and select paste).

  • Give it a description and then hit submit at the bottom.

  • Then, restart Firefox, as soon as it is started, go to the three bar menu -> Web Developer -> Browser Console (not the Web Console).

  • Right-click and Select All

  • Then Right-click again and Copy Message.

  • Paste that into another new attachment as per the above instructions.

That will hopefully give us enough information, but we may need a little more. Either way, we'll hopefully be able to get you running again.

Flags: needinfo?(ulli.conrad)
Flags: needinfo?(ulli.conrad)
Attached file Browser console

Thank you, for some reason the local database that we use to store some data is not working. This errors are particularly pointing to that:

IndexedDB UnknownErr: ActorsParent.cpp:17128
UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code.
IndexedDB UnknownErr: ActorsParent.cpp:17128
UnknownError: IndexedDB: main/anti-tracking-url-decoration getLastModified() IndexedDB:   The operation failed for reasons unrelated to the database itself and not covered by any other error code. IDBHelpers.jsm:24:5
IndexedDB: main/hijack-blocklists getLastModified() IndexedDB:   The operation failed for reasons unrelated to the database itself and not covered by any other error code. IDBHelpers.jsm:24

Unfortunately our search service has just switched to using that local database, and as a result isn't working properly.

I'll see if I can find some people to have a look at it.

Status: UNCONFIRMED → NEW
Component: Search → Storage: IndexedDB
Ever confirmed: true
Product: Firefox → Core

Thank you, at least a quickt idea what's goin' wrong :-)

Jan/Baku, please could you take a look at this? The search service is now using remote settings in 78, and if indexed DB isn't working, that stops that and a lot of other things from working.

Flags: needinfo?(jvarga)
Flags: needinfo?(amarchesini)

I'm looking at this, the attached browser console output is very useful.

Flags: needinfo?(jvarga)
Flags: needinfo?(amarchesini)
Assignee: nobody → jvarga
Status: NEW → ASSIGNED

Ulli, from your other bug, you implied that you've restored an old profile from backup (not downgraded) and you're using that with 77 now.

Do you get similar error messages on the browser console with that older profile? Obviously the search issue won't be there, but I'm wondering about the IndexedDB side.

Flags: needinfo?(ulli.conrad)

Doesn't look so, but it is a newly created profile with all needed files (places.sqlite,prefs.js and so on) copied back from backup. All the rubbish collected over the years of course is gone now...

Flags: needinfo?(ulli.conrad)

The line would be https://searchfox.org/mozilla-release/rev/e9fe0d92b780775234419a0651fef84f6e8311f2/dom/indexedDB/ActorsParent.cpp#17128, which is within QuotaClient::UpgradeStorageFrom1_0To2_0, which would be consistent with this happening with an old profile.

Severity: -- → S4
Priority: -- → P5
Severity: S4 → S3
Priority: P5 → P2

The good news is that remote settings code doesn't depend on temporary storage initialization (storage for the web). The relevant IDB files live in permanent/chrome/idb/, 3870112724rsegmnoittet-es.files and 3870112724rsegmnoittet-es.sqlite

We have a telemetry for initialization of directories like this [1]
The relevant keys for this bug are "Storage" and "UpgradeStorageFrom1_0To2_0"
"Storage" is at 99.99% for Nightly which roughly matches Release.
The failure at [2] is covered by the "UpgradeStorageFrom1_0To2_0" key and there are currently 194 failures out of 280 tries on Nightly for last month (we track it only once per FF session).
We can also provide exact data for Release using custom queries.
There are other upgrades with relevant telemetry, for example "UpgradeStorageFrom2_0To2_1", but they all look good.
So there's a high probability that fixing the problem at [2] will improve overall initialization success rate and mitigate the problem with search engines.

I'm now preparing a custom FF 78 build which can be used to identify precise directory in user's profile which causes the upgrade to fail.
We will then ask the reporter to provide full recursive listing of the directory.

[1] https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2020-06-28&include_spill=0&keys=UpgradeStorageFrom1_0To2_0!PersistentOrigin!Storage!TemporaryStorage&max_channel_version=nightly%252F79&measure=QM_FIRST_INITIALIZATION_ATTEMPT&min_channel_version=null&processType=*&product=Firefox&sanitize=0&sort_by_value=0&sort_keys=submissions&start_date=2020-06-01&table=1&trim=1&use_submission_date=0

[2] https://searchfox.org/mozilla-release/rev/e9fe0d92b780775234419a0651fef84f6e8311f2/dom/indexedDB/ActorsParent.cpp#17128

Here is data for Release 78:

Storage key
number of all storage initializations (roughly matches number of FF sessions): 27925
number of failed storage initializations: 9
success rate: 99.97%

UpgradeFromIndexedDBDirectory key
number of all storage upgrades: 1
success rate: 100%

UpgradeFromPersistentStorageDirectory
number of all storage upgrades: 13
success rate: 100%

UpgradeStorageFrom0_0To1_0
number of all storage upgrades: 35
success rate: 100%

UpgradeStorageFrom1_0To2_0
number of all storage upgrades: 46
number of failed storage initializations: 7
success rate: 84.78%

UpgradeStorageFrom2_0To2_1
number of all storage upgrades: 46
success rate: 100%

UpgradeStorageFrom2_1To2_2
number of all storage upgrades: 83
success rate: 100%

UpgradeStorageFrom2_2To2_3
number of all storage upgrades: 116
success rate: 100%

There are only 7 failed initializations for UpgradeStorageFrom1_0To2_0, but number of total failures is 9.
It seems there can be something else which breaks the initialization and is not covered by individual keys.

Sorry, I made a mistake in the last comment, it's now fixed.

Attached file storage.zip

I attached a zip file (storage.zip) which can be used to simulate a profile with broken storage (leading to broken search).

Steps for creating broken profile and reproducing the search issue:

  1. Create a new profile
  2. Close FF
  3. Remove or rename the "storage" folder and the "storage.sqlite" database in the new profile (you can find location for it in about:profiles)
  4. Unzip storage.zip in the new profile (it will create the "storage" folder and the "storage.sqlite" database)
  5. Run FF for given profile

Let me know if it doesn't work for you.

Search engines are gone Yandex ?:C why?what for ?

Here are custom FF 78 release builds with additional debugging info (reported to browser console):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=42699d073a2a44ac1846da87266afd09ff45c1ac

Ulli, here's a custom build for x64 windows (win64):
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ic3gHdMuQuKAES04xMTjgQ/runs/0/artifacts/public/build/target.zip

Can you run your profile with the custom build ?

You should see something like this in the browser console:
IndexedDB Directory (<path_to_profile>\storage\default\<some_origin>\idb\<basename>..files) already exists!: ActorsParent.cpp:17130

Can you send me full recursive listing of the <some_origin> directory ?
Thank you!

Flags: needinfo?(ulli.conrad)

Wow you guys are quick :-) Thank you very much indeed! No default storage message, I've send the permanent folder listing instead

Flags: needinfo?(ulli.conrad)

Ok, thanks.

So you probably see messages like this:
IndexedDB Directory (<path_to_profile>\storage\permanent\moz-safe-about+home\idb\818200132aebmoouht.files) already exists!: ActorsParent.cpp:17130

I suspect this was caused by running an older FF (version between 36 and 48 probably) with a profile which was already upgraded by newer FF.
These days, something like that is not possible (fortunately).

Working on a fix and a test now.

The search issue is fixed for 78.0.1 via bug 1649558.

Depends on: 1649700
Duplicate of this bug: 1649394

Just updated to 78.0.1
Everything seems fine now :-) Thank you very much indeed!

If an old IDB files directory exists, it no longer causes a failure during the
storage upgrade. The directory is removed instead.

Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c609868d51f
Remove old IDB files directories during 1.0 to 2.0 storage upgrade; r=dom-workers-and-storage-reviewers,ttung
Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla80

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Please nominate this for Beta and ESR78 approval when you get a chance. Also, should we throw together a test build for the reporter to try out to confirm that this works with their profile as well?

Yeah, I'll do that once telemetry data is available (should be tomorrow).

Here are custom FF 78 release builds with the fix for the upgrade function:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d19f3c2f7aa7c8ed9fee89a2809235018c15584d

Ulli, here's a custom build for x64 windows (win64):
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZWv_IPQiRCSCqwgtOEx2Kg/runs/0/artifacts/public/build/target.zip

Would you have time to run your profile with this custom build ?

You can either set the preference browser.search.modernConfig to true to test if the new search works for you
or go to https://firefox-storage-test.glitch.me/ to see if your storage is all good.

Flags: needinfo?(ulli.conrad)
Attached file Storage debug output

Search engines are back and work with that build although storage seems to be broken

Flags: needinfo?(ulli.conrad)

Ok, there must be something else broken, but the issue reported in comment 4 seems to be fixed.
I might ping you in a different bug about the other issues.
Thank you!

Thank >you<, keep up the great work ;-)

Comment on attachment 9160680 [details]
Bug 1649393 - Remove old IDB files directories during 1.0 to 2.0 storage upgrade; r=#dom-workers-and-storage-reviewers

Beta/Release Uplift Approval Request

  • User impact if declined: Quota Manager storage initialization breaks for some users which may lead to browser usability problems.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch has been tested on Nightly. The success rate of storage initialization significantly improved after landing.
  • String changes made/needed: None

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enabling of search modernization depends on this patch.
  • User impact if declined: Quota Manager storage initialization breaks for some users which may lead to browser usability problems.
  • Fix Landed on Version: 80
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch has been tested on Nightly. The success rate of storage initialization significantly improved after landing.
  • String or UUID changes made by this patch: None
Flags: needinfo?(jvarga)
Attachment #9160680 - Flags: approval-mozilla-esr78?
Attachment #9160680 - Flags: approval-mozilla-beta?

tested fix so far using comment#17 in:

Win10 and Ubuntu 18.04 64bit:
FX Nightly 80.0a1 > modernization is ON. i was able to see search engines.
FX Release 78.0.2 > with custom FF 78 release build having user.js with modernization turned on, i saw the search engines.

Mac0s 10.15
Nightly 80 80.0a1 > modernization is ON. i was able to see search engines.

Comment on attachment 9160680 [details]
Bug 1649393 - Remove old IDB files directories during 1.0 to 2.0 storage upgrade; r=#dom-workers-and-storage-reviewers

Sounds like there's still some follow-up work to do, but this still strictly sounds like an improvement over the status quo and helps avoid a totally broken browser for some users with corrupted profiles. Approved for 79.0b5 and 78.1esr.

Attachment #9160680 - Flags: approval-mozilla-esr78?
Attachment #9160680 - Flags: approval-mozilla-esr78+
Attachment #9160680 - Flags: approval-mozilla-beta?
Attachment #9160680 - Flags: approval-mozilla-beta+

Proceeded to verify the uplift, with the following steps:

  1. Created broken profiles with help of the steps on comment 17.
  2. forced an user.js with user_pref("browser.search.modernConfig", true);
  3. Reproduced the broken index db and seach initialization, outputting no search engines
  4. Update to versions that contained the fix.

with the following results:

79.0a1 2020-06-03(broken) updated to 80.0a1 2020-07-10(fixed)
79.0b2 2020-06-30 (broken) updated to 79.0b5 2020-07-08(fixed)
79.0b2 2020-06-30 (broken) updated to 79.0b6 2020-07-09 (fixed)
78.0.1esr 2020-06-30(broken) updated to 78.0.2esr 20200708170510(broken) - paste bin reflecting the index db is still broken: https://pastebin.com/grfak1TN

The above scenarios and results are the same across the tested platforms: Windows10, Ubuntu 20 and Mac 10.13.6 .

Jan, can you take a look over the results, since my assumption is that for esr 78.0.2 there is something missing for this fix to function as expected.

Note:
This verification tries to exclude bug 1649700, hence the upgrade to 79beta5 which does not contain it.

Flags: needinfo?(jvarga)

This patch is not in 78.0.2esr, it's only on the default branch (future 78.1esr).

Flags: needinfo?(jvarga)

thanks @Julien, I was a bit confused by the 78.0.2 branching; adding reminder to re-check this on 78.1esr when it is near release to reduce any risk from other uplifts regression.

Flags: needinfo?(adrian.florinescu)

This issue is verified fixed using Windows 10 64bit, macOS 10.14 and Ubuntu 20.

Updated from 78.0esr (broken) to 78.1.0esr (fixed). (using the same scenario as in comment 40)

Status: RESOLVED → VERIFIED
Flags: needinfo?(adrian.florinescu)
You need to log in before you can comment on or make changes to this bug.