1649981 - address bar doesn't work if it's impossible to search for results (corrupt or locked database)

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P1)

Description

[marcel.dietzmann]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

installed the latest ESR version 78.0.1 for macOS

Actual results:

the address bar does not work for search engines, the issue seems to be not fixed with ESR 78.0.1 update on macOS

Expected results:

typing something in the address bar and press enter should automatically use my preferres search engine

Comment 1

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[marcel.dietzmann]

Oh, and in addition, I have to clarify – when I put an internal address into the address bar like 192.168.1.1 for a home router, this also doesn't work anymore … I have to put the http:// or https:// in front and in addition if no standard port like 80 or 443 is used I have to add the special port too!

This is really annoying as it worked before (with Firefox 77.x.x) without any problems.

The same problem happens with 'normal websites'. I have to use http://macon.de to access my site, only macon.de doesn't work …

And of course if I put something like "firefox esr address bar bug" (as an example) into the address bar, it should start searching after pressing ENTER, but nothing happens …

Comment 3

[marcel.dietzmann]

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #1)

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Mmh sorry, don't understand how this is connected to "Bugbug", never heard about that before.

I'm a normal user, no developer or engineer. Have never used such tool …

Comment 4

[Francesco Lodolo [:flod]]

Bugbug is a bot that tries to automatically triage bugs.

Can you try this as a start?

• Go to the three bar menu -> Help -> Troubleshooting Information.
• Click the "Copy raw data to clipboard" Button.
• Back on this bug, just above your first comment, click "Attach new file", click in the box at the top and do ctrl-v (or right-click and select paste).
• Give it a description and then hit submit at the bottom.
• Then, restart Firefox, as soon as it is started, go to the three bar menu -> Web Developer -> Browser Console (not the Web Console).
• Right-click and Select All
• Then Right-click again and Copy Message.
• Paste that into another new attachment as per the above instructions.

Comment 5

[marcel.dietzmann]

Attachment: attached Troubleshooting Information from my Firefox ESR 78.0.1 (macOS)

Comment 6

[marcel.dietzmann]

Attachment: attached Browser Console information from my Firefox ESR 78.0.1 (macOS)

Comment 7

[Dale Harvey (:daleharvey)]

Hi, trying to figure out whats up here. Could you possibly:

1. Go to about:config
2. Toggle "browser.search.log" to true
3. Restart
4. Follow the above steps to paste the console in here

It would be a huge help, thank you

Comment 8

[Dale Harvey (:daleharvey)]

Attachment: Console log from reddit user with same issue

Comment 9

[Dale Harvey (:daleharvey)]

We had another report of this issue from reddit, in this case the SearchService completed initialisation but still wouldnt load urls, after the SearchService initialisation there are various SQLLite errors to suggest that SQLLite is soemhow corrupt, both logs have Error: Can't find profile directory. XULStore.jsm:66:15 as well

Comment 10

[marcel.dietzmann]

(In reply to Dale Harvey (:daleharvey) from comment #7)

Hi, trying to figure out whats up here. Could you possibly:

1. Go to about:config
2. Toggle "browser.search.log" to true
3. Restart
4. Follow the above steps to paste the console in here

It would be a huge help, thank you

Will do so as fast as possible, happy if I can help in any way to fix this issue.

PS: If it helps, I have also another Mac (10.15) and a lot of "Test machines" with 10.14.6, 10.15.5 and also 11 beta …

Comment 11

[Marco Bonardo [:mak]]

Hello, I have a few other questions:

1. Could you please tell me the file size of the places.sqlite file in your profile folder? (you can reach it from about:support)
2. Did you ever run SQL queries on it from some external software? Did you ever use other software to cleanup Firefox Data (Examples are Cookies on Mac or CCleaner on Windows).
3. Were you getting normal history/bookmarks results when typing in the urlbar before the update? Did bookmarks/history/downloads UI work correctly, or did you have problems with them, or you don't use them?
4. Which version did you upgrade from, was it Firefox 77 or previous ESR?

Comment 12

[marcel.dietzmann]

(In reply to Marco Bonardo [:mak] from comment #11)

1. Could you please tell me the file size of the places.sqlite file in your profile folder? (you can reach it from about:support)

storage.vacuum.last.places.sqlite = 1587937777

(but this is on my iMac, which has exactly the same issue, not on my MBP)

2. Did you ever run SQL queries on it from some external software? Did you ever use other software to cleanup Firefox Data (Examples are Cookies on Mac or CCleaner on Windows).

I do actively use the following add-ons in Firefox:

1Password extension
Decentraleyes
HTTPS Everywhere
Pricavy Badger
uBlock Origin
uMatrix

And on top, I use the "Cookie.app" (latest version 6.0.16) from the Mac AppStore, yes … As I'am interested in a solution to "manage" cookies for 'all' my Browser (Safari, Brave, Chrome, Opera and Firefox) in one place. But maybe the Cookie.app is a problem?

3. Were you getting normal history/bookmarks results when typing in the urlbar before the update? Did bookmarks/history/downloads UI work correctly, or did you have problems with them, or you don't use them?

All worked fine before, yes. Sometimes the Bookmarks couldn't be edited/deleted, but another restart of Firefox helped normally … this behaviour was mostly a problem directly after an update of Firefox.

4. Which version did you upgrade from, was it Firefox 77 or previous ESR?

Yes, I've upgraded on both machines from the normal Firefox 77 to the new 78 ESR version. Maybe the profile from the normal Firefox is a problem for the ESR version?

Comment 13

[marcel.dietzmann]

PS: I don't use any other "Cleanup"-Tool, no CleanMyMac or such things, as these tools are mostly 'bad' and make more problems than fixing things … ;)

Comment 14

[Marco Bonardo [:mak]]

(In reply to marcel.dietzmann from comment #12)

And on top, I use the "Cookie.app" (latest version 6.0.16) from the Mac AppStore, yes … As I'am interested in a solution to "manage" cookies for 'all' my Browser (Safari, Brave, Chrome, Opera and Firefox) in one place. But maybe the Cookie.app is a problem?

Ah, unfortunately we are aware that Cookie.app in certain cases breaks our databases (or at least prevents Firefox from accessing them).

All worked fine before, yes. Sometimes the Bookmarks couldn't be edited/deleted, but another restart of Firefox helped normally … this behaviour was mostly a problem directly after an update of Firefox.

This seems to confirm my theory, the problem is Cookie.app doesn't let Firefox use its databases, and that in Firefox 78 breaks the urlbar too, not just bookmarks/history.

Comment 15

[Marco Bonardo [:mak]]

Could you please terminate Cookie.app, launch firefox, go to about:config and create a new boolean pref with the name storage.multiProcessAccess.enabled set it to true. Then restart everything (included Cookie.app) and let me know if things are better.

Comment 16

[marcel.dietzmann]

(In reply to Marco Bonardo [:mak] from comment #15)

Could you please terminate Cookie.app, launch firefox, go to about:config and create a new boolean pref with the name storage.multiProcessAccess.enabled set it to true. Then restart everything (included Cookie.app) and let me know if things are better.

Did it, and then came a "warning" from Cookie.app to better not delete cookies when any Browser is active. Was wondering about this, as I'am only let delete cookies automatically when quitting the Browser – but the option "delete every 10 minutes" was activated!

Of course I've deactivated this now, but sadly still the same issue.

Comment 17

[marcel.dietzmann]

Attachment: It's a bad idea to use this option of Cookie.app on macOS …

Comment 18

[marcel.dietzmann]

Oh, what I realise now too is, I can't delete any bookmarks. So my profile / database is somehow corrupt? :(

PS: Started using "Sync" for Bookmarks syncing yesterday, but this is not a problem hopefully …

Maybe I should somehow backup / export my database, create a fresh one and try to import my data again?

Comment 19

[Marco Bonardo [:mak]]

My plan here may involved multiple fixes:

1. We should add back some fallback to the urlbar code, when it's not possible to get a heuristic result we should still search with the default engine or visit a url at least.
2. We may investigate ways to notify when the urlbar can't search... We could maybe use a Tip? Though it may not be great to tell the user "The urlbar doesn't work properly, something may be breaking it" without pointing out a solution
3. We should investigate whether we can a fallback in Sqlite where we try to get an exclusive lock, but if we fail getting it, we continue with the normal non-exclusive behavior. This may require a unix-try-excl vfs or some very smart handling on our side.

Comment 20

[Marco Bonardo [:mak]]

(In reply to marcel.dietzmann from comment #18)

Oh, what I realise now too is, I can't delete any bookmarks. So my profile / database is somehow corrupt? :(

If you can load about:config somehow, try my suggestion from comment 15 about creating a boolean storage.multiProcessAccess.enabled set to true and restarting Firefox.

Comment 21

[Marco Bonardo [:mak]]

I'm marking this as regressed by bug 1636583, because that's the behavior change that made broken databases "visible" in the urlbar and not just in bookmarks/history views. It's not exactly a direct regression, but the effect is similar.

I figured that without the Slack discussion what I said here is unclear.

What happens is that bug 1636583 changed the urlbar behavior a bit to make it more consistent (always obtain the same result when typing the same string) and safer (stop passing raw strings to the docshell). Now the urlbar always executes a search and picks the first result out of it.
But, if the places.sqlite database is corrupt or locked by a third party, we can't get that first result, thus we end up doing nothing, while the previous version was just passing the raw string to the docshell.
This means even if history and bookmarks were broken, and the urlbar was likely to not return results at all, it was still possible to type a string, press Enter and visit a page or search.
This is indeed very useful for support reasons and it should keep working regardless of other broken things.

Comment 22

[marcel.dietzmann]

Ok, as I'am no coder, I don't know why or how this helped, but I've found a solution – now everything works fine again. :)

https://superuser.com/questions/111998/how-do-i-repair-a-corrupted-firefox-places-sqlite-database

The last comment from Daniel (6th June '18) helped successfully!

Comment 23

[Marco Bonardo [:mak]]

Attachment: Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ecc00c97f3dd
Address bar should still allow to search or navigate even if it can't get results. r=harry

Comment 25

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/ecc00c97f3dd

Comment 26

[Marco Bonardo [:mak]]

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

Beta/Release Uplift Approval Request

• User impact if declined: If places.sqlite (or some other database used by the urlbar) is corrupt, the urlbar doesn't allow to search or navigate to typed strings. This disallows also searching for support.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Can be reproduced by corrupting the database through a third party Sqlite app, for example by issuing a "DROP TABLE moz_bookmarks" SQL query. This won't be exactly match in the wild corruption but should be enough to simulate the before/after state.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is providing a fallback code path that in most cases is not hit.
• String changes made/needed:

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: The urlbar should stay workable in most cases, to be able to look for help.
• User impact if declined: If places.sqlite (or some other database used by the urlbar) is corrupt, the urlbar doesn't allow to search or navigate to typed strings.
• Fix Landed on Version: 80
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is providing a fallback code path that in most cases is not hit.
• String or UUID changes made by this patch:

Comment 27

[[Back August 16] Ryan VanderMeulen]

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

Makes us a bit more robust in case of database corruption. Let's start with uplifting this to Beta and go from there with respect to other branches. Approved for 79.0b5.

Comment 28

[[Back August 16] Ryan VanderMeulen]

https://hg.mozilla.org/releases/mozilla-beta/rev/95e348e21b40

Comment 29

[Hani Yacoub]

I'm trying to reproduce and verify the bug on Firefox but I'm having difficulties corrupting the database through Sqlite app.
Could you please provide more details on how exactly could I issue a "DROP TABLE moz_bookmarks" SQL query?
Thanks.

Comment 30

[Marco Bonardo [:mak]]

(In reply to Hani Yacoub from comment #29)

I'm trying to reproduce and verify the bug on Firefox but I'm having difficulties corrupting the database through Sqlite app.
Could you please provide more details on how exactly could I issue a "DROP TABLE moz_bookmarks" SQL query?
Thanks.

You can use the official Sqlite command line tools (see Precompiled Binaries at https://www.sqlite.org/download.html), then open the database with "sqlite3 path_to/places.sqlite" and then issue "DROP TABLE moz_bookmarks;" and ".exit"

Comment 31

[Marco Bonardo [:mak]]

Sorry for the late question, when you had the problem, did you see any warning/notification bar from Firefox about the bookmarks and history system being non functional?

Comment 32

[Hani Yacoub]

Thanks for the details. Actually I'm still trying to figure out how to unlock the database in order to "DROP TABLE moz_bookmarks;".
Is there any other way rather than creating a backup of the database?

Comment 33

[Julien Cristau [:jcristau] (away, back August 16)]

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

make address bar more resilient to places.sqlite corruption, approved for 78.0.2

Comment 34

[Julien Cristau [:jcristau] (away, back August 16)]

for 78.0.2:
https://hg.mozilla.org/releases/mozilla-release/rev/33f017b08942dd3116c57810e67464fd409e77e1

for 78.0.2esr:
https://hg.mozilla.org/releases/mozilla-esr78/rev/2e228bcae46e0851a006c10d119200493996937d

for 78.1esr:
https://hg.mozilla.org/releases/mozilla-esr78/rev/6b778891f6bc5e305ce5325f9b90698ff23ad5ad

Comment 35

[Marco Bonardo [:mak]]

(In reply to Hani Yacoub from comment #32)

Thanks for the details. Actually I'm still trying to figure out how to unlock the database in order to "DROP TABLE moz_bookmarks;".
Is there any other way rather than creating a backup of the database?

Do it when Firefox is closed.

There is also another alternative that I'm testing on Mac/Linux that is, before opening firefox, run sqlite3 places.sqlite and then issue a "BEGIN EXCLUSIVE;", then launch Firefox.

Comment 36

[Hani Yacoub]

Thanks for your help.
Verified as fixed on MacOS 10.14 on the latest Firefox Nightly 80.0a1 and on Firefox 79.0b5.

Comment 37

[Hani Yacoub]

Verified as fixed on MacOS 10.14 on the Firefox 78.0.2 and on Firefox 78.0.2esr.

Comment 38

[Hani Yacoub]

Verified as fixed on MacOS 10.14 on the Firefox 78.1.0esr also.