Closed Bug 1649981 Opened 1 year ago Closed 1 year ago

address bar doesn't work if it's impossible to search for results (corrupt or locked database)

Categories

(Firefox :: Address Bar, defect, P1)

78 Branch
defect
Points:
3

Tracking

()

VERIFIED FIXED
Firefox 80
Iteration:
80.1 - June 29 - July 12
Tracking Status
firefox-esr68 --- unaffected
firefox-esr78 --- verified
firefox77 --- unaffected
firefox78 --- verified
firefox79 --- verified
firefox80 --- verified

People

(Reporter: marcel.dietzmann, Assigned: mak, NeedInfo)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(5 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

installed the latest ESR version 78.0.1 for macOS

Actual results:

the address bar does not work for search engines, the issue seems to be not fixed with ESR 78.0.1 update on macOS

Expected results:

typing something in the address bar and press enter should automatically use my preferres search engine

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Address Bar

Oh, and in addition, I have to clarify – when I put an internal address into the address bar like 192.168.1.1 for a home router, this also doesn't work anymore … I have to put the http:// or https:// in front and in addition if no standard port like 80 or 443 is used I have to add the special port too!

This is really annoying as it worked before (with Firefox 77.x.x) without any problems.

The same problem happens with 'normal websites'. I have to use http://macon.de to access my site, only macon.de doesn't work …

And of course if I put something like "firefox esr address bar bug" (as an example) into the address bar, it should start searching after pressing ENTER, but nothing happens …

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #1)

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Mmh sorry, don't understand how this is connected to "Bugbug", never heard about that before.

I'm a normal user, no developer or engineer. Have never used such tool …

Bugbug is a bot that tries to automatically triage bugs.

Can you try this as a start?

  • Go to the three bar menu -> Help -> Troubleshooting Information.
  • Click the "Copy raw data to clipboard" Button.
  • Back on this bug, just above your first comment, click "Attach new file", click in the box at the top and do ctrl-v (or right-click and select paste).
  • Give it a description and then hit submit at the bottom.
  • Then, restart Firefox, as soon as it is started, go to the three bar menu -> Web Developer -> Browser Console (not the Web Console).
  • Right-click and Select All
  • Then Right-click again and Copy Message.
  • Paste that into another new attachment as per the above instructions.
Component: Address Bar → Search
Flags: needinfo?(marcel.dietzmann)
Flags: needinfo?(marcel.dietzmann)

Hi, trying to figure out whats up here. Could you possibly:

  1. Go to about:config
  2. Toggle "browser.search.log" to true
  3. Restart
  4. Follow the above steps to paste the console in here

It would be a huge help, thank you

Flags: needinfo?(marcel.dietzmann)

We had another report of this issue from reddit, in this case the SearchService completed initialisation but still wouldnt load urls, after the SearchService initialisation there are various SQLLite errors to suggest that SQLLite is soemhow corrupt, both logs have Error: Can't find profile directory. XULStore.jsm:66:15 as well

(In reply to Dale Harvey (:daleharvey) from comment #7)

Hi, trying to figure out whats up here. Could you possibly:

  1. Go to about:config
  2. Toggle "browser.search.log" to true
  3. Restart
  4. Follow the above steps to paste the console in here

It would be a huge help, thank you

Will do so as fast as possible, happy if I can help in any way to fix this issue.

PS: If it helps, I have also another Mac (10.15) and a lot of "Test machines" with 10.14.6, 10.15.5 and also 11 beta …

Flags: needinfo?(marcel.dietzmann)

Hello, I have a few other questions:

  1. Could you please tell me the file size of the places.sqlite file in your profile folder? (you can reach it from about:support)
  2. Did you ever run SQL queries on it from some external software? Did you ever use other software to cleanup Firefox Data (Examples are Cookies on Mac or CCleaner on Windows).
  3. Were you getting normal history/bookmarks results when typing in the urlbar before the update? Did bookmarks/history/downloads UI work correctly, or did you have problems with them, or you don't use them?
  4. Which version did you upgrade from, was it Firefox 77 or previous ESR?
Flags: needinfo?(marcel.dietzmann)

(In reply to Marco Bonardo [:mak] from comment #11)

  1. Could you please tell me the file size of the places.sqlite file in your profile folder? (you can reach it from about:support)

storage.vacuum.last.places.sqlite = 1587937777

(but this is on my iMac, which has exactly the same issue, not on my MBP)

  1. Did you ever run SQL queries on it from some external software? Did you ever use other software to cleanup Firefox Data (Examples are Cookies on Mac or CCleaner on Windows).

I do actively use the following add-ons in Firefox:

1Password extension
Decentraleyes
HTTPS Everywhere
Pricavy Badger
uBlock Origin
uMatrix

And on top, I use the "Cookie.app" (latest version 6.0.16) from the Mac AppStore, yes … As I'am interested in a solution to "manage" cookies for 'all' my Browser (Safari, Brave, Chrome, Opera and Firefox) in one place. But maybe the Cookie.app is a problem?

  1. Were you getting normal history/bookmarks results when typing in the urlbar before the update? Did bookmarks/history/downloads UI work correctly, or did you have problems with them, or you don't use them?

All worked fine before, yes. Sometimes the Bookmarks couldn't be edited/deleted, but another restart of Firefox helped normally … this behaviour was mostly a problem directly after an update of Firefox.

  1. Which version did you upgrade from, was it Firefox 77 or previous ESR?

Yes, I've upgraded on both machines from the normal Firefox 77 to the new 78 ESR version. Maybe the profile from the normal Firefox is a problem for the ESR version?

Flags: needinfo?(marcel.dietzmann)

PS: I don't use any other "Cleanup"-Tool, no CleanMyMac or such things, as these tools are mostly 'bad' and make more problems than fixing things … ;)

(In reply to marcel.dietzmann from comment #12)

And on top, I use the "Cookie.app" (latest version 6.0.16) from the Mac AppStore, yes … As I'am interested in a solution to "manage" cookies for 'all' my Browser (Safari, Brave, Chrome, Opera and Firefox) in one place. But maybe the Cookie.app is a problem?

Ah, unfortunately we are aware that Cookie.app in certain cases breaks our databases (or at least prevents Firefox from accessing them).

All worked fine before, yes. Sometimes the Bookmarks couldn't be edited/deleted, but another restart of Firefox helped normally … this behaviour was mostly a problem directly after an update of Firefox.

This seems to confirm my theory, the problem is Cookie.app doesn't let Firefox use its databases, and that in Firefox 78 breaks the urlbar too, not just bookmarks/history.

Could you please terminate Cookie.app, launch firefox, go to about:config and create a new boolean pref with the name storage.multiProcessAccess.enabled set it to true. Then restart everything (included Cookie.app) and let me know if things are better.

Flags: needinfo?(marcel.dietzmann)
Summary: address bar still gives zero results / doesn't work → address bar doesn't work if it's impossible to search for results (corrupt or locked database)

(In reply to Marco Bonardo [:mak] from comment #15)

Could you please terminate Cookie.app, launch firefox, go to about:config and create a new boolean pref with the name storage.multiProcessAccess.enabled set it to true. Then restart everything (included Cookie.app) and let me know if things are better.

Did it, and then came a "warning" from Cookie.app to better not delete cookies when any Browser is active. Was wondering about this, as I'am only let delete cookies automatically when quitting the Browser – but the option "delete every 10 minutes" was activated!

Of course I've deactivated this now, but sadly still the same issue.

Flags: needinfo?(marcel.dietzmann)

Oh, what I realise now too is, I can't delete any bookmarks. So my profile / database is somehow corrupt? :(

PS: Started using "Sync" for Bookmarks syncing yesterday, but this is not a problem hopefully …

Maybe I should somehow backup / export my database, create a fresh one and try to import my data again?

My plan here may involved multiple fixes:

  1. We should add back some fallback to the urlbar code, when it's not possible to get a heuristic result we should still search with the default engine or visit a url at least.
  2. We may investigate ways to notify when the urlbar can't search... We could maybe use a Tip? Though it may not be great to tell the user "The urlbar doesn't work properly, something may be breaking it" without pointing out a solution
  3. We should investigate whether we can a fallback in Sqlite where we try to get an exclusive lock, but if we fail getting it, we continue with the normal non-exclusive behavior. This may require a unix-try-excl vfs or some very smart handling on our side.

(In reply to marcel.dietzmann from comment #18)

Oh, what I realise now too is, I can't delete any bookmarks. So my profile / database is somehow corrupt? :(

If you can load about:config somehow, try my suggestion from comment 15 about creating a boolean storage.multiProcessAccess.enabled set to true and restarting Firefox.

I'm marking this as regressed by bug 1636583, because that's the behavior change that made broken databases "visible" in the urlbar and not just in bookmarks/history views. It's not exactly a direct regression, but the effect is similar.

I figured that without the Slack discussion what I said here is unclear.

What happens is that bug 1636583 changed the urlbar behavior a bit to make it more consistent (always obtain the same result when typing the same string) and safer (stop passing raw strings to the docshell). Now the urlbar always executes a search and picks the first result out of it.
But, if the places.sqlite database is corrupt or locked by a third party, we can't get that first result, thus we end up doing nothing, while the previous version was just passing the raw string to the docshell.
This means even if history and bookmarks were broken, and the urlbar was likely to not return results at all, it was still possible to type a string, press Enter and visit a page or search.
This is indeed very useful for support reasons and it should keep working regardless of other broken things.

Assignee: nobody → mak
Severity: -- → S2
Status: UNCONFIRMED → ASSIGNED
Iteration: --- → 80.1 - June 29 - July 12
Component: Search → Address Bar
Ever confirmed: true
Priority: -- → P1
Regressed by: 1636583
Depends on: 1650201
Blocks: 1650201
No longer depends on: 1650201

Ok, as I'am no coder, I don't know why or how this helped, but I've found a solution – now everything works fine again. :)

https://superuser.com/questions/111998/how-do-i-repair-a-corrupted-firefox-places-sqlite-database

The last comment from Daniel (6th June '18) helped successfully!

Blocks: 1650205
Pushed by htwyford@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ecc00c97f3dd
Address bar should still allow to search or navigate even if it can't get results. r=harry
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 80

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

Beta/Release Uplift Approval Request

  • User impact if declined: If places.sqlite (or some other database used by the urlbar) is corrupt, the urlbar doesn't allow to search or navigate to typed strings. This disallows also searching for support.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: Can be reproduced by corrupting the database through a third party Sqlite app, for example by issuing a "DROP TABLE moz_bookmarks" SQL query. This won't be exactly match in the wild corruption but should be enough to simulate the before/after state.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is providing a fallback code path that in most cases is not hit.
  • String changes made/needed:

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: The urlbar should stay workable in most cases, to be able to look for help.
  • User impact if declined: If places.sqlite (or some other database used by the urlbar) is corrupt, the urlbar doesn't allow to search or navigate to typed strings.
  • Fix Landed on Version: 80
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is providing a fallback code path that in most cases is not hit.
  • String or UUID changes made by this patch:
Attachment #9161196 - Flags: approval-mozilla-release?
Attachment #9161196 - Flags: approval-mozilla-esr78?
Attachment #9161196 - Flags: approval-mozilla-beta?
Flags: qe-verify+

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

Makes us a bit more robust in case of database corruption. Let's start with uplifting this to Beta and go from there with respect to other branches. Approved for 79.0b5.

Attachment #9161196 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
QA Whiteboard: [qa-triaged]

I'm trying to reproduce and verify the bug on Firefox but I'm having difficulties corrupting the database through Sqlite app.
Could you please provide more details on how exactly could I issue a "DROP TABLE moz_bookmarks" SQL query?
Thanks.

Flags: needinfo?(mak)

(In reply to Hani Yacoub from comment #29)

I'm trying to reproduce and verify the bug on Firefox but I'm having difficulties corrupting the database through Sqlite app.
Could you please provide more details on how exactly could I issue a "DROP TABLE moz_bookmarks" SQL query?
Thanks.

You can use the official Sqlite command line tools (see Precompiled Binaries at https://www.sqlite.org/download.html), then open the database with "sqlite3 path_to/places.sqlite" and then issue "DROP TABLE moz_bookmarks;" and ".exit"

Flags: needinfo?(mak)

Sorry for the late question, when you had the problem, did you see any warning/notification bar from Firefox about the bookmarks and history system being non functional?

Flags: needinfo?(marcel.dietzmann)

Thanks for the details. Actually I'm still trying to figure out how to unlock the database in order to "DROP TABLE moz_bookmarks;".
Is there any other way rather than creating a backup of the database?

Flags: needinfo?(mak)

Comment on attachment 9161196 [details]
Bug 1649981 - Address bar should still allow to search or navigate even if it can't get results. r=harry

make address bar more resilient to places.sqlite corruption, approved for 78.0.2

Attachment #9161196 - Flags: approval-mozilla-release?
Attachment #9161196 - Flags: approval-mozilla-release+
Attachment #9161196 - Flags: approval-mozilla-esr78?
Attachment #9161196 - Flags: approval-mozilla-esr78+

(In reply to Hani Yacoub from comment #32)

Thanks for the details. Actually I'm still trying to figure out how to unlock the database in order to "DROP TABLE moz_bookmarks;".
Is there any other way rather than creating a backup of the database?

Do it when Firefox is closed.

There is also another alternative that I'm testing on Mac/Linux that is, before opening firefox, run sqlite3 places.sqlite and then issue a "BEGIN EXCLUSIVE;", then launch Firefox.

Flags: needinfo?(mak)
Points: --- → 5

Thanks for your help.
Verified as fixed on MacOS 10.14 on the latest Firefox Nightly 80.0a1 and on Firefox 79.0b5.

Verified as fixed on MacOS 10.14 on the Firefox 78.0.2 and on Firefox 78.0.2esr.

Status: RESOLVED → VERIFIED
QA Whiteboard: [qa-triaged]
Flags: qe-verify+

Verified as fixed on MacOS 10.14 on the Firefox 78.1.0esr also.

Points: --- → 3
You need to log in before you can comment on or make changes to this bug.