1763894 - Places database verification hangs

Status: NEW

(Toolkit :: Places, defect, P5)

Description

[MrOtherGuy]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0

Steps to reproduce:

Go to about:support#place-database

Click Verify Integrity

Actual results:

Verification never completes, and CPU usage is very high

Expected results:

Verification should complete in couple of seconds

Comment 1

[MrOtherGuy]

I don't know what exactly triee

Comment 2

[MrOtherGuy]

Pardon the previous incomplete comment.

While CPU use remains high and verification hangs, the browser appears completely usable otherwise.

Based on what I could gather, this bug seems to be caused by a particular favicons.sqlite file. Either because it is very large (about 650MB) or for some other reason. I don't know exactly when this happened, but I recall experiencing shutdown hangs happening sometime in the past 2 weeks.

I have tried to create new profiles and copying the affected favicons.sqlite to them with following results:

Firefox 99 with new profile: Verification is successful
DE 100b.3 with new profile: Verification is successful
Nightly 101a.1 with new profile: Verification hangs indefinitely (or at least +30 minutes)

Verification on DE and release does take some time for such large database (about 10s).

Further information:

• This bug is very likely to cause the following shutdown crash (the process hangs on shutdown): https://crash-stats.mozilla.org/report/index/8ec7f4e3-bd45-4e09-86cc-2c1ac0220409 (this is how I first noticed the issue)
• When the database hangs, the urlbar won't respond correctly to typing.
• New empty favicons.sqlite can be verified just fine on Nightly.
• The affected favicons.sqlite works on Nightly after it has been successfully verified once in DE

Comment 3

[MrOtherGuy]

After some more investigation I can say that the above is not entirely accurate. I found out that to reproduce the issue I need to have BOTH favicons.sqlite AND places.sqlite from in their corrupted state. But I can reproduce for me on Firefox 99, DE 100 and Nightly 101

A very step-by-step description of what I did:

1 Create new profile by running firefox -profile "C:\test\profile"
2 Close that instance so profile files get generated
3 Replace ONLY places.sqlite with my "corrupted" places.sqlite
4 Start firefox with the above command-line
5 about:support and verify integrity
6 It works fine
7 Close firefox and delete the temporary profile folder
8 Create new profile by running firefox -profile "C:\test\profile"
9 Close that instance so profile files get generated
10 Replace ONLY favicons.sqlite with my "corrupted" favicons.sqlite
11 Start firefox with the above command-line
12 about:support and verify integrity
13 It works fine
14 Close firefox and delete the temporary profile folder
15 Create new profile by running firefox -profile "C:\test\profile"
16 Close that instance so profile files get generated
17 Replace BOTH places.sqlite and favicons.sqlite with their "corrupted" versions
18 Start firefox with the above command-line
19 about:support and verify integrity
20 Verification hangs and I get a shutdown hang (which eventually crashes)
21 Delete the temporary profile folder
22 Create new profile by running firefox -profile "C:\test\profile"
23 Close that instance so profile files get generated
24 Replace ONLY places.sqlite with my "corrupted" places.sqlite
25 Start firefox with the above command-line
26 about:support and verify integrity
27 It works fine
28 Close firefox (don't delete profile, places.sqlite is now verified)
29 Replace the favicons.sqlite with its "corrupted version"
30 Start firefox with the above command-line
31 about:support and verify integrity
32 Verification hangs and I get a shutdown hang (which eventually crashes)
33 Delete the temporary profile folder
34 Create new profile by running firefox -profile "C:\test\profile"
35 Close that instance so profile files get generated
36 Replace ONLY favicons.sqlite with my "corrupted" favicons.sqlite
37 Start firefox with the above command-line
38 about:support and verify integrity
39 It works fine
40 Close firefox (don't delete profile, favicons.sqlite is now verified)
41 Replace the places.sqlite with its "corrupted version"
42 Start firefox with the above command-line
43 about:support and verify integrity
44 It works fine

I tried the above in both Firefox 99 and Nightly 101 and the results are the same.

In short, if I verify files separately it works fine. However, the hang appears when I try to verify corrupt favicons.sqlite at the same time as "non-empty places.sqlite" whether that places.sqlite is corrupt or not.

For the record, the file size of my "corrupt versions" are:

• places.sqlite - 61440KiB
• favicons.sqlite - 654752KiB

On new profile the generated database sizes are 5120KiB which might or might not have anything to do with why the issue doesn't reproduce when either one is brand new.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:mak, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 5

[Marco Bonardo [:mak]]

This is very strange, in particular a 650MB favicons.sqlite is very unexpected, it seems to be 10 times bigger than expected. That seems to point out to some kind of file corruption, and such a large size, depending on the PC specs, may also explain why it hangs and uses so much cpu.

copying just favicons.sqlite (without place.sqlite) to a new profile will likely just discard most entries because they won't have a match in places.sqlite. That would explain why then it works, removing orphans is easier and cheaper.

Now the problem is understanding what is taking space into that database... Do you still have a copy of the corrupt favicons.sqlite to analyze?
Could you check it with this tool https://www.sqlite.org/sqlanalyze.html and post results here?

I may also ask you to run some queries to better understand what happens, I wonder mostly if this is effectively corruption or if somehow a site was able to store thousands (or a few) oversized icons.

Comment 6

[MrOtherGuy]

Attachment: sqlite3_analyzer_output_favicons.sqlite

Comment 7

[MrOtherGuy]

Attachment: sqlite3_analyzer_output_places.sqlite

Comment 8

[MrOtherGuy]

Right, I've attached output of the analyzer.

I wonder mostly if this is effectively corruption or if somehow a site was able to store thousands (or a few) oversized icons.

This got me thinking, how would favicons database react to a scenario where a website has a dynamic favicon? I do have an extension (written by me as a test, it's not publicly available). The extension draws a "progress-arc" into the website favicon when the page is playing sound, and the image is updated quite often.

So, if such dynamic changes are stored into the database, then that could very well be bloating it since, as it appears, I've been testing the extension since 2019.

Comment 9

[Marco Bonardo [:mak]]

(In reply to MrOtherGuy from comment #8)

So, if such dynamic changes are stored into the database, then that could very well be bloating it since, as it appears, I've been testing the extension since 2019.

Ideally we should not store favicons that are served with a no-store header (that should be the case for most pages using indicators on favicons), or after the "pageshow" event happened. But I cannot exclude you somehow bypassed that check through the use of an extension, maybe your extension is generating lots of favicons before "pageshow" already? Can you share how the extension works, or even the extension itself?
Your database moz_icons table has clearly a lot of entries, a lot more than I'd expect for that places.sqlite, so it looks like something is indeed adding favicons for each page.

Comment 10

[MrOtherGuy]

Actually, it seems I have the extension source available here. The interesting bits for this issue are probably in content_script.js since that's where the favicon is created. Much of complexity is because the extension can update the top-level tab favicon with the progress of media progress in a sub-frame.

In short the extension:

1. store a copy of the original favicon
2. get an average color from it
3. get media playback progress from playing media
4. draw arc representing the media progress to <canvas>
5. create a new <link> element with canvas image as data-uri (or replace href of one previously generated by the extension)

Anyway, the extension generates bunch of data-uri favicons. Still, none should be created before the tab actually starts producing sound, but perhaps that could happen before pageshow event, I don't know.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:mak, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 12

[Marco Bonardo [:mak]]

OK, this is something to investigate, but off-hand I don't think it's a common problem for most users.
It's likely the extension is somehow overloading the favicons storage system, that causes the db to grow a lot more than expected and may explain the long time it takes to complete the verification.
We should check if we can introduce some sort of protection, I was actually assuming what we have (not storing favicons after pageshow) was enough, but maybe it's not. The addon is a content script that pulls the favicon from the page, when the background script calls into it, it applies some transformations to the favicon and changes the href of the <link>.
On your side, you could maybe try changing the add-on to not act before pageshow has been received.

Comment 13

[MrOtherGuy]

Just some findings here, I made few changes to the extension such that it doesn't create new icons before pageshow event and added some logging to count whether the tab had begun producing sound before pageshow had happened. I've been running a couple of days with this setup and unless my logging logic is somehow totally wrong, pageshow has always happened before the extension creates any new favicons.

That is not very surprising IMO, since the injected content-script that would create new favicon is only injected to content when Firefox notifies the background-script that the tab starts producing sound.

Regardless, if the issue is indeed triggered by this extension, then what I'm more worried about is that web-sites could do the same sort of thing and break favicon database. Even more worrying is if favicon database getting bloated somehow affects places database then it could potentially cause data-loss.

A month back I removed the old favicons.sqlite from this profile and Firefox created new one. Since then, favicons.sqlite has grown to about 112MB in size. I wouldn't worry about disk-space use otherwise, but if 650MB is about 10x what you say it should be, then 112MB is already about 2x in a months time.