Closed Bug 857888 Opened 12 years ago Closed 3 months ago

webappsstore.sqlite is larger than it should be

Categories

(Core :: Storage: localStorage & sessionStorage, defect, P2)

Product:
Core
Component:
Storage: localStorage & sessionStorage
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
fennec + ---

People

(Reporter: rnewman, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: dom-lws-bugdash-triage)

I don't use the Marketplace, or pin web apps to my home screen; I browse. My Aurora profile on my phone has webappsstore.sqlite 4.1MB webappsstore.sqlite-wal 1.5MB chromeappsstore.sqlite 96KB That's bigger than browser.db, and I have a lot of bookmarks and history! The webappsstore2 table has 498 rows containing what looks like junk: page javascript, piles of JSON, random fragments of URLs. What is this thing, and why is it hanging around in my profile directory?
More importantly, is a sqlite database with a big WAL the best storage system for this? And how does it clean itself up?
These files are related to DOMStorage and have nothing to do with webapps. They're just badly named. chromeappsstore is deprecated. I'm not sure why it stays around.
Ok, let's leave this bug around as a general bucket for annoyances with DOMStorage space. I'll file a bug to have chromeappstore deleted after migration.
On my system, webappstore.sqlite* uses nearly 10MB of space. It's huge! On my low-end ARMv6 phone, I already have a big enough problem with Firefox being over 30MB, but this 10MB worth of files uses way too much space. # pwd /data/data/org.mozilla.firefox/files/mozilla/foo.default # du -ks * | sort -n ... 1049 startupCache 3549 safebrowsing 4096 webappsstore.sqlite 6072 adblockplus 6149 webappsstore.sqlite-wal # # du -ks /data/data/org.mozilla.firefox/files/mozilla* 34777 /data/data/org.mozilla.firefox/files/mozilla
tracking-fennec: --- → ?
Doug, any plans to limit the space used by DOMStorage?
tracking-fennec: ? → +
Flags: needinfo?(doug.turner)
honza, could you take a look at this on mobile? I think ben fixed indexed db to not pre-allocate.
Assignee: nobody → honzab.moz
Flags: needinfo?(doug.turner)
(In reply to Doug Turner (:dougt) from comment #6) > I think ben fixed indexed db to not pre-allocate. Well, sorta... The change I made was to allow a preference to control the default page_size, so you could change that from the default to something else on fennec. SQLite requires one page for each table and index that are created, even if they're empty. Our default page_size is set to 32k, so obviously that can quickly grow bigger than you'd want. However, IndexedDB also compresses its data, something DOMStorage doesn't do. And, if these databases really do have "page javascript, piles of JSON" (comment 0) in them then maybe they're actually going to require that much space even with a smaller page size...
This DB was uncompressed, as far as I can tell -- I was browsing through it with SQLite Debugger on the phone. I think the solution here will have to involve cleanup, possibly linked to Fennec's existing memory pressure hooks, and perhaps with a UI component for management -- right now the only way to get this space back is to use Android Settings to blow away all of Fennec's data, which is pretty harsh. Phrasing this as a user story: as a user, I expect that visiting Fast Company's site once on my phone should not permanently deprive me of a measurable amount of storage.
Do we have any quota or expiration system for DOMStorage?
(In reply to Mark Finkle (:mfinkle) from comment #9) > Do we have any quota or expiration system for DOMStorage? We have quota (5MB per eTLD+1). We don't have any expiration or eviction mechanism.
Blocks: fatfennec
Blocks: 1042372
No longer blocks: fatfennec
I've see that webappstore.sqlite and netpredictions.sqlite have more than 100mb. Maybe is strange?
That is horrible. Maybe we can try taking a look at your databases here.
I'm uploading the files on my owncloud private server. webappsstore.sqlite - 103.8mb netpredictions.sqlite - 118.6mb Zipped are 56 mb file. Funny. LOL
Bug 1045884 tracks eliminating netpredictions.sqlite entirely. It's no longer used.
Uhm i'm using the 33 version, i wait the new version for remove this file. Maybe an autoremove on the upgrade?
(In reply to Daniele "Mte90" Scasciafratte from comment #16) > Uhm i'm using the 33 version, i wait the new version for remove this file. > Maybe an autoremove on the upgrade? We should remove or clean up the db in that bug, but there's no fix yet.
For my huge webappsstore.sqlite any news? PS: i see the ticket is for Firefox for android but my file are of Firefox desktop.
There is unfortunately no progress on this bug.
Thank you for sharing your data. Your (desktop) webappsstore.sqlite contains 7,834 entries for 1,625 unique scopes. On average, each entry is 12KB, totaling about 95MB. Data includes mediawiki module storage, JS file URLs, Twitter user hashes, YouTube settings, Chartbeat values from Delicious, LinkedIn JS caches, etc. What's most interesting, beyond the long tail, are the biggest entries: scope key length(value) ----------------------- ---------- ------------- moc.ksidllif.2.:http:80 filldisk 2500000 moc.ksidllif.1.:http:80 filldisk 2500000 moc.mvhh.scod.:http:80 search-en 2068396 ten.php.:http:80 search-en 2008770 ten.php.1ti.:http:80 search-it 2004574 ten.php.2ti.:http:80 search-it 2004574 moc.xoferif.ecalptekram 0::model_c 1376262 moc.rettiwt.:https:443 __typeahea 969012 gro.aidemikiw.snommoc.: MediaWikiM 799795 moc.kcedteewt.bew.:http typeaheadU 771099 So filldisk.com is using 5MB alone. Then there are a bunch of sites using 1-2MB, and a long tail down from there. There are over five hundred entries larger than 50KB. marketplace.firefox.com is taking up 1.2MB. About 3MB was recovered by vacuuming from the 104MB. The only general solution to fat webappsstore.sqlite is eviction and expiration. It is unacceptable for thousands of sites to be forever storing hundreds of KB each, particularly on a phone. I'll leave this assigned to Honza, and move it to Core :: DOM. I'll file another bug to make marketplace.firefox.com less bloaty.
Status: NEW → ASSIGNED
Component: General → DOM
OS: Android → All
Product: Firefox for Android → Core
Hardware: ARM → All
Maybe a setting for set the timeout or a page for manually delete the datas.
filter on [mass-p5]
Priority: -- → P5
Priority: P5 → --
No.
Status: ASSIGNED → NEW
Flags: needinfo?(honzab.moz)
FYI: My webappsstore.sqlite is 57mb, my places.sqlite is 71mb. We need a utility reduce the size of these, delete old history or keywords.
Note: Firefox 46 will be doing a vacuum during the database update.
Bug 1058934 just got WONTFIXed, so this bug could expand to "throw away everything stored for marketplace.firefox.com". Re Comment 28: note that a sqlite VACUUM won't recover any space unless database content is first deleted or replaced by smaller values. (Indeed, temporarily it will double our space usage as sqlite rewrites the entire DB.) The bulk of the issue here is that we don't ever delete DOM storage, so if you visit a site that stores something, you will lose that space until you uninstall Firefox from your phone or wipe your desktop profile. Honza, Olli: do you have a plan for making progress here?
Flags: needinfo?(honzab.moz)
Flags: needinfo?(bugs)
See Also: → 1058934
I'll leave storage handling (in its various forms) to mayhemer and janv. (Happy to review patches anyhow.)
Flags: needinfo?(bugs) → needinfo?(jvarga)
> Note: Firefox 46 will be doing a vacuum during the database update. That doesn't fix the problem. VACUUM only releases space for *records that were deleted* in the database. We need a way to delete old records from within Firefox. I don't need stuff in these sqlite dbs from 3 years ago for a website I only went to once. I'd like to see each record in each of those databases, and the date it was last used or created. I'd like to be able to sort these records by that date, and select multiple records (it could be 100+ records) and delete the old ones. I've never gone to marketplace.firefox.com, I don't even know what it is. For now there is a 3rd party utility, and even a Firefox plugin (https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search) to deal with these sqlite databases, but that's too much trouble for the average user. Windows 3rd party util: Sqlite manager: https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search Wikipedia page: https://en.wikipedia.org/wiki/SQLite
****, I can't edit my comment. When doing a VACUUM the db doubles in size temporarily. Make sure the user has enough disk space to do this, when doing a VACUUM on each db. And what are my formatting options in Bugzilla? lol. I can't seem to find if it supports any markup of any kind. :) Also, the Sqlite Manager above is not free. SQDatabase is free: http://sqdatabase.com/
I think we need bug 527667 here.
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(honzab.moz)
Resolution: --- → DUPLICATE
To explain: there is currently no data about how old a record is. That bug is about to add it. We can then add a mechanism pruning old data automatically. OTOH, localStorage has no expiration mechanism by default. So hard to say how 'old' something has to be to remove it on users behalf.
The problem described in this bug is one of the reasons why we are moving localStorage to QuotaManager and temporary storage in bug 742822. It means that when temporary storage is full, quota manager starts evicting data for origins using LRU policy. Also, at some point, we can also add a new feature that would delete data for origins that were not used for X days/months/years.
Flags: needinfo?(jvarga)
I closed firefox and deleted webappsstore.sqlite from the profile folder. No side effect.
Thanks for the pointers, Honza and Jan. Flipping this to depend on those two bugs -- one to add timestamping, one to add quotas. Comment 34 and Comment 35 don't sound like they will directly solve this problem, but they're prerequisites, so un-duping this.
Status: RESOLVED → REOPENED
Depends on: 527667, 742822
Resolution: DUPLICATE → ---
(In reply to Richard Newman [:rnewman] from comment #37) > Thanks for the pointers, Honza and Jan. > > Flipping this to depend on those two bugs -- one to add timestamping, one to > add quotas. > > Comment 34 and Comment 35 don't sound like they will directly solve this > problem, but they're prerequisites, so un-duping this. If bug 742822 won't directly solve this, why it would directly solve bug 1238771?
(In reply to Richard Newman [:rnewman] from comment #37) > Thanks for the pointers, Honza and Jan. > > Flipping this to depend on those two bugs -- one to add timestamping, one to > add quotas. > > Comment 34 and Comment 35 don't sound like they will directly solve this > problem, but they're prerequisites, so un-duping this. Yep, it's better. There is still more work to do when we have the timing info on each data row. I agree.
Status: REOPENED → NEW
+1, as Firefox on a new install slows down within one month to such an extent that many sites are hardly navigatable.
Depends on: 1286798
See Also: → 1296572
Priority: -- → P5
See Also: → 1477890
Priority: P5 → P2
Whiteboard: DWS_NEXT
Blocks: 1504142
I was having this problem as well in SeaMonkey v2.49.4 (User agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.4). It was 33.2 MB since October 2016! I found out because https://flickmetrix.com/ was not giving me results until I discovered this file. I assume it was corrupted since (delet/renam)ing it fixed my issue.
Component: DOM → DOM: Core & HTML
Component: DOM: Core & HTML → DOM: Web Storage
Severity: normal → S3

I checked and mine has 285mb of stuff with 36274 lines.

I am wondering if there is a way to clean up data from an year ago as example, the columns in the table doesn't have this kind of data.

Now I am noticing that the file last change is 19/05/22 and the size didn't changed.
Maybe the files is not used anymore and it is safe to remove it?

So the file doesn't changed in the meantime (still 19/05/22) but the webappsstore.sqlite-wal file instead was touched on 05/04/2024.
The file can be removed or not?

(In reply to Daniele "Mte90" Scasciafratte from comment #45)

So the file doesn't changed in the meantime (still 19/05/22) but the webappsstore.sqlite-wal file instead was touched on 05/04/2024.
The file can be removed or not?

This can happen when the storage/ls-archive.sqlite has been manually removed or entire storage/ folder has been removed.
The archive needs to be created again and for that webappsstore.sqlite serves as a data source (hence the modified time of the wal file). If webappsstore.sqlite is unusable or empty, everything should work normally, the ls-archive.sqlite will end up being empty and that's ok as well.
So I would say (for now unofficially) that removing webappsstore.sqlite and webappsstore.sqlite-wal manually while the app is fully closed is ok. These files may reappear (if ls-archive.sqlite needs to be recreated), but they will be rather small and will stay small.

I removed the file and everything works with no issues.
I see that the file was generated again and that on storage there is ls-archive.sqlite that is like 200~ mb

I guess the next question which is coming is whether you can remove ls-archive.sqlite :)

I understand there to have been 2 primary concerns here, both having to do with the amount of disk usage:

  • Potential inefficiency in the SQLite database page size / growth factor.
  • Data clearing policy for web sites that haven't been visited in a while.

The single database file has been replaced by QuotaManager-managed individual files and we were sensitive to growth factors in the choice of page sizes there. Data clearing policy is still an active issue for enhancement, but is somewhat difficult because it's hard to tell a website a user will want to visit again from one they won't want to visit. The anti-tracking team does have a subsystem that tries to at least determine sites that were tracker-y in nature and purge those.

Status: NEW → RESOLVED
Closed: 9 years ago → 3 months ago
Resolution: --- → INCOMPLETE
Whiteboard: DWS_NEXT → dom-lws-bugdash-triage
You need to log in before you can comment on or make changes to this bug.