More importantly, is a sqlite database with a big WAL the best storage system for this? And how does it clean itself up?
These files are related to DOMStorage and have nothing to do with webapps. They're just badly named. chromeappsstore is deprecated. I'm not sure why it stays around.
Ok, let's leave this bug around as a general bucket for annoyances with DOMStorage space. I'll file a bug to have chromeappstore deleted after migration.
On my system, webappstore.sqlite* uses nearly 10MB of space. It's huge! On my low-end ARMv6 phone, I already have a big enough problem with Firefox being over 30MB, but this 10MB worth of files uses way too much space. # pwd /data/data/org.mozilla.firefox/files/mozilla/foo.default # du -ks * | sort -n ... 1049 startupCache 3549 safebrowsing 4096 webappsstore.sqlite 6072 adblockplus 6149 webappsstore.sqlite-wal # # du -ks /data/data/org.mozilla.firefox/files/mozilla* 34777 /data/data/org.mozilla.firefox/files/mozilla
Doug, any plans to limit the space used by DOMStorage?
honza, could you take a look at this on mobile? I think ben fixed indexed db to not pre-allocate.
This DB was uncompressed, as far as I can tell -- I was browsing through it with SQLite Debugger on the phone. I think the solution here will have to involve cleanup, possibly linked to Fennec's existing memory pressure hooks, and perhaps with a UI component for management -- right now the only way to get this space back is to use Android Settings to blow away all of Fennec's data, which is pretty harsh. Phrasing this as a user story: as a user, I expect that visiting Fast Company's site once on my phone should not permanently deprive me of a measurable amount of storage.
Do we have any quota or expiration system for DOMStorage?
(In reply to Mark Finkle (:mfinkle) from comment #9) > Do we have any quota or expiration system for DOMStorage? We have quota (5MB per eTLD+1). We don't have any expiration or eviction mechanism.
I've see that webappstore.sqlite and netpredictions.sqlite have more than 100mb. Maybe is strange?
That is horrible. Maybe we can try taking a look at your databases here.
I'm uploading the files on my owncloud private server. webappsstore.sqlite - 103.8mb netpredictions.sqlite - 118.6mb Zipped are 56 mb file. Funny. LOL
This are the sqlite files https://it.owncube.com/public.php?service=files&t=215a555facdb716c7046d37ad114daee
Bug 1045884 tracks eliminating netpredictions.sqlite entirely. It's no longer used.
Uhm i'm using the 33 version, i wait the new version for remove this file. Maybe an autoremove on the upgrade?
(In reply to Daniele "Mte90" Scasciafratte from comment #16) > Uhm i'm using the 33 version, i wait the new version for remove this file. > Maybe an autoremove on the upgrade? We should remove or clean up the db in that bug, but there's no fix yet.
For my huge webappsstore.sqlite any news? PS: i see the ticket is for Firefox for android but my file are of Firefox desktop.
There is unfortunately no progress on this bug.
Thank you for sharing your data. Your (desktop) webappsstore.sqlite contains 7,834 entries for 1,625 unique scopes. On average, each entry is 12KB, totaling about 95MB. Data includes mediawiki module storage, JS file URLs, Twitter user hashes, YouTube settings, Chartbeat values from Delicious, LinkedIn JS caches, etc. What's most interesting, beyond the long tail, are the biggest entries: scope key length(value) ----------------------- ---------- ------------- moc.ksidllif.2.:http:80 filldisk 2500000 moc.ksidllif.1.:http:80 filldisk 2500000 moc.mvhh.scod.:http:80 search-en 2068396 ten.php.:http:80 search-en 2008770 ten.php.1ti.:http:80 search-it 2004574 ten.php.2ti.:http:80 search-it 2004574 moc.xoferif.ecalptekram 0::model_c 1376262 moc.rettiwt.:https:443 __typeahea 969012 gro.aidemikiw.snommoc.: MediaWikiM 799795 moc.kcedteewt.bew.:http typeaheadU 771099 So filldisk.com is using 5MB alone. Then there are a bunch of sites using 1-2MB, and a long tail down from there. There are over five hundred entries larger than 50KB. marketplace.firefox.com is taking up 1.2MB. About 3MB was recovered by vacuuming from the 104MB. The only general solution to fat webappsstore.sqlite is eviction and expiration. It is unacceptable for thousands of sites to be forever storing hundreds of KB each, particularly on a phone. I'll leave this assigned to Honza, and move it to Core :: DOM. I'll file another bug to make marketplace.firefox.com less bloaty.
Maybe a setting for set the timeout or a page for manually delete the datas.
filter on [mass-p5]
Any update on this bug?
FYI: My webappsstore.sqlite is 57mb, my places.sqlite is 71mb. We need a utility reduce the size of these, delete old history or keywords.
Note: Firefox 46 will be doing a vacuum during the database update.
Bug 1058934 just got WONTFIXed, so this bug could expand to "throw away everything stored for marketplace.firefox.com". Re Comment 28: note that a sqlite VACUUM won't recover any space unless database content is first deleted or replaced by smaller values. (Indeed, temporarily it will double our space usage as sqlite rewrites the entire DB.) The bulk of the issue here is that we don't ever delete DOM storage, so if you visit a site that stores something, you will lose that space until you uninstall Firefox from your phone or wipe your desktop profile. Honza, Olli: do you have a plan for making progress here?
I'll leave storage handling (in its various forms) to mayhemer and janv. (Happy to review patches anyhow.)
> Note: Firefox 46 will be doing a vacuum during the database update. That doesn't fix the problem. VACUUM only releases space for *records that were deleted* in the database. We need a way to delete old records from within Firefox. I don't need stuff in these sqlite dbs from 3 years ago for a website I only went to once. I'd like to see each record in each of those databases, and the date it was last used or created. I'd like to be able to sort these records by that date, and select multiple records (it could be 100+ records) and delete the old ones. I've never gone to marketplace.firefox.com, I don't even know what it is. For now there is a 3rd party utility, and even a Firefox plugin (https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search) to deal with these sqlite databases, but that's too much trouble for the average user. Windows 3rd party util: Sqlite manager: https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search Wikipedia page: https://en.wikipedia.org/wiki/SQLite
****, I can't edit my comment. When doing a VACUUM the db doubles in size temporarily. Make sure the user has enough disk space to do this, when doing a VACUUM on each db. And what are my formatting options in Bugzilla? lol. I can't seem to find if it supports any markup of any kind. :) Also, the Sqlite Manager above is not free. SQDatabase is free: http://sqdatabase.com/
I think we need bug 527667 here.
To explain: there is currently no data about how old a record is. That bug is about to add it. We can then add a mechanism pruning old data automatically. OTOH, localStorage has no expiration mechanism by default. So hard to say how 'old' something has to be to remove it on users behalf.
The problem described in this bug is one of the reasons why we are moving localStorage to QuotaManager and temporary storage in bug 742822. It means that when temporary storage is full, quota manager starts evicting data for origins using LRU policy. Also, at some point, we can also add a new feature that would delete data for origins that were not used for X days/months/years.
I closed firefox and deleted webappsstore.sqlite from the profile folder. No side effect.
Thanks for the pointers, Honza and Jan. Flipping this to depend on those two bugs -- one to add timestamping, one to add quotas. Comment 34 and Comment 35 don't sound like they will directly solve this problem, but they're prerequisites, so un-duping this.
(In reply to Richard Newman [:rnewman] from comment #37) > Thanks for the pointers, Honza and Jan. > > Flipping this to depend on those two bugs -- one to add timestamping, one to > add quotas. > > Comment 34 and Comment 35 don't sound like they will directly solve this > problem, but they're prerequisites, so un-duping this. If bug 742822 won't directly solve this, why it would directly solve bug 1238771?
(In reply to Richard Newman [:rnewman] from comment #37) > Thanks for the pointers, Honza and Jan. > > Flipping this to depend on those two bugs -- one to add timestamping, one to > add quotas. > > Comment 34 and Comment 35 don't sound like they will directly solve this > problem, but they're prerequisites, so un-duping this. Yep, it's better. There is still more work to do when we have the timing info on each data row. I agree.
+1, as Firefox on a new install slows down within one month to such an extent that many sites are hardly navigatable.