Closed Bug 673409 Opened 9 years ago Closed 9 years ago
"It's All Text" causes memory usage increases gradually to multiple GB; a huge places DB is involved
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0 Build ID: 20110622131247 Steps to reproduce: Right after start, Firefox uses between 300 and 500 MB of memory. After a while (a few hours, a couple of days at most), memory usage jumps to the 2-4 GB range. Firefox then starts being slower and slower, and pauses every now and then (as I'm typing this, it's a 2 second freeze every 5 seconds). I've seen memory usage ramping up to 6-8 GB (that was 6 GB RSS and 8 GB VIRT as given by top/ps) but I generally restart Firefox before that point because it's then unusable. This is with ~10 tabs open (around 20 to the max). I've noticed this issue with all the versions of Firefox I tried since FF3. My setup is a 64 bits Debian GNU/Linux. I tried with Debian packages, Ubuntu packages, and Mozilla binaries (the latter in 32 bits mode), and saw no difference. The following extensions are active: - AdBlock Plus - Delicious (~100 bookmarks there) - Flashblock - It's All Text - Nagios Checker - Speed Dial - Tree Style Tab The biggest files in my profile are: -rw-r--r-- 1 jpetazzo jpetazzo 1081344 Jul 22 07:46 formhistory.sqlite -rw-r--r-- 1 jpetazzo jpetazzo 1572864 Jul 22 07:56 cookies.sqlite -rw-r--r-- 1 jpetazzo jpetazzo 2907263 Jul 22 07:54 XUL.mfasl -rw-r--r-- 1 jpetazzo jpetazzo 5272576 Jul 22 07:57 webappsstore.sqlite -rw-r--r-- 1 jpetazzo jpetazzo 5669888 Mar 17 21:12 urlclassifier2.sqlite -rw-r--r-- 1 jpetazzo jpetazzo 9053516 Mar 17 21:12 history.dat -rw-r--r-- 1 jpetazzo jpetazzo 11774992 Jul 22 07:57 places.sqlite-wal -rw-r--r-- 1 jpetazzo jpetazzo 41943040 Jul 22 07:48 places.sqlite -rw-r--r-- 1 jpetazzo jpetazzo 57671680 Jul 22 07:53 urlclassifier3.sqlite I attached the output of about:memory. The only exotic settings I have is a 999 days history. (Yup, that's a lot, but that worked out pretty well with Netscape 4 on a 486 with 8 MB RAM, so I hope I can still use it on a quad core i7 with 8 GB :-))
It seems that the bugtracker incorrectly detects the type of the attachment (by default, when saving about:memory, firefox uses .xhtml); so I'm re-attaching with the hopefully correct mime type here.
Attachment #547691 - Attachment is obsolete: true
Attachment #547694 - Attachment mime type: text/plain → text/html
Do you have the same behavior with a Nightly build? You can find them here: http://nightly.mozilla.org/
I'm currently testing without extensions, and will switch to Nightly in a couple of days. Sometimes memory ramps up in just hours; but without extensions, I juggle with much less tabs, so I would rather wait a bit to have a firm answer.
Summary: Extremely high memory usage (4~8 GB RAM) and permanent freezes with less than 10 tabs) → Memory usage increases gradually to multiple GB; a huge places DB is involved
Jérôme, to check if your places.sqlite is still sane you can either run Components.utils.import("resource://gre/modules/PlacesDBUtils.jsm"); PlacesDBUtils.checkAndFixDatabase(); in the Error Console or, alternatively, run the "Places Maintenance" Extension by Marco: https://addons.mozilla.org/en-US/firefox/addon/places-maintenance/ (doing a Backup of places.sqlite beforehand).
I highly recommend to make a backup of places.sqlite before. That may be useful to determine why it was corrupted in the first place.
Alright, after a whole week-end running without any extension nor plugin, here goes the results: - much lower memory usage (started at 300 MB, now at 800 MB) - the "2 seconds freeze every 5 seconds" effect is gone In one hand, that's excellent news. On the other hand: - I notice some extended periods of CPU use (e.g. for the last 10 minutes, Firefox has been using 200% CPU, according to "top") - it seems to be scheduling a lot of I/O; well, I don't know how to make sure that Firefox is the process responsible for the I/O, but I killed every other process that showed up in top, and according to vmstat, I've been writing to disk at ~2 MB/s for the last hour - I guess I now have to re-enable each extension one by one, and run for a couple of days, until I find the one that blows things up :-) Unless instructed otherwise, I will: - re-enable extensions and plugins, and run the "Places Maintenance" extension to see if it changes anything; - then switch to Nightly and test with extensions; - then disable some extensions until I find the culprit(s). Thanks a lot for the insights in any case.
storage/places.sqlite/Cache_Used seems much better now (59MB). Have you done what comment 5 says? Yes, it may be some extension that is using the history much. Try to pinpoint it, thanks.
Ah, correction, it is 53MB (59 is the storage/sqlite/pagecache). The js/string-data value of 4,290,854,378 seems bogus (Bug 648490). The report seems to be from FF5, linux, marking that.
OS: Other → Linux
Hardware: All → x86_64
(Sorry about the MIME type of the attachment — I'm pretty sure I did manually enter text/html, but I somehow managed to get it borked. Anyway!) For the record, restarting Firefox took quite a long time (~5 mins). It usually takes some time to shutdown, but less than 1 minute with my old box (core 2 duo with 4 GB RAM and old-style HDD), and less than 10 seconds with the new box (quad core i7 with 8 GB and SSD). I just ran the "Places Maintenance" extension. Just in case it's of any use, here's the output: > Integrity check + The database is sane > Reindex + The database has been reindexed > Orphans expiration + Database cleaned up > Coherence check + The database is coherent > Vacuum Initial database size is 51200 KiB + The database has been vacuumed Final database size is 51200 KiB > Statistics Database size is 51200 KiB user_version is 11 page_size is 4096 cache_size is 121166 journal_mode is wal synchronous is 1 History can store a maximum of 248149 unique pages Table moz_bookmarks has 51 records Table moz_bookmarks_roots has 5 records Table moz_keywords has 5 records Table sqlite_sequence has 1 records Table moz_favicons has 3968 records Table moz_annos has 24 records Table moz_anno_attributes has 10 records Table moz_items_annos has 24 records Table moz_places has 56187 records Table moz_historyvisits has 282555 records Table moz_inputhistory has 1084 records Table sqlite_stat1 has 15 records Index sqlite_autoindex_moz_bookmarks_roots_1 Index sqlite_autoindex_moz_keywords_1 Index sqlite_autoindex_moz_favicons_1 Index sqlite_autoindex_moz_anno_attributes_1 Index sqlite_autoindex_moz_inputhistory_1 Index moz_bookmarks_itemindex Index moz_bookmarks_parentindex Index moz_bookmarks_itemlastmodifiedindex Index moz_places_faviconindex Index moz_places_hostindex Index moz_places_visitcount Index moz_places_frecencyindex Index moz_historyvisits_placedateindex Index moz_historyvisits_fromindex Index moz_historyvisits_dateindex Index moz_places_lastvisitdateindex Index moz_annos_placeattributeindex Index moz_items_annos_itemattributeindex Index moz_places_url_uniqueindex Index moz_bookmarks_guid_uniqueindex Index moz_places_guid_uniqueindex Trigger moz_bookmarks_beforedelete_v1_trigger The size of the file hasn't changed a single bit (but I cannot tell if that's meaningful at all). I will now run with all extensions again, and if memory usage gets back to "high" again, switch to Nightly. And then to some "bissect" on my extension set to find out which one is nibbling at my precious RAM :-) Thanks!
it's not natural for a Storage cache to be larger than the database itself (if not by a few amount, due to fragmentation), it shouldn't really ever happen. Which add-ons related to history or bookmarks did you have? Maybe they attached their own DBs to the Places connection.
I suggest we close this bug, because there's no clear action for Firefox developers to take at the moment. Jérôme, if you find that one of your add-ons is responsible for unreasonable memory usage, could you please open a new bug report about that. And thanks for the detailed data you've provided and your willingness to try various things, that's very helpful.
Status: UNCONFIRMED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Re-enabling all extensions caused the memory usage to skyrocket again (2.7G right now, with 700 MB just for sqlite pagecache). Unfortunately, most of the extensions I use don't seem to work with Nightly. So I will start the "extension bissect of doom" with my current version, and open a new bug as soon as I find the extension responsible for it! The only obvious extension using bookmarks would be Delicious, but I have very few bookmarks in it and I don't really use it. Others might be using the Places DB, but I don't know how to guess that for now. Thanks!
Status: RESOLVED → UNCONFIRMED
Resolution: FIXED → ---
Nicholas, probably rather INCOMPLETE instead of FIXED :) Jerome, thanks, go ahead with your working version of FF and extensions and find which one it does. Then you can test it with Nightly later when it becomes compatible. The Delicious one is most interesting as you say it has bookmark functions.
Status: UNCONFIRMED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → INCOMPLETE
For the record — the problem was caused by It's All Text. It's too bad that I missed Nicholas' note about it (and reference to bug 672619), it would have saved me the thorough investigation :-) Anyway: with "It's All Text", memory usage skyrockets. Without, I can run for days without exceeding 1 GB RAM usage. I don't think I understand all the technical implications of bug 672619; should I open a separate bug? Or report there? Or contact It's All Text author in the first place?
Jérôme, the key question is whether you still see the problem with It's All Text version 1.6.0, which is the version and the one that contains the fix for the problem in bug 672619? (Oh, it looks like there's a version 1.6.1 now.) (BTW, Christian Höltje (IAT's author) is already CC'd to this bug.)
Summary: Memory usage increases gradually to multiple GB; a huge places DB is involved → "It's All Text" causes memory usage increases gradually to multiple GB; a huge places DB is involved
You need to log in before you can comment on or make changes to this bug.