471084 - Importing a large bookmarks JSON backup makes Firefox crash (out of memory)

Status: VERIFIED FIXED

(Firefox :: Bookmarks & History, defect, P1)

Description

[lacavernedelex]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20
Build Identifier: Firefox 3.0.5 downloaded at http://www.mozilla-europe.org/fr/firefox/ on Decmber 24th of 2008

Hi.

I updated (almost mechanically) my Firefox to the last version few days ago (from 3.0.4 to 3.0.5). I met some "slow moments" (CPU used at 100% for several seconds...). Because I had several versions of Firefox installed since my last installation of Windows XP, I decided to uninstall Firefox, clean all my Firefox folders and reinstall it in a clean place. I thought it may solve these issues.

Before to do this, I exported my bookmarks as a JSON file. I also saved the backups it had made in my profile folder. I uninstalled Firefox, rebboted my computer and installed the last version of Firefox (3.0.5).

Then, I tried to reimport my backup bookmarks in Firefox.
Some details about this backup file. It's a JSON file and it's 3.3 Mb (I got a lot of bookmarks). So, the processing starts, almost freezes it all, until it crashes (after about 10 minutes of processing). After the crash, when I try to restart Firefox, it uses the CPU at 100% and doesn't allow me to use it (the UI does not appear).

I reinstalled it and gave another try.
I monitored what happened with the "task manager of Windows". When it loads the backup file, it tries to load it in memory and uses all the memory (I have 2 Gb of RAM, it was fully used, and it started to use the swap space). The reason why it crashes is pretty obviousd, there is not enough memory. But come on, 2 Gb to process a file of 3Mb... 

I guess the reason why I can restart my Firefox after this crash is that the embedded DB is corrupted. So, I don't know if this is a bug, or an enhancement, but currently, I can't restore my bookmarks and this is really annoying.

And in terms of performances, use so much memory to process a small file (even compressed), that's surprising.

Reproducible: Always

Steps to Reproduce:
1. Use a bookmarks JSON backup file with a size of more than 3.3 Mb, with different addresses in it (no duplicate).
2. Try to restore your bookmarks from this file with the version 3.0.5 of Firefox.
3. Monitor the memory space used for the operation.
Actual Results:  
Full RAM memory used, swap space started to be used.
Firefox crashes and can't be run again.

Expected Results:  
Use a correct amount of memory, show a progress dialog while importing / processing the bookmarks, import correctly all of the bookmarks.

The Firefox I used to export my bookmarks had the "no duplicate bookmark" (that might not be the exact name) plug-in installed. This extension is not installed on the newly installed Firefox.

Comment 1

[Nochum Sossonko [:Natch]]

Can you try this on a new profile, to rule out any external deficiencies: http://support.mozilla.com/en-US/kb/Basic+Troubleshooting#Make_a_new_profile .

Also can you follow instructions at the following page to get a stack trace: https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report

Comment 2

[lacavernedelex]

I created a new profile and imported my bookmarks again.
I got a different behavior.

Firefox is still using the whole memory, but instead of crashing, it opens a popup saying it could not process the JSON file completely. And I got "some" bookmarks only that were imported (thousands, but not all apparently).

So it does not crashes anymore but the performance issue is still here about memory use. Is there any way to get access to some log in order to have details about where the problem occured at the end?

Comment 3

[timeless]

If your bookmarks aren't private, could you possibly attach the file you're importing?

Comment 4

[Shawn Wilsher :sdwilsh]

(In reply to comment #3)
> If your bookmarks aren't private, could you possibly attach the file you're
> importing?
Alternatively, you can send it to one of the places developers (I am one) via e-mail.  We won't disclose this information publicly, but will use it to try and debug your issue.

Comment 5

[Marco Bonardo [:mak]]

3.05 contains fixes for some JSON bookmarks backup issue, but if you have exported bookmarks using 3.0.0-3.0.4 there is some chance you hit one of those bugs fixed in next version, in such a case your json backup could be incomplete :\
do you still have your old profiles folders?

Comment 6

[lacavernedelex]

(In reply to comment #5)
> 3.05 contains fixes for some JSON bookmarks backup issue, but if you have
> exported bookmarks using 3.0.0-3.0.4 there is some chance you hit one of those
> bugs fixed in next version, in such a case your json backup could be incomplete
> :\
> do you still have your old profiles folders?

No. :|
What if I re-install the same version I had?  Would the restoration work? Then with a reexport in html for the next versions...

Comment 7

[lacavernedelex]

But in fact, I don't think it is a problem of corrupted backup.
IMO, the reason why several bookmars are missing is that the entire backup could not be processed. Because I have 2 Gb of memory and when all of this is used by Firefox, it opens a dialog saying it could not import all the bookmarks.

The real question I think, is why it needs so much memory.

Comment 8

[Marco Bonardo [:mak]]

well, you are right, i was reading this as a bug due to a corrupt json backup, but sounds like the backup is not corrupt, it is only quite big, so you should be able to restore from it. Hwv while trying importing it my system hanged ( i have 2GB memory too), so, definately, a perf issue needs to be addressed, and i'm confirming for investigating that.

Comment 9

[Dietrich Ayala (:dietrich)]

blocking 3.1 for this. 3.3mb of bookmarks backup should not have such a deleterious effect.

Comment 10

[Marco Bonardo [:mak]]

the backup has:
- 8295 entries in bookmarks menu
- 94 entries in bookmarks toolbar
- no tags
- 1 entry in unsorted

Comment 11

[Marco Bonardo [:mak]]

correction: bookmarks menu contains an import of an old toolbar with 1100 items, so it contains almost 10000 entries.

Hwv i originally tested this on 3.0.x, sounds like trunk is doing really better, i can restore in less than a minute, with one slow script warning though.

Comment 12

[Dietrich Ayala (:dietrich)]

(In reply to comment #11)
> Hwv i originally tested this on 3.0.x, sounds like trunk is doing really
> better, i can restore in less than a minute, with one slow script warning
> though.

better, but still not so good. what's the bottleneck?

Comment 13

[Shawn Wilsher :sdwilsh]

If need be, I can get a shark profile

Comment 14

[Marco Bonardo [:mak]]

Attachment: patch

So, this is really better on trunk, i think fixes to removeFolderChildre, to observers, tag service changes, and other perf fixes, have helped a lot.

this patch clean up the code a bit, trying to gain some minor perf gain, the biggest one is changing the read buffer on the json file, for reading the above json i've moved from about 13s -> 3s. The restore code is quite straight-forward, so we can't reduce it more than this, apart having a faster backend, i've filed bug 473664 to investigate setting additional params on insertBookmark.

Globally in a debug build in a vm the restore took less than 40s.

Comment 15

[Marco Bonardo [:mak]]

actual unit tests are already covering restore and queries fixup, so i did not write new ones.

Comment 16

[Dietrich Ayala (:dietrich)]

Comment on attachment 357064 [details] [diff] [review]
patch

r=me

Comment 17

[Marco Bonardo [:mak]]

http://hg.mozilla.org/mozilla-central/rev/f8642a26ef86

Comment 18

[Marco Bonardo [:mak]]

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/b59c44ccc6e3

Comment 19

[Tracy Walker [:tracy]]

reporter can you verify this is fixed in latest nightly builds of 3.1beta3pre

Comment 20

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv

Comment 21

[lacavernedelex]

> reporter can you verify this is fixed in latest nightly builds of 3.1beta3pre

Almost one year later. Sorry for the delay. I didn't try to import my backup since I posted this bug.

I tried a restoration on my new install, with an even bigger backup (I think) and it perfectly worked. I monitored the used memory and it remained pretty stable. I'm running version 3.5.5.

Thank you, and once again, sorry for the delay.

Comment 22

[Dietrich Ayala (:dietrich)]

verified by reporter, thanks!