Importing a large bookmarks JSON backup makes Firefox crash (out of memory)

VERIFIED FIXED in Firefox 3.5

Status

()

Firefox
Bookmarks & History
P1
major
VERIFIED FIXED
9 years ago
8 years ago

People

(Reporter: lacavernedelex, Assigned: mak)

Tracking

({crash, fixed1.9.1, hang})

3.0 Branch
Firefox 3.5
x86
Windows XP
crash, fixed1.9.1, hang
Points:
---
Bug Flags:
blocking-firefox3.5 +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mostly fixed by other perf fixes in 3.1])

Attachments

(1 attachment)

(Reporter)

Description

9 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20
Build Identifier: Firefox 3.0.5 downloaded at http://www.mozilla-europe.org/fr/firefox/ on Decmber 24th of 2008

Hi.

I updated (almost mechanically) my Firefox to the last version few days ago (from 3.0.4 to 3.0.5). I met some "slow moments" (CPU used at 100% for several seconds...). Because I had several versions of Firefox installed since my last installation of Windows XP, I decided to uninstall Firefox, clean all my Firefox folders and reinstall it in a clean place. I thought it may solve these issues.

Before to do this, I exported my bookmarks as a JSON file. I also saved the backups it had made in my profile folder. I uninstalled Firefox, rebboted my computer and installed the last version of Firefox (3.0.5).

Then, I tried to reimport my backup bookmarks in Firefox.
Some details about this backup file. It's a JSON file and it's 3.3 Mb (I got a lot of bookmarks). So, the processing starts, almost freezes it all, until it crashes (after about 10 minutes of processing). After the crash, when I try to restart Firefox, it uses the CPU at 100% and doesn't allow me to use it (the UI does not appear).

I reinstalled it and gave another try.
I monitored what happened with the "task manager of Windows". When it loads the backup file, it tries to load it in memory and uses all the memory (I have 2 Gb of RAM, it was fully used, and it started to use the swap space). The reason why it crashes is pretty obviousd, there is not enough memory. But come on, 2 Gb to process a file of 3Mb... 

I guess the reason why I can restart my Firefox after this crash is that the embedded DB is corrupted. So, I don't know if this is a bug, or an enhancement, but currently, I can't restore my bookmarks and this is really annoying.

And in terms of performances, use so much memory to process a small file (even compressed), that's surprising.

Reproducible: Always

Steps to Reproduce:
1. Use a bookmarks JSON backup file with a size of more than 3.3 Mb, with different addresses in it (no duplicate).
2. Try to restore your bookmarks from this file with the version 3.0.5 of Firefox.
3. Monitor the memory space used for the operation.
Actual Results:  
Full RAM memory used, swap space started to be used.
Firefox crashes and can't be run again.

Expected Results:  
Use a correct amount of memory, show a progress dialog while importing / processing the bookmarks, import correctly all of the bookmarks.

The Firefox I used to export my bookmarks had the "no duplicate bookmark" (that might not be the exact name) plug-in installed. This extension is not installed on the newly installed Firefox.
Can you try this on a new profile, to rule out any external deficiencies: http://support.mozilla.com/en-US/kb/Basic+Troubleshooting#Make_a_new_profile .

Also can you follow instructions at the following page to get a stack trace: https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report
Component: Bookmarks & History → Places
Keywords: crash, hang
QA Contact: bookmarks → places
Version: unspecified → 3.0 Branch
(Reporter)

Comment 2

9 years ago
I created a new profile and imported my bookmarks again.
I got a different behavior.

Firefox is still using the whole memory, but instead of crashing, it opens a popup saying it could not process the JSON file completely. And I got "some" bookmarks only that were imported (thousands, but not all apparently).

So it does not crashes anymore but the performance issue is still here about memory use. Is there any way to get access to some log in order to have details about where the problem occured at the end?

Comment 3

9 years ago
If your bookmarks aren't private, could you possibly attach the file you're importing?
(In reply to comment #3)
> If your bookmarks aren't private, could you possibly attach the file you're
> importing?
Alternatively, you can send it to one of the places developers (I am one) via e-mail.  We won't disclose this information publicly, but will use it to try and debug your issue.
(Assignee)

Comment 5

9 years ago
3.05 contains fixes for some JSON bookmarks backup issue, but if you have exported bookmarks using 3.0.0-3.0.4 there is some chance you hit one of those bugs fixed in next version, in such a case your json backup could be incomplete :\
do you still have your old profiles folders?
(Reporter)

Comment 6

9 years ago
(In reply to comment #5)
> 3.05 contains fixes for some JSON bookmarks backup issue, but if you have
> exported bookmarks using 3.0.0-3.0.4 there is some chance you hit one of those
> bugs fixed in next version, in such a case your json backup could be incomplete
> :\
> do you still have your old profiles folders?

No. :|
What if I re-install the same version I had?  Would the restoration work? Then with a reexport in html for the next versions...
(Reporter)

Comment 7

9 years ago
But in fact, I don't think it is a problem of corrupted backup.
IMO, the reason why several bookmars are missing is that the entire backup could not be processed. Because I have 2 Gb of memory and when all of this is used by Firefox, it opens a dialog saying it could not import all the bookmarks.

The real question I think, is why it needs so much memory.
(Assignee)

Comment 8

9 years ago
well, you are right, i was reading this as a bug due to a corrupt json backup, but sounds like the backup is not corrupt, it is only quite big, so you should be able to restore from it. Hwv while trying importing it my system hanged ( i have 2GB memory too), so, definately, a perf issue needs to be addressed, and i'm confirming for investigating that.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Importing bookmarks JSON backup makes Firefox crash (out of memory) → Importing a large bookmarks JSON backup makes Firefox crash (out of memory)
(Assignee)

Updated

9 years ago
Target Milestone: --- → Firefox 3.1
blocking 3.1 for this. 3.3mb of bookmarks backup should not have such a deleterious effect.
Flags: blocking-firefox3.1+
Priority: -- → P1
(Assignee)

Updated

9 years ago
Assignee: nobody → mak77
Status: NEW → ASSIGNED
Whiteboard: [investigating on causes]
(Assignee)

Comment 10

9 years ago
the backup has:
- 8295 entries in bookmarks menu
- 94 entries in bookmarks toolbar
- no tags
- 1 entry in unsorted
(Assignee)

Comment 11

9 years ago
correction: bookmarks menu contains an import of an old toolbar with 1100 items, so it contains almost 10000 entries.

Hwv i originally tested this on 3.0.x, sounds like trunk is doing really better, i can restore in less than a minute, with one slow script warning though.
(In reply to comment #11)
> Hwv i originally tested this on 3.0.x, sounds like trunk is doing really
> better, i can restore in less than a minute, with one slow script warning
> though.

better, but still not so good. what's the bottleneck?
If need be, I can get a shark profile
(Assignee)

Comment 14

9 years ago
Created attachment 357064 [details] [diff] [review]
patch

So, this is really better on trunk, i think fixes to removeFolderChildre, to observers, tag service changes, and other perf fixes, have helped a lot.

this patch clean up the code a bit, trying to gain some minor perf gain, the biggest one is changing the read buffer on the json file, for reading the above json i've moved from about 13s -> 3s. The restore code is quite straight-forward, so we can't reduce it more than this, apart having a faster backend, i've filed bug 473664 to investigate setting additional params on insertBookmark.

Globally in a debug build in a vm the restore took less than 40s.
(Assignee)

Updated

9 years ago
Attachment #357064 - Flags: review?(dietrich)
(Assignee)

Comment 15

9 years ago
actual unit tests are already covering restore and queries fixup, so i did not write new ones.
Attachment #357064 - Flags: review?(dietrich) → review+
(Assignee)

Comment 17

9 years ago
http://hg.mozilla.org/mozilla-central/rev/f8642a26ef86
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Whiteboard: [investigating on causes] → [mostly fixed by other perf fixes in 3.1]
(Assignee)

Updated

9 years ago
Whiteboard: [mostly fixed by other perf fixes in 3.1] → [mostly fixed by other perf fixes in 3.1][baking on trunk]
(Assignee)

Comment 18

9 years ago
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/b59c44ccc6e3
Keywords: fixed1.9.1
Whiteboard: [mostly fixed by other perf fixes in 3.1][baking on trunk] → [mostly fixed by other perf fixes in 3.1]
reporter can you verify this is fixed in latest nightly builds of 3.1beta3pre
Target Milestone: Firefox 3.1 → Firefox 3.5
Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv
Component: Places → Bookmarks & History
QA Contact: places → bookmarks
(Reporter)

Comment 21

8 years ago
> reporter can you verify this is fixed in latest nightly builds of 3.1beta3pre

Almost one year later. Sorry for the delay. I didn't try to import my backup since I posted this bug.

I tried a restoration on my new install, with an even bigger backup (I think) and it perfectly worked. I monitored the used memory and it remained pretty stable. I'm running version 3.5.5.

Thank you, and once again, sorry for the delay.
verified by reporter, thanks!
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.