Closed Bug 490831 Opened 15 years ago Closed 10 years ago

Ridiculously slow bookmarks export when generating a 16MB .JSON backup

Categories

(Firefox :: Bookmarks & History, defect)

Product:
Firefox
Component:
Bookmarks & History
Version:
3.5 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: geos651, Unassigned)

References

Details

(Keywords: perf, Whiteboard: [TSnappiness]) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4

It takes 20 (twenty) minutes to export or import a large Firefox bookmarks file from Bookmarks.JSON or bookmarks.html files.

Reproducible: Always

Steps to Reproduce:
1. Use a large Bookmarks file in Firefox
2. Organize Bookmarks (Ctrl+Shift+B)
3. Import and Backup
   Select Backup/Restore or
   Import HTML.../Export HTML...
Actual Results:  
It takes 20 minutes to Backup or Restore bookmarks! Also it takes approximately the same amount of time to Import/Export to HTML.
Moreover, a warning message pops up 'Script is busy. Do you want to continue?'

Expected Results:  
Less than ONE minute.

Google Chrome 2 Beta takes less than one minute to export or import the same Bookmarks.html. Thus it is 20 times faster than Firefox. Moreover, the user can INSTANTLY save or restore a single file, BOOKMARKS, and thus save/restore bookmarks in the native browser format.
Apparently, Firefox uses a JavaScipt script for Bookmarks Backup/Restore/Import/Export, which is painfully slow for these features.
It should use a custom sed script or an executable module written in C/C++. Then these tasks could be accomplished in seconds!
A simple way to preclude these problems would be to offer users an option to use internally bookmarks.html file format instead of PLACES.SQLITE, as was done in Firefox 2.x. Both Orca and K-Meleon Gecko-based browsers avoid these problems using Booknarks.html internallt.
do you have a guess of how many bookmarks are we talking about?
could you please send us by mail a json backup of your bookmarks?
Whiteboard: [TSnappiness]
Summary: Ridiculously slow boobmarks export and import both from .JSON and Bookmarks.html files. → Ridiculously slow bookmarks export and import both from .JSON and Bookmarks.html files.
bug 493731 is about restore only. let's make this one about backup/export. morphing.

Reporter, we really need to know the file-size of the html and json files, at the least. Since structure (folder depth, for example) can affect import/export times, it'd be great to get information about that as well.
Version: unspecified → 3.5 Branch
Summary: Ridiculously slow bookmarks export and import both from .JSON and Bookmarks.html files. → Ridiculously slow bookmarks export for both .JSON and Bookmarks.html files.
Cannot reproduce. 

With my Compaq 6910p notebook both Backup than Restore are immediate (JSON file backup is 283 KB). Same for Export HMTL (Bookmarks.html is 558 KB).

Firefox 3.5.2 on Windows XP.
Additional Details:

Computer Pentium 3, 566 MhZ, 327 MB RAM
Bookmark size: 10 MB HTML file, 
   a. without any favicons and
   b. without any bookmark Descripion field
both of which dramaticlly affect the size and are useless for most users.

It takes 20 minutes and requires huge amount of RAM to export bookmarks to either .JSON or HTMl format.
In Google Chrome the export to HTML is very quick, since it does not use SQLITE to store user bookmarks. Chrome's internal bookmarks format (file Bookmarks) is similar to that of JSON format, except that it does NOT include favicons.
use current firefox 3.6 nightly to check these measurements please.

And stop pointing to not using SQLite as a solution, nobody would ever do that, and if they do is because they don't expect large amounts of bookmarks. We have users with ten thousands bookmarks or more, and nothing could handle that better than SQLite. If there are problems they are NO in SQLite use but in actual code.

I'll wait with interest your measurements on Firefox 3.6 to evaluate if that got better.
Using Firefox 3.6b1, the bookmark export to .JSON is still very slow, and Firefox memory consumption gradually increases from less than 70MB to 250MB!!!.
Please note that the memory consumption oscillates (increases and than decreases) several times until the bookmark export is completed. 
Since .JSON is supposed to be Firefox 'native' bookmark format, 
   a. It should not take a very long time to export bookmarks to this format.
   b. The memory consumption should not be much higher than it is before the export.
   c. The Firefox CPU load should not increase significantly during the export.
   d. The user should be able to close Firefox during this very long process.

Thus, even in the Firefox 3.6b1 version (latest nightly build), this problem has not been fixed.
please define "very slow" comparing with the previously reported 20 minutes, that will be useful to understand how much the fixes we did helped.
PC1:  Pentium 3,  500Mhz, 256MB of RAM
PC2:  Pentium 4, 2.00GHz, 512MB of RAM

1. Data from Windows XP Task Manager during the backup to .JSON after new bookmarks were added on PC2:
RAM Used Before the Bookmark Backup (to .JSON)  36,096KB
Typical RAM Used During the Bookmark Backup    114,444KB 
Maximum RAM Used During the Bookmark Backup    171,924KB

Notes: 
a. Time required to backup bookmarks to .JSON increases if additional bookmarks have been added before the backup.
b. The maximum RAM used during the backup is much higher on a PC1 with 256MB RAM. It was 250MB of RAM!
c. The time required to backup bookmarks is much higher on a slow PC1 (500Mhz Pentium 3, with 256MB RAM). 
d. After the backup the Firefox RAM used is much higher than before the backup.

2. Attached please find the memory snapshots taken on PC2 during the bookmark backup when no new bookmarks were added before the backup.
The snapshots were taken every 2 seconds. The size of the Bookmarks.JSON file is shown after each snapshot.

PsList -m Usage Summary
a. All memory values are displayed in KB.
b. Abbreviation key:
   Pid         Process ID
   VM          Virtual Memory
   WS          Working Set
   Priv        Private Virtual Memory
   Priv Pk     Private Virtual Memory Peak
   Faults      Page Faults
   NonP        Non-Paged Pool
   Page        Paged Pool
-------------------------------------------------------------
Name                Pid      VM      WS    Priv Priv Pk   Faults   NonP Page
firefox            2984  168172   75116   62412   63040    20794     15   87
10/25/2009  05:58 AM         2,660,984 bookmarks.json
firefox            2984  234732  111760   99048  130112    39663     18   87
10/25/2009  05:58 AM         4,825,471 bookmarks.json
firefox            2984  233708  132332  119512  130112    45295     18   87
10/25/2009  05:58 AM         6,379,326 bookmarks.json
firefox            2984  245996  153668  140832  142468    50627     18   87
10/25/2009  05:58 AM         7,801,396 bookmarks.json
firefox            2984  244972  126776  114024  144420    55164     18   87
10/25/2009  05:58 AM         9,387,933 bookmarks.json
firefox            2984  244972  149256  136516  144420    60784     18   87
10/25/2009  05:58 AM        11,060,184 bookmarks.json
firefox            2984  254188  132868  120072  152532    65882     19   87
10/25/2009  05:58 AM        12,455,717 bookmarks.json
firefox            2984  253164  151784  139024  152532    70622     18   86
10/25/2009  05:59 AM        14,117,052 bookmarks.json
--------------------------------------------------------------------------
firefox            2984  253476   88784   75288  159160    77354     20   87
10/25/2009  05:59 AM        14,117,052 bookmarks.json
firefox            2984  253476   88784   75288  159160    77354     20   87
10/25/2009  05:59 AM        14,117,052 bookmarks.json
geos, can you still reproduce?
WFM with 5,000 bookmarks
Keywords: perf
Whiteboard: [TSnappiness] → [closeme 2011-11-25][TSnappiness]
Closing bug as WFM - if you are still experiencing this issue or have more information to provide feel free to post back here and we can re-open the bug. You can also get assistance by visiting the Firefox help site -> https://support.mozilla.com/en-US/kb/ask
Status: UNCONFIRMED → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
Whiteboard: [closeme 2011-11-25][TSnappiness] → [TSnappiness]
As I have mentioned initially the export is very slow ONLY for large JSON files (greater than 16 MB without any comments).
Currently, JavaScript is used for the export to either JSON or HTML. Both are still VERY slow in Firefox 10 Beta!
If the export function were an EXE procedure, it would be 100 times faster!

Note: The developers should consider adopting the Google Chrome implementation, where the bookmarks (internal format) file is ALWAYS up-to-date, without ANY export!
This would obviate the need to export to JSON file!
The import/export time has been improved in FF11, though a 16MB JSON is clearly still an issue, luckily an edge case, often caused by bad add-ons doing automatic tagging or things like those. As such, it should live in its own bug and not being reported in others, so please file one for your specific large json case.
ehr, that bug may be this one since you reported it, so I'm just reopening
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: WORKSFORME → ---
Summary: Ridiculously slow bookmarks export for both .JSON and Bookmarks.html files. → Ridiculously slow bookmarks export when generating a 16MB .JSON backup
Did you already sent me a copy of this 16MB json backup? would you mind doing that? In a previous case I was able to track 10 perf improvements that made a 2MB one much faster, so that would help more than guessing what's happening without having a test case.
1. You can select all the bookmarks from your 2 MB bookmark file and paste it 8 times. Then retest with a new 16 MB file.
2. Interpreted programs (e.g.. JS scripts) are much slower than the compiled ones.
Thus, the correct way to accelerate "slow" conversion is to recode in a compiled language, say C or C++.
3. "Works for me" is NOT the correct fix. You have to duplicate the test scenario.
   a. Bookmark file size.
   b. Test PC speed 600 mHz (Pentium 3).
1. You can select all the bookmarks from your 2 MB bookmark file and paste it 8 times. Then retest with a new 16 MB file.
2. Interpreted programs (e.g.. JS scripts) are much slower than the compiled ones.
Thus, the correct way to accelerate "slow" conversion is to recode in a compiled language, say C or C++.
3. "Works for me" is NOT the correct fix. You have to duplicate the test scenario.
   a. Bookmark file size.
   b. Test PC speed 600 mHz (Pentium 3).
The bug is not about making this possible on outdated hardware, miracles are hard to do, sorry. Even phones today are over 1GHz.
1. Google Chrome does NOT require any export to save its internal Bookmarks database, Bookmarks is always up-to-date !!! (Some designs are superior to others.)
2. Google Chrome exports its bookmarks to a HTML file extremely quickly, on any hardware, unlike Mozilla Firefox. Presumably, by not using JavaScript.
(In reply to geos651 from comment #21)
> 1. Google Chrome does NOT require any export to save its internal Bookmarks
> database, Bookmarks is always up-to-date !!! (Some designs are superior to
> others.)

You are doing the wrong comparison, our bookmarks are in places.sqlite, bookmarks.html or bookmarks.json are not our bookmarks store.

> 2. Google Chrome exports its bookmarks to a HTML file extremely quickly, on
> any hardware, unlike Mozilla Firefox. Presumably, by not using JavaScript.

Sure, they keep all bookmarks in memory, they don't support tags, they don't have a Library... I can ensure you that's not hard to do, unfortunately we have more stuff to migrate than Chrome, and we don't plan to keep everything in memory.
It is a design issue though. We could break up the work into chunks and show progress meter, and stream the data to disk, and probably many other things that would allow us to export 16mb json of bookmarks data without taking 20 minutes and without filling up RAM at all.
Facing the exact problem Still
Any timeline when will this be fixed?
(In reply to magnumarchonbasileus from comment #24)
> Facing the exact problem Still
> Any timeline when will this be fixed?

how big is your bookmarks json? How much time does it take? On which version of Firefox?
Does safe mode make things any better? http://support.mozilla.org/kb/safe+mode
in my bugreport i have the time and file size as the 'backup' process proceeded. 
https://bugzilla.mozilla.org/show_bug.cgi?id=945120

Landis
bug 824433 should have fixed this, if you can still reproduce please file a bug apart and mail me a json that takes huge times to be created. We now also have telemetry about the time we take so we'll monitor that.
Status: REOPENED → RESOLVED
Closed: 13 years ago → 10 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.