380134 - It takes long to import history with only a history.dat file in the profile

Status: RESOLVED WORKSFORME

(Firefox :: Bookmarks & History, defect, P4)

Description

[Ria Klaassen (not reading all bugmail)]

When I put a large history.dat file (about 4 MB) in an empty profile folder and start it up, it takes over a full minute (with 100 CPU usage) before Firefox starts up.
When I also put a compatibility.ini in it, it starts up quickly.
I doubt if anyone will ever do this, but it is a (recent) regression, so I'm reporting it.

Regression range for this is http://bonsai.mozilla.org/cvsquery.cgi?module=PhoenixTinderbox&date=explicit&mindate=1178037420&maxdate=1178040899
with two bugs in it, bug 378975 and bug 378038.

Comment 1

[Ria Klaassen (not reading all bugmail)]

Oh well, my computer is cheating me about the compatibility.ini file. It has nothing to do with it I think. There must be another file in the profile except places.sqlite that causes the bug not to appear for the second time.

The regression range is correct in any case.

Comment 2

[Boris Zbarsky [:bzbarsky]]

Honestly, I can't see what either of those bugs would have to do with history import.  But nothing else around there seems relevant either.

I'd be happy to look into this if you can describe in a little more detail the way to set this up; I'm not that familiar with the profile business with Firefox...

Also, can that history.dat compress enough to attach to this bug?  If not, or if you don't want to make it public, can you mail it to me?

Comment 3

[Boris Zbarsky [:bzbarsky]]

Well, I think I can reproduce this...  but I have no idea what's going on here.  :(  For some reason, finishing up the places async IO, and shutting down that thread, takes forever, with 100% CPU during that time.  But all the time seems to be spent in things like gtk event code and waiting on the X server... which is not likely to be an issue on Win XP.

I'll attach a profile; it covers part of startup, from when things really should have appeared but hadn't to when I got bored (about 50 seconds all told).

Benjamin, Seth, do you know what might be up here?

Comment 4

[Boris Zbarsky [:bzbarsky]]

Attachment: Zipped-up profile

Comment 5

[Boris Zbarsky [:bzbarsky]]

For what it's worth, I tried reverting the main change from bug 378975 (made InitialReflow sync again), and this problem did not go away.

Comment 6

[Boris Zbarsky [:bzbarsky]]

So... this is very interesting...  I'm seeing us hitting the history from the end of the load for resource://gre/res/hiddenWindow.html.  The reflow at the end of that load does the scrollbar guessing thing, which gets the global history service, which does nsMorkHistoryImporter::ImportHistory... which takes a while, presumably.  Not sure that would be enough to account for the 1-minute thing.  This is all _before_ the EM restart.

Not sure what happens after that, to be honest, but I bet that explains the regression range.  That said, the profile I posted was started _after_ the history import is all done, and we're still chewing tons of CPU there.

Now I bet we could change the scrollbar code to not look in the history for XUL documents or chrome documents or whatever... but I'm not sure why this is being a problem; doesn't Places import stuff on startup anyway?

Comment 7

[(not reading, please use [e-mail] instead)]

> Benjamin, Seth, do you know what might be up here?

sorry for the delay boris.  As for perf problems with nsMorkHistoryImporter::ImportHistory() on startup, I haven't profiled it yet.

as for after start up, so some thoughts:

1)  do you have lots of livebookmarks?  (see bug #329534 for example).  That might explain the chewing of CPU after startup.
2)  we have other bugs about perf problems with lots of history, for example see bug #223476.

Comment 8

[Boris Zbarsky [:bzbarsky]]

> 1)  do you have lots of livebookmarks?

Seth, this is a profile that just contains a history.dat...

And yes, I realize that there are existing performance problems with history.  But why did they not appear here until we started getting the history service during the EM startup?

Comment 9

[Ria Klaassen (not reading all bugmail)]

I can't reproduce the problem on Vista. It opens within 5 seconds. Although my XP laptop isn't slow at all, the other one is much faster. The bug was tested on both computers under exactly the same conditions. 

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

Blocking, see also bug 329736

Comment 11

[Boris Zbarsky [:bzbarsky]]

Maybe we should just back out bug 378975 or something... I really don't have the time to dig into this bug before 1.9, and given bug 380515 it seems there's more to things than just the Places history stuff?

Of course that would regress Tp/Tp2...  The other option is to not do this reflow that we're 

Of course what I really don't understand is why we're bothering to reflow the hidden window on non-Mac platforms.... and why we bother looking at the history for the hidden window on any platform.  We really shouldn't do that, probably.

Comment 12

[Mike Connor [:mconnor]]

_need_ this for b1

Comment 13

[(not reading, please use [e-mail] instead)]

a while back shawn and I were looking into bug #386739 about while migration from IE was taking so long, which is related.

some thoughts we had:

1) for history, bookmark, import, run the import as a transaction [see runInBatchMode()]

2) we also noticed a big perf gain when we did:

 rv = mDBConn->ExecuteSimpleSQL(
   NS_LITERAL_CSTRING("PRAGMA synchronous = OFF"));
 NS_ENSURE_SUCCESS(rv, rv);

moving off to m8

Comment 14

[(not reading, please use [e-mail] instead)]

> 1) for history, bookmark, import, run the import as a transaction [see
> runInBatchMode()]

I think this is the right thing to do first, before b1.

in migration.js, we have onMigratingMigrate()

If we could do that work as the callback for nsINavHistoryBatchCallback, we would wrap all the history and bookmark import in a transaction, and this should make initial import (and manual import) much faster.

note, initial profile import from IE is slow, due to importing history.  we end up calling AddPageWithDetails() for every item in the IE history.

Comment 15

[(not reading, please use [e-mail] instead)]

> in migration.js, we have onMigratingMigrate()

that will not work, as I think we have to wait until after we call DoStartup() in nsIEProfileMigrator::Migrate() before we can instantiate the history or bookmark service.

Comment 16

[(not reading, please use [e-mail] instead)]

looking at the mork importer code, importing history.dat (which this bug is about) already uses a transaction.

but first run migration / import from ie (and other browsers), does not, which I'll spin off to another bug.

Comment 17

[Dietrich Ayala (:dietrich)]

We've dropped the index on title, which will mitigate this a bit. Talked to seth the other day about turning off PRAGMA synchronous, or setting to NORMAL during import operations, which could provide up to 50% performance boost. The risk is that if the OS crashes or power is turned of while file operations are occurring, db corruption might occur.

Comment 18

[Ria Klaassen (not reading all bugmail)]

I could still reproduce it in 9 May 2007 build, but I don't see this problem anymore with the latest trunk build. Tried it a couple of times but it appears to be fixed. 

Comment 19

[Dietrich Ayala (:dietrich)]

Resolving WFM as the reporter is saying that's the case. Please re-open if you're experiencing extended delays on first run when migrating from Fx2 to Fx3.

Comment 20

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv