374532 - [meta] Improve performance (as measured by memory use, transactional speed and Ts) of bookmark and history storage and retrieval operations

Status: RESOLVED INCOMPLETE

(Firefox :: Bookmarks & History, defect, P2)

Description

[Dietrich Ayala (:dietrich)]

This is a tracking bug for item PLCS-005a in the Firefox 3 PRD.

Comment 1

[Mike Beltzner [:beltzner, not reading bugmail]]

We need some good metrics to act as measures of success, here. Any ideas?

Comment 2

[Dietrich Ayala (:dietrich)]

Ben Hearsum has created builds for no-Places, Places-history and Places-history+bookmarks. Alice and Rob said they'd be able run these builds under the new performance testing framework, the results of which should be able to give us data for memory use, ts, tp.

We may need to implement a separate test to target transactional speed of specific operations.

Comment 3

[Dietrich Ayala (:dietrich)]

the builds:

http://hera.senecac.on.ca:9080/GP-A5-no-places.zip
http://hera.senecac.on.ca:9080/GP-A5-places.zip
http://hera.senecac.on.ca:9080/GP-A5-places-bookmarks.zip

Comment 4

[Rob Campbell [:rc] (:robcee)]

it'd be nice to have a debugging mode in storage (and elsewhere) to time things like this. At worst we could time the transactions in javascript with some wrappers around entries/exits, though really we're going to be bound by sqlite ultimately.

We talked briefly with Bob Clary about using the history data from his spider runs as a potential datasource for a large dataset.

Comment 5

[Dietrich Ayala (:dietrich)]

retargeting bugs that don't meet the alpha release-blocker criteria at http://wiki.mozilla.org/Firefox3/Schedule.

Comment 6

[Dietrich Ayala (:dietrich)]

from Alice:

Using those builds I ran the tests last night.  Each build was tested
four times with the results posted to the graph server.

http://graphs.mozilla.org/graph.html
http://graphs.mozilla.org/dgraph.html

The machines are called qm-pxp03-GP-A5-*.

The comparison of load times is here:
http://graphs.mozilla.org/graph.html#spst=range&spstart=1183068341&spend=1183102872&bpst=cursor&bpstart=1183068341&bpend=1183102872&m1tid=2082&m1bl=0&m1avg=0&m2tid=2092&m2bl=0&m2avg=0&m3tid=2102&m3bl=0&m3avg=0

The comparison of ts times is here:
http://graphs.mozilla.org/graph.html#spst=range&spstart=1183068341&spend=1183102872&bpst=cursor&bpstart=1183068341&bpend=1183102872&m1tid=2081&m1bl=0&m1avg=0&m2tid=2091&m2bl=0&m2avg=0&m3tid=2101&m3bl=0&m3avg=0

The numbers aren't all that consistent, so it might be worth doing more
runs to get a larger test set.  On the plus side, I'm not seeing much
difference at all in memory usage for the three builds.

Comment 7

[Dietrich Ayala (:dietrich)]

> The numbers aren't all that consistent, so it might be worth doing more
> runs to get a larger test set. 

They are mostly within a percentage point apart, which seems normal relative to the fluctuations on some of our tinderboxen.

I put all the Ts and pageload runs from 7/3 - 7/5 in a spreadsheet, and got the mean for both Places-Bookmarks and non-Places, and then calculated the percentage difference:
 
http://spreadsheets4.google.com/ccc?key=pwYmMOzoFPJU3MSzrHmwoFg&hl=en

These numbers show a 2+% regression in both Ts and Tp. (Though, please take a look and see if there are any mathematical errors.)

1. These numbers are trunk vs trunk, not Fx2 vs Fx3. This was the only way to reasonably test Places specifically.

2. This was done with an empty profile, the same as our current Ts and Tp tests. We'd like to do a run of Fx2 vs Fx3 with large history and bookmarks sets at some point, but again that tests the whole app, not Places in particular.

Comment 8

[Mike Connor [:mconnor]]

Discussed this with schrep, and I'm going to call this complete (measured against the PRD requirement).  While there is a small (~2%) hit with an empty profile, in the real world with real history and bookmark sets its my understanding that we're faster across the board, especially where the old history impl hits the wall on performance.

I'm not saying there isn't room to tune and tweak performance, but the point of the requirement was to improve history performance for end users, and I believe we've hit that goal already (as evidenced by our ability to bump the history expiration default 20x).

Comment 9

[(not reading, please use [e-mail] instead)]

> I'm going to call this complete

I feel uncomfortable calling this complete.  

2% hit for an empty profile is good, I agree.  I think both dietrich and I were surprised it was so small.  (kudos to the original places team, who I think deserve the credit there.)

here are my concerns:

1)  "with real history and bookmark sets its my understanding that we're faster across the board".  

There are definitely specific cases (example, the history menu) where we're faster than fx 2.  

but we haven't done full apples-to-apples with real world history and bookmarks (not sure what's good here) memory measurements and performance wall clock tests on a target machine.

2)  "as evidenced by our ability to bump the history expiration default 20x"

we've bumped to 180 days, but I think that needs to come way down.  see bug #332748 for a discussion of this issue.

a user could hit some serious performance issues with a large places.sqlite as soon as they are bigger than the cache size we specify (see bug #394079 for a discussion of this issue.) 

Comment 10

[Mike Connor [:mconnor]]

Needs more verification, not clear whether we need more work here, gut feeling is still no.

Comment 11

[Dietrich Ayala (:dietrich)]

Pushing to M10. There's still no clear "apples to apples" comparison between Fx3 and Fx2.

Comment 12

[Dietrich Ayala (:dietrich)]

We should be able to get some numbers on transactional speed on branch, and then on trunk with a large places db, for some basic activities:

- history menu query
- add a folder
- add a bookmark (with various properties, eg keyword, description)
- remove a bookmark

etc.

Comment 13

[Ondrej Brablc]

I'm not sure if it makes sense to prepare a big test case, which would compare performance between Fx2 and Fx3 at this moment. This would take definitely some time and would become obsolete after Fx3 release. At the moment we have quite a lot performance related bugs. I suggest first fixing them, and only afterwards if there is still time and we still have complains that places are slow to build a comparison test:

Marked as blocking:
  Bug 385245 history sidebar very slow
  Bug 364272 Ts performance regression on 12/15/06 from enabling places ...
  Bug 399566 Bookmarks and History menus open slower than other menus every time
  Bug 405765 Opening folder in the right pane of the Places Organizer is slow

Not marked as blocking:
  Bug 416988 Backup of bookmarks to HTML on shutdown does too many queries
  Bug 417262 History sidebar slow for "By Most Visited" and "By Last Visited"
  Bug 417729 Livemark refreshing inefficient
  Bug 418166 Long sqlite database shutdown times + 100 % CPU on browser exit 
  Bug 409723 when clearing all history, can we avoid calling FindVisits()?
  Bug 412758 Erase functions in nsNavHistoryExpire iterate all ...
  Bug 390614 Split up the "Older than 6 days" history folder
  Bug 417042 Moving mouse over bookmark folders in Places Library ...

Comment 14

[Mike Connor [:mconnor]]

I think we've done a lot of good work here, everything else will need to wait for the next release.

Comment 15

[Shawn Wilsher :sdwilsh]

I'm going through and marking old performance regression bugs as INCOMPLETE that are likely too old to be valid or get any traction on them.

Please re-open if you have more information or can demonstrate the regression still exists.

Comment 16

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv