117863 - Improve addressbook loading performance.

Status: RESOLVED EXPIRED

(SeaMonkey :: MailNews: Address Book & Contacts, defect)

Description

[[e-mail] (gone - use [e-mail] instead)]

Build ID: 2001-01-02-03, Windows 98.

Summary: Improve addressbook loading performance.

We're still about 5x slower than our competitor when it comes to:

* Focusing
* Loading
* Display the contents of:

a 1,000 entry addressbook.

The latest numbers are posted at
http://www.mozilla.org/mailnews/performance/abookperf.html, so as not to clutter
this bug.

I will attach two files:

1,000 message .msf folder locally (which takes us ~ 1 second to load).
1,000 entry impbook.mab (which takes us around 6 seconds to load).

Comment 1

[[e-mail] (gone - use [e-mail] instead)]

Attachment: 1000 message local folder

Comment 2

[[e-mail] (gone - use [e-mail] instead)]

Attachment: Addressbook.mab (1,000 entry)

Comment 3

[Henrik Gemal]

dupe of 9012 rite?

Comment 4

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 9012 has been marked as a duplicate of this bug. ***

Comment 5

[(not reading, please use [e-mail] instead)]

Something things about loading and addressbook into a view (which is different 
than opening it up for use with just autocomplete).

1)  loading causes us to copy all the cards out of mork and into a giant array. 
(lots of allocs going on here.)
2)  then we build up two collation keys (i18n code) for each card (one for the 
primary sort, one for the secondary sort).  note, the secondary sort key is 
always email.
3)  then we sort the array, the compare function uses primary sort key, and if 
they are equal, then uses the secondary sort key to decide sort order.

My first optimization, as you can probably guess, is to not generate the 
secondary sort keys until we need them.  

The reason I made this "optimization" and generated the sort keys was:

a) if you sort by name, and then sort by phone number, the secondary sort key 
stay the same, and do not need to be regenerated.  of course, you still need to 
generate primary sort keys for all the phone numbers.

b) if you sort by email, you don't need to generate a primary sort key, since 
we can just use the secondary sort key.  and, for our secondary sort, we can 
just use the previous primary sort key, if we have one.

c) but primarily, it made the code simpler.

I'll fix my "optimization" (which helps subsequent sorts, but hurts loading), 
and see what it gets us.  Beyond that, I'll dive into quantify and see what's 
going on.

Large local addressbook performance comes into play with LDAP replication.  
when offline, we use local addressbook.  

But, we have an advantage here, because unlike large local addressbooks, large 
replicated addressbooks show up as blank (like LDAP addressbooks do) when you 
click on them.

So, I plan to make it so we won't even open the local mab file until you do a 
quick search (or advanced search) because until then, you won't need it.
(note, it might already be open due to autocomplete.)

Opening and loading the mab file, generating all the keys and doing all the 
sorting is much more work than just opening, doing a search, and doing the keys 
and sorting on the search results.  (given that the search results are much 
smaller than the total.)

Rajiv, this is why we need to treat offline LDAP addressbooks like online LDAP 
addressbooks.  (like 4.x did)

One day, I might even put a hidden per addressbook pref (defaulted to false) to 
treat normal local addressbooks like this, as a possible work around for 
someone who has HUGE a local addressbooks.  That trick should make them more 
usuable, at the cost of not requiring you to quick search or advance search 
to "see" what's in your addressbook.  (note, you could easily create an  
advanced search for email = "foo" or email != "foo" to see them all.  you get 
my point.)

Comments?

Comment 6

[[e-mail] (gone - use [e-mail] instead)]

All (especially Seth), please ignore comment #1.  Current metrics posted at :
http://www.mozilla.org/mailnews/performance/abookperf.html

Comment 7

[[e-mail] (gone - use [e-mail] instead)]

Err..comment 0?  Does that work?  Anyways, the 1st entry in bugzilla ;-)

Comment 8

[(not reading, please use [e-mail] instead)]

whoa, looking at the new metrics, we're decent (for 1000 card addressbooks) 
except for batch delete.

Some comments on the big differences:

New Card, Edit Card:  my gut tells me this is new window time.
 
Batch Delete (1,000 cards):  this is because we don't do any batch delete in AB.

Application Launch Time (Empty Address Book): was this comparing 
mozilla -addressbook to something similar for OE, or was this comparing 
launching AB from mail, compared to something similar for OE?

based on those numbers, I would move this bug off to 1.0, and make batch delete 
something for 0.9.9.

Comments?
 

Comment 9

[(not reading, please use [e-mail] instead)]

of course, for large local AB, we're slow.

see http://www.mozilla.org/mailnews/performance/LDIF-Addressbook-performance-
testing.html

donner, can you get some OE numbers for those addressbooks, to see where we are?

also, does OE block the UI like we do?  (I doubt it, they are smart up there in 
Redmond.)

Given how slow we get with 10,000+, this bug should probably stay open for 0.9.9

Comment 10

[[e-mail] (gone - use [e-mail] instead)]

<snip>Application Launch Time (Empty Address Book): was this comparing 
mozilla -addressbook to something similar for OE, or was this comparing 
launching AB from mail, compared to something similar for OE?
</snip>

This was launching addressbook from the Tasks menu after launching Navigator. 
OE 6's addressbook is launched from the main client window, just like ours.

Comment 11

[[e-mail] (gone - use [e-mail] instead)]

I've uploaded the latest metrics to the url above
(http://www.mozilla.org/mailnews/performance/abookperf.html), but 4.7x shows us
at .57 seconds to launch from the Communicator menu (similar to our
implementation of the Tasks menu).  Outlook Express takes roughly 0.60 seconds.

Comment 12

[[e-mail] (gone - use [e-mail] instead)]

Sigh, that comment was meant for bug 9012 (too many AB bugs to comment in!)

Comment 13

[scottputterman]

Marking as P2 because if I'm reading the #'s right, we're better than our
competition up to 5000 cards.  Hopefully it'll be possible to get the 10,000
card numbers for other products so we can see where we stand.

Comment 14

[[e-mail] (gone - use [e-mail] instead)]

http://www.mozilla.org/mailnews/performance/LDIF-Addressbook-performance-testing.html
contains the latest testing for huge amounts of cards (in LDIF format, but
should give us an estimate of similar performance for .mab, etc).

Comment 15

[Michael Buckland]

Discussed in 2/27/02 Mail & News bug meeting.  Decisions was to minus and Future
this bug.  Please renominate if you get data showing we are slower than the
competition.

Comment 16

[[e-mail] (gone - use [e-mail] instead)]

Reducing severity to normal, since I've done the metrics and this isn't that far
off.

Comment 17

[Robert Kaiser]

MASS-CHANGE:
This bug report is registered in the SeaMonkey product, but has been without a comment since the inception of the SeaMonkey project. This means that it was logged against the old Mozilla suite and we cannot determine that it's still valid for the current SeaMonkey suite. Because of this, we are setting it to an UNCONFIRMED state.

If you can confirm that this report still applies to current SeaMonkey 2.x nightly builds, please set it back to the NEW state along with a comment on how you reproduced it on what Build ID, or if it's an enhancement request, why it's still worth implementing and in what way.
If you can confirm that the report doesn't apply to current SeaMonkey 2.x nightly builds, please set it to the appropriate RESOLVED state (WORKSFORME, INVALID, WONTFIX, or similar).
If no action happens within the next few months, we move this bug report to an EXPIRED state.

Query tag for this change: mass-UNCONFIRM-20090614

Comment 18

[Robert Kaiser]

MASS-CHANGE:
This bug report is registered in the SeaMonkey product, but still has no comment since the inception of the SeaMonkey project 5 years ago.

Because of this, we're resolving the bug as EXPIRED.

If you still can reproduce the bug on SeaMonkey 2 or otherwise think it's still valid, please REOPEN it and if it is a platform or toolkit issue, move it to the according component.

Query tag for this change: EXPIRED-20100420