Egregious memory usage and load times for large address book

RESOLVED WORKSFORME

Status

Thunderbird
Address Book
--
major
RESOLVED WORKSFORME
8 years ago
6 years ago

People

(Reporter: David Taber, Unassigned)

Tracking

({memory-leak, perf})

Trunk
x86
Windows XP
memory-leak, perf

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [str: comment 11])

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)
Build Identifier: Eudora 8Beta9

I admit it:  I'm a info pack rat.  7500 contacts, nearly 100 total mailboxes.

Even so, there's something silly going on w/Eudora when it's been running for a while.  RAM usage of >200 MB in *both* windows' "Mem Usage" and "VM Size".  And when I try to open the address book, it can take over 4 minutes of CPU time.  It seems to take longer every time I open the address book.  There also seems to be a memory leak w/ addresses, because the memory usage goes up by at least 10 MB every time I open the address book.

Reproducible: Always

Steps to Reproduce:
1. Click the "address book" icon or switch between address books
2.
3.
Actual Results:  
Takes minimum 15 seconds to load a 7500 contact address book, but the fully populated address book seems to take longer with each open (can grow to minutes).

Expected Results:  
Took the same address book in Outlook...it takes almost no time to open, and has a total memory footprint of less than 30% of T'Bird/Eudora.

Plugins:  Penelope, Lightning, Zindus, and a few tiny utilities

Tried the same address book operations in Safe Mode, it's no where near as bad (I can't seem to cause it to take more than 90 seconds to load the address book).  But the basic symptom is the same -- maybe this is a T'Bird bug?

Comment 1

8 years ago
I'm not aware of any open bugs in this area. 

Exactly what was the different in number of seconds between safe mode and non-safe?  Is address autocomplete also affected? What is the size of largest address book file in your profile directory?
  http://support.mozillamessaging.com/en-US/kb/Profiles
Severity: normal → major
Component: General → Address Book
Keywords: perf
QA Contact: general → address-book
(Reporter)

Comment 2

8 years ago
The amount of time to load the address book seems to grow monotonically in both safe mode and Eudora-with-all-the-fixin's, and the delta between their address book load times is not all that consistent.  I'll guess that Eudora is 20-40 seconds longer than Safe mode, but I haven't been running Safe mode for hours and hours to make a solid comparison.

As I said earlier, this may well be a T'Bird bug that is merely exacerbated by Penelope and Lightning.

My largest abook.mab is just under 4 MB, and has been over 3 MB for the last few versions of Eudora beta.  This performance problem has been with us for a while, but seems to have gotten a little worse.  That could be because my address book has grown by maybe 20% in the last year or so.

Address autocomplete performance used to be horrendous, but got way way better with Eudora Beta 8.  Most of the time, the autocomplete is a couple of seconds...but every once in a while it can be 30 s or more of 100% CPU utilization.

System is a 1.6 GHz laptop with 2 GB of RAM and 4 GB swap space.  Resident set for all apps in RAM appears to be less than 1 GB total, so I'm not losing much to page faults.  While the address book is loading, Eudora's resident set size grows by as much as 25 MB so much of the delay is likely coming from thrashing memory.

Comment 3

8 years ago
(In reply to comment #2)
> 
> My largest abook.mab is just under 4 MB, and has been over 3 MB for the last
> few versions of Eudora beta.  

how many contacts for that 4MB address book?
do you delete a lot of AB entries in a year?

> This performance problem has been with us for a while

please define a while. since version 3.0? version 2.0.something?


> Address autocomplete performance used to be horrendous, but got way way better
> with Eudora Beta 8.  Most of the time, the autocomplete is a couple of
> seconds...but every once in a while it can be 30 s or more of 100% CPU
> utilization.

autocomplete should be on the order of a second or less.


note: no performance bugs on file for the types of things being mentioned ...
https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=anywordssubstr&keywords=perf&keywords_type=allwords&field0-0-0=short_desc&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&type0-0-0=nowords&value0-0-0=count%20counts&component=Address%20Book&component=Composition&component=Message%20Compose%20Window&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird
(Reporter)

Comment 4

8 years ago
> > My largest abook.mab is just under 4 MB, and has been over 3 MB for the last
> > few versions of Eudora beta.  
> 
> how many contacts for that 4MB address book?

7500

> do you delete a lot of AB entries in a year?

Not really.  There are ~100 new addresses and ~150 edits of one kind or another per month in the main address book.

> 
> > This performance problem has been with us for a while
> 
> please define a while. 

Picky, picky, picky ;-)

I've been on Penelope for 3 years -- I guess that's Penelope a11 and Eudora 8 B2 or 3.  I've been a user of original Eudora since 96, but that's a different story.


> autocomplete should be on the order of a second or less.

most of the time, it is.  but if you do a typo, or erase some of the characters you've just entered in the address bar it can really get wrapped around the axle.  I have no idea if this is unique to Eudora, but I would guess not.

> 

> note: no performance bugs on file for the types of things being mentioned ...

Yeah, I was a good boy and searched before submitting.  

I had submitted a related bug (particularly on autocomplete) maybe 18 months ago, but the bugmaster got back to me with an "already fixed in the next release" message...so that one was closed out long ago.

Comment 5

8 years ago
> 100 total mailboxes

mailboxes = folders?  or inboxes?


(In reply to comment #4)
> > > My largest abook.mab is just under 4 MB, and has been over 3 MB for the last
> > > few versions of Eudora beta.  
> > 
> > how many contacts for that 4MB address book?
> 
> 7500
> 
> > do you delete a lot of AB entries in a year?
> 
> Not really.  There are ~100 new addresses and ~150 edits of one kind or another
> per month in the main address book.

that may kill my idea, but let's try it anyway. 
backup (or copy) your address book file
export it to ldif file
delete the address book
import it back in.
(or use some alternative process to accomplish the same results)

results?


> > > This performance problem has been with us for a while
> > 
> > please define a while. 
> 
> Picky, picky, picky ;-)
> 
> I've been on Penelope for 3 years -- I guess that's Penelope a11 and Eudora 8
> B2 or 3.  I've been a user of original Eudora since 96, but that's a different
> story.

clarity is crucial :)
so, to connect the thoughts, you've seen this performance problem for 3 years, to an increasing degree.


> Yeah, I was a good boy and searched before submitting.  
> 
> I had submitted a related bug (particularly on autocomplete) maybe 18 months
> ago, but the bugmaster got back to me with an "already fixed in the next
> release" message...so that one was closed out long ago.

if you still see some of the impact of what you wrote about in bug 472576, then please reopen the bug.
(Reporter)

Comment 6

8 years ago
> > 100 total mailboxes
> 
> mailboxes = folders?  or inboxes?

sorry for my imprecision.

100 total folders (nested as much as 3 levels) with 10 inboxes.


> > > do you delete a lot of AB entries in a year?
> > 
> > Not really.  There are ~100 new addresses and ~150 edits of one kind or another
> > per month in the main address book.
> 
> that may kill my idea, but let's try it anyway. 
> backup (or copy) your address book file

OK, so some interesting tidbits here:
  * the exported ldif file was <3 MB
  * when re-importing it in, the resulting mab file was ~1.6 MB

> delete the address book

Well, this is my "personal address book" which can't be deleted from the GUI.  So I shut down Eudora, renamed abook.mab as OLDabook.mab and restarted. This gave me an empty personal address book.

Copied the contacts (through UI drag/drop) from the Imported folder back into the Personal Address Book...this resulted in an abook.mab file of ~2 MB.

Addressbook load speed in the GUI is somewhat better, but still 30-60 seconds for each load (where a "load" is defined as close the address book window and then re-open it). Memory appears to have a nice, solid 20 MB leak with each load cycle.
(Reporter)

Comment 7

8 years ago
Note that since I'm using Zindus to synchronize the main address book with Google, I expect there are some extra goodies in the .mab file.  

To ANYBODY in the same situation, be very careful about removing/replacing your main address book file, as Zindus will politely offer to delete all your contacts in Google.  You can decline the offer...but pay attention!

The solution is to click Zindus' "reset" button (wait for it to process that -- it can take a while) before you click the "sync".  This puts you in slow-sync mode, which is very accurately named.  Think "dinner-time".

Comment 8

8 years ago
thunderbird caches addresses for address autocomplete, so ABs are loaded into memory, but you should not see increasing mem usage over time and you should not be experiencing delays with ever AB load.

Can you reproduce the issue in a bare Thunderbird profile using version 3.1.1, no eudora?
(Reporter)

Comment 9

8 years ago
OK, so I took the following measurements with T'bird 3.1.1 with no plug-ins (safe mode).

I'm running winxp-sp3 on a 32-bit intel laptop with 2 GB RAM.  My address book contains 7500 entries (mab file less than 3 MB)

*Every time* I click on the address book, something like 30 MB of additional RAM is consumed (as measured by "Mem Usage" in windows task manager tool).  If I open the address book, immediately close it, and repeat 10x...I get a T'Bird memory footprint of 411 MB.

If I then start to write a mail and enter some names for quick auto-complete, every new name adds about another MB.  If I enter a few characters of a common name (like "dav") and then hit backspace a couple of letters, the memory footprint grows by maybe 20 MB and the CPU utilization goes to 99% for a few seconds.

Even if I close that message and don't save it, the memory footprint doesn't shrink appreciably.  Right now, I'm at 450 MB and I've only had T'Bird open long enough to fill out this bug report.

Seems like a confirmed sighting of a memleak to me...
(Reporter)

Comment 10

8 years ago
Oh yeah...and thanks to installing TB 3.1.1 my Eudora/Lightning calendar database is now corrupted.

Grrrrrrrr.  I know it's my fault, I should have done a backup of everything.  But it really bothers me that the bug *poster* has to take all the risk here.  Like it's a better idea to risk user data than to have the developer try the multiple-version test themselves.

Comment 11

8 years ago
sorry, I don't know calendar. If you need assistance with calendar please try on IRC channels #calendar and #maildev, #penelope for eudora - irc tips at https://wiki.mozilla.org/Thunderbird:Bugdays#Where

I was able to reproduce the big memory issue. I tested with 11k contacts in the address book, 2MB file. I don't see the autocomplete issue, nor the long load times - it's just a couple seconds on my laptop. 
1. open AB window
2. click AB - memory increases 30MB
3. close aB window
4. repeat 1-3

I can't reproduce this with smaller AB of 800.

list of open memory issues related to addresses and address book - https://bugzilla.mozilla.org/buglist.cgi?bug_id=503174%2C366457%2C103083%2C126859%2C106857%2C445507%2C570062&bug_id_type=anyexact&query_format=advanced
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: mlk
Version: unspecified → Trunk
(Reporter)

Comment 12

8 years ago
The long load times (>5 sec) seem to be Eudora only (or maybe it's TB 3.0), but I was able to get all the other effects with TB 3.1.1 w/no plug ins.

The autocomplete issue seems to occur when there are a lot of similar names.  Unique strings make it easier for the AC algorithm to do its work quickly and with little memory use.
(Reporter)

Comment 13

8 years ago
Hi.

I realized I'd left out an important clues here:  CPU utilization and I/O.

The first time I try to open my address book, it might take 15 seconds or even less.  But with each subsequent opening, the CPU crunch time expands (sometimes to well over a minute of 90%+ CPU utilization).  This means that something is using as much as 100,000,000,000 CPU cycles with each address book open.  During the high CPU usage/memory expansion there are essentially no I/Os consumed by the Eudora process and the disk is idle.  

So, the bigger the memory footprint, the longer some loop is searching, or incrementing, or something.  This occurs every time I load the AB...even if I just switch between one AB and another (all of them are local).  Same behavior (although different numbers) in Safe mode (with Eudora and all other plug ins turned off).  I can get TB 3.0 to spend over a minute of CPU time opening my address book.

Comment 14

8 years ago
David, 

Can you put a short recap on a) current situation using version 3.1.7 b) differences between what you see in TB and OSE, c) the effect of Norton enabled and disabled [1]

and, can you confirm that comment 11 steps are complete to reproduce the issue for you?

[1] In email correspondence David wrote "Tried it with Norton AV on and off. For some reason, turning it off made the abook performance much more erratic -- sometimes 5 seconds, sometimes 5 minutes. Memory usage patterns were different, but not much better."
Whiteboard: [str: comment 11]
(Reporter)

Comment 15

8 years ago
Hi Wayne,

a) I'm on Eudora OSE rc2, which I believe is TB 3.0
b) For a number of reasons (data corruption), I can't go back and forth between OSE and TB
c) Norton off didn't seem to run much faster, but there was a much wider range of load times

UPDATE:  I'm off XP and running on Win7 now.  With OSE, memory footprint problem is essentially gone.  Seems to have been some interaction with the OS.  On my old XP system, problem persists unabated.

Comment 16

8 years ago
(In reply to comment #15)> 
> UPDATE:  I'm off XP and running on Win7 now.  With OSE, memory footprint
> problem is essentially gone.  Seems to have been some interaction with the OS. 
> On my old XP system, problem persists unabated.

Thanks for the update.

is there any notable difference in file size of the address books between the two OS?
Target Milestone: --- → Thunderbird 3.3a2
(Reporter)

Comment 17

8 years ago
No difference.

In fact, I took the XP file and applied it to my profile on the W7 machine and it worked instantly.  So, essentially, it's the same file.

FWIW my W7 machine is a 64-bit 4-core and the XP box was a 32 bit uni.

Comment 18

7 years ago
David,

I hope I haven't led us astray with so many comments. To recap 
- your issues didn't end with Bug 450134 being fixed (performance improvement for autocomplete), and 
- memory usage is better in safe mode, and
- "With OSE [and new profile on win7], memory footprint problem is essentially gone."
- "(although different numbers) in Safe mode (with Eudora and all other plug ins turned off). I can get TB 3.0 to spend over a minute of CPU time opening my address book."

Is that a fair recap?

For AB load time, suggest you mitigate by moving/archiving a large number of your contacts into a couple other ABs so that the main one has lots less. And unless the memory grows higher every time you enter, there's nothing more that will help except fix Bug 382876 - conversion from mork db to sqlite for addressbooks.

footnote - be aware that TB 3.0.4 has many performance issues and unfortunately you are stuck there. Most notable for you I would think is Bug 542234 - Thunderbird 3 - memory increases about 12MB on each check for new mail / get new messages. all add-ons are disabled [pop]

Updated

7 years ago
Target Milestone: Thunderbird 3.3a2 → ---
(Reporter)

Comment 19

7 years ago
Your recap is fair.  The good news is that with a 4-core 2.4 GHz system and 8 GB RAM, I hardly notice load time at all in W7 ;-)  Plus, there doesn't seem to be a memory leak in either address book load or auto-complete.  So I'm a happy camper.

As an aside, the 12 MB memory leak for each new get-mail cycle doesn't happen on W7 either, even though I'm still on TB 3.0.4 code base.

Anyone stuck on XP is not likely to be very happy, though.

Comment 20

7 years ago
> System is a 1.6 GHz laptop with 2 GB of RAM

In my testing, on Dell D531 running vista pro and 1.8 GHz AMD Thurion 64 with ~800MB DDR2 memory and Fujitsu MHY2080BH 80GB 5400 RPM 8MB - which I would estimate to be perhaps 1.5x faster than your 1.6 GHz laptop setup

AB gobbles about 2MB memory per 1,000 contacts, give or take 20%. 16k contact AB loads in 3-5 sec, 8k contact AB in ~2 sec. So unless your old laptop had a failing hard disk, a hard disk that was highly fragmented (not adequately defraged) or something else weird going on then you should not have experienced AB load times of 30-60 sec, or even more. 

FTR, the autocomplete speed improvement was via bug 450134 (performance improvement for autocomplete by using the directory search facilities), mentioned in your bug 472576  

You are working OK on the new machine so I'm closing this WORKSFORME even though I'm conflicted because we're not 100% certain that the problem on the old laptop wasn't caused by thunderbird. OTOH, good odds that it wasn't TB.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.