521312 - Huge memory consumption while determining which messages to index in inbox

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[shopping_mozilla]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Build Identifier: 3.0b4

After opening Thunderbird (in safe-more or otherwise) a message appears in the status bar stating "Determining which messages to index in inbox". When watching the memory in task manager I can see it reach between 500MB and 700MB while the VM jumps from 500MB up to 1.3GB. Since I only have 2GB of ram there is plenty of hard drive thrashing as the numbers jump up and down. The VM can be seen going from 300MB to 700MB and back numerous times within several seconds. I have several inboxs with thousands of email but Thunderbird 2.0 handled them just fine.

Reproducible: Always

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

This suggests we need to be triggering garbage collections more frequently rather than deferring them until the database commit.  We should look into this once the correctness issues are dealt with.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

The gloda correctness patch adds logic to initiate garbage collections when the number of headers 'touched' crosses a threshold.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Reporter, the gloda correctness patch has landed and is available in current nightlies.  If you could please give things another try, that will help me understand if your problem was what I thought it was.

It would be ideal if you could run without any extensions for this.

You can find the latest nightlies here:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.1/

Comment 4

[Wayne Mery (:wsmwk)]

shopping, please test and report success!  (hopefully)

Note, there have been other checkins since 2009-11-08 that you will want to have tested, so please the topmost item on try http://www.mozillamessaging.com/en-US/thunderbird/early_releases/

Comment 5

[shopping_mozilla]

Tested the most recent release (RC2) and things are much better. Physical RAM usage never went over 160MB during indexing. CPU usage peaked around 40% but dropped to zero after indexing was completed. This is on a T5600 Core 2 laptop with 2GB of RAM.

Comment 6

[Wayne Mery (:wsmwk)]

thanks for the feedback

Comment 7

[Wayne Mery (:wsmwk)]

asuth if you have a preferred disposition for this bug, eg duplicate, feel free to change my closing WFM per reporter.