554559 - Improperly closing Thunderbird forces a gloda index rebuild

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Jesus Cea]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100214 Ubuntu/9.10 (karmic) Firefox/3.5.8
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100322 Shredder/3.0.5pre

If I close Thunderbird 3 "improperly" (a "kill", a "core dump", a server reboot), it will regenerate the gloda index again. In my environment (>25 GB or email), it takes a lot of time, performance and harddisk bandwidth. In fact it can take a week.

Reproducible: Always

Steps to Reproduce:
1.Open Thunderbird
2.Kill it abruptly (unix "kill", machine switch off)
3.Restart Thunderbird
Actual Results:  
Thunderbird will recreate the gloda index database. This take a lot of time and resources.

Expected Results:  
The gloda index must only index the new email coming. The index for old messages should be still valid.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

This is probably evidence of why we should do
https://bugzilla.mozilla.org/show_bug.cgi?id=467688

Comment 2

[Jesus Cea]

I created this entry a week ago. The GLODA reconstruction is still going in my laptop, and I keep Thunderbird open 12-16 hours per day, six days per week...

When it is done, if this happen again, I will disable indexing :-(

Comment 3

[Jesus Cea]

A month and still going. The indexing is taking quite a time.

My email partition is only about 20 GB.

Comment 4

[Jesus Cea]

Just finished!. Three months to index 20GB!. The index size is 7.7 GB.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

How many messages did you delete a day?  This is probably bug 530098 which is only fixed in 3.0.5 on the 3.0.x branch.

Comment 6

[Jesus Cea]

Hundreds.

And yes, I noticed a huge speed improvement in indexing lately (going from indexing a message every few minutes, to hundreds of messages per second). I use thunderbird nighties.

Comment 7

[Wayne Mery (:wsmwk)]

Andrew, comment 6 is good news. But global-messages-db.sqlite of 7.7GB sounds too big to me.  would bug 467688 account for that?

Jesus, what kind of message do you get on a daily basis, that are not from "humans"?  Or, do you get lots of messages with attachments?

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

If the problem was slow deletion processing, and it definitely sounds like it was, then the global-message-db.sqlite could easily be bloated and full of empty space.  To fix that, you are going to want to delete the database.  If you are disabling gloda, turn it off before quitting and deleting the database.  Hopefully gloda is usable for you now, though.

Comment 9

[Wayne Mery (:wsmwk)]

Perhaps I'm in a poor position to judge and I guess this confuses me, because I delete probably a thousand or more messages a week but have never seen major impact - I've only seen minor short freezes, no endless indexing nor over sized index.

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

The deletion problem caused what amounts to a table scan which was a function of the total number of messages in your database.  Presumably your have one or more of: fewer messages, faster hard disk, more RAM, faster processor, less fragmented datastore.

Comment 11

[Wayne Mery (:wsmwk)]

I guess a 10k rpm disk can cover a multitude of ills. And perhaps 1/10 the number of Jesus' messages, my total messages size is 2GB vs his 25GB

Comment 12

[Jesus Cea]

I have 3.2 millions of messages in my storage, 25GB. The gloda index is 7.7GB. I am working in a laptop, so the HD is probably pretty slow too. I receive around 4-5.000 msgs per day, deleting 1/3 of them.

Nevertheless in the last couple of weeks, gloda activity is pretty small. I use nightly, so I guess somebody patched something :).