All users were logged out of Bugzilla on October 13th, 2018

Improperly closing Thunderbird forces a gloda index rebuild

RESOLVED WORKSFORME

Status

--
major
RESOLVED WORKSFORME
9 years ago
8 years ago

People

(Reporter: jcea, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

9 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100214 Ubuntu/9.10 (karmic) Firefox/3.5.8
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100322 Shredder/3.0.5pre

If I close Thunderbird 3 "improperly" (a "kill", a "core dump", a server reboot), it will regenerate the gloda index again. In my environment (>25 GB or email), it takes a lot of time, performance and harddisk bandwidth. In fact it can take a week.

Reproducible: Always

Steps to Reproduce:
1.Open Thunderbird
2.Kill it abruptly (unix "kill", machine switch off)
3.Restart Thunderbird
Actual Results:  
Thunderbird will recreate the gloda index database. This take a lot of time and resources.

Expected Results:  
The gloda index must only index the new email coming. The index for old messages should be still valid.
(Reporter)

Updated

9 years ago
Version: unspecified → 3.0
Component: Search → Database
Product: Thunderbird → MailNews Core
QA Contact: search → database
Version: 3.0 → Trunk
This is probably evidence of why we should do
https://bugzilla.mozilla.org/show_bug.cgi?id=467688
(Reporter)

Comment 2

9 years ago
I created this entry a week ago. The GLODA reconstruction is still going in my laptop, and I keep Thunderbird open 12-16 hours per day, six days per week...

When it is done, if this happen again, I will disable indexing :-(
Depends on: 467688
(Reporter)

Comment 3

9 years ago
A month and still going. The indexing is taking quite a time.

My email partition is only about 20 GB.
(Reporter)

Comment 4

8 years ago
Just finished!. Three months to index 20GB!. The index size is 7.7 GB.
How many messages did you delete a day?  This is probably bug 530098 which is only fixed in 3.0.5 on the 3.0.x branch.
(Reporter)

Comment 6

8 years ago
Hundreds.

And yes, I noticed a huge speed improvement in indexing lately (going from indexing a message every few minutes, to hundreds of messages per second). I use thunderbird nighties.

Comment 7

8 years ago
Andrew, comment 6 is good news. But global-messages-db.sqlite of 7.7GB sounds too big to me.  would bug 467688 account for that?

Jesus, what kind of message do you get on a daily basis, that are not from "humans"?  Or, do you get lots of messages with attachments?
Severity: normal → major
If the problem was slow deletion processing, and it definitely sounds like it was, then the global-message-db.sqlite could easily be bloated and full of empty space.  To fix that, you are going to want to delete the database.  If you are disabling gloda, turn it off before quitting and deleting the database.  Hopefully gloda is usable for you now, though.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME

Comment 9

8 years ago
Perhaps I'm in a poor position to judge and I guess this confuses me, because I delete probably a thousand or more messages a week but have never seen major impact - I've only seen minor short freezes, no endless indexing nor over sized index.
The deletion problem caused what amounts to a table scan which was a function of the total number of messages in your database.  Presumably your have one or more of: fewer messages, faster hard disk, more RAM, faster processor, less fragmented datastore.

Comment 11

8 years ago
I guess a 10k rpm disk can cover a multitude of ills. And perhaps 1/10 the number of Jesus' messages, my total messages size is 2GB vs his 25GB
(Reporter)

Comment 12

8 years ago
I have 3.2 millions of messages in my storage, 25GB. The gloda index is 7.7GB. I am working in a laptop, so the HD is probably pretty slow too. I receive around 4-5.000 msgs per day, deleting 1/3 of them.

Nevertheless in the last couple of weeks, gloda activity is pretty small. I use nightly, so I guess somebody patched something :).
You need to log in before you can comment on or make changes to this bug.