531012 - search doesn't work when sqlite global-messages-db.sqlite reaches 2 GB

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Andrea Zucchelli]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5) Gecko/20091106 Firefox/3.5.1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20091118 Lightning/1.0pre Thunderbird/3.0b4

I've about 6 GiB of messages, when global-messages-db.sqlite hits 2 GiB searching breaks and I'm not able to "search everywhere", I get an empty tab.


Reproducible: Always

Steps to Reproduce:
1. Open an account with lot of messages
2. Wait for the indexing process to create a 2GB global-messages-db.sqlite
3. Try to search anything
Actual Results:  
Searching doesn't work

Expected Results:  
Search working

Comment 1

[Wayne Mery (:wsmwk)]

wow, 2 gig index on 6gig message store. sounds odd. oh...gmail related? :)

Comment 2

[Andrea Zucchelli]

No, it's just a pop account here at work. 
I still have all messages since 2002 :)
I have a lot of long messages very similar, reports from database backups, server backups etc. so same subject, same from, almost same message.
I don't know if this kind of message pattern can inflate in some way the size of the database.

Comment 3

[Shawn Wilsher :sdwilsh]

It'd be interesting to know if there are any errors in the error console.  Also, is this a mozilla.org build, custom build, or a build from your distro?

Comment 4

[Andrea Zucchelli]

(In reply to comment #3)
> It'd be interesting to know if there are any errors in the error console.

Yes the console had errors. I need some time to reproduce them because in the meantime I deleted some old logs to fit 2 GiB of database file.
Anyway I can remember 2 distinct conditions:
- during the first startup when the index process hanged I had errors related to disk I/O (the filesystem was fine and I had free space).
- Since that time every time I launched thunderbird I had an exception in gloda.js or datastore.js something like this.asyncConnection is null ( I'm trying to remember ).

> Also, is this a mozilla.org build, custom build, or a build from your distro?

I built it from source using the PKGBUILD here: http://aur.archlinux.org/packages.php?ID=16963
here's the mozconfig http://aur.archlinux.org/packages/thunderbird3/thunderbird3/mozconfig

At the moment they've just updated to 3.0rc... If I've some times tomorrow morning (UTC+1) I'm going to try with this version..

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Did your disk fill up?  The fact that you had to delete log files suggests perhaps it did.

If the disk filled up, it might be conceivable that the database became corrupted.  Gloda doesn't automatically recover from corrupt databases (bug 520139), so that could be why search would become broken.  The asyncConnection is null certainly sounds exactly like that.

Comment 6

[Andrea Zucchelli]

Nope, I wish... I had to delete email messages (which where log messages form various server) to keep the database size under 2 GiB in order to be able to search through messages.

Sorry if I'm not clear, English is not my native language :/

Comment 7

[Andrea Zucchelli]

I've upgraded to 3.0rc and tried to open the original profile with all the messages.
The index process worked flawlessly, now my global-messages-db.sqlite is 3.8 GiB and everything seems to work.
Let's wait to hit the 4 GiB barrier :)
Did the underlying sqlite got update between 3.0b4 and 3.0rc ?

As far as I'm concerned the bug can be closed...

Comment 8

[[:Aureliano Buendía]]

(In reply to comment #7)
> Let's wait to hit the 4 GiB barrier :)
> Did the underlying sqlite got update between 3.0b4 and 3.0rc ?

Maybe fixed by Gloda correctness fix, tracked by bug #465618

> As far as I'm concerned the bug can be closed...
Closing WFM per comment #7.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Andrea, we did indeed upgrade the SQLite between 3.0b4 and 3.0rc.

I'm still a little concerned about the size of the gloda database, but full-text indexing of log file contents would indeed explain things.  You may want to install the GlodaQuilla extension that allows you to mark certain folders or accounts as not to be indexed:
https://addons.mozilla.org/en-US/thunderbird/addon/9873

You probably want to delete your global-messages-db.sqlite after configuring GlodaQuilla to ignore your log files.

Comment 10

[Andrea Zucchelli]

(In reply to comment #9)
> Andrea, we did indeed upgrade the SQLite between 3.0b4 and 3.0rc.
This would probally explains why now it works...

> I'm still a little concerned about the size of the gloda database, but
> full-text indexing of log file contents would indeed explain things.  You may
> want to install the GlodaQuilla extension that allows you to mark certain
> folders or accounts as not to be indexed:
> https://addons.mozilla.org/en-US/thunderbird/addon/9873

Thank you very much for the suggestion ! now my database is just 88 MiB !