550764 - (gloda?) indexing fails to complete on large installation updates

Status: RESOLVED INCOMPLETE

(Thunderbird :: General, defect)

Description

[william.allen.simpson]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3

Just upgraded from Thunderbird 2.  All messages are on a single MacBook Pro, no external drives, no IMAP.

Indexing starts up by default.  Constantly spinning work indicator.  Indexing status messages at bottom of window.

Left running overnight.  Failed to finish in 16 hours.  Only 224 of 2000+ folders completed.

Slower and slower.  Indexing folders with ~300 messages initially took 18-21 seconds (IndexingBegin.png), now taking 120+ seconds (IndexingStall.png).



Reproducible: Didn't try

Steps to Reproduce:
1. ~5GB email total (6.4+ GB including .msf files), ~2000 folders
2. upgrade 2.0.0.23 to 3.0.3
3. indexing starts by default (bad enough).

Actual Results:  
4. never completes


Expected Results:  
Finish indexing in a timely manner....

Otherwise, for large installations (say, greater than 2 GB), don't start by default!  Give warnings that this doesn't work for > 4 GB files!


Current size of global-messages-db.sqlite is 1.14 GB.

Began 4:15 pm Sat, not completed by 8:49 am Sun.

Comment 1

[william.allen.simpson]

Attachment: (start) screen shot of activity manager

Comment 2

[william.allen.simpson]

Attachment: (stall) screen shot of activity manager

Comment 3

[zug_treno]

Duplicate of/related to MailNews Core bug 543737?

Comment 4

[william.allen.simpson]

No, that's about IMAP message bodies in accounts that are compacted.

This is about POP3 accounts, all on one computer.  The messages are rarely large in and of themselves.  The folders can be large, especially for some very active mailing lists, as I usually archive per month, per quarter, or per year.

Comment 5

[Ludovic Hirlimann [:Usul]]

Ok so how large is the global-messages-db.sqlite file in your profile ?

Do you have error messages in Tools -> Error console ?

Comment 6

[william.allen.simpson]

On 4/22/10 1:55 PM, Wayne Mery (vn) wrote:
> question for you in comment 5.
> possible to respond?
>
Yet another person that didn't actually *read* my reports?  I didn't 
bother answering the first time around....  But repeating the obvious:


# Ludovic Hirlimann (:_Tsk_ )      2010-03-10 08:53:16 PST
#
# Ok so how large is the global-messages-db.sqlite file in your profile ?
#
Go back and look at my report again.  It clearly says:

Current size of global-messages-db.sqlite is 1.14 GB.

Began 4:15 pm Sat, not completed by 8:49 am Sun.


# Do you have error messages in Tools -> Error console ?
#
I halted the indexing.  I quit and restarted (see bug 550770).  There are 
no error messages, because error messages are cleared on quit.

Comment 7

[Ludovic Hirlimann [:Usul]]

(In reply to comment #6)
> On 4/22/10 1:55 PM, Wayne Mery (vn) wrote:
> > question for you in comment 5.
> > possible to respond?
> >
> Yet another person that didn't actually *read* my reports?  I didn't 
> bother answering the first time around....  But repeating the obvious:

Yes and I apologize, I read very quickly ....

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to comment #6)
> On 4/22/10 1:55 PM, Wayne Mery (vn) wrote:
> > question for you in comment 5.
> > possible to respond?
> >
> Yet another person that didn't actually *read* my reports?  I didn't 
> bother answering the first time around.... 

your right, I didn't read it. My purpose here was only to handle comment 5, and took your non-response as walking away from the bug. Making progress in bugs is very much a collaborative process. :)

are you in a position to try v3.1 beta?  ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.2/ - take appropriate precautions to protect your profile data via backups.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Expected Results:  
> Finish indexing in a timely manner....

I think developers are already aware of this kind of issues.
Read next document first, please.
> https://developer.mozilla.org/en/Thunderbird/About_gloda#Indexing

Install GlodaQuilla, please, if you need to report problem of Gloda.
"onDisk", "gloda id", "gloda dirty" column is added.  
> https://addons.mozilla.org/en-US/thunderbird/addon/9873/
> http://mesquilla.com/extensions/glodaquilla/
Enable Gloda debugging please, if you want to provide data for problem analysis to developers.
> https://wiki.mozilla.org/Thunderbird:Debugging_Gloda
As Trunk(Tb 3.2pre) still has many problems which affect on Gloda, please check with Tb 3.1pre(1.9.2 branch).
As schema is changed by Tb 3.1, re-indexing may occur upon both changing to Tb 3.1pre and changing back to Tb 3.0.x, if same profile is used. If you don't want it, check with Tb 3.1pre using new profile first, please.

> Otherwise, for large installations (say, greater than 2 GB), don't start by default!

We requested don't force (a) Gloda use since just after migration, don't force (b) IMAP auto-sync for all IMAP folders, don't force (c) "Smart Folders" when migration to Tb 3.0, which can be enabled only by single click at option setting or UI after migration, but ...
With Tb 3.1, (b) is now user selectable and (c) is now never forced.

Comment 10

[Wayne Mery (:wsmwk)]

william, v3.1.3 becomes available soon. please test and comment in the bug. 
Note: the index may rebuild during this update.

Comment 11

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.