Last Comment Bug 529684 - Gloda indexer hangs if it needs to initiate a local folder reparse [Error: this.callbackDriver is not a function]
: Gloda indexer hangs if it needs to initiate a local folder reparse [Error: th...
[gloda key][tb3ride-along][fixed RC1 ...
: fixed-seamonkey2.0.3
Product: MailNews Core
Classification: Components
Component: Database (show other bugs)
: Trunk
: All All
-- normal (vote)
: Thunderbird 3
Assigned To: Andrew Sutherland [:asuth]
Depends on:
  Show dependency treegraph
Reported: 2009-11-18 16:07 PST by Andrew Sutherland [:asuth]
Modified: 2010-02-05 15:14 PST (History)
5 users (show)
standard8: blocking‑thunderbird3-
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

fix refactoring glitch and add unit test that would have caught it v1 (2.17 KB, patch)
2009-11-20 08:39 PST, Andrew Sutherland [:asuth]
mozilla: review+
standard8: approval‑thunderbird3+
Details | Diff | Splinter Review

Description User image Andrew Sutherland [:asuth] 2009-11-18 16:07:09 PST
When I refactored index_msg.js out of indexer.js a 'this' relative statement did not get updated correctly.

The specific situation is when we are entering a local folder to index it and call getDatabaseWithReparse and the reparse is actually required.

In theory, at one point our unit tests exercised this, but I think compensating for intermittent oranges caused an updateFolder abstraction to be interposed.  That change has been entirely mooted by our subsequent change to use addMessage to inject local messages rather than building mboxes from full cloth and then trying to parse them up.

I will introduce a new unit test that forces a folder database closed, touches its timestamp or otherwise marks it as requiring a rebuild, and then has us index it again.
Comment 1 User image Dan Mosedale (:dmose) 2009-11-19 09:57:27 PST
It sounded like Dawn's instance of this problem went away of its own accord.  Dawn, is that true, or did you have to take any actions before that happened?
Comment 2 User image Dawn Endico 2009-11-19 11:47:15 PST
I think what happened is that when I first upgraded Thunderbird, it re-indexed a local folder and then hung. Then when I re-started, it indexed the second folder and hung. When I re-started the third time it indexed the last folder and finally finished.

So yes, it eventually fixed itself but the only reason I restarted it was because you guys were helping me diagnose the problem. Under normal circumstances I only restart the client when there's a macos update and I have to reboot the machine. Under normal circumstances it could take weeks or months before I would have restarted the client 3 times. For people with lots of local folders it might take a long time before everything is re-indexed. I bet you had the problem too but didn't notice because you restart the client several times a day. 

If this affects everybody with local folders when they upgrade to 3.1 then it could affect a lot of people. What happens if you try to open an un-indexed folder?  What happens when you try to move a message to an un-indexed folder? Will there be data loss?

It sounds like Andrew understands the problem and a fix should be easy. I hope this gets fixed before the release even if the unit test isn't updated.
Comment 3 User image Andrew Sutherland [:asuth] 2009-11-19 20:20:33 PST
This should only affect people who do not already have up-to-date .msf files when gloda gets to the folder to index it.

As I understand things, normal operation of Thunderbird should not leave it in such a state.  The user or some program acting on their behalf would likely have to have touched the mbox file.  Although nsMsgDatabase has a mechanism for compelling a reindexing of .msf files if their version is too old, we have been at version 1 since CVS revision 1.94 of June 27, 2001.  If I read the CVS Tags correctly, there has never been a release of "Thunderbird" that pre-dates the introduction of the version marker.

Gloda is unique along with the Spotlight/Vista Indexer in poking its nose into almost every folder it can find.  This means that timestamp mismatches (or missing .msf files) on folders that users never go into can linger forever and that gloda will trip over them.

Since restarting programs and rebooting computers is a common way to attempt to troubleshoot problems and gloda will not trip up on each folder a second time (bar really meddlesome interference with the .msf files and mbox files) and the general problem is (allegedly per me) not very likely, I do not think this merits a re-spin but obviously we want this in for 3.0.1.

Bienvenu, knower of the horrible realities with which Thunderbird is assaulted, could you verify my sincere hope that what I'm saying is true or true enough?
Comment 4 User image Dawn Endico 2009-11-19 20:48:08 PST
Hmm, ok. I assumed that the re-index was because of some recently introduced feature. I wonder why I had to re-index then. I haven't used any other clients or touched the mail files recently.  A few weeks ago I started using the archive button and started using my local folders more often but I've updated Shredder a couple times since then and didn't notice any re-indexing. I guess the archive button is why I switched to the daily shredder builds.
Comment 5 User image Andrew Sutherland [:asuth] 2009-11-20 08:39:32 PST
Created attachment 413621 [details] [diff] [review]
fix refactoring glitch and add unit test that would have caught it v1

Test fails without the change in the patch, succeeds with it.  (And it doesn't require a timeout either; thanks to our having subscribed an error console listener, the unit test fails immediately.)
Comment 6 User image Mark Banner (:standard8) 2009-11-20 10:08:54 PST
We don't feel this will affect many users but we'll take this if we do a respin.
Comment 7 User image Andrew Sutherland [:asuth] 2009-11-20 10:55:37 PST
pushed to comm-central (trunk):

pushed to comm-1.9.1 default:

pushed to comm-1.9.1 relbranch:
Comment 8 User image Mark Banner (:standard8) 2009-11-20 11:05:13 PST
Comment on attachment 413621 [details] [diff] [review]
fix refactoring glitch and add unit test that would have caught it v1

(just to note verbal a=Standard8 over phone).

Note You need to log in before you can comment on or make changes to this bug.