529684 - Gloda indexer hangs if it needs to initiate a local folder reparse [Error: this.callbackDriver is not a function]

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

When I refactored index_msg.js out of indexer.js a 'this' relative statement did not get updated correctly.

The specific situation is when we are entering a local folder to index it and call getDatabaseWithReparse and the reparse is actually required.

In theory, at one point our unit tests exercised this, but I think compensating for intermittent oranges caused an updateFolder abstraction to be interposed.  That change has been entirely mooted by our subsequent change to use addMessage to inject local messages rather than building mboxes from full cloth and then trying to parse them up.

I will introduce a new unit test that forces a folder database closed, touches its timestamp or otherwise marks it as requiring a rebuild, and then has us index it again.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

It sounded like Dawn's instance of this problem went away of its own accord.  Dawn, is that true, or did you have to take any actions before that happened?

Comment 2

[Dawn Endico]

I think what happened is that when I first upgraded Thunderbird, it re-indexed a local folder and then hung. Then when I re-started, it indexed the second folder and hung. When I re-started the third time it indexed the last folder and finally finished.

So yes, it eventually fixed itself but the only reason I restarted it was because you guys were helping me diagnose the problem. Under normal circumstances I only restart the client when there's a macos update and I have to reboot the machine. Under normal circumstances it could take weeks or months before I would have restarted the client 3 times. For people with lots of local folders it might take a long time before everything is re-indexed. I bet you had the problem too but didn't notice because you restart the client several times a day. 

If this affects everybody with local folders when they upgrade to 3.1 then it could affect a lot of people. What happens if you try to open an un-indexed folder?  What happens when you try to move a message to an un-indexed folder? Will there be data loss?

It sounds like Andrew understands the problem and a fix should be easy. I hope this gets fixed before the release even if the unit test isn't updated.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

This should only affect people who do not already have up-to-date .msf files when gloda gets to the folder to index it.

As I understand things, normal operation of Thunderbird should not leave it in such a state.  The user or some program acting on their behalf would likely have to have touched the mbox file.  Although nsMsgDatabase has a mechanism for compelling a reindexing of .msf files if their version is too old, we have been at version 1 since CVS revision 1.94 of June 27, 2001.  If I read the CVS Tags correctly, there has never been a release of "Thunderbird" that pre-dates the introduction of the version marker.

Gloda is unique along with the Spotlight/Vista Indexer in poking its nose into almost every folder it can find.  This means that timestamp mismatches (or missing .msf files) on folders that users never go into can linger forever and that gloda will trip over them.

Since restarting programs and rebooting computers is a common way to attempt to troubleshoot problems and gloda will not trip up on each folder a second time (bar really meddlesome interference with the .msf files and mbox files) and the general problem is (allegedly per me) not very likely, I do not think this merits a re-spin but obviously we want this in for 3.0.1.

Bienvenu, knower of the horrible realities with which Thunderbird is assaulted, could you verify my sincere hope that what I'm saying is true or true enough?

Comment 4

[Dawn Endico]

Hmm, ok. I assumed that the re-index was because of some recently introduced feature. I wonder why I had to re-index then. I haven't used any other clients or touched the mail files recently.  A few weeks ago I started using the archive button and started using my local folders more often but I've updated Shredder a couple times since then and didn't notice any re-indexing. I guess the archive button is why I switched to the daily shredder builds.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Attachment: fix refactoring glitch and add unit test that would have caught it v1

Test fails without the change in the patch, succeeds with it.  (And it doesn't require a timeout either; thanks to our having subscribed an error console listener, the unit test fails immediately.)

Comment 6

[Mark Banner (:standard8)]

We don't feel this will affect many users but we'll take this if we do a respin.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

pushed to comm-central (trunk):
http://hg.mozilla.org/comm-central/rev/6d19e94e8f97

pushed to comm-1.9.1 default:
http://hg.mozilla.org/releases/comm-1.9.1/rev/b640179e483d

pushed to comm-1.9.1 relbranch:
http://hg.mozilla.org/releases/comm-1.9.1/rev/c8ef70c02240

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 413621 [details] [diff] [review]
fix refactoring glitch and add unit test that would have caught it v1

(just to note verbal a=Standard8 over phone).