471307 - "Rebuild Index" feature creating corrupted .msf index files

Status: VERIFIED FIXED

(Thunderbird :: General, defect, P1)

Description

[IU]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081227 Lightning/1.0pre

Folder index files are being corrupted when right-clicking a folder and doing "Properties... --> General Information --> Rebuild Index".  The messages are still listed, but their content is no longer viewable.  Once this occurs, the only solution is to shutdown Shredder and manually delete the associated .msf file.

Regression Range:
Works: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081219 Shredder/3.0b2pre

Broken: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081220 Shredder/3.0b2pre


Reproducible: Always

Steps to Reproduce:
1. Right-click a folder (containing messages) and select "Properties... --> General Information --> Rebuild Index"
2. Click a different folder then reselect the folder in Step #1 and attempt to view any of the messages.  Notice that no message content can be displayed.
3. Shutdown Shredder and delete the corresponding .msf file for the folder
4. Start Shredder and notice that you can now view the message content.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Confirmed with Tb trunk on MS Win-XP SP3.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081227 Shredder/3.0b2pre

(1) Rebuild Index(of a local mail folder)
(2) Click some mails in the folder => Message pane was not correctly updated.
    Following error in Error Console when mail click.
> Error: uncaught exception: [Exception... "Component returned failure code: 0x80550008 [nsIMsgDBHdr.markHasAttachments]" nsresult: "0x80550008 (<unknown>)"
>  location: "JS frame :: chrome://messenger/content/msgHdrViewOverlay.js :: anonymous :: line 520"  data: no]
(3) Move a mail to the troubled mail folder from other mail folder
    => Problem disappeared. Mails were displayed normally.
       Above error message was not issued, until Rebuid Index again.

Comment 2

[Valery Kondakoff]

Confirmed on Win Vista HP SP1 (Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b3pre) Gecko/20081228 Lightning/1.0pre Shredder/3.0b2pre).

The index corruption happens most often on account's Sent folder, after sendin a message.

Comment 3

[David :Bienvenu]

yikes, I wonder if this had to do with the changes to the db opening api...I'll try to figure this out.

Comment 4

[David :Bienvenu]

In my test case, just clicking away to a different folder and then back to the original folder fixes the problem. I'm not sure if the db is getting regenerated in that case. The message key in the view is the same in both cases, so it's not that we have the wrong offset into the mailbox.

Comment 5

[Joe Sabash [:JoeS1]]

Not so here, I have the sent folder in the "locked up" state and get this:
Error: Component returned failure code: 0x80550005 [nsIMsgFolder.getMsgDatabase] = <unknown>
Source file: chrome://messenger/content/mailWidgets.xml
Line: 1929

Probably trying to open a message in a new tab, at that point.

Comment 6

[Joe Sabash [:JoeS1]]

Here is another related error:
Error: Component returned failure code: 0x80550005 [nsIMessenger.msgHdrFromURI] = <unknown>
Source file: chrome://messenger/content/messageWindow.js
Line: 382

Double clicking to open in a new window

Comment 7

[David :Bienvenu]

Two things are going wrong, from what I can tell:

CloseAndBackupDB is not backing up the db, because mFlags is 0.
GetDatabaseWithReparse is failing because OpenFolderDB is returning NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE and a null mDatabase. In that case, GetDatabaseWithReparse does nothing useful.

Sometimes OpenFolderDB returns NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE and a non-null mDatabase, and then GetDatabaseWithReparse works and the bug doesn't occur.

Comment 8

[David :Bienvenu]

Joe, you can probably discover about 50 things that go wrong if we can't open the database for the folder. We need to fix the underlying problem.

Comment 9

[IU]

(In reply to comment #2)
> The index corruption happens most often on account's Sent folder, after sendin
> a message.

This is how I first ran into the problem, but didn't realize that the corruption occurred after sending messages, because I didn't observe the state of the Sent folder before sending a message.  It was only afterward, when I tried to double-check what I had sent, that I found my Sent messages were inaccessible.

Thanks for pointing this out.

Comment 10

[David :Bienvenu]

I believe the problem is that we never delete the .msf file in this case, so we reparse the folder and add headers to an existing mdb, where everything is duplicated. But debugging this is a bit challenging on the mac...

Comment 11

[Joe Sabash [:JoeS1]]

For a while now, simply opening the sent folder has resulted in an auto reindex.
Looking at the regression range, I'd suspect:
http://hg.mozilla.org/comm-central/rev/e10a3baac8d7

Comment 12

[Valery Kondakoff]

I'm not sure if this can help, but I can provide two Sent.msf files: the 'broken' one (after sending a message it was impossible to view the message bodies) and the second, which was recreated after TB restart. 

The files are zipped: http://www.nncron.ru/temp/sent.zip (~540kB)

Comment 13

[David :Bienvenu]

Attachment: proposed fix

proposed fix - this fixes it for me. Asking jcranmer for review since I think this was broken in the db open reworking code.

Comment 14

[Joshua Cranmer [:jcranmer]]

Comment on attachment 354770 [details] [diff] [review]
proposed fix

I swore I changed all of those...

From grep, this is the last one.

Comment 15

[David :Bienvenu]

fix checked in.

Comment 16

[IU]

"Rebuild Index" is fixed, but the corruption on sending that was mentioned in Comment #2, and which I acknowledged in Comment #9 remains.

I have filed Bug 471682 for the corruption on sending.

Comment 17

[Phil Lacy]

if you look far enough into the openfolderdb I think it can return a NS_ERROR_FILE_TARGET_DOES_NOT_EXIST so you may want to || it.

Comment 18

[Phil Lacy]

if you look far enough into the openfolderdb I think it can return a NS_ERROR_FILE_TARGET_DOES_NOT_EXIST so you may want to || it.

Comment 19

[Wayne Mery (:wsmwk)]

is this still considered to be a regression?  of Bug 437886?

Comment 20

[David :Bienvenu]

yes, correct, it's a regression from bug 437886