619035 - All old mail in inbox no longer had a body after a certain mail was received

Status: RESOLVED INCOMPLETE

(Thunderbird :: General, defect)

Description

[cybermonktitan]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13
Build Identifier: 2.0.0.24 (20100623)

A big inbox (about 900K MSF file, local folders) received a mail (using POP) with an attachment (standard encoding for attachments, base-64 if I'm not mistaken). After receiving this mail the user noted that the mail appeared empty and after some time suddenly completely disappeared from the listing of the mails. The next received mail the user saw was corrupted instead and all later received mail seemed normal.

Upon investigation the Inbox.msf file seemed to contain all the headers of the mails, including the presumably rogue mail, which explains why all the headers were visible in the inbox overview. The actual content of the mails received before the rogue mail was completely absent from the Inbox file. The Inbox file started with a whole bunch of Base64 encoded data followed by the mails received after the first mail after the rogue mail (so if after the rogue mail mails A B C etc were received, the Inbox file contained base64 data followed by mails B C etc).

It may be noteworthy that while the Inbox file was at this point only 505K the MSF file was almost a megabyte large.

This was clearly not tested/retested on a newer version, since it is hopefully (for the user, that is) completely not reproducible.

Reproducible: Didn't try




Profile is placed in the default folder (~/.mozilla-thunderbird/....). The home dirs on these computers are stored on a 64-bit linux server (ext3 filesystem) mapped using NFS (the actual mapping from fstab being
the_server_ip:/home     /home   nfs     nolock,nouser,atime,auto,rw,nodev,noexec,nosuid 0       0
and the server and clients have the same user data)

A previous measurement of the size of the mail folder was over 7G, after the bug had appeared this was only 3.6G. The user had not done any cleanup in the meantime. It is known that the inbox prior to the bug was very big, so the size of the Inbox file might actually have been in the order of gigabytes.

This bug might have to do with hitting file size limits. How does Thunderbird react to hitting that limit while writing an email? And how does NFS fix into that mix?

Comment 1

[WADA:World Anti-bad-Duping Agency]

What is file size of file named "Inbox"?
(not "Inbox.msf", not directory size which contains "Inbox" and "Inbox.msf")

Comment 2

[WADA:World Anti-bad-Duping Agency]

My question was; Next is Inbox=a megabyte large, Inbox.msf=505K, isn't it? If so, "more than 2GB or 4GB" instead of "a megabyte large", isn't it?
> It may be noteworthy that while the Inbox file was at this point only 505K
> the MSF file was almost a megabyte large.
Show "Order Recceived" column(offset in file if local mail folder), and sort by the column. What is largest value?

Comment 3

[cybermonktitan]

The file named Inbox was, at the moment of investigation, only 505K large. The Inbox.msf was almost a megabyte. (Yes, the Inbox.msf file *was* larger than the Inbox file.) At this point, however, the problem had already occurred and the old mail had already disappeared. It was not in the Inbox file, although it was still indexed in the Inbox.msf file.

The size of several gigabytes is purely inferred from the differences of the total profile size before and after the problem (no better data available, alas). Point is that, between the previous measurement of the complete profile (over 7G) and the measurement of the complete profile after the problem occurred (3.6G), the only important change was the problem itself: The user has a tendency not to throw away anything ;). Also, no maintenance operations have been executed manually and none should run automatically other than those that come as defaults with Thunderbird.

As for the "Order Received" column, I could do that but preferrably using just a text editor to inspect the files. I only have SSH access to the data and can't really view them in Thunderbird anymore. (I could, but I'd have to travel a couple of hours to get to the site.)

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> The file named Inbox was, at the moment of investigation, only 505K large. The
> Inbox.msf was almost a megabyte. (Yes, the Inbox.msf file *was* larger than the
> Inbox file.)

Keep back up of the Inbox and Inbox.msf on NSF ASAP.
Inbox.msf may be real Inbox and Inbox may be real Inbox.msf before problem.
"Real Inbox(Unix Mbox file) or not" is easily checked by first part of file.
   From - ... (... is date format data)
   X-xxx...: headers
   X-Mozilla-Status: ...
   X-Mozilla-Status-2: ...
   Other header lines follows
"Real Inbox.msf(Mork DB) or not" is easily checked by first part of file.
See content of normal .msf file by text editor.

Comment 5

[cybermonktitan]

Thanks for the heads-up. I had already checked, though, and the Inbox file was indeed the actual inbox and the Inbox.msf file was the database file. Both had already been backed up.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> The Inbox file was indeed the actual inbox (snip)
> Both had already been backed up.

If so, "delete Inbox.msf, restart Tb, click Inbox folder(rebuild-index is automatically executed)" is one of simplest ways to recover, if file named Inbox is healthy.

Note: You(and we) can do further analysis with backup files at your local Thunderbird, as far as backup files are kept.

Comment 7

[cybermonktitan]

Thanks for the suggestion, but I'm afraid that is the very problem: the Inbox file was completely broken, as I described in my original report. It started with some Base64 garbage and then only contained the mails that came in after the mail that apparently broke it. Rebuilding the indices has already been done and completely removed all traces from the old mails.

The backup I made was after a few days and some diagnostic steps after the problem had occurred. I would love to have the original inbox available, for debugging and more even for the user, but I don't. Debugging in a local Thunderbird is possible, of course, were it not for the sheer volume of data I'd have to download (these folders, as stated, are over 3G in size; it'd take me about half a day to get that transferred). Or can debugging be done with just those two files, ignoring all the other folders?

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> Rebuilding the indices has already been done
> and completely removed all traces from the old mails.

Difference between (a) "rebuild-index with currently used Inbx.msf" and (b) "delete Inbox.msf, restart Tb, click folder to force rebuild-index" is;
(a) : currently used Inbx.msf is red, and some of already held data is re-used.
(b) : as no Inbox.msf, Inbox.msf is re-created from absolutely scratch.

As file size of Inbox is 505KB, download from NFS server won't take long, and "rebuild-index with Inbox.msf deleted" can be checked locally using your local Tb.
1. Create dummy pop3 account. (specify z@z.z.z as mail address, click "Stop" twice, one for account, one for SMTP, change fields for dummy POP3 account, non-existent server, "Manual Setup").
2. Change Server Settings/Local Directory: to a local directory on local HDD.
3. Copy backup or downloaded Inbox of 505KB in the local directory.
   File name like Inbox-Corrupted may be better for testing.
4. Restart Tb, click folder and force rebuild-index.
Is Inbox.msf of several giga bytes generated?

Comment 9

[Wayne Mery (:wsmwk)]

t.m. please reply to comment 8

Comment 10

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.