269554 - going from tbird 0.8 -> 0.9 or 1.0 hangs with huge POP mail folders

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[sairuh (rarely reading bugmail)]

filing this to keep track of, as it has cropped up in the Mozillazine tbird
forums: http://forums.mozillazine.org/viewtopic.php?t=155230

a friend of mine had huge (~1.3 GB) POP folders in tbird 0.8. when he installed
and ran tbird 0.9 for the the first time, the app just hanged: he got hourglass
cursor and the CPU shot up to near-100%.

the workaround suggested in the forum link above worked for him: remove the .msf
files for the local files, restart and tbird 0.9 ran happily.

I couldn't really find an open tbird (or mailnews) bug which would cover this.
would bug 254947 be related? also sounds similar to bug 249506, but that seemed
limited to tbird 0.7.1.

Comment 1

[John Kondis]

(In reply to comment #0)

> a friend of mine had huge (~1.3 GB) POP folders in tbird 0.8. when he installed
> and ran tbird 0.9 for the the first time, the app just hanged: he got hourglass
> cursor and the CPU shot up to near-100%.

I believe may I have experienced the same bug.  I start tbird, it just hangs and
get the eternal "waiting" icon.  Meanwhile top reports

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8738 xxxxxxx   25   0 49068  23m  31m R 99.6  2.4   0:15.49 thunderbird-bin

When I start tbird from the command line, this hanging behavior correlates with
the message

"need to reset thread root key"

at the shell.

This hanging behavior is repeatable, as long as I start up with the same files
in "Mail/Local Folders".  I can make it go away by deleting the single file
Inbox.msf; removing the other .msf files don't prevent the hanging.  Once
Inbox.msf is removed and tbird restarted, everything seems to run fine.

My Inbox file isn't "huge" per se, but it is about 1.5 megs.  I have always
closed tbird normally - it hasn't crashed with open files that I know of.

I'm using tbird 0.9, from the binary install downloaded from Mozilla.org. 
System is SuSE Linux 9.1 Pro, on i386.

Wish I had something more helpful to say about the bug, but this is all I know.

Comment 2

[John Kondis]

(In reply to comment #1)

> My Inbox file isn't "huge" per se, but it is about 1.5 megs.  I have always
> closed tbird normally - it hasn't crashed with open files that I know of.

Ack!  My Inbox file isn't 1.5 megs, it's 15 megs.  Sorry.

Comment 3

[Dave Miller [:justdave]]

I just reproduced this trying to upgrade from 0.8 to 1.0.  I couldn't use 0.9
because of bug 268185 (but interestingly enough I didn't hit this bug when I
upgraded to 0.9).

I'm not convinced that it has anything to do with the size of the folder, as I
only have a couple folders that fit the definition of huge as described here,
and deleting the .msf file for those folders didn't help.  I finally just
deleted ALL .msf files in the Mail tree and then it worked.  I *do* have a
backup copy of the profile in question if anyone wants me to do any specific
experimenting on it.  Let me know.

Comment 4

[Dave Miller [:justdave]]

Switching os/platform to all/all.  I'm on Mac OS X.

Comment 5

[Dave Miller [:justdave]]

OK, one more profile nuke/backup restore determined that it was actually the
Inbox file on the first-listed account was the only .msf file it didn't like. 
Deleting just that one .msf file made everything run.

-rw-r--r--   1 dave  admin       32M  7 Dec 22:31 Inbox
-rw-r--r--   1 dave  admin      680K  7 Dec 22:33 Inbox.msf
drwxr-xr-x   2 dave  admin       68B  7 Dec 23:36 Inbox.sbd

Comment 6

[David :Bienvenu]

Dave, if you want to e-mail me your Inbox.msf, I can have a look at it.

Comment 7

[Dave Miller [:justdave]]

(In reply to comment #6)
> Dave, if you want to e-mail me your Inbox.msf, I can have a look at it.

hmm, quick look at it, it does appear to contain subject lines and so forth,
some of which do reveal information from a former employer that I can't share. 
Anything in particular I should look for if I try to dissect it myself?

Comment 8

[Antti Penttala]

I had this happen to me also when upgrading from 0.8 to 0.9. First I thought
that deleting the .msf file helped but after working for a couple of hours large
folders were slow again and CPU at 100%.

However, on my computer it was not Thunderbird that was using the processor
time. It was the (f-secure) virus scanner. For some reason, when using
Thunderbird 0.9, 1.0RC1 or 1.0, F-secure decides to constantly scan the complete
mail folder file which in my case was 400 MB. I think that this has changed
since version 0.8 when everything worked very fast even with a large mail
folder. Maybe the mail folder file is handled somehow differently now. I'm
running Win XP+SP2 and F-Secure 5.42.

Comment 9

[David :Bienvenu]

Attachment: fix for dave's problem

This should fix Dave's problem - as I suspected, the msg db stored invalid info
in the .msf file, such that a msg was its own parent. The check for that
invalid situation only worked the first time through the loop. The fix makes it
work for each time through the loop. This is unrelated to the issues other
folks are seeing, but I believe those are either fixed, or the result of virus
scanners.

Comment 10

[Patrick Reiter Horn]

I experienced this problem as well on Thunderbird 1.0, which was released
slightly before you made this patch. So when I just upgraded to 1.0.2, I was
suprised to find that this patch still hadn't put in, and I was forced to
compile from source to fix this problem.  It makes a few annoying beeps each
time the fix gets run (whenever I click on my Inbox), but that is a lot better
than crashing. :-)

Is there a specific reason that this patch didn't get applied into the main
tree?  I don't see how it could hurt to prevent an infinite loop through this
method.

It seems like there is something that causes this "parent==this" bug.  It isn't
just a corrupt mail index (It comes back after deleting the .msf file).  Is it
related to the size of my POP mailbox? (I keep all my mail on the server as a
backup ever since my ISP raised the limit to 100MB.)  Is the POP server sending
bad data?

I have never experienced this bug on windows, strangely enough, even though they
share mail files and pop servers. This does not cause the bug, since I
originally did not share mail, and only after it initially corrupted itself did
I share the mail between two OS's.

I have tried removing/renaming the .msf file for the folder with this bug a
number of times, and it continues to come back.

It appears to happen when a new message comes in automatically (I set it to one
minute) while I left another message open. (Note that it doesn't have to appear
in the inbox; merely checking for mail when a new mail is waiting is enough.)

While this bug happens, I notice total and unread message totals adding the
totals each time, so the number clains that there are 2,4,6,8,... unread
messages in the folder every time it checks for mail on the POP server.

When clicking away and back, I can be sure that it will freeze.

This stops happening for a while after resetting the .msf file, but it comes
back and the totals is always an indicator of the bug happening (It might follow
a pointer to itself when calculating totals.)

When this patch is applied, all symptoms occur, except that it doesn't freeze. 
Sometimes it goes slower than normal when changing folders.  But this patch
doesn't completely solve the bigger problem that causes this.

I don't know if this helps, but I now have a debug version, and I can get a
backtrace when it initially gets corrupted if you tell me where to break it.
I also have a backtrace of the infinite loop location, but since this patch
clearly fixes it, there is no reason to post it.

Until the problem that causes the parent in the .msf file to be set wrong is
fixed, I'd at least like to see your fix in the next release so that I don't
have to use the ugly/slow debug version forever.

Comment 11

[David :Bienvenu]

this is checked in on the trunk, in nightly builds; I just forgot to mark it
fixed. 1.02 is not the trunk - 1.02 just contains security fixes. I can ask for
approval for the next security spin, if we have one before 1.1, but I don't know
if approval will be granted, since we try to limit the security releases to
security fixes.

As to why this is happening, I'm not sure. How big is your inbox? Have you ever
compacted it? One possibility is that when the .msf file is regenerated, it's in
some sort of "pre-corrupt" state, and all it takes is adding another message to
a thread causes the problem. Does this happen to widely disparate threads, or
usually the same threads? Do you often undo deletes of messages? I could try to
figure out some code to generate an assertion when this happens, and catch it in
the act, instead of waiting until the view is re-opened to notice the problem.