406851 - When folders get compacted, Thunderbird crashes [@ nsFolderCompactState::EndCopy(nsISupports*, unsigned int)] and index gets corrupted when BitDefender is running

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Don Daniels]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
Build Identifier: version 2.0.0.9 (20071031)

Whenever this new version 2.0.0.9 (20071031) compacts my folders, I loose the majority of my e-mails. Fortunately I had a good backup, but even after re-installing that mail file and cleaning up spam and duplicates, it automatically started compacting and all my mail went away again. All that was left were the brand new downloads from probably after the compacting today. I loaded the mail file again from my backup, and only lost whatever I had downloaded since this morning, and sent my options to NOT automatically compact the folders until you find the bug and fix it. 

Yesterday, when it tried to compact the program crashed, sent a bug report (twice) and my mail folders were empty other then NEW mail retrieved after my crash. Today, it just started auto-compacting and crashed again, and when I checked one of my folders later it only contained about 20 notes, or just the time since it had compacted. This is getting annoying. I have to do backup's several times a day until we figure this out. 

I do have Bit Defender 2007 with it's spam filter which is new within the last couple of weeks, so it might be an interaction with that also.


Reproducible: Always

Steps to Reproduce:
1.Cleaning folders and reach auto-compact limit
2.Unable to manually initiate folder compaction.
3.
Actual Results:  
Make a backup of your mail file. (Seems to only trash my Inbox's in the two accounts where they are over 600 Mb.
Set your Thunderbird to auto-compact when over (5000Kb in my case).
Clean out enough junk to trigger the compact folder. Sometimes it started compacting when I had re-booted due to an update (There was something from Windows Live this morning that required a restart, I think it happened on the re-boot after that. 

Expected Results:  
There was a long compact as the bars slowly moved across the bottom, but the next time I looked in that folder about 3 years worth of e-mail was gone and I only had the last 10 minutes or so of e-mails. Perhaps it is somehow marking good mail as deleted and cleaning the whole file, especially if it crashes in the process. 

Come up with some way (option) to save the most recent version as a backup file so if the compact crashes you can re-install the older version. I am having to do several backup's a day until I figure this out.

I updated my BitDefender to version 2007, and it now integrates with Thunderbird with it's spam filter, which is very effective. However, it may be clashing with this version of Thunderbird.

Comment 1

[Magnus Melin [:mkmelin]]

Talkback crash ids could be useful; <http://kb.mozillazine.org/Talkback>

Comment 2

[Don Daniels]

Have had no further crashes or losses since turning off Auto-Compact. I have worked with the spam filter in BitDefender several times also, and it has not crashed. It seemed to crash in the middle of the file compaction, and upon restart the mail was missing. I'm not doing any file compacting unless I back up the Inbox files first.

You can download the new BitDefender 2007 Internet Security and use it free for 30 days for bug testing. 

Just backed up my Inbox's and tried a manual compact. Initially got a message that "The folder "Inbox" cannot be compacted because another operation is in progress. Please try again later." I recall seeing this or a similar message along with the crash report. I just finished a manual compact and it worked fine. I think the main difference is that I had not just downloaded a large number of files and thus BitDefender was not still processing spam when the compaction started.

I am wondering if BitDefender does not understand when a folder is compacting, and is trying to run it's spam filter at the same time, and that might be causing a problem. Conversely, Thunderbird might not recognize that BitDefender Internet Security is running spam filtering when it starts auto-compacting of my folders, and again that might cause a clash. Thunderbird probably started the auto-compact immediately after finishing the download, and BitDefender runs for a few seconds more as it identifies and moves the spam to the spam bucket under the trash folder. 

Comment 3

[Don Daniels]

(In reply to comment #1)
> Talkback crash ids could be useful; <http://kb.mozillazine.org/Talkback>
> 
Where would I find that information? The crashes occurred on Monday and Tuesday, Dec 3rd and 4th.

Comment 4

[Don Daniels]

(In reply to comment #3)
> (In reply to comment #1)
> > Talkback crash ids could be useful; <http://kb.mozillazine.org/Talkback>
> > 
> Where would I find that information? The crashes occurred on Monday and
> Tuesday, Dec 3rd and 4th.
> 
I found the Error Console. Is there an easy way to send that info and big files to you?

Comment 5

[Don Daniels]

(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #1)
> > > Talkback crash ids could be useful; <http://kb.mozillazine.org/Talkback>
> > > 
> > Where would I find that information? The crashes occurred on Monday and
> > Tuesday, Dec 3rd and 4th.
> > 
> I found the Error Console. Is there an easy way to send that info and big files
> to you?
> 

I just checked my BitDefender version information. 

BitDefender Internet Defender Internet Security 2008  Build 11.0.15

www.BitDefender.com

BitDefender Tech Support. http://www.bitdefender.com/site/KnowledgeBase/browseProducts/2195/

This version's page.

http://www.bitdefender.com/PRODUCT-2195-en--BitDefender-Internet-Security-2008.html

Hope this helps.

Comment 6

[Magnus Melin [:mkmelin]]

(In reply to comment #3)
> Where would I find that information? 

See http://kb.mozillazine.org/Talkback#Getting_an_incident_ID

Error Console *Errors* could also be helpful, if you get them when all extensions are disabled / in -safe-mode. Just paste them here.

Comment 7

[Don Daniels]

Found the Talkback crash ID's 
TB38773292Y, TB38772414Z, TB38712422Q, TB38712329H, TB38701619Z, TB38611573X, TB38611571G

Let me know if you still want the Error Console reports, and which parts you want.

Comment 8

[Don Daniels]

Here is the info from the Error Console:  (Let me know if you what the big text files related to the links. I can't find an easy way to copy all of that without making a very large text file.)

BDTB: bitdefender antispam toolbar v11.0.0.25
[onLoad]: loading bitdefender toolbar...
check for thunderbird window: yes
UniversalXPConnect: OK
messenger: <[xpconnect wrapped nsIMessenger]> : OK
create BD instance: <[xpconnect wrapped nsISupports]> : OK
get interface: <[xpconnect wrapped IBDThunderbird]> : OK
Initialize: ret code<0> : OK
GetSettings: code<2> : bMoveToTrash : OK

Error: [Exception... "'JavaScript component does not have a method named: "NotifyComposeBodyReady"' when calling method: [nsIMsgComposeStateListener::NotifyComposeBodyReady]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "JS frame :: chrome://messenger/content/messengercompose/MsgComposeCommands.js :: InitEditor :: line 3586"  data: no]
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 3586
add observer: OK

Error: gAccountManager has no properties
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 2384

Error: gAccountManager has no properties
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 2384

Error: gAccountManager has no properties
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 2384

Error: gAccountManager has no properties
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 2384

Error: [Exception... "'JavaScript component does not have a method named: "NotifyComposeBodyReady"' when calling method: [nsIMsgComposeStateListener::NotifyComposeBodyReady]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "JS frame :: chrome://messenger/content/messengercompose/MsgComposeCommands.js :: InitEditor :: line 3586"  data: no]
Source File: chrome://messenger/content/messengercompose/MsgComposeCommands.js
Line: 3586

There are many more.

Comment 9

[Magnus Melin [:mkmelin]]

TB38611571 seems to be the one where compact is involved

nsFolderCompactState::EndCopy  [mozilla/mailnews/base/src/nsMsgFolderCompactor.cpp, line 995]
nsMailboxProtocol::OnStopRequest  [mozilla/mailnews/local/src/nsMailboxProtocol.cpp, line 301]
nsInputStreamPump::OnStateStop  [mozilla/netwerk/base/src/nsInputStreamPump.cpp, line 564]
0x067f5360
nsMailboxProtocol::`vftable'
nsNotifyAddrListener::AddRef  [mozilla/netwerk/system/win32/nsNotifyAddrListener.cpp, line 272]

Leads to http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/mailnews/base/src/nsMsgFolderCompactor.cpp&mark=995&rev=MOZILLA_1_8_BRANCH#995

m_db is null?

Comment 10

[Don Daniels]

Big Clue!

I am finding that I generally don't really loose the data, it just has to rebuild the index of the open inbox after each crash, and with a 600MB file that takes a while.

I have also been suspecting that BitDefender antivirus was part of the problem. I just got a big clue on that. Thunderbird has been crashing so fast that often I could not get the index rebuilt and the mail downloaded before the next crash. 

This morning, Thunderbird was running normally for a considerable amount of time, and I was baffled about what was different. I was reading e-mail, downloading new notes, clicking on links, and it just kept running. Then, underneath that window I found a warning that BitDefender protection had been disabled somehow. Opened BitDefender, and the PC Security and Network Security shields were Red with problems, which when opened showed that the protection was turned off. I couldn't fix it at their control panel, so I had to reboot to re-enable BitDefender Anti-virus and Anti-Spam protection.

I started having these crash problems about the time BitDefender interfaced directly with Thunderbird for anti-spam (and about the time of the latest Thunderbird version upgrade), rather than BitDefender requiring manual Friends list maintenance. I don't think BitDefender has quite figured out how to play nice together with Thunderbird yet, and I especially suspect their Spam Filter and Thunderbird's Spam filter working at the same time may clash. I suggest you work with them to see if you can resolve the conflict. In the mean time, I will contact them and see if there is a way to turn off their spam filter (Which works very well)and see if that solves the crashes with the rest of their security features still running.

This is the version I am using now.
 http://www.bitdefender.com/PRODUCT-2195-en--BitDefender-Internet-Security-2008.html

You can download a free 30 day trial, and I'm sure they will work with you beyond that if necessary to resolve this problem.

Don Daniels

United States:

BitDefender LLC
6301 NW 5th Way, Suite 3500
Fort Lauderdale, Florida 33309

Phone: [phone]
Fax: [phone]

Comment 11

[Don Daniels]

Here is the current Talkback crash record list to help research the above note:
TB39277868M, TB39277670M, TB39277574Y, TB39277117W, TB39263596X, TB39263376M, TB39263364E, TB39263115W, TB39258826Q, TB39255372K, TB39223163Q, TB39222679G, TB39221563Y, TB39191641Y, TB39185440K, TB39184033X, TB39183658H, TB39163909K, TB39163337X, TB39162890W, TB39162604G, TB39162416E, TB39162221M, TB39162008Q, TB39161792Q, TB39161765G, TB39160897K, TB39160712M, TB39160514M, TB39086386Y, TB39086202X, TB39085949M, TB39085693Y, TB39085673E, TB39085652M, TB39085371G, TB39084990Z, TB39084805Z, TB39084789Q, TB39084750X, TB39084480G, TB39084108Y, TB39082635H, TB39082634Q, TB39081944Z, TB39079497H, TB39072168W, TB39071802G, TB39071382M, TB39055722G, TB39053998Z, TB39052955E, TB39052651Q, TB39052456Z, TB39052232M, TB39051980Z, TB38773292Y, TB38772414Z, TB38712422Q, TB38712329H, TB38701619Z, TB38611573X, TB38611571G

As you can see, this is getting old, but I think I have narrowed it down a lot.

Thanks,

Don

Comment 12

[timeless]

OK, I just checked all your stacks, here's the analysis (no counts, but sadly the mail related crashes listed at the top of this comment are dwarfed by gc crashes and of course "no stack"). I've included in the Incident ID line a reference where possible, the ones labeled timeless: are based on crashes that I experienced around 2004.

Incident ID: 38611573 (comment 9)
Stack Signature	nsFolderCompactState::EndCopy b3e4faf9
Product ID	Thunderbird2
Build ID	2007103104
Trigger Time	2007-11-29 19:30:55.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	thunderbird.exe + (0047198a)
URL visited	
User Comments	Compacting folders
Since Last Crash	115182 sec
Total Uptime	115182 sec
Trigger Reason	Access violation
Source File, Line No.	e:/builds/tinderbox/Tb-Mozilla1.8-Release/WINNT_5.0_Depend/mozilla/mailnews/base/src/nsMsgFolderCompactor.cpp, line 995
Stack Trace 	
nsFolderCompactState::EndCopy  [mozilla/mailnews/base/src/nsMsgFolderCompactor.cpp, line 995]
nsMailboxProtocol::OnStopRequest  [mozilla/mailnews/local/src/nsMailboxProtocol.cpp, line 301]
nsInputStreamPump::OnStateStop  [mozilla/netwerk/base/src/nsInputStreamPump.cpp, line 564]
0x067f5360
nsMailboxProtocol::`vftable'
nsNotifyAddrListener::AddRef  [mozilla/netwerk/system/win32/nsNotifyAddrListener.cpp, line 272]
0x8b560c45

Incident ID: 39184033 (bug 284876)
Stack Signature	nsMsgLocalMailFolder::WriteStartOfNewMessage b27f28b8
Product ID	Thunderbird2
Build ID	2007103104
Trigger Time	2007-12-15 14:44:27.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	thunderbird.exe + (005107a4)
URL visited	
User Comments	
Since Last Crash	569 sec
Total Uptime	264237 sec
Trigger Reason	Access violation
Source File, Line No.	e:/builds/tinderbox/Tb-Mozilla1.8-Release/WINNT_5.0_Depend/mozilla/mailnews/local/src/nsLocalMailFolder.cpp, line 2321
Stack Trace 	
nsMsgLocalMailFolder::WriteStartOfNewMessage  [mozilla/mailnews/local/src/nsLocalMailFolder.cpp, line 2321]
nsMsgLocalMailFolder::StartMessage  [mozilla/mailnews/local/src/nsLocalMailFolder.cpp, line 2871]
nsInputStreamPump::OnStateStop  [mozilla/netwerk/base/src/nsInputStreamPump.cpp, line 564]
0x06164670
nsMailboxProtocol::`vftable'
nsNotifyAddrListener::AddRef  [mozilla/netwerk/system/win32/nsNotifyAddrListener.cpp, line 272]
0x8b560c45

Incident ID: 39277117 timeless:CompositeDataSourceImpl::OnChange
Stack Signature	MSVCR80.dll + 0x149d1 (0x781449d1) a89f2f99
Product ID	Thunderbird2
Build ID	2007103104
Trigger Time	2007-12-18 07:37:46.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	MSVCR80.dll + (000149d1)
URL visited	
User Comments	
Since Last Crash	21 sec
Total Uptime	327432 sec
Trigger Reason	Access violation
Source File, Line No.	N/A
Stack Trace 	
MSVCR80.dll + 0x149d1 (0x781449d1)
XPCWrappedNative::CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappednative.cpp, line 2169]
XPC_WN_CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp, line 1455]
js_Invoke  [mozilla/js/src/jsinterp.c, line 1375]
js_Interpret  [mozilla/js/src/jsinterp.c, line 3946]
js_Invoke  [mozilla/js/src/jsinterp.c, line 1394]
nsXPCWrappedJSClass::CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappedjsclass.cpp, line 1453]
nsXPCWrappedJS::CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappedjs.cpp, line 468]
SharedStub  [mozilla/xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp, line 147]
CompositeDataSourceImpl::OnChange  [mozilla/rdf/base/src/nsCompositeDataSource.cpp, line 1514]

Incident ID: 39258826 (bug 320256)
Stack Signature	js_AllocStack 807a72b0

(bug 272286 isn't what I'm looking for)
Incident ID: 39222679 (timeless:nsXULElement::IsFocusable)
Stack Signature	 nsScriptSecurityManager::GetScriptPrincipal 8e447184
Product ID	Thunderbird2
Build ID	2007103104
Trigger Time	2007-12-16 19:44:19.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	thunderbird.exe + (000da16b)
URL visited	
User Comments	Cleaned up inbox at server with web-mail, and tried again
Since Last Crash	1552 sec
Total Uptime	287855 sec
Trigger Reason	Access violation
Source File, Line No.	e:/builds/tinderbox/Tb-Mozilla1.8-Release/WINNT_5.0_Depend/mozilla/caps/src/nsScriptSecurityManager.cpp, line 2029
Stack Trace 	
nsScriptSecurityManager::GetScriptPrincipal  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 2029]
nsScriptSecurityManager::GetPrincipalAndFrame  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 2138]
nsScriptSecurityManager::GetSubjectPrincipal  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 2180]
nsScriptSecurityManager::CheckPropertyAccessImpl  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 641]
nsScriptSecurityManager::CheckPropertyAccess  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 521]
nsScriptSecurityManager::CheckObjectAccess  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 505]
js_InternalGetOrSet  [mozilla/js/src/jsinterp.c, line 1536]
js_NativeGet  [mozilla/js/src/jsobj.c, line 3460]
js_GetProperty  [mozilla/js/src/jsobj.c, line 3607]
JS_GetProperty  [mozilla/js/src/jsapi.c, line 2850]
nsXPCWrappedJSClass::CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappedjsclass.cpp, line 1432]
nsXPCWrappedJS::CallMethod  [mozilla/js/src/xpconnect/src/xpcwrappedjs.cpp, line 468]
SharedStub  [mozilla/xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp, line 147]
nsXULElement::IsFocusable  [mozilla/content/xul/content/src/nsXULElement.cpp, line 645]

Incident ID: 39185440 (bug 281984)
Stack Signature	Detecting c8f723cb

Incident ID: 39084750 (timeless:nsContentUtils::CanCallerAccess)
Stack Signature	JS_GetFrameFunctionObject 4bff6e7d
Product ID	Thunderbird2
Build ID	2007103104
Trigger Time	2007-12-12 21:06:54.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	js3250.dll + (0000f919)
URL visited	
User Comments	Editing e-mail I was writing
Since Last Crash	1254 sec
Total Uptime	233169 sec
Trigger Reason	Access violation
Source File, Line No.	e:/builds/tinderbox/Tb-Mozilla1.8-Release/WINNT_5.0_Depend/mozilla/js/src/jsdbgapi.c, line 918
Stack Trace 	
JS_GetFrameFunctionObject  [mozilla/js/src/jsdbgapi.c, line 918]
nsScriptSecurityManager::GetPrincipalAndFrame  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 2138]
nsScriptSecurityManager::GetSubjectPrincipal  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 2180]
nsScriptSecurityManager::doGetSubjectPrincipal  [mozilla/caps/src/nsScriptSecurityManager.cpp, line 1778]
nsContentUtils::CanCallerAccess  [mozilla/content/base/src/nsContentUtils.cpp, line 669]

Comment 13

[Don Daniels]

I have not had a crash in over a month. I get a message that "the procedure has failed because it is already in use by another process", but no lockups. I think Thunderbird and BitDefender have learned to work together better. 

Comment 14

[Magnus Melin [:mkmelin]]

FIXED is only used when known code changes resolved the issue (WORKSFORME would have been appropriate). 
I think this is probably still valid though so let's keep it open -even though you haven't been exposed to it lately.

Comment 15

[Wayne Mery (:wsmwk)]

nsFolderCompactState::EndCopy is fairly high, #50 crasher, on talkback.

Don, have you seen this again in the last 5 months?

Comment 16

[Don Daniels]

(In reply to comment #15)
> nsFolderCompactState::EndCopy is fairly high, #50 crasher, on talkback.
> 
> Don, have you seen this again in the last 5 months?
> 
I have not seen the crash problem in quite some time. Actually, I think it was corrupting my index file, and if I didn't notice that the little bar at the bottom of the window was moving slowly indicating it was working (In this case rebuilding the index files) and I then shut down or quit the program, it caused further problems. I have not seen even that in some time. 

HOWEVER! I do still have a problem with the BDToolbar add-on to Thunderbird. When it is on, it causes Thunderbird to run VERY slowly, consuming 100% CPU almost continuously, slowing the whole computer to a crawl, and causing the CPU to heat up, turning the laptop fan to high speed until it is finished downloading the mail, which comes in one message every 3-4 seconds, while my other computers (like my wife's laptop with McAfee (I'm replacing that as soon as the subscription runs out)) will download the mail multiple messages per second unless it runs into a big file. I download the same batch of mail close to 100 times faster with Bitdefender turned off than when it is on, or on my Desktop with eEye's Blink security . 

When I Disable BDTool, things run a bit faster, it doesn't max out the CPU or crank the fan to high, but it still downloads the mail much slower than otherwise, 2-3 seconds per small e-mail. I can't seem to activate the option to un-install the BDTool in Thunderbird, so I am going to try installing the BETA of BitDefender 2009 and see if that is any better, but BitDefender and Thunderbird still don't seem to play well together, probably the two Anti-Spam modules are clashing.

Comment 17

[Wayne Mery (:wsmwk)]

Don, what did you find from comment 16?


this is topcrash  now for 2.0.0.18

I see one recent crash on trunk 3.0b2pre bp-4d74d38c-c794-436a-a0ad-29bad2090122
nsFolderCompactState::EndCopy	nsMsgFolderCompactor.cpp:1021
nsMailboxProtocol::OnStopRequest	nsMailboxProtocol.cpp:294
nsInputStreamPump::OnStateStop	netwerk/base/src/nsInputStreamPump.cpp:576
nsInputStreamPump::OnInputStreamReady	netwerk/base/src/nsInputStreamPump.cpp:401
nsOutputStreamReadyEvent::Run	xpcom/io/nsStreamUtils.cpp:111
nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:510
NS_ProcessNextEvent_P	nsThreadUtils.cpp:227
nsBaseAppShell::Run	widget/src/xpwidgets/nsBaseAppShell.cpp:170
nsAppStartup::Run	toolkit/components/startup/src/nsAppStartup.cpp:192
XRE_main	toolkit/xre/nsAppRunner.cpp:3279
NS_internal_main	nsMailApp.cpp:103
wmain	toolkit/xre/nsWindowsWMain.cpp:87
__tmainCRTStartup	crtexe.c:594
kernel32.dll@0x17066

Comment 18

[Wayne Mery (:wsmwk)]

:(  Don appears to be gone

in comment 16 he said "I am going to try installing the BETA
of BitDefender 2009 and see if that is any better, but BitDefender and
Thunderbird still don't seem to play well together, probably the two Anti-Spam
modules are clashing."

Comment 19

[Wayne Mery (:wsmwk)]

3.0b2 topcrash, so this should block 3.0.
hopefully, solution will also apply to 1.8.

example:
bp-4afa5b63-6694-437e-b4a9-7ce932090629
when marking SPAM email as JUNK; Mozilla-Eudora crashed
Frame	Module	Signature [Expand]	Source
0	eudora.exe	nsFolderCompactState::EndCopy	nsMsgFolderCompactor.cpp:1021
1	eudora.exe	nsMailboxProtocol::OnStopRequest	nsMailboxProtocol.cpp:294
2	eudora.exe	nsInputStreamPump::OnStateStop	netwerk/base/src/nsInputStreamPump.cpp:576
3	eudora.exe	nsInputStreamPump::OnInputStreamReady	netwerk/base/src/nsInputStreamPump.cpp:401
4	xpcom_core.dll	nsInputStreamReadyEvent::Run	xpcom/io/nsStreamUtils.cpp:111
5	xpcom_core.dll	nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:510
6	xpcom_core.dll	NS_ProcessNextEvent_P	nsThreadUtils.cpp:227
7	eudora.exe	nsBaseAppShell::Run	widget/src/xpwidgets/nsBaseAppShell.cpp:170
8	eudora.exe	nsAppStartup::Run	toolkit/components/startup/src/nsAppStartup.cpp:192
9	eudora.exe	XRE_main	toolkit/xre/nsAppRunner.cpp:3279
10	eudora.exe	NS_internal_main	nsMailApp.cpp:103
11	eudora.exe	wmain	toolkit/xre/nsWindowsWMain.cpp:87
12	eudora.exe	__tmainCRTStartup	crtexe.c:594 

plenty of comments http://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=exact&query=nsFolderCompactState%3A%3AEndCopy%28nsISupports*%2C%20unsigned%20int%29&date=&range_value=31&range_unit=days&do_query=1&signature=nsFolderCompactState%3A%3AEndCopy%28nsISupports*%2C%20unsigned%20int%29

Comment 20

[Kent James (:rkent)]

The folder compactor seems to routinely use a reference to the message database without checking that it is null, which is risky since in many cases we try our hardest to make it null in order to force closed databases.

Comment 21

[Wayne Mery (:wsmwk)]

I should have paid attention to version.  topcrash in 3.0b2, but seems gone from trunk except for 2009040800 crashes.

But it's common on 3.0b2 but rare in 3.0b2pre - so I'm not confident that it's gone. Indeed only 2 crashes in the 2 months leading up to 3.0b2 - 20090121031845 (bp-4d74d38c-c794-436a-a0ad-29bad2090122) and 20090121031845 (bp-4d74d38c-c794-436a-a0ad-29bad2090122)

leaving blocking? to get analysis, & needs more eyes than I have

Comment 22

[WADA:World Anti-bad-Duping Agency]

FYI.
I experienced crash once(I didn't send Crash report) during test of Bug 498185(compact silently fails, and .msf is corrupted). As seen in Bug 498814 Comment #5, current logic of "Compact Folder" doen't care for interfere by other program(by Tb's other task and by external program). If internal pointer somehow becomes ZERO due to interfere by others, crash can occur.
See also Bug 494706 for severe problem when interfered by external program.  
I think current coding is dangerous.

Comment 23

[WADA:World Anti-bad-Duping Agency]

I think already fixed Bug 392015(crash when interfered by rebuild-index) is sibling of this bug.

Comment 24

[David Ascher (:davida)]

accepting as a blocker because topcrasher of some kind.  May review later if the stats change.

Comment 25

[David :Bienvenu]

I wonder if this is happening in b4...

Comment 26

[Wayne Mery (:wsmwk)]

(In reply to comment #25)
> I wonder if this is happening in b4...

a lot in b2 (was b2 topcrasher on 2009-01-28), but only a few in b3 and none so far in b4. so topcrash-, unblocking and removing TM.

FWIW there appears to be a new, rare compact crash, - nsFolderCompactState::CleanupTempFilesAfterError()
http://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=contains&query=FolderCompact&date=&range_value=4&range_unit=weeks&do_query=1&signature=nsFolderCompactState%3A%3ACleanupTempFilesAfterError%28%29

Comment 27

[Wayne Mery (:wsmwk)]

this is the most common crash signature, no longer rare.

bp-2614796e-e128-4278-adc6-da4822110712 (ma28180) one of his crashes is "Crashed after a compact foldeers command"
EXCEPTION_ACCESS_VIOLATION_READ
0x0

0	xul.dll	nsFolderCompactState::EndCopy	mailnews/base/src/nsMsgFolderCompactor.cpp:1123
1	xul.dll	nsMailboxProtocol::OnStopRequest	mailnews/local/src/nsMailboxProtocol.cpp:286
2	xul.dll	nsInputStreamPump::OnStateStop	netwerk/base/src/nsInputStreamPump.cpp:578
3	xul.dll	nsInputStreamPump::OnInputStreamReady	netwerk/base/src/nsInputStreamPump.cpp:403

Comment 28

[David :Bienvenu]

I tried compacting with the mailbox file locked by a text editor - we handled it correctly, deleted the temp file, but left the original folder and db alone, and didn't crsah.

Comment 29

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #28)
> I tried compacting with the mailbox file locked by a text editor - we
> handled it correctly, deleted the temp file, but left the original folder
> and db alone, and didn't crsah.

must be something else then, but I haven't figured it out

Jerry from bp-2614796e-e128-4278-adc6-da4822110712 (comment 27) says he no longer sees the crash. but crashes still continue for current releases

Comment 30

[Wayne Mery (:wsmwk)]

nsFolderCompactState::EndCopy is Mac

Comment 31

[Wayne Mery (:wsmwk)]

will contact more crash reporters from https://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=contains&query=nsFolderCompactState&reason_type=contains&date=03%2F09%2F2012%2018%3A42%3A21&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&admin=1&signature=nsFolderCompactState%3A%3AEndCopy%28nsISupports*%2C%20unsigned%20int%29

#70 crash for version 10.0.2 (the highest rank of any compact signature)

Comment 32

[Wayne Mery (:wsmwk)]

crash is on line 1127 
1125    nsMsgKey key = m_startOfNewMsg > 0xFFFFFF00 ? nsMsgKey_None : (nsMsgKey) m_startOfNewMsg;
1126  m_db->CopyHdrFromExistingHdr(key, m_curSrcHdr, true,
1127 getter_AddRefs(newMsgHdr));

Comment 33

[Makoto Kato [:m_kato]]

Attachment: fix

Comment 34

[Makoto Kato [:m_kato]]

Should we get m_db again instead of?

Comment 35

[David :Bienvenu]

Comment on attachment 650759 [details] [diff] [review]
fix

you can try to get the db again, but there's no guarantee that you'll be able to.

Comment 36

[Hiroyuki Ikezoe (:hiro)]

Attachment: Fix for some of the crashes

I'd suggest we have to investigate how the crash happens before applying NULL check barrier.

I found a reason for some of the crashes and wrote a xpcshell test to reproduce the crash.

I do not think this patch fixes all of the crashes because the crash had reported before the problem in nsMsgFolderCompactor.cpp has been introduced by this commit http://hg.mozilla.org/comm-central/diff/097bc36f180d/mailnews/base/src/nsMsgFolderCompactor.cpp

Though we have to investigate other crash reasons, I hope this patch will reduce crash reports.

Note: The test in this patch depends on the fix for bug 782868 and the utility for bug 786936.

Comment 37

[Hiroyuki Ikezoe (:hiro)]

I should note the test might be crashing other places. One place I know is at the assertion in nsMailboxProtocol::SetupMessageExtraction(). See https://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsMailboxProtocol.cpp#435

Removing the assertion makes the crash in EndCopy at least on my local linux.

Comment 38

[David :Bienvenu]

Hiro, thx for the patch. Did you forget to hg add a file?

TEST-INFO | c:\builds\tbirdhq\obj-i686-pc-mingw32\mozilla\_tests\xpcshell\mailnews\base\test\unit\test_compactFailure.js | running test ...
TEST-UNEXPECTED-FAIL | c:\builds\tbirdhq\obj-i686-pc-mingw32\mozilla\_tests\xpcshell\mailnews\base\test\unit\test_compactFailure.js | test failed (with xpcshell return code: 3), see following log:
>>>>>>>
### XPCOM_MEM_LEAK_LOG defined -- logging leaks to c:\users\davidb~1\appdata\local\temp\tmpbz2zlk\runxpcshelltests_leaks.log
c:/builds/tbirdhq/obj-i686-pc-mingw32/mozilla/_tests/xpcshell/mailnews/base/test/unit/test_compactFailure.js:4: Error: cannot open file '../../../resources/testDoubleUtils.js' for reading
WARNING: nsExceptionService ignoring thread destruction after shutdown: file c:/builds/tbirdhq/mozilla/xpcom/base/nsExceptionService.cpp, line 166
WARNING: OOPDeinit() without successful OOPInit(): file c:/builds/tbirdhq/mozilla/toolkit/crashreporter/nsExceptionHandler.cpp, line 2219
nsStringStats
 => mAllocCount:           2973
 => mReallocCount:          427
 => mFreeCount:            2962  --  LEAKED 11 !!!
 => mShareCount:           9181
 => mAdoptCount:            109
 => mAdoptFreeCount:        109
<<<<<<<

== BloatView: ALL (cumulative) LEAK STATISTICS, default process 7016

     |<----------------Class--------------->|<-----Bytes------>|<----------------Objects---------------->|<--------------References-------------->|
                                              Per-Inst   Leaked    Total      Rem      Mean       StdDev     Total      Rem      Mean       StdDev
   0 TOTAL                                          26     2872     8627       73 (  475.57 +/-   577.96)    25296       59 ( 1607.12 +/-  2121.43)
   3 BackstagePass                                  24       24        1        1 (    1.00 +/-     0.00)      128        4 (   21.78 +/-    10.64)
   7 DirectoryProvider                              12       12        1        1 (    1.00 +/-     0.00)       12        1 (    2.35 +/-     0.78)
  28 Mutex                                          12       12       70        1 (   35.18 +/-    20.02)        0        0 (    0.00 +/-     0.00)
  51 XPCNativeScriptableShared                     232      232       87        1 (    8.75 +/-     2.65)        0        0 (    0.00 +/-     0.00)
  53 XPCWrappedNative                               52      312      110        6 (   56.47 +/-    31.01)      692        6 (   56.07 +/-    31.88)
  54 XPCWrappedNativeProto                          32       96       42        3 (   21.74 +/-    11.79)        0        0 (    0.00 +/-     0.00)
  56 mozHunspellDirProvider                         12       12        1        1 (    1.00 +/-     0.00)       12        1 (    2.35 +/-     0.78)
  58 nsAppFileLocationProvider                      16       16        1        1 (    1.00 +/-     0.00)       10        1 (    1.47 +/-     0.51)
  66 nsCStringKey                                   20      200       58       10 (    7.71 +/-     3.25)        0        0 (    0.00 +/-     0.00)
  70 nsConsoleService                               72       72        1        1 (    1.00 +/-     0.00)        9        1 (    2.65 +/-     1.17)
  77 nsDirectoryService                             68       68        1        1 (    1.00 +/-     0.00)       64        1 (    4.38 +/-     1.82)
  93 nsHashKey                                       8       80       64       10 (    8.96 +/-     4.02)        0        0 (    0.00 +/-     0.00)
  95 nsHashtable                                    44       44        7        1 (    3.77 +/-     1.96)        0        0 (    0.00 +/-     0.00)
 109 nsLocalFile                                    88     1056      531       12 (   57.39 +/-    47.72)     3648       12 (   62.26 +/-    44.55)
 110 nsMailDirProvider                              12       12        1        1 (    1.00 +/-     0.00)       12        1 (    2.35 +/-     0.78)
 118 nsObserverService                              48       48        1        1 (    1.00 +/-     0.00)       86        1 (    3.58 +/-     1.36)
 126 nsScriptError                                  96       96        1        1 (    1.00 +/-     0.00)        6        1 (    1.82 +/-     0.75)
 133 nsStringBuffer                                  8       88     3399       11 ( 1070.95 +/-   489.23)    12580       15 ( 3189.94 +/-  2013.97)
 138 nsSupportsArray                                56       56        1        1 (    1.00 +/-     0.00)        4        1 (    1.43 +/-     0.53)
 140 nsSystemPrincipal                              16       16        1        1 (    1.00 +/-     0.00)       77        1 (    5.55 +/-     1.31)
 161 nsXPCWrappedJS                                 64      192        5        3 (    3.14 +/-     1.35)       30        7 (    7.26 +/-     3.18)
 162 nsXPCWrappedJSClass                            44       88        4        2 (    2.50 +/-     1.05)       16        3 (    3.62 +/-     1.24)
 172 xptiInterfaceInfo                              20       40       64        2 (   20.77 +/-    10.31)      280        2 (   30.79 +/-    16.54)

Comment 39

[Hiroyuki Ikezoe (:hiro)]

Oops! I am really sorry for taking your time to waste on the review.

I should have requested the review after dependency bugs gets closed.

Comment 40

[Giorgio]

hi, are about five years that I submit this bug with the automatic "Mozilla crash report"...

well... I'm really tired.

is there a workaround, a manual fix? a divine hope of god hand?

seriously.... I'm tired now.

Comment 41

[Wayne Mery (:wsmwk)]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #39)
> I should have requested the review after dependency bugs gets closed.

hiro, what do you mean?  No bug is currently blocking this one. Do we need to add one?

Comment 42

[WADA:World Anti-bad-Duping Agency]

FTI.

Bug 498814 is for "corruption of Mbox" and/or "Deletion of mail store file" by Compact due to interfere by other software.
As written in that bug, Tb went ahead even when "write open of mail folder file for Compact" failed due to "read or write open of same file by other software". This kind of problem perhaps puts ZERO/neative-value/garbage in file handle, pointer etc. So, crash may occur in special situation.
And, as reported in Bug 674742, nsFolderCompactState::EndCopy didn't process error return code while Compact well, and Tb might delete original mail folder file in worst case.

Recently, Bug 674742 has been fixed(fixed by Tb 18.0), and error return code is now correctly processed. It may resolve this bug's crash problem.

Comment 43

[Wayne Mery (:wsmwk)]

starting in TB19, it the signature may have changed to nsFolderCompactState::EndCopy(nsISupports*, tag_nsresult) eg. bp-0ef06260-0395-40de-93c6-1a2162130125

bp-1e255c9d-794a-4e6f-9b0a-aa7bf2130118 (admin) is a frequent crasher

Comment 44

[Scoobidiver (away)]

The topcrash- keyword is not actively maintained and pollutes queries with topcrash.

Comment 45

[Wayne Mery (:wsmwk)]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #36)
> Created attachment 656767 [details] [diff] [review]
> Fix for some of the crashes
> 
> I'd suggest we have to investigate how the crash happens before applying
> NULL check barrier.
> 
> I found a reason for some of the crashes and wrote a xpcshell test to
> reproduce the crash.
> 
> I do not think this patch fixes all of the crashes because the crash had
> reported before the problem in nsMsgFolderCompactor.cpp has been introduced
> by this commit
> http://hg.mozilla.org/comm-central/diff/097bc36f180d/mailnews/base/src/
> nsMsgFolderCompactor.cpp
> 
> Though we have to investigate other crash reasons, I hope this patch will
> reduce crash reports.
> 
> Note: The test in this patch depends on the fix for bug 782868 and the
> utility for bug 786936.

So these bugs are blockers to this?

Is topcrash bug 673904 related?

Comment 46

[Wayne Mery (:wsmwk)]

definitely new signature in TB24
- to nsFolderCompactState::EndCopy(nsISupports*, tag_nsresult) version 24 at roughly same crash rate as TB17
- from nsFolderCompactState::EndCopy(nsISupports*, unsigned int) version 17

Perhaps we should throw up a try build and see what sticks. Do patches still apply?

Comment 47

[Hiroyuki Ikezoe (:hiro)]

I will rewrite Attachment #656767 [details] [diff] once bug 786936 has been closed.

Comment 48

[Hiroyuki Ikezoe (:hiro)]

Attachment: Fix some of crashes v2

Rewrite xpcshell test with MockFactory.

Try server results:
https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=df68e02b4d35

Comment 49

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I see this crash all the time with the latest official Thunderbird release on my brothers machine. Here a crash report: bp-c191ddef-c2b2-4cf3-8942-246c32140104. He is not running BitDefender but Avira. So any kind of antivirus could most likely cause this? Is there an option to prevent it? At least disabling the live scanner of Avira doesn't solve it.

Comment 50

[Wayne Mery (:wsmwk)]

(In reply to Henrik Skupin (:whimboo) from comment #49)
> So any kind of antivirus could most likely cause this? Is there an
> option to prevent it? At least disabling the live scanner of Avira doesn't
> solve it.

No one has identified a workaround. However, I suggest that Avira or some other software may still be involved, despite the live scanner being "disabled".

Note also this occurs on Mac, so it's clearly not 100% dependent on having AV installed.

Perhaps he'd be a good tester for the patch once it lands. How often does he crash?

Comment 51

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This crashed happened on the WinXP machine of my brother. We are going to kill this machine soon, so no idea how long I would be able to help out. At least the crash happened each time I tried to compact all folders for an POP3 account. The next time I can have a look at is next week on Friday as earliest.

Comment 52

[Mark Banner (:standard8)]

Comment on attachment 8356973 [details] [diff] [review]
Fix some of crashes v2

Review of attachment 8356973 [details] [diff] [review]:
-----------------------------------------------------------------

Ok, looks good, thanks for the patch.

Just to note, I definitely want this to ride the trains and not be pushed out early.

Comment 53

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/4cde42d4a1df