1506613 - Crashing in MaildirStoreParser::TimerCallback on grouped folder view with maildir storage (archives folder, SIGINT)

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Marius Burkard]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0

Steps to reproduce:

I have a really big profile (about 25G of mails) and converted to maildir from mbox.
When now using the grouped folder view and open the archives main folder, Thunderbird crashes with a SIGINT.

Currently I use Thunderbird 64b2, but this also occured with Thunderbird 60.3.


Actual results:

Thunderbird receives a SIGINT.

Assertion failure: mRawPtr != nullptr (You can't dereference a NULL nsCOMPtr with operator->().), at /home/downloads/thunderbird/source/source/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:940

Thread 1 "thunderbird" received signal SIGSEGV, Segmentation fault.
0x00007fffe7b95aad in nsCOMPtr<nsIMsgDatabase>::operator-> (this=<optimized out>)
    at /home/downloads/thunderbird/source/source/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:939
939	    MOZ_ASSERT(mRawPtr != nullptr,
(gdb)  bt
#0  0x00007fffe7b95aad in nsCOMPtr<nsIMsgDatabase>::operator-> (this=<optimized out>)
    at /home/downloads/thunderbird/source/source/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:939
#1  0x00007fffe7dc0bf9 in MaildirStoreParser::TimerCallback (aTimer=<optimized out>, aClosure=0x7fffc7ad5b50)
    at /home/downloads/thunderbird/source/source/comm/mailnews/local/src/nsMsgMaildirStore.cpp:1251
#2  0x00007fffe7f7ea56 in nsTimerImpl::Fire (this=0x7fffc1ac0e20, aGeneration=1)
    at /home/downloads/thunderbird/source/source/xpcom/threads/nsTimerImpl.cpp:684
#3  0x00007fffe7f6881d in nsTimerEvent::Run (this=0x7fffdcdd9530)
    at /home/downloads/thunderbird/source/source/xpcom/threads/TimerThread.cpp:297
#4  0x00007fffe7f6dea9 in nsThread::ProcessNextEvent (this=0x7fffe0c29340, aMayWait=<optimized out>, 
    aResult=0x7fffffffc67f) at /home/downloads/thunderbird/source/source/xpcom/threads/nsThread.cpp:1246
#5  0x00007fffe7f6f8a3 in NS_ProcessNextEvent (aThread=0x7ffff7116520 <_IO_2_1_stderr_>, aMayWait=false)
    at /home/downloads/thunderbird/source/source/xpcom/threads/nsThreadUtils.cpp:530
#6  0x00007fffe84b868f in mozilla::ipc::MessagePump::Run (this=0x7fffe1cdaec0, aDelegate=0x7fffe1cc2040)
    at /home/downloads/thunderbird/source/source/ipc/glue/MessagePump.cpp:97
#7  0x00007fffe845dbed in MessageLoop::RunHandler (this=0x7fffe1cc2040)
    at /home/downloads/thunderbird/source/source/ipc/chromium/src/base/message_loop.cc:318
#8  MessageLoop::Run (this=0x7fffe1cc2040)
    at /home/downloads/thunderbird/source/source/ipc/chromium/src/base/message_loop.cc:298
#9  0x00007fffea495407 in nsBaseAppShell::Run (this=0x7fffe0ca5dd0)
    at /home/downloads/thunderbird/source/source/widget/nsBaseAppShell.cpp:158
#10 0x00007fffeb5ec781 in nsAppStartup::Run (this=0x7fffe1cd66a0)
    at /home/downloads/thunderbird/source/source/toolkit/components/startup/nsAppStartup.cpp:290
#11 0x00007fffeb6ab52d in XREMain::XRE_mainRun (this=0x7fffffffc988)
    at /home/downloads/thunderbird/source/source/toolkit/xre/nsAppRunner.cpp:4791
#12 0x00007fffeb6abe7e in XREMain::XRE_main (this=0x7fffffffc988, argc=<optimized out>, argv=<optimized out>, aConfig=...)
    at /home/downloads/thunderbird/source/source/toolkit/xre/nsAppRunner.cpp:4936
#13 0x00007fffeb6ac604 in XRE_main (argc=-149850288, argv=0x0, aConfig=...)
    at /home/downloads/thunderbird/source/source/toolkit/xre/nsAppRunner.cpp:5028
#14 0x0000555555565316 in do_main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at /home/downloads/thunderbird/source/source/comm/mail/app/nsMailApp.cpp:232
#15 main (argc=3, argv=<optimized out>, envp=0x7fffffffdcb8)
    at /home/downloads/thunderbird/source/source/comm/mail/app/nsMailApp.cpp:311
(gdb) quit
A debugging session is active.




Expected results:

Thunderbird should not crash.

I debugged this a bit and found that in file 
/home/downloads/thunderbird/source/source/comm/mailnews/local/src/nsMsgMaildirStore.cpp line 1251
the line
parser->m_db->SetSummaryValid(true);
is called but parser->m_db is NULL at that point.

I wrapped this line into
if(parser->m_db) { parser->m_db->SetSummaryValid(true);}
to temporarily solve the problem for me, but I cannot tell why it is NULL at that point.

I also added a print to STDOUT every time parser->m_db is null and would cause a SIGINT and my console is hammered by that outputs as soon as I open the archives folder.

Maybe this is related to bug id 578148, too?

Comment 1

[Marius Burkard]

The output to the console stops as soon as I change the folder, even if the new folder is a subfolder of "Archives". As soon as I switch back to the Archives main grouped folder, the output starts again.

Comment 2

[Marius Burkard]

Same occurs in 
/home/downloads/thunderbird/source/source/comm/mailnews/local/src/nsMsgMaildirStore.cpp line 1198:

  rv = m_db->CreateNewHdr(nsMsgKey_None, getter_AddRefs(newMsgHdr));

in method
nsresult MaildirStoreParser::ParseNextMessage(nsIFile *aFile)

and of course a few lines below
m_db->AddNewHdrToDB(newMsgHdr, true);


This is happening temporarily (could not find the root cause, yet) and m_db is NULL then and causing TB to SIGINT.

Comment 3

[Marius Burkard]

I have checked that it at least occurs in Sent folder (grouped main folder) and Archives (different SIGINT crashes, see above).
The crash in Sent folder group occurs when parsing maildir files from "Local Folder-maildir".
I could not see anything special in the mail files themselves, though.

Comment 4

[Marius Burkard]

Further maybe interesting notes:
When I open the different archive folders e.g. 2018-10 before opening the grouped view main archives folder, the first SIGINT goes away and the second one occurs instead. So maybe it has something to do with a missing index file for that grouped folder that causes the crashes.
To get rid of the second SIGINT I simply set a "return rv" in the method if m_db is null so it does not process the messages but does not crash, either.
Interesting: the second SIGINT went away as soon as all messages have been "parsed" (or not parsed) this way. This strongens the assumption that it has to do with index files.

Comment 5

[Marius Burkard]

It must have something to do with the msf files. When I open a crashing folder, let it run through with the 'return rv' "fix", then navigate away and back, the error is gone. As soon as I open the folder preferences and click "repair folder", the errors flood my console.

Comment 6

[Marius Burkard]

Currently the SIGINT crashes seem to only occur on everything related to "Local Folders" and "indexing (.msf)".
I have re-generated all .msf files for "Local Folders" and each subfolder using my modified developer version of TB (trapping the null m_db) and now the crashes are gone even on my TB 64b2 that I use normally. I can navigate to the previously crashing folders without problem and the .msf files for all other folders (non-local) are generated flawlessly.

Comment 7

[Wayne Mery (:wsmwk)]

Same signature as Bug 1317066 - Crash in MaildirStoreParser::TimerCallback

Comment 8

[Wayne Mery (:wsmwk)]

What is the sort column?

(It is speculated that bug 1317066 is fixed by bug 906469 which has a potential patch you can view)

Comment 9

[:aceman]

This still happens, I just hit it on trunk (69), thus it is not fixed by bug 906469. Notice that bug only touched one of the places, but the use of m_db in TimerCallback is still hitting null unchecked.

Comment 10

[:aceman]

Steps to reproduce:

1. have an account with maildir
2. delete .msf file of a folder, e.g. Inbox
3. start TB
4. go into Filter list
5. choose that account
6. select a filter that can run manually
7. run the filter on the folder in step 2 (e.g. Inbox)

Crash as m_db (the message database) is null.

With these steps, the constructor of MaildirStoreParser is called with a null aMsgDB, then
for each message in the folder (which still exist, only the .msf is deleted) parseNextMessage() is called, again with null m_db.
Then at last TimerCallback is called last ONCE for the folder, still with m_db being null.

Comment 11

[:aceman]

Attachment: 1506613.patch

With this patch the crash is avoided. I also am not sure if we should bail out completely or just skil setting the 'valid' flag on the database, but this seems to work for me. The filter run does nothing, but if you visit the folder by clicking it, then it rebuilds its .msf and all is fine. The filters then work on the folder. This is exactly what the reporter also tried and worked.

Comment 12

[Jorg K (CEST = GMT+2)]

Comment on attachment 9067494 [details] [diff] [review]
1506613.patch

Review of attachment 9067494 [details] [diff] [review]:
-----------------------------------------------------------------

You can have my drive-by OK if you want.

::: mailnews/local/src/nsMsgMaildirStore.cpp
@@ +1230,1 @@
>      //    store->SetSummaryFileValid(parser->m_folder, parser->m_db, true);

Please fix the ugly comment (remove spaces) while you're here.
Note: The // was at the start of the line and reformatting indented it maintaining the spaces :-(

Comment 13

[:aceman]

Thanks.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0820221247ce
Check parser_>m_db for null in MaildirStoreParser::TimerCallback to avoid crash. r=mkmelin

Comment 15

[Jorg K (CEST = GMT+2)]

Please keep Google formatting in mind for C++ patches: if (xxx) foo(xxx); - even if you don't like it.

Comment 16

[Jorg K (CEST = GMT+2)]

TB 68 beta:
https://hg.mozilla.org/releases/comm-beta/rev/67b44c4fe3fd89a245620394fac000587fa78735

Comment 17

[Jorg K (CEST = GMT+2)]

TB 60.7.1 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/a9f99cffa752340d368ddf90ba813c44c82eafce

Comment 18

[Wayne Mery (:wsmwk)]

No crashes in beta 68 and 60.7.1