536960 - crash [@ sqlite3DbMallocRaw]

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Wayne Mery (:wsmwk)]

crash [@ sqlite3DbMallocRaw]

topcrash for 3.0.0. 

doesn't appear in 3.0b4 or nightlies prior to 20091224 (only in 3.0 and 3.0.1pre) so this may be a regression ... from  Bug 523405 / bug 525539 (which landed on 1.9.1 branch 2009-11-09)

nightly build crashes 
https://crash-stats.mozilla.com/report/list?product=Thunderbird&branch=1.9.1&branch=1.9.2&branch=1.9.3&version=Thunderbird%3A3.0.1pre&version=Thunderbird%3A3.0a3&version=Thunderbird%3A3.0b1pre&version=Thunderbird%3A3.0b2pre&version=Thunderbird%3A3.0b3pre&version=Thunderbird%3A3.0b4pre&version=Thunderbird%3A3.0pre&version=Thunderbird%3A3.1a1pre&query_search=signature&query_type=exact&query=sqlite3DbMallocRaw&date=&range_value=4&range_unit=weeks&do_query=1&signature=sqlite3DbMallocRaw&page=1
2009122400
2009112500 bp-dc24fb39-9e37-4631-8959-33b792091125
2009121400
2009120800
2009120600
2009120300
2009120200
2009112900


bp-dc24fb39-9e37-4631-8959-33b792091125
0	sqlite3.dll	sqlite3DbMallocRaw	 db/sqlite3/src/sqlite3.c:16236
1	sqlite3.dll	sqlite3SrcListAppend	db/sqlite3/src/sqlite3.c:69077
2	sqlite3.dll	sqlite3SrcListAppendFromTerm	db/sqlite3/src/sqlite3.c:69172
3	sqlite3.dll	yy_reduce	db/sqlite3/src/sqlite3.c:92563
4	sqlite3.dll	sqlite3Parser	db/sqlite3/src/sqlite3.c:93435
5	sqlite3.dll	sqlite3RunParser	db/sqlite3/src/sqlite3.c:94269
6	sqlite3.dll	sqlite3Prepare	db/sqlite3/src/sqlite3.c:78423
7	sqlite3.dll	sqlite3LockAndPrepare	db/sqlite3/src/sqlite3.c:78522
8	sqlite3.dll	segdir_max_index	db/sqlite3/src/sqlite3.c:99937
9	sqlite3.dll	writeZeroSegment	db/sqlite3/src/sqlite3.c:103621
10	sqlite3.dll	fulltextSync	db/sqlite3/src/sqlite3.c:103787
11	sqlite3.dll	sqlite3VtabSync	db/sqlite3/src/sqlite3.c:85926
12	sqlite3.dll	vdbeCommit	db/sqlite3/src/sqlite3.c:48801
13	sqlite3.dll	sqlite3VdbeHalt	db/sqlite3/src/sqlite3.c:49258
14	sqlite3.dll	sqlite3VdbeExec	db/sqlite3/src/sqlite3.c:54716
15	mozcrt19.dll	arena_dalloc_small	objdir-tb/mozilla/memory/jemalloc/src/jemalloc.c:4442
16	sqlite3.dll	sqlite3Step	db/sqlite3/src/sqlite3.c:50609
17	nspr4.dll	PR_Lock	
18	thunderbird.exe	mozilla::storage::AsyncExecuteStatements::executeStatement	storage/src/mozStorageAsyncStatementExecution.cpp:354

Comment 1

[Wayne Mery (:wsmwk)]

#7 crash for Thunderbird 3.0.0, 2.2% of crashes.

oh, and there are still FF crashes for sqlite3DBMallocRaw despite the checkin

Comment 2

[Wayne Mery (:wsmwk)]

~50 crashes per day for 3.0.0. one email addresses in ~1,350 crashes

Comment 3

[Mark Banner (:standard8)]

This sounds like something we need to have for 3.0.x. Assigning to asuth so he can take a look in the first instance.

Comment 4

[Mark Banner (:standard8)]

(In reply to comment #0)
> doesn't appear in 3.0b4 or nightlies prior to 20091224 (only in 3.0 and
> 3.0.1pre) so this may be a regression ... from  Bug 523405 / bug 525539 (which
> landed on 1.9.1 branch 2009-11-09)

That may be a mis-guidance - we locked builds on gecko 1.9.1.5 for quite a while over the RCs. We've now moved them forward to gecko 1.9.1.7.

Whether or not bug 523405 was included in asuth's sqlite patch that we added for gloda, I don't know, but it is worth bearing in mind that this could already be fixed.

Comment 5

[Mark Banner (:standard8)]

Ok, so we've had at least one crash since we swapped to 1.9.1.7:

bp-6ad1e90f-a7e9-4e0a-8475-dc1602091225

Comment 6

[Wayne Mery (:wsmwk)]

hmm, and hard to tell from https://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=exact&query=sqlite3DbMallocRaw&date=&range_value=4&range_unit=weeks&do_query=1&signature=sqlite3DbMallocRaw&page=1 whether this has diminished any.

will help to have users experiencing this crash to test 3.0.1pre ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.1/ 

will try pinging a few.

Comment 7

[Wayne Mery (:wsmwk)]

PMed 3 crash reporters in last 10 days. no Cc: so far :(

Comment 8

[Randy Smith]

I was contacted and asked to run a nightly build and attempt to reproduce this crash.  I have been running a nightly build for about a week and have been unable to create this crash.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Randy, thank you for trying out the nightly!  A few questions:

1) What specific revision nightly did you test with?  If you go to the Help menu and choose About, the last 2-3 lines in the box provide detailed version information and can be selected and then copied and pasting (via hot-key or context-menu).

I am mainly interested because we have 2 or 3 different sets of nightlies going on right now.

2) What version of Thunderbird were you using previously? 3.0?

3) How reliably would it crash for you previously (on the old version)?

Comment 10

[Randy Smith]

1)  I'm not sure of the original nightly build number any more, but I also ran 3.0.1pre and 3.0.2pre and I am currently running 3.0.1 release.  None of those versions have generated this crash.

2) When the crash occurred, I was running 3.0 release.

3) I only had this crash occur once.  At the time it crashed, I was being a bit impatient with it.  I checked a mailbox that had about 100 new messages.  Then, while it was indexing those, I deleted them.  Then I closed the program while the indexing was still occurring.  I've tried that same sequence with the newer versions but I have not been able to reproduce this crash.

Since I've only been able to create the crash one time, I don't know how helpful any of this is, but hopefully it helps at least a little...

Thanks!
Randy

Comment 11

[Mark Banner (:standard8)]

For the record, this is still happening in 3.0.2/3.0.3 as can be seen via crash-stats:

https://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=exact&query=sqlite3DbMallocRaw&date=&range_value=4&range_unit=weeks&do_query=1&signature=sqlite3DbMallocRaw&page=1

There's quite a few comments there that may or may not help.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

All of our 3.0.2 top-crashers look like they have to do with memory corruption, either victims of it or causing it.  I'd be more inclined to suspect the MailNews stuff already on the list for causing this.  For example, the "nsImapMailCopyState::`scalar deleting destructor''(unsigned int)" crash is more likely to be the source of all our problems...

https://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=nsImapMailCopyState%3A%3A%60scalar%20deleting%20destructor%27%27%28unsigned%20int%29&version=Thunderbird%3A3.0.2

Of course, it could also just be a victim.  We probably would do well to write a mozmill test that just tools around Thunderbird doing stuff and run that under valgrind.

Comment 13

[Wayne Mery (:wsmwk)]

perhaps a victim, but one thing that bothers me that a high percentage of these crashes (25-50%) appear to be during startup, and perhaps not happening as a result of some user action/interaction. (unfortuantely our list of crashes identified as being related to startup may not be very illuminating [1]) 

two interesting crash comments in the past month:
* Long boot up time, over 20 minutes) ever since I installed the latest Thunderbird.  bp-6e9555ca-b0ee-44dd-9ead-49a562100320 
* This was first run after installation, and cancel of [account] wizard bp-dec8ff6c-5574-449d-8b70-2b8f12100322 (strikes me as not being related to other common crashes)

Unfortunately there are very few comments in the crashes and even fewer emailaddresses (in fact none for the past month). I did a spot check but didn't find any crashes without comments that have email addresses.

[1] startup crashes https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%20&query_format=advanced&keywords_type=allwords&short_desc=startup%20starting&short_desc_type=anywordssubstr&type0-0-0=nowords&value0-0-0=count%20counts&resolution=---&product=MailNews%20Core&product=Thunderbird

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #13)
> perhaps a victim, but one thing that bothers me that a high percentage of these
> crashes (25-50%) appear to be during startup, and perhaps not happening as a
> result of some user action/interaction. (unfortuantely our list of crashes
> identified as being related to startup may not be very illuminating [1]) 

Gloda is most active at startup because of the initial indexing sweep.  Since it likes to open every folder that had any activity since the last indexing sweep[1], this results in a lot of mailnews activity too.

1: When gloda gets a notification that something has happened to a message and its event-driven processing queue has not overflowed, it marks the folder as dirty.  If it successfully processes the message it does not mark the folder as clean, even though with sufficient tracking and processing we could assert that it is.  (It is important that the folder be marked as dirty at least until we process the message; otherwise if Thunderbird crashed we might never go in and (re)index that message).

Comment 15

[Wayne Mery (:wsmwk)]

Unfortunately still precious few reporters to contact (noted below).  

These crashes run the gamut from 1 second to a couple days of uptime. Some even on shutdown.  Also, not as many crash comments compared to most crash sigs, which may indicate a) it doesn't repeat on people b) it just hits people out of the blue.  I'm not sure what that tells us about what it is or is not.


the shortest of the stacks is 
bp-9033de76-29c0-42bb-b2cd-ba7a22100330 (welland...)
0	sqlite3.dll	sqlite3DbMallocRaw	 db/sqlite3/src/sqlite3.c:16147
1	sqlite3.dll	sqlite3DbRealloc	db/sqlite3/src/sqlite3.c:16177
2	sqlite3.dll	sqlite3ExprListAppend	db/sqlite3/src/sqlite3.c:61222
3	sqlite3.dll	yy_reduce	db/sqlite3/src/sqlite3.c:93028
4	sqlite3.dll	sqlite3Parser	db/sqlite3/src/sqlite3.c:93930
5	sqlite3.dll	sqlite3RunParser	db/sqlite3/src/sqlite3.c:94756
6	sqlite3.dll	sqlite3Prepare	db/sqlite3/src/sqlite3.c:78728
7	sqlite3.dll	sqlite3LockAndPrepare	db/sqlite3/src/sqlite3.c:78827
8	sqlite3.dll	sqlite3_prepare_v2	db/sqlite3/src/sqlite3.c:78902
9	thunderbird.exe	mozilla::storage::Statement::getAsyncStatement	storage/src/mozStorageStatement.cpp:328
10	thunderbird.exe	mozilla::storage::StatementData::getSqliteStatement	storage/src/mozStorageStatementData.h:80
11	thunderbird.exe	mozilla::storage::AsyncExecuteStatements::executeAndProcessStatement	storage/src/mozStorageAsyncStatementExecution.cpp:285
12	thunderbird.exe	mozilla::storage::AsyncExecuteStatements::Run	storage/src/mozStorageAsyncStatementExecution.cpp:579
13	xpcom_core.dll	nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:521 


bp-1a535136-5aa4-4a70-ae28-ec44e2100330 (craig)
On opening TB update confirmation dialog appeared. I selected the 'later' option, intending to backup first. On attempting to close TB, update download launched (!?!). After download complete selected 'restart TB' option. This Crash Reporter dialog then appeared.


1 second, one of the shortest uptimes
bp-b18ca0d1-9c63-4303-aae4-44ce32100405

Comment 16

[Wayne Mery (:wsmwk)]

also contacting 
bp-eb20cb1e-92c9-45aa-b6c9-a933b2100310  (takaaki on Mac)
bp-75d2bf45-f8a0-4aa4-96f1-190f22100404  (aesqe on windows 3.1b1)

Comment 17

[Wayne Mery (:wsmwk)]

bienvenu, an early comment from andrew, this "is likely Thunderbird corrupting memory somewhere.  For example, one of bienvenu's recent libmime fixes."

I found a "first crash" in 3.0b4 20090915181920 bp-087ef1c6-f1c0-4d8a-a429-667912091015, so I was incorrect in comment 0.

Comment 18

[Craig]

Greetings. I'm the "craig" of the crash report quoted above. I cannot really provide much more commentary than that already quoted. Basically, as I recall, on attempting TBird launch an update download confirmation dialog appeared. I selected the 'later' option, intending to close and MozBack before updating. As I recall, update download launched anyway, so I awaited completion. After completion I selected the restart TBird option, interacted with Crash Reporter, and awaited TB restart. TB restarted with no problem. Let me know if any other contextual information is needed (system config or whatever). Thanks for working to improve TBird.

Comment 19

[Wayne Mery (:wsmwk)]

3.1 has but 3 crashes in 2 weeks.
3.1.2 so far has none.
WFM per crash-stats