1192778 - networking hang? crash in shutdownhang | libsystem_kernel.dylib@0x16136 [Mac OS 10.10]

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking, defect)

Description

[Wayne Mery (:wsmwk)]

Thunderbird 38.1.0 #1 Mac crash, #17 overall (no windows crashes)
Some comments suggest hanging on network issues

report bp-90c08e30-3e54-430a-824f-aa63e2150803
 0 	XUL	mozilla::(anonymous namespace)::RunWatchdog(void*)	toolkit/components/terminator/nsTerminator.cpp
1 	libnss3.dylib	_pt_root	nsprpub/pr/src/pthreads/ptthread.c
2 	libsystem_pthread.dylib	_pthread_body	
3 	libsystem_pthread.dylib	_pthread_start	
4 	libsystem_pthread.dylib	thread_start	
5 	libnss3.dylib	libnss3.dylib@0x20255f

Comment 1

[Wayne Mery (:wsmwk)]

I'm guessing related to bug 1197214

Comment 2

[Chris Peterson [:cpeterson]]

I hit this shutdown crash in Firefox Aurora 42:

bp-641dcde0-7ced-46f5-96c9-598b62150905

Comment 3

[Minh Nguyễn]

Via bug 1199957, I believe this is related to Spotlight indexing.

Comment 4

[Wayne Mery (:wsmwk)]

Nomis, are any stacks [1] useful in determining the cause? Or do we need more info from reporters?

Reporter of bp-03d68f13-e990-4c74-bedf-5681e2160525 is notable of also having this crash signature nsStreamConverter::OnDataAvailable bp-2bb4c0f1-5e54-44ed-be6c-c56dd2160527

[1] https://crash-stats.mozilla.com/signature/?product=Thunderbird&date=%3E2016-06-01&signature=shutdownhang%20%7C%20libsystem_kernel.dylib%400x16136#comments


(In reply to Minh Nguyễn from comment #3)
> Via bug 1199957, I believe this is related to Spotlight indexing.
From bug 1199957 comment 5
"The CPU spike, hang, and subsequent crash on quit is much less common with Spotlight indexing turned off. I noticed that there’s always a stalled indexing activity in the Activity Manager, under the account “null”. Judging from the number of total messages to index, the indexing activity is from a different account each time. I do have an IMAP account with an absurdly large number of messages in one folder (on the order of tens of thousands). However, this folder wasn’t causing problems until recently."

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Minh Nguyễn from comment #3)
> Via bug 1199957, I believe this is related to Spotlight indexing.

Or perhaps mail summaries addon?

Comment 6

[Nomis101]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #4)
> Nomis, are any stacks [1] useful in determining the cause? Or do we need
> more info from reporters?
> 
> Reporter of bp-03d68f13-e990-4c74-bedf-5681e2160525 is notable of also
> having this crash signature nsStreamConverter::OnDataAvailable
> bp-2bb4c0f1-5e54-44ed-be6c-c56dd2160527
> 
> [1]
> https://crash-stats.mozilla.com/signature/?product=Thunderbird&date=%3E2016-
> 06-01&signature=shutdownhang%20%7C%20libsystem_kernel.
> dylib%400x16136#comments
> 
> 
> (In reply to Minh Nguyễn from comment #3)
> > Via bug 1199957, I believe this is related to Spotlight indexing.
> From bug 1199957 comment 5
> "The CPU spike, hang, and subsequent crash on quit is much less common with
> Spotlight indexing turned off. I noticed that there’s always a stalled
> indexing activity in the Activity Manager, under the account “null”. Judging
> from the number of total messages to index, the indexing activity is from a
> different account each time. I do have an IMAP account with an absurdly
> large number of messages in one folder (on the order of tens of thousands).
> However, this folder wasn’t causing problems until recently."
More informations are always good, maybe somebody with a bit more time can find similarities in the stacks that could help, I couldn't in my quick look.
But nsStreamConverter::OnDataAvailable seems like its related to the part in nsStreamConverter.cpp which is 'called whenver data is written into the input stream by the networking library'.

Comment 7

[Minh Nguyễn]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> (In reply to Minh Nguyễn from comment #3)
> > Via bug 1199957, I believe this is related to Spotlight indexing.
> 
> Or perhaps mail summaries addon?

Yes, that’s entirely possible. I had forgotten that the add-on was installed. (Came to think of it as a built-in part of Thunderbird! :-D)

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to Minh Nguyễn from comment #7)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > (In reply to Minh Nguyễn from comment #3)
> > > Via bug 1199957, I believe this is related to Spotlight indexing.
> > 
> > Or perhaps mail summaries addon?
> 
> Yes, that’s entirely possible. I had forgotten that the add-on was
> installed. (Came to think of it as a built-in part of Thunderbird! :-D)

Can you confirm this theory with your testing?

Comment 9

[Wayne Mery (:wsmwk)]

Adam, you've had a couple crashes, like bp-73e4995a-86f9-4641-83dd-a81820170912.  Do you have spotlight enabled?

Comment 10

[Minh Nguyễn]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #8)
> (In reply to Minh Nguyễn from comment #7)
> > (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > > (In reply to Minh Nguyễn from comment #3)
> > > > Via bug 1199957, I believe this is related to Spotlight indexing.
> > > 
> > > Or perhaps mail summaries addon?
> > 
> > Yes, that’s entirely possible. I had forgotten that the add-on was
> > installed. (Came to think of it as a built-in part of Thunderbird! :-D)
> 
> Can you confirm this theory with your testing?

Belated update: after that comment, I disabled Mail Summaries and haven’t seen the crashes since.

Comment 11

[Adam Roach [:abr]]

(In reply to Wayne Mery (:wsmwk) from comment #9)
> Adam, you've had a couple crashes, like
> bp-73e4995a-86f9-4641-83dd-a81820170912.  Do you have spotlight enabled?

Yes, I have "Allow Spotlight to search messages" checked. I do _not_ have the "Mail Summaries" add-on installed.

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Adam Roach [:abr] from comment #11)
> (In reply to Wayne Mery (:wsmwk) from comment #9)
> > Adam, you've had a couple crashes, like
> > bp-73e4995a-86f9-4641-83dd-a81820170912.  Do you have spotlight enabled?
> 
> Yes, I have "Allow Spotlight to search messages" checked. I do _not_ have
> the "Mail Summaries" add-on installed.

Adam, have you learned anything in the past year?

Crash rate has gone down slightly in the past 5 months but not great progress.

Comment 13

[Adam Roach [:abr]]

I haven't really learned much about this problem. I seem to crash less during normal, everyday use, although I'm having a lot of shutdown crashes now. It's probably not related.

Sorry; I wish I had something more substantive to report.

Comment 14

[Adam Roach [:abr]]

Sorry, I got confused about which bug this was. :)

I'm still seeing a lot of shutdown crashes, but they're in XPCOM shutdown rather than apparently being related to networking. They're probably unrelated, but in case there's some connection, here are some typical stacks:

bp-f52b9d82-2171-483d-876c-9b1930180910
bp-236f41a9-3df4-47c0-9320-460750180908
bp-d0b1815d-3438-4a9d-b371-a5cf70180908
bp-fc49d097-50e6-498f-aac6-e5c4c0180908

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to Adam Roach [:abr] from comment #14)
> Sorry, I got confused about which bug this was. :)
> 
> I'm still seeing a lot of shutdown crashes, but they're in XPCOM shutdown
> rather than apparently being related to networking. They're probably
> unrelated, but in case there's some connection, here are some typical stacks:
> 
> bp-f52b9d82-2171-483d-876c-9b1930180910
> bp-236f41a9-3df4-47c0-9320-460750180908
> bp-d0b1815d-3438-4a9d-b371-a5cf70180908
> bp-fc49d097-50e6-498f-aac6-e5c4c0180908

These point to bug 1461731 (which should be in  NSS 3.38 Firefox 62, NSS 3.37.2 for Firefox 61, NSS 3.36.3 for Firefox 60 ESR and potentially 60.0.2 - and corresponding versions of Thunderbird?) ... except that bug 1461731 was a more recent regression. Still, one question is whether your original report (all OSX 10.10) has a cause related to NSS issues - if yes then your original issue morphed to shutdownhang | libsystem_kernel.dylib@0x1ca1e

What happens for you with newer version 60.3.0?  Or a current beta?

Comment 16

[Wayne Mery (:wsmwk)]

Adam,  What happens for you with newer version 60.3.0?  Or a current beta?

Comment 17

[Adam Roach [:abr]]

(In reply to Wayne Mery (:wsmwk) from comment #16)
> Adam,  What happens for you with newer version 60.3.0?  Or a current beta?

The problem is resolved under 60.3.0. Thanks!

Comment 18

[Wayne Mery (:wsmwk)]

Thanks for the report. Unfortunately, still plenty of crashes and no perceptible change in the graph.

Comment 19

[Wayne Mery (:wsmwk)]

but no longer a topcrash

Comment 20

[Wayne Mery (:wsmwk)]

Oct 26 the crash rate dropped basically to 1-2 per day [1]. Comparing the previous months period [2] it would seem this reduction is across all versions of Thunderbird (at least 60 and 68 - the majority being 60.9.0). I don't see anything on my relman folder that we changed anything signficant in your release channels. Either tons of users moved off of MacOS 10.10 or this morphed to something else. Oddly enough [3] shutdownhang | __psynch_cvwait | _pthread_cond_wait | mozilla::detail::MutexImpl::unlock | pthread_cond_signal_thread_np comes into existence at that exact time - for both 10.10 and 10.11.

[1] Oct 17 - Nov 17 https://crash-stats.mozilla.org/signature/?product=Thunderbird&signature=shutdownhang%20%7C%20libsystem_kernel.dylib%400x16136&date=%3E%3D2019-10-17T02%3A38%3A00.000Z&date=%3C2019-11-17T02%3A38%3A00.000Z#graphs

[2] sept 17 - Oct 17 https://crash-stats.mozilla.org/signature/?product=Thunderbird&signature=shutdownhang%20%7C%20libsystem_kernel.dylib%400x16136&date=%3E%3D2019-09-17T02%3A38%3A00.000Z&date=%3C2019-10-17T02%3A38%3A00.000Z#summary

[3] https://crash-stats.mozilla.org/signature/?product=Thunderbird&signature=shutdownhang%20%7C%20__psynch_cvwait%20%7C%20_pthread_cond_wait%20%7C%20mozilla%3A%3Adetail%3A%3AMutexImpl%3A%3Aunlock%20%7C%20pthread_cond_signal_thread_np&date=%3E%3D2019-08-17T02%3A53%3A00.000Z&date=%3C2019-11-17T02%3A53%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#graphs

Comment 21

[Wayne Mery (:wsmwk)]

shutdownhang | libsystem_kernel.dylib@0x16136 is gone, roughly as of Dec 1 2019.
Not sure what it morphed to

Comment 22

[Calum Mackay]

Perhaps this, which I saw a few days ago: https://crash-stats.mozilla.com/report/index/24d3dce4-4c3e-4895-aabe-86ef90200115