1335769 - Crash in shutdownhang | PR_JoinThread | mozilla::dom::DOMStorageDBThread::Shutdown

Status: RESOLVED WORKSFORME

(Core :: SQLite and Embedded Database Bindings, defect, P3)

Description

[Ben Kelly [:bkelly, not reviewing]]

This bug was filed from the Socorro interface and is 
report bp-59ee959b-0c7c-45b2-bca0-d9a962170201.
=============================================================

We have a fair number of these in release/beta.  About 200+ crashes per day.

AFAICT, though, we're hitting the shutdown hang timer just trying to flush an sqlite database to disk.  See thread 42 here:

https://crash-stats.mozilla.com/report/index/59ee959b-0c7c-45b2-bca0-d9a962170201#allthreads

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

Andrew, do you know where the shutdownhang timer is defined?  Is there any way storage could somehow extend it if it has to flush to disk?  Seems bad to force a crash while we are trying to write out an sqlite database.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

This is a similar issue to bug 1186276 where mozStorage spins an event loop at shutdown and encounters apparently slow I/O.  There I proposed (and am still assigned, will abandon now) to switch us over to using nsIAsyncShutdown.  Crash-stats seems to not want to show any signature reports for me right now, but "super search" on "mozilla::storage::Service::Observe" shows a slightly higher crash rate for that bug than this one over the last week.

The localstorage case may be slightly unique in that it has a clever batching implementation with a 5 second flushing interval where shutdown compels a flush that will have a tendency to incur I/O debt at the worst time.  It might be appropriate to recognize that localStorage makes no transactional guarantees and reduce shutdown I/O load by causing DOMStorage to simply discard its batched data at shutdown if we suspect our system has I/O throughput issues.  At least after we've tried other enhancements that I'll post about on bug 1186276 now.

Comment 3

[Andrew McCreight [:mccr8]]

(In reply to Ben Kelly [:bkelly] from comment #1)
> Andrew, do you know where the shutdownhang timer is defined?  Is there any
> way storage could somehow extend it if it has to flush to disk?  Seems bad
> to force a crash while we are trying to write out an sqlite database.

It looks like maybe the main process hang monitor stuff is in here somewhere:

https://hg.mozilla.org/releases/mozilla-beta/annotate/0f339c1e154f/xpcom/threads/HangMonitor.cpp#l287

I don't know anything about it, sorry.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Hmm, if I'm reading it right that defaults to 5 seconds.  Which does seem a bit low if you have a slow disk or network drive you are trying to write to.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'shutdownhang | PR_JoinThread | mozilla::dom::DOMStorageDBThread::Shutdown':
 - nightly (version 54): 0 crashes from 2017-01-23.
 - aurora  (version 53): 0 crashes from 2017-01-23.
 - beta    (version 52): 368 crashes from 2017-01-23.
 - release (version 51): 1535 crashes from 2017-01-16.
 - esr     (version 45): 1678 crashes from 2016-08-10.

Crash volume on the last weeks (Week N is from 02-06 to 02-12):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       0       0
 - aurora        0       0
 - beta        217      66
 - release    1001     200       0
 - esr          78     102      89      69      76      57      68

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly
 - aurora
 - beta    #50
 - release #34
 - esr     #178

Comment 6

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.