1185223 - crash at [@ mozilla::dom::quota::QuotaObject::Release() ]

Status: RESOLVED FIXED

(Core :: Storage: IndexedDB, defect)

Description

[Willy Aguirre [:marti1125]]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Build ID: 20150630154324

Steps to reproduce:

I found in the list of top crashes for Firefox Nightly (42.0a1) 
https://crash-stats.mozilla.com/report/index/62ac50a6-64bc-42ec-abc8-c5f6a2150713

This happens in Windows 8.1 (70.59 %) mostly. the first  signature appears in 2015-07-11

More reports can be found at:
https://crash-stats.mozilla.com/report/list?product=Firefox&signature=mozilla%3A%3Adom%3A%3Aquota%3A%3AQuotaObject%3A%3ARelease%28%29

0 	xul.dll 	mozilla::dom::quota::QuotaObject::Release() 	dom/quota/QuotaManager.cpp
1 	xul.dll 	`anonymous namespace'::xClose(sqlite3_file*) 	storage/TelemetryVFS.cpp
2 	nss3.dll 	sqlite3OsClose 	db/sqlite3/src/sqlite3.c
3 	nss3.dll 	sqlite3WalClose 	db/sqlite3/src/sqlite3.c
4 	nss3.dll 	sqlite3PagerClose 	db/sqlite3/src/sqlite3.c
5 	nss3.dll 	sqlite3BtreeClose 	db/sqlite3/src/sqlite3.c
6 	nss3.dll 	sqlite3LeaveMutexAndCloseZombie 	db/sqlite3/src/sqlite3.c
7 	nss3.dll 	sqlite3Close 	db/sqlite3/src/sqlite3.c
8 	xul.dll 	mozilla::storage::Connection::internalClose(sqlite3*) 	storage/mozStorageConnection.cpp
9 	xul.dll 	mozilla::storage::Connection::Close() 	storage/mozStorageConnection.cpp
10 	xul.dll 	mozilla::storage::Connection::~Connection() 	storage/mozStorageConnection.cpp
11 	xul.dll 	mozilla::storage::Connection::Release() 	storage/mozStorageConnection.cpp
12 	xul.dll 	NS_ProxyRelease(nsIEventTarget*, nsISupports*, bool) 	xpcom/glue/nsProxyRelease.cpp
13 	xul.dll 	mozilla::storage::Service::unregisterConnection(mozilla::storage::Connection*) 	storage/mozStorageService.cpp
14 	xul.dll 	mozilla::storage::Connection::Release() 	storage/mozStorageConnection.cpp
15 	xul.dll 	nsCOMPtr_base::~nsCOMPtr_base() 	xpcom/glue/nsCOMPtr.h
16 	xul.dll 	mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::PerformIdleMaintenanceOnDatabaseInternal(unsigned int, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo const&) 	dom/indexeddb/ActorsParent.cpp
17 	xul.dll 	mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::PerformIdleMaintenanceOnDatabase(unsigned int, nsACString_internal const&, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo&&) 	dom/indexeddb/ActorsParent.cpp
18 	xul.dll 	nsRunnableMethodArguments<unsigned int, nsCString, mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::SingleMaintenanceInfo&&>::apply<mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient, void ( mozilla::dom::indexedDB::A0x409682f2::QuotaClient::*)(unsigned int, nsACString_internal const&, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo&&)>(mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient*, void ( mozilla::dom::indexedDB::A0x409682f2::QuotaClient::*)(unsigned int, nsACString_internal const&, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo&&)) 	xpcom/glue/nsThreadUtils.h
19 	xul.dll 	nsRunnableMethodImpl<void ( mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::*)(unsigned int, nsACString_internal const&, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo&&), 1, unsigned int, nsCString, mozilla::dom::indexedDB::A0x409682f2::QuotaClient::SingleMaintenanceInfo&&>::Run() 	xpcom/glue/nsThreadUtils.h
20 	xul.dll 	nsThreadPool::Run() 	xpcom/threads/nsThreadPool.cpp
21 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp
22 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/glue/nsThreadUtils.cpp
23 	xul.dll 	mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp
24 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
25 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc
26 	xul.dll 	nsThread::ThreadFunc(void*) 	xpcom/threads/nsThread.cpp
27 	nss3.dll 	PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c
28 	nss3.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c
29 	msvcr120.dll 	_callthreadstartex 	f:\dd\vctools\crt\crtw32\startup\threadex.c:376
30 	msvcr120.dll 	_threadstartex 	f:\dd\vctools\crt\crtw32\startup\threadex.c:354
31 	kernel32.dll 	BaseThreadInitThunk 	
32 	ntdll.dll 	RtlUserThreadStart 	
33 	kernel32.dll 	BasepReportFault 	
34 	kernel32.dll 	BasepReportFault


Actual results:

It crashed


Expected results:

It should not crash

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

Is IDB trying to do maintenance while "idle" at the same time quota system is shutting down?

Comment 2

[Jan Varga [:janv]]

(In reply to Ben Kelly [:bkelly] from comment #1)
> Is IDB trying to do maintenance while "idle" at the same time quota system
> is shutting down?

It seems the locking for idle maintenance is not right or there's a bug in locking itself.
I need to find a way how to reproduce, probably by triggering the maintenance explicitly and deleting origin directories at the same time.

Comment 5

[Jan Varga [:janv]]

Attachment: possible fix

The NS_ProxyRelease() in the stack trace indicates that the storage connection is not closed in PerformIdleMaintenanceOnDatabaseInternal(). So the directory lock might get released before the connection is closed. If an origin clear operation was queued and the clear op is executed quickly then OriginInfo data structure gets deleted and QuotaObject now has a dangling reference to OriginInfo. The connection is then closed and it tries to release QuotaObject ...

I tested this scenario locally and I got the same crash and the patch fixes it for me. However I had to artificially modify the code to trigger this scenario. I don't know how to reproduce the crash w/o this modification.
Anyway, I think the patch improves correctness and there's a good chance it will fix the crash reported here.

Comment 6

[Jan Varga [:janv]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c3f5b5ca5087009bc5348def25a5fa8a870a3724
Bug 1185223 - crash at [@ mozilla::dom::quota::QuotaObject::Release() ]; r=khuey

Comment 7

[Jan Varga [:janv]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/bfc4a9432e12a483af442050b4332d065fc34290
Followup build fix for bug 1185223; r=buildbustage

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c3f5b5ca5087
https://hg.mozilla.org/mozilla-central/rev/bfc4a9432e12

Comment 9

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

[Tracking Requested - why for this release]:

No crashes for this signature on 43 in the last month, but there are some on 42 and 41. It is not very high volume on beta 42; around 30 crashes for each beta cycle so far.

Comment 10

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Tracking for 44, in case this reopens.

Comment 11

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Adding the crash signature from  bug 1212941, since this is a duplicate. That may also mean that 43 is affected. I can check back on crash-stats later in the week and follow up.

Comment 12

[Sylvestre Ledru [:Sylvestre]]

Jan, do you want to uplift this to 42? It is low volume but if it is super safe, why not taking it.

Comment 13

[Jan Varga [:janv]]

Yes, it's super safe, I'll request aurora and beta approval.

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Jan, could you request it now? Beta 8 gtb is next Monday.
Thanks

Comment 15

[Jan Varga [:janv]]

Comment on attachment 8671602 [details] [diff] [review]
possible fix

Approval Request Comment
[Feature/regressing bug #]: Bug 858680.

[User impact if declined]: Occasional crashes.

[Describe test coverage new/current, TreeHerder]: There are no tests for this. We just saw it in the crash reports.

[Risks and why]: The patch is super safe.

[String/UUID change made/needed]: None.

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8671602 [details] [diff] [review]
possible fix

Taking it as it fix a crash, thanks.
Should be in 42 beta 8.

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/85ec9994546f
https://hg.mozilla.org/releases/mozilla-aurora/rev/863e41485ec1

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1153ec762010
https://hg.mozilla.org/releases/mozilla-beta/rev/2d497358081c