1712118 - Crash in [@ shutdownhang | NtWriteFile]

Status: RESOLVED DUPLICATE of bug 1738984

(Core :: Security: PSM, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

68 Nightly crashes since 89.0a1 20210413093459, all on Windows 10. Beta doesn't seem to be affected (only single instances per version)

Crash report: https://crash-stats.mozilla.org/report/index/214a829f-fa49-4a92-8e52-379860210519

MOZ_CRASH Reason: Shutdown hanging at step quit-application. Something is blocking the main-thread.

Top 10 frames of crashing thread:

0 ntdll.dll NtWriteFile 
1 kernelbase.dll WriteFile 
2 nss3.dll winWrite third_party/sqlite3/src/sqlite3.c:44963
3 nss3.dll pagerAddPageToRollbackJournal third_party/sqlite3/src/sqlite3.c:58075
4 nss3.dll pager_write third_party/sqlite3/src/sqlite3.c:58151
5 nss3.dll allocateBtreePage third_party/sqlite3/src/sqlite3.c:71175
6 nss3.dll sqlite3BtreeCreateTable third_party/sqlite3/src/sqlite3.c:74376
7 nss3.dll sqlite3VdbeExec third_party/sqlite3/src/sqlite3.c:92253
8 nss3.dll sqlite3_step third_party/sqlite3/src/sqlite3.c:84388
9 nss3.dll sqlite3_exec third_party/sqlite3/src/sqlite3.c:125282

 0 	ntdll.dll	NtWriteFile		context
1 	kernelbase.dll	WriteFile		cfi
2 	nss3.dll	winWrite(sqlite3_file*, void const*, int, long long)	third_party/sqlite3/src/sqlite3.c:44963 	cfi
3 	nss3.dll	pagerAddPageToRollbackJournal(PgHdr*)	third_party/sqlite3/src/sqlite3.c:58075 	cfi
4 	nss3.dll	pager_write(PgHdr*)	third_party/sqlite3/src/sqlite3.c:58151 	cfi
5 	nss3.dll	allocateBtreePage(BtShared*, MemPage**, unsigned int*, unsigned int, unsigned char)	third_party/sqlite3/src/sqlite3.c:71175 	cfi
6 	nss3.dll	sqlite3BtreeCreateTable(Btree*, unsigned int*, int)	third_party/sqlite3/src/sqlite3.c:74376 	cfi
7 	nss3.dll	sqlite3VdbeExec(Vdbe*)	third_party/sqlite3/src/sqlite3.c:92253 	cfi
8 	nss3.dll	sqlite3_step(sqlite3_stmt*)	third_party/sqlite3/src/sqlite3.c:84388 	cfi
9 	nss3.dll	sqlite3_exec(sqlite3*, char const*, int (*)(void*, int, char**, char**), void*, char**)	third_party/sqlite3/src/sqlite3.c:125282 	cfi
10 	softokn3.dll	sdb_PutMetaData(SDBStr*, char const*, SECItemStr const*, SECItemStr const*)	security/nss/lib/softoken/sdb.c:1769 	cfi
11 	softokn3.dll	sftkdb_ChangePassword(SFTKDBHandleStr*, char*, char*, int*)	security/nss/lib/softoken/sftkpwd.c:1422 	cfi
12 	softokn3.dll	NSC_InitPIN(unsigned long, unsigned char*, unsigned long)	security/nss/lib/softoken/pkcs11.c:4033 	cfi
13 	nss3.dll	PK11_InitPin(PK11SlotInfoStr*, char const*, char const*)	security/nss/lib/pk11wrap/pk11auth.c:456 	cfi
14 	xul.dll	mozilla::psm::InitializeNSS(nsTSubstring<char> const&, mozilla::psm::NSSDBConfig, mozilla::psm::PKCS11DBConfig)	security/certverifier/NSSCertDBTrustDomain.cpp:1561 	cfi
15 	xul.dll	nsNSSComponent::InitializeNSS()	security/manager/ssl/nsNSSComponent.cpp:1902 	cfi
16 	xul.dll	nsNSSComponent::Init()	security/manager/ssl/nsNSSComponent.cpp:2191 	cfi
17 	xul.dll	mozilla::xpcom::CreateInstanceImpl(mozilla::xpcom::ModuleID, nsISupports*, nsID const&, void**)	xpcom/components/StaticComponents.cpp:12073 	cfi
18 	xul.dll	nsGetServiceByContractID::operator()(nsID const&, void**) const	xpcom/components/nsComponentManagerUtils.cpp:245 	cfi
19 	xul.dll	nsCOMPtr_base::assign_from_gs_contractid(const nsGetServiceByContractID, nsID const&)	xpcom/base/nsCOMPtr.cpp:82 	cfi
20 	xul.dll	EnsureNSSInitializedChromeOrContent()	security/manager/ssl/nsNSSComponent.cpp:129 	cfi
21 	xul.dll	mozilla::net::nsHttpHandler::NewProxiedChannel(nsIURI*, nsIProxyInfo*, unsigned int, nsIURI*, nsILoadInfo*, nsIChannel**)	netwerk/protocol/http/nsHttpHandler.cpp:2111 	cfi
22 	xul.dll	mozilla::net::nsIOService::NewChannelFromURIWithProxyFlagsInternal(nsIURI*, nsIURI*, unsigned int, nsILoadInfo*, nsIChannel**)	netwerk/base/nsIOService.cpp:1122 	cfi
23 	xul.dll	mozilla::net::nsIOService::NewChannelFromURIWithClientAndController(nsIURI*, nsINode*, nsIPrincipal*, nsIPrincipal*, mozilla::Maybe<mozilla::dom::ClientInfo> const&, mozilla::Maybe<mozilla::dom::ServiceWorkerDescriptor> const&, unsigned int, nsIContentPolicy::nsContentPolicyType, unsigned int, nsIChannel**)	netwerk/base/nsIOService.cpp:1066 	cfi
24 	xul.dll	NS_NewChannelInternal(nsIChannel**, nsIURI*, nsINode*, nsIPrincipal*, nsIPrincipal*, mozilla::Maybe<mozilla::dom::ClientInfo> const&, mozilla::Maybe<mozilla::dom::ServiceWorkerDescriptor> const&, unsigned int, nsIContentPolicy::nsContentPolicyType, nsICookieJarSettings*, mozilla::dom::PerformanceStorage*, nsILoadGroup*, nsIInterfaceRequestor*, unsigned int, nsIIOService*, unsigned int)	netwerk/base/nsNetUtil.cpp:425 	cfi
25 	xul.dll	NS_NewChannel(nsIChannel**, nsIURI*, nsIPrincipal*, unsigned int, nsIContentPolicy::nsContentPolicyType, nsICookieJarSettings*, mozilla::dom::PerformanceStorage*, nsILoadGroup*, nsIInterfaceRequestor*, unsigned int, nsIIOService*, unsigned int)	netwerk/base/nsNetUtil.cpp:366 	cfi
26 	xul.dll	mozilla::dom::XMLHttpRequestMainThread::Open(nsTSubstring<char> const&, nsTSubstring<char16_t> const&, bool, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, mozilla::ErrorResult&)	dom/xhr/XMLHttpRequestMainThread.cpp:1389 	cfi
27 	xul.dll	mozilla::dom::XMLHttpRequest_Binding::open(JSContext*, JS::Handle<JSObject*>, void*, JSJitMethodCallArgs const&)	dom/bindings/XMLHttpRequestBinding.cpp:1044 	cfi
28 	xul.dll	mozilla::dom::binding_detail::GenericMethod<mozilla::dom::binding_detail::NormalThisPolicy, mozilla::dom::binding_detail::ThrowExceptions>(JSContext*, unsigned int, JS::Value*)	dom/bindings/BindingUtils.cpp:3297 	cfi
29 	xul.dll	js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason)	js/src/vm/Interpreter.cpp:512 	cfi
30 	xul.dll	Interpret(JSContext*, js::RunState&)	js/src/vm/Interpreter.cpp:3227 	cfi
31 	xul.dll	js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason)	js/src/vm/Interpreter.cpp:544 	cfi
32 	xul.dll	JS_CallFunctionValue(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>)	js/src/jsapi.cpp:2733 	cfi
33 	xul.dll	nsXPCWrappedJS::CallMethod(unsigned short, nsXPTMethodInfo const*, nsXPTCMiniVariant*)	js/xpconnect/src/XPCWrappedJSClass.cpp:971 	cfi
34 	xul.dll	PrepareAndDispatch(nsXPTCStubBase*, unsigned int, unsigned long long*, unsigned long long*, double*)	xpcom/reflect/xptcall/md/win32/xptcstubs_x86_64.cpp:168 	cfi
35 	xul.dll	SharedStub		cfi
36 	xul.dll	mozilla::net::CaptivePortalService::Prepare()	netwerk/base/CaptivePortalService.cpp:365 	cfi
37 	xul.dll	XPTC__InvokebyIndex		cfi
38 	xul.dll	js::NativeObject::shrinkElements(JSContext*, unsigned int)	js/src/vm/NativeObject.cpp:998 	cfi
39 	xul.dll	_tailMerge_d3dcompiler_47.dll		cfi
40 	xul.dll	_tailMerge_d3dcompiler_47.dll		scan
41 	xul.dll	DefinePropertyIfFound(XPCCallContext&, JS::Handle<JSObject*>, JS::Handle<JS::PropertyKey>, XPCNativeSet*, XPCNativeInterface*, XPCNativeMember*, XPCWrappedNativeScope*, bool, XPCWrappedNative*, XPCWrappedNative*, nsIXPCScriptable*, unsigned int, bool*)	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:499 	scan
42 	xul.dll	_tailMerge_d3dcompiler_47.dll		cfi

Comment 1

[Dana Keeler (she/her) [:keeler]]

Ryan - could this be due to the recent sqlite3 update in bug 1698592?

Comment 2

[Dana Keeler (she/her) [:keeler]]

Benjamin - did something change in softoken recently?

Comment 3

[Benjamin Beurdouche [:beurdouche]]

If we are looking for something specific to Fx89 here, NSS 3.64 is likely the target.
We had a few patches [0] affecting softoken and pkcs11 around that timeframe that could have caused trouble, but I can't immediately think of one in particular that would produce such trace. Maybe Bob will have a more precise idea.

[0] https://github.com/nss-dev/nss/commits/master/lib

Comment 4

[Robert Relyea]

It seems more likely to be an issue with the sqlite update. None of those patches affect the dbm code. What is surprising to me is that you are getting a crash in nss3.dll in sqlite code. Usually NSS references sqlite from an external library. I would expect it to find sqlite somewhere in mozilla, not in NSS, so it could be a result of a change to gyp building.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #1)

Ryan - could this be due to the recent sqlite3 update in bug 1698592?

Comment 0 suggests that this spike started about a week and a half after that update landed. In general, I'm having a difficult time discerning much of a spike though looking at the trends in prior releases.

(In reply to Robert Relyea from comment #4)

What is surprising to me is that you are getting a crash in nss3.dll in sqlite code.

This sounds bad. Are we really shipping two copies of sqlite these days? I guess the stacks at least show the correct copy in third_party being used.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

FWIW, bug 1695974 landed shortly before the reported regression range.

Comment 7

[Dana Keeler (she/her) [:keeler]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #6)

FWIW, bug 1695974 landed shortly before the reported regression range.

The changes there were internal to osclientcerts - I doubt it would have an impact on softoken/sqlite.