1381148 - deadlock between nssPKIObject lock and NSSTrustDomain lock

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Say you have two threads: one calling SECMOD_DeleteModule and another calling CERT_FindCertByIssuerAndSN. They can deadlock with the following stacks:

#0  0x00007f2568ddb460 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f256a019cee in PR_EnterMonitor (mon=0x60f00001fc60) at ../../../../pr/src/pthreads/ptsynch.c:602
#2  0x00007f256a4236d8 in nssPKIObject_Lock (object=0x61d0000fab58) at ../../lib/pki/pkibase.c:22
#3  0x00007f256a42fbd5 in remove_token_certs (k=0x61d0000fab58, v=0x61d0000fb160, a=0x7f2563a326c0) at ../../lib/pki/tdcache.c:385
#4  0x00007f256a455531 in nss_hash_enumerator (he=0x604000002060, index=0, arg=0x7f2563a32580) at ../../lib/base/hash.c:291
#5  0x00007f25681e2121 in PL_HashTableEnumerateEntries (ht=0x60700000d3f0, f=0x7f256a455440 <nss_hash_enumerator>, arg=0x7f2563a32580) at ../../../lib/ds/plhash.c:374
#6  0x00007f256a4553a1 in nssHash_Iterate (hash=0x61d0000078f0, fcn=0x7f256a42fb90 <remove_token_certs>, closure=0x7f2563a326c0) at ../../lib/base/hash.c:309
#7  0x00007f256a42f4f7 in nssTrustDomain_RemoveTokenCertsFromCache (td=0x61d0000082b0, token=0x61d0001112b0) at ../../lib/pki/tdcache.c:429
#8  0x00007f256a313555 in nssToken_NotifyCertsNotVisible (tok=0x61d0001112b0) at ../../lib/pk11wrap/dev3hack.c:278
#9  0x00007f256a41aaf8 in STAN_RemoveModuleFromDefaultTrustDomain (module=0x61500002dd20) at ../../lib/pki/pki3hack.c:187
#10 0x00007f256a3ceb48 in SECMOD_DeleteModuleEx (name=0x519a40 <.str> "Builtin Roots Module", mod=0x0, type=0x7f2563a32c20, permdb=1) at ../../lib/pk11wrap/pk11util.c:371
#11 0x00007f256a3cf24b in SECMOD_DeleteModule (name=0x519a40 <.str> "Builtin Roots Module", type=0x7f2563a32c20) at ../../lib/pk11wrap/pk11util.c:418

#0  0x00007f2568ddb460 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f256a0181ab in PR_WaitCondVar (cvar=0x60600000e2a0, timeout=4294967295) at ../../../../pr/src/pthreads/ptsynch.c:395
#2  0x00007f25696c3f7e in NSSRWLock_LockRead_Util (rwlock=0x60600000e300) at ../../lib/util/nssrwlk.c:165
#3  0x00007f256a434717 in nssTrustDomain_GetActiveSlots (td=0x61d0000082b0, updateLevel=0x7f2564433160) at ../../lib/pki/trustdomain.c:116
#4  0x00007f256a43633a in nssTrustDomain_FindTrustForCertificate (td=0x61d0000082b0, c=0x61d0000fab58) at ../../lib/pki/trustdomain.c:1054
#5  0x00007f256a41da71 in nssTrust_GetCERTCertTrustForCert (c=0x61d0000fab58, cc=0x61d0001284a0) at ../../lib/pki/pki3hack.c:611
#6  0x00007f256a4232e8 in fill_CERTCertificateFields (c=0x61d0000fab58, cc=0x61d0001284a0, forced=0) at ../../lib/pki/pki3hack.c:817
#7  0x00007f256a41ed7d in stan_GetCERTCertificate (c=0x61d0000fab58, forceUpdate=0) at ../../lib/pki/pki3hack.c:891
#8  0x00007f256a41ef47 in STAN_GetCERTCertificateOrRelease (c=0x61d0000fab58) at ../../lib/pki/pki3hack.c:937
#9  0x00007f256a331873 in PK11_FindCertByIssuerAndSN (slotPtr=0x7f2564433a80, issuerSN=0x7f2564433b80, wincx=0x0) at ../../lib/pk11wrap/pk11cert.c:1924
#10 0x00007f256a40ef57 in CERT_FindCertByIssuerAndSN (handle=0x0, issuerAndSN=0x7f2564433b80) at ../../lib/certdb/stanpcertdb.c:468

Thread A is attempting to lock an nssPKIObject while holding the NSSTrustDomain's lock (acquired in frame 9).
Thread B is attempting to lock the NSSTrustDomain's tokenLock while holding the nssPKIObject's lock (acquired in frame 7).

Comment 1

[Franziskus Kiefer [:franziskus]]

This might help, or not https://nss-review.dev.mozaws.net/D378

Comment 2

[Franziskus Kiefer [:franziskus]]

Let's see if this helps https://hg.mozilla.org/projects/nss/rev/5c1d693a5f7a56e24bdaff60b8787cb0b627ba86