Intermittent node moz-http2 [stdout] [child stderr] SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_mutex_lock
Categories
(NSS :: Libraries, defect, P5)
Tracking
(nss 3.125, firefox153 fixed)
People
(Reporter: intermittent-bug-filer, Assigned: leggert)
References
(Blocks 2 open bugs)
Details
(Keywords: intermittent-failure, Whiteboard: [nss-nofx])
Attachments
(2 files)
Filed by: ishikawa [at] yk.rim.or.jp
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=564492213&repo=try-comm-central&task=RSfrCb_cQl6-IpoMtA_IFQ.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RSfrCb_cQl6-IpoMtA_IFQ/runs/0/artifacts/public/logs/live_backing.log
ThreadSanitizer reports a lock-order inversion in NSS softoken:
M0 → M1 → M0 cycle detected
The inversion occurs in:
sftk_hasAttribute
sftk_CopyObject
→ NSC_CopyObject → PK11_CopyKey → SSL
This indicates inconsistent lock acquisition ordering between
session/object (or related) mutexes inside softoken.
When invoked concurrently (e.g. via QUIC/HTTP3 server using NSS),
this can lead to deadlock.
Fix would be:
Establish and enforce a consistent global lock ordering across
softoken code paths, or restructure code to avoid nested locks
with conflicting order.
A bit more detail
Cycle in lock order graph:
M0 → M1 → M0
Thread A: lock(M0) → lock(M1)
Thread B: lock(M1) → lock(M0)
2. Where it happens (critical path)
From the stack:
PR_Lock
→ sftk_hasAttribute
→ sftk_CopyObject
→ NSC_CopyObject
→ PK11_CopyKey
→ SECKEY_CopyPrivateKey
→ ssl_* (self encrypt / session ticket)
→ neqo (Rust HTTP/3 server)
The test timed out.
| Comment hidden (Intermittent Failures Robot) |
|
||
Comment 2•2 months ago
•
|
||
This is in a server that is used in tests, so I'm marking the bug as not affecting Firefox. I also don't think deadlock is possible here---one of the locks being taken is in a new object that is not yet visible to any other thread.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
|
Assignee | |
Comment 6•2 months ago
|
||
ssl_GetSelfEncryptKeyPair holds the RWLock while calling SECKEY_CopyPrivateKey (which acquires the slot monitor). ssl_SetSelfEncryptKeyPair did the same copy outside the lock, creating the opposite ordering. Moving the copies inside the write lock makes both paths consistent: RWLock → slot monitor.
Pushed by jschanck@mozilla.com:
https://hg.mozilla.org/projects/nss/rev/9474d9443b1c
acquire RWLock before key copies in ssl_SetSelfEncryptKeyPair r=nss-reviewers,bbeurdouche
|
||
Updated•2 months ago
|
|
|
Assignee | |
Comment 8•2 months ago
|
||
sftk_CopyObject holds src->attributeLock while calling sftk_hasAttribute,
sftk_NewAttribute, and sftk_AddAttribute on a freshly created destObject,
each of which acquires dest->attributeLock. TSAN flags this as a potential
deadlock. Since destObject is not yet published, extract lock-free
sftk_{has,New,Add}AttributeUnsafe helpers (taking SFTKSessionObject*
directly) and use them in sftk_CopyObject.
|
||
Comment 9•2 months ago
|
||
A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)
|
||
Comment 10•2 months ago
|
||
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Description
•