1808816 - Intermittent headless-spi SUMMARY: ThreadSanitizer: data race /builds/worker/checkouts/gecko/security/nss/lib/pki/pki3hack.c:750:15 in fill_CERTCertificateFields

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Treeherder Bug Filer]

Filed by: rvandermeulen [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=401517600&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DRzPh0-DSB2taJPxRenN2g/runs/0/artifacts/public/logs/live_backing.log

---

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This hits about 50% of the time when running mochitest-plain in headless-spi (socketprocess) mode (--headless --setpref=network.process.enabled=true --setpref=network.http.network_access_on_socket_process.enabled=true) under TSAN. Seems to hit consistently when running dom/base/test/test_bug466080.html. This isn't currently enabled by default in CI, but I've been running Try pushes in the hopes of getting it green enough to do so.

Log link:
https://treeherder.mozilla.org/logviewer?job_id=401496165&repo=try&lineNumber=3120

Pernosco trace:
https://pernos.co/debug/m_IHUARELHepEATm7BITwQ/index.html#f{m[A98G,AV0_,t[CQ,CFU_,f{e[A8zk,zt4_,s{afv/yALAA,bAdc,u7As,o8ao___/

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Attachment: tsan log

Comment 3

[Andrew McCreight [:mccr8]]

This test does change the value of the pref network.cors_preflight.allow_client_cert. I don't know if that might be causing the race or not.

Comment 4

[John Schanck [:jschanck]]

Looks like FindClientCertificatesWithPrivateKeys can create CERTCertificates in some cases, and it is being called off the socket thread in SelectClientAuthCertificate::DoSelectClientAuthCertificate. Arguably an NSS bug, but I think our current approach is to fix this sort of issue by dispatching to the socket thread, so I'm moving this to PSM. I don't think it's a security issue---there's no attacker-controllable data and the two threads should write identical data to the CERTCertificate's fields.

Comment 5

[Intermittent Failures Robot]

30 failures in 2607 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 30

Platform and build breakdown:

• linux1804-64-tsan-qr: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1808816&startday=2023-01-02&endday=2023-01-08&tree=all

Comment 6

[Dana Keeler (she/her) [:keeler]]

Attachment: Bug 1808816 - find potential client certificates on the socket thread rather than the main thread r?jschanck

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78e0947c2257
find potential client certificates on the socket thread rather than the main thread r=jschanck

Comment 8

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/78e0947c2257

Comment 9

[Ryan VanderMeulen [:RyanVM]]

This appears to still be happening on m-c tip:
https://treeherder.mozilla.org/logviewer?job_id=402780486&repo=try&lineNumber=3131

Comment 10

[Dana Keeler (she/her) [:keeler]]

Attachment: Bug 1808816 - socket process follow-up: find potential client certificates on the socket thread r?jschanck

Comment 11

[Intermittent Failures Robot]

13 failures in 3717 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 13

Platform and build breakdown:

• linux1804-64-tsan-qr: 13
  • opt: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1808816&startday=2023-01-16&endday=2023-01-22&tree=all

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4f69a02ee8f9
socket process follow-up: find potential client certificates on the socket thread r=jschanck

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4f69a02ee8f9