Open Bug 1026805 Opened 6 years ago Updated Last year

Intermittent test_bug324121.js,test_bug394300.js,test_addons_store.js,test_AddonRepository_cache.js | Test timed out (then [@ libsystem_c.dylib + 0x2f8e] with psm::OCSPCache::MakeMostRecentlyUsed(), psm::OCSPCache::Put() or psm::DoOCSPRequest() on stack)

Categories

(Core :: Security: PSM, defect, P3)

x86_64
macOS
defect

Tracking

()

Tracking Status
firefox32 --- disabled
firefox33 --- disabled
firefox34 --- disabled
firefox-esr24 --- unaffected
firefox-esr31 --- unaffected
b2g-v1.3 --- unaffected
b2g-v1.3T --- unaffected
b2g-v1.4 --- unaffected
b2g-v2.0 --- disabled
b2g-v2.1 --- disabled

People

(Reporter: KWierso, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: crash, intermittent-failure, Whiteboard: [tests disabled on OSX][psm-intermittent])

https://tbpl.mozilla.org/php/getParsedLog.php?id=41895978&tree=Mozilla-Inbound
Rev5 MacOSX Mountain Lion 10.8 mozilla-inbound debug test xpcshell on 2014-06-17 13:11:01 PDT for push b981b1d06b71

slave: talos-mtnlion-r5-089



13:25:03     INFO -  TEST-INFO | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_bug324121.js | running test ...
13:30:03  WARNING -  TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_bug324121.js | Test timed out
13:30:03  WARNING -  TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_bug324121.js | test failed (with xpcshell return code: 5), see following log:
13:30:03     INFO -  >>>>>>>
13:30:03     INFO -  DMD[4462] $DMD = '(null)'
13:30:03     INFO -  DMD[4462] DMD is not enabled
13:30:03     INFO -  [4462] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/slave/m-in-osx64-d-00000000000000000/build/toolkit/crashreporter/nsExceptionHandler.cpp, line 2158
13:30:03     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/head_addons.js:1217 - in strict mode code, functions may be declared only at top level or immediately within another function
13:30:03     INFO -  System JS : ERROR (null):0 - uncaught exception: 2147500037
13:30:03     INFO -  TEST-INFO | (xpcshell/head.js) | test MAIN run_test pending (1)
13:30:03     INFO -  TEST-INFO | (xpcshell/head.js) | test pending (2)
13:30:03     INFO -  1403036703510	addons.manager	DEBUG	Application has been upgraded
13:30:03     INFO -  1403036703552	addons.manager	DEBUG	Loaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]
13:30:03     INFO -  1403036703556	addons.manager	DEBUG	Loaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]


.....


13:30:03     INFO -  TEST-INFO | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/head_addons.js | "Mock profileBeforeChange blocker for 'AddonManager: shutting down providers'"
13:30:03     INFO -  1403036704087	addons.xpi	DEBUG	startup
13:30:03     INFO -  1403036704089	addons.xpi	INFO	Mapping bug324121_1@tests.mozilla.org">bug324121_1@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_1@tests.mozilla.org.xpi">bug324121_1@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704089	addons.xpi	INFO	Mapping bug324121_2@tests.mozilla.org">bug324121_2@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_2@tests.mozilla.org.xpi">bug324121_2@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704089	addons.xpi	INFO	Mapping bug324121_3@tests.mozilla.org">bug324121_3@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_3@tests.mozilla.org.xpi">bug324121_3@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704089	addons.xpi	INFO	Mapping bug324121_4@tests.mozilla.org">bug324121_4@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_4@tests.mozilla.org.xpi">bug324121_4@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704089	addons.xpi	INFO	Mapping bug324121_5@tests.mozilla.org">bug324121_5@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_5@tests.mozilla.org.xpi">bug324121_5@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704090	addons.xpi	INFO	Mapping bug324121_6@tests.mozilla.org">bug324121_6@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_6@tests.mozilla.org.xpi">bug324121_6@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704090	addons.xpi	INFO	Mapping bug324121_7@tests.mozilla.org">bug324121_7@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_7@tests.mozilla.org.xpi">bug324121_7@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704090	addons.xpi	INFO	Mapping bug324121_8@tests.mozilla.org">bug324121_8@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_8@tests.mozilla.org.xpi">bug324121_8@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704091	addons.xpi	INFO	Mapping bug324121_9@tests.mozilla.org">bug324121_9@tests.mozilla.org to /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_9@tests.mozilla.org.xpi">bug324121_9@tests.mozilla.org.xpi
13:30:03     INFO -  1403036704091	addons.xpi	DEBUG	checkForChanges
13:30:03     INFO -  1403036704095	addons.xpi	DEBUG	Directory state JSON differs: cache [{"name":"app-profile","addons":{"bug324121_9@tests.mozilla.org">bug324121_9@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_9@tests.mozilla.org.xpi">bug324121_9@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_8@tests.mozilla.org">bug324121_8@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_8@tests.mozilla.org.xpi">bug324121_8@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_7@tests.mozilla.org">bug324121_7@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_7@tests.mozilla.org.xpi">bug324121_7@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_6@tests.mozilla.org">bug324121_6@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_6@tests.mozilla.org.xpi">bug324121_6@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_5@tests.mozilla.org">bug324121_5@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_5@tests.mozilla.org.xpi">bug324121_5@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_4@tests.mozilla.org">bug324121_4@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_4@tests.mozilla.org.xpi">bug324121_4@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_3@tests.mozilla.org">bug324121_3@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_3@tests.mozilla.org.xpi">bug324121_3@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_2@tests.mozilla.org">bug324121_2@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_2@tests.mozilla.org.xpi">bug324121_2@tests.mozilla.org.xpi","mtime":1403036704000},"bug324121_1@tests.mozilla.org">bug324121_1@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_1@tests.mozilla.org.xpi">bug324121_1@tests.mozilla.org.xpi","mtime":1403036704000}}}] state [{"name":"app-profile","addons":{"bug324121_1@tests.mozilla.org">bug324121_1@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_1@tests.mozilla.org.xpi">bug324121_1@tests.mozilla.org.xpi","mtime":1403036704000},"bug324121_2@tests.mozilla.org">bug324121_2@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_2@tests.mozilla.org.xpi">bug324121_2@tests.mozilla.org.xpi","mtime":1403036704000},"bug324121_3@tests.mozilla.org">bug324121_3@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_3@tests.mozilla.org.xpi">bug324121_3@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_4@tests.mozilla.org">bug324121_4@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_4@tests.mozilla.org.xpi">bug324121_4@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_5@tests.mozilla.org">bug324121_5@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_5@tests.mozilla.org.xpi">bug324121_5@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_6@tests.mozilla.org">bug324121_6@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_6@tests.mozilla.org.xpi">bug324121_6@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_7@tests.mozilla.org">bug324121_7@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_7@tests.mozilla.org.xpi">bug324121_7@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_8@tests.mozilla.org">bug324121_8@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_8@tests.mozilla.org.xpi">bug324121_8@tests.mozilla.org.xpi","mtime":1403036703000},"bug324121_9@tests.mozilla.org">bug324121_9@tests.mozilla.org":{"descriptor":"/var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions/bug324121_9@tests.mozilla.org.xpi">bug324121_9@tests.mozilla.org.xpi","mtime":1403036703000}}}]
13:30:03     INFO -  1403036704102	addons.xpi-utils	DEBUG	Opening XPI database /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpOT7dSn/extensions.json
13:30:03     INFO -  1403036704105	addons.xpi-utils	DEBUG	Successfully read XPI database
13:30:03     INFO -  1403036704111	addons.xpi	DEBUG	No changes found
13:30:03     INFO -  TEST-INFO | (xpcshell/head.js) | test delayed callback finished (2)
13:30:03     INFO -  1403036704136	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121.rdf
13:30:03     INFO -  System JS : ERROR (null):0 - uncaught exception: 2147500037
13:30:03     INFO -  1403036704262	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121.rdf
13:30:03     INFO -  1403036704266	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121_4.rdf
13:30:03     INFO -  1403036704271	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121.rdf
13:30:03     INFO -  1403036704275	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121.rdf
13:30:03     INFO -  1403036704278	addons.update-checker	DEBUG	Requesting http://localhost:4444/data/test_bug324121_8.rdf
13:30:03     INFO -  1403036704340	addons.update-checker	DEBUG	Found an update entry for bug324121_2@tests.mozilla.org">bug324121_2@tests.mozilla.org version 2
13:30:03     INFO -  1403036704433	addons.update-checker	DEBUG	Found an update entry for bug324121_3@tests.mozilla.org">bug324121_3@tests.mozilla.org version 2
13:30:03     INFO -  1403036704494	addons.update-checker	WARN	Request failed: http://localhost:4444/data/test_bug324121_4.rdf - 404: Not Found
13:30:03     INFO -  1403036704555	addons.update-checker	DEBUG	Found an update entry for bug324121_6@tests.mozilla.org">bug324121_6@tests.mozilla.org version 2
13:30:03     INFO -  1403036704615	addons.update-checker	DEBUG	Found an update entry for bug324121_7@tests.mozilla.org">bug324121_7@tests.mozilla.org version 2
13:30:03     INFO -  1403036704675	addons.update-checker	WARN	Request failed: http://localhost:4444/data/test_bug324121_8.rdf - 404: Not Found
13:30:03     INFO -  TEST-PASS | /builds/slave/talos-slave/test/build/tests/xpcshell/head.js | [do_check_eq : 762] 0 == 0
13:30:03     INFO -  TEST-INFO | (xpcshell/head.js) | test finished (1)
13:30:03     INFO -  TEST-INFO | (xpcshell/head.js) | exiting test
13:30:03     INFO -  1403036704733	addons.manager	DEBUG	shutdown
13:30:03     INFO -  1403036704734	addons.xpi	DEBUG	shutdown
13:30:03     INFO -  1403036704734	addons.xpi-utils	DEBUG	shutdown
13:30:03     INFO -  1403036704738	addons.xpi	DEBUG	Notifying XPI shutdown observers
13:30:03     INFO -  1403036704740	addons.manager	DEBUG	Async provider shutdown done
13:30:03     INFO -  TEST-INFO | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/head_addons.js | "gXPISaveError set to: null"
13:30:03     INFO -  <<<<<<<
13:30:18  WARNING -  PROCESS-CRASH | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_bug324121.js | application crashed [@ libsystem_c.dylib + 0x2fa5]
13:30:18     INFO -  Crash dump filename: /var/folders/xw/5dh24hb11nvbw7vd2_gyr_5400000w/T/tmpNdrXdA/23CADF3F-A252-4871-B4C6-A2B77122EC55.dmp
13:30:18     INFO -  Operating system: Mac OS X
13:30:18     INFO -                    10.8.0 12A269
13:30:18     INFO -  CPU: amd64
13:30:18     INFO -       family 6 model 42 stepping 7
13:30:18     INFO -       8 CPUs
13:30:18     INFO -  Crash reason:  EXC_SOFTWARE / SIGABRT
13:30:18     INFO -  Crash address: 0x7fff9565ffa5
13:30:18     INFO -  Thread 0 (crashed)
13:30:18     INFO -   0  libsystem_c.dylib + 0x2fa5
13:30:18     INFO -      rbx = 0x0000000106300040   r12 = 0x000000010d800178
13:30:18     INFO -      r13 = 0x000000010d80c001   r14 = 0x000000010d80d970
13:30:18     INFO -      r15 = 0x0000000000000070   rip = 0x00007fff9565ffa5
13:30:18     INFO -      rsp = 0x00007fff5fbfbf18   rbp = 0x00007fff5fbfbfc0
13:30:18     INFO -      Found by: given as instruction pointer in context
13:30:18     INFO -   1  libmozglue.dylib!arena_dalloc [jemalloc.c:b981b1d06b71 : 1688 + 0xf]
13:30:18     INFO -      rip = 0x00000001056778d7   rsp = 0x00007fff5fbfbf20
13:30:18     INFO -      rbp = 0x00007fff5fbfbfc0
13:30:18     INFO -      Found by: stack scanning
13:30:18     INFO -   2  libmozglue.dylib!je_free [jemalloc.c:b981b1d06b71 : 6503 + 0x7]
13:30:18     INFO -      rbx = 0x000000010d80d970   r12 = 0x00000001056943f0
13:30:18     INFO -      r13 = 0x0000000000000000   r14 = 0x000000010d80d970
13:30:18     INFO -      r15 = 0x0000000000000070   rip = 0x0000000105677826
13:30:18     INFO -      rsp = 0x00007fff5fbfbfd0   rbp = 0x00007fff5fbfbff0
13:30:18     INFO -      Found by: call frame info
13:30:18     INFO -   3  libsystem_c.dylib + 0x2d8c7
13:30:18     INFO -      rbx = 0x000000010d91d1c0   r12 = 0x00000001056943f0
13:30:18     INFO -      r13 = 0x0000000000000000   r14 = 0x000000010d80d970
13:30:18     INFO -      r15 = 0x0000000000000070   rip = 0x00007fff9568a8c8
13:30:18     INFO -      rsp = 0x00007fff5fbfc000   rbp = 0x00007fff5fbfc040
13:30:18     INFO -      Found by: call frame info
13:30:18     INFO -   4  XUL!mozilla::psm::DoOCSPRequest(PLArenaPool*, char const*, SECItemStr const*, unsigned int, bool) [nsCOMPtr.h:b981b1d06b71 : 536 + 0xa]
13:30:18     INFO -      rip = 0x0000000100010000   rsp = 0x00007fff5fbfc0a8
13:30:18     INFO -      Found by: stack scanning
Looking lower down the stack shows variations of OCSPCache frames in every failure - hg log:
http://hg.mozilla.org/mozilla-central/filelog/bdac18bd6c74/security/certverifier/OCSPCache.cpp

...shows bug 1017826 landing hours before this started.
Blocks: 1017826
Component: Add-ons Manager → Security: PSM
Product: Toolkit → Core
Summary: Intermittent test_bug324121.js | Test timed out followed by a crash → Intermittent test_bug324121.js | Test timed out (followed by a crash with mozilla::psm::OCSPCache::MakeMostRecentlyUsed(), mozilla::psm::OCSPCache::Put() or mozilla::psm::DoOCSPRequest() on the stack)
Version: unspecified → Trunk
Would you mind taking a look at this, and if it's going to be a more involved fix, backing out bug 1017826 in the meantime - so we can stop these intermittent failures in automation? :-)
Flags: needinfo?(hpathak)
This happens during test_bug394300.js too - which was previously being put under bug 1018984 (12 failures noted in that bug).
Summary: Intermittent test_bug324121.js | Test timed out (followed by a crash with mozilla::psm::OCSPCache::MakeMostRecentlyUsed(), mozilla::psm::OCSPCache::Put() or mozilla::psm::DoOCSPRequest() on the stack) → Intermittent test_bug324121.js,test_bug394300.js | Test timed out (followed by a crash [@ libsystem_c.dylib + 0x2f8e] with mozilla::psm::OCSPCache::MakeMostRecentlyUsed(), mozilla::psm::OCSPCache::Put() or mozilla::psm::DoOCSPRequest() on the stack)
Duplicate of this bug: 1018984
Duplicate of this bug: 1022199
Also test_addons_store.js from bug 1022199, where there were 8 failures.
Summary: Intermittent test_bug324121.js,test_bug394300.js | Test timed out (followed by a crash [@ libsystem_c.dylib + 0x2f8e] with mozilla::psm::OCSPCache::MakeMostRecentlyUsed(), mozilla::psm::OCSPCache::Put() or mozilla::psm::DoOCSPRequest() on the stack) → Intermittent test_bug324121.js, test_bug394300.js, test_addons_store.js | Test timed out (followed by crash [@ libsystem_c.dylib + 0x2f8e] with psm::OCSPCache::MakeMostRecentlyUsed(), psm::OCSPCache::Put() or psm::DoOCSPRequest() on the stack)
Those stacks are most likely corrupted:

08:05:43     INFO -   4  XUL!mozilla::psm::OCSPCache::Get(CERTCertificateStr const*, CERTCertificateStr const*, int&, long long&) [Mutex.h:4cffdbeec391 : 177 + 0x12]
08:05:43     INFO -      rip = 0x0000000100010000   rsp = 0x00007fff5fbfc128
08:05:43     INFO -      Found by: stack scanning
08:05:43     INFO -   5  XUL!mozilla::layers::LayerPropertiesBase::OldTransformedBounds() [LayerTreeInvalidation.cpp:4cffdbeec391 : 194 + 0x3]

I don't know what LayerTreeInvalidation.cpp does, but it has nothing to do with the OCSP cache. Only NSSCertDBTrustDomain.cpp calls mozilla::psm::OCSPCache::Get.
Flags: needinfo?(hpathak)
Ah yeah that is weird :-s
I've requested some retriggers on the push where bug 1017826 landed, as well as the one before - so we can at least see if it is the cause (I think likely, given the timing of that landing and first seeing this).
https://tbpl.mozilla.org/php/getParsedLog.php?id=42150534&tree=Mozilla-Inbound
Summary: Intermittent test_bug324121.js, test_bug394300.js, test_addons_store.js | Test timed out (followed by crash [@ libsystem_c.dylib + 0x2f8e] with psm::OCSPCache::MakeMostRecentlyUsed(), psm::OCSPCache::Put() or psm::DoOCSPRequest() on the stack) → Intermittent test_bug324121.js,test_bug394300.js,test_addons_store.js,test_AddonRepository_cache.js | Test timed out (then [@ libsystem_c.dylib + 0x2f8e] with psm::OCSPCache::MakeMostRecentlyUsed(), psm::OCSPCache::Put() or psm::DoOCSPRequest() on stack)
After quite a few retriggers, it seems that this occurs before bug 1017826 landed too:
https://tbpl.mozilla.org/php/getParsedLog.php?id=42260044&tree=Mozilla-Inbound
(In reply to David Keeler (:keeler) [use needinfo?] from comment #25)
> Those stacks are most likely corrupted:
> 
> 08:05:43     INFO -   4  XUL!mozilla::psm::OCSPCache::Get(CERTCertificateStr
> const*, CERTCertificateStr const*, int&, long long&) [Mutex.h:4cffdbeec391 :
> 177 + 0x12]
> 08:05:43     INFO -      rip = 0x0000000100010000   rsp = 0x00007fff5fbfc128
> 08:05:43     INFO -      Found by: stack scanning
> 08:05:43     INFO -   5 
> XUL!mozilla::layers::LayerPropertiesBase::OldTransformedBounds()
> [LayerTreeInvalidation.cpp:4cffdbeec391 : 194 + 0x3]
> 
> I don't know what LayerTreeInvalidation.cpp does, but it has nothing to do
> with the OCSP cache. Only NSSCertDBTrustDomain.cpp calls
> mozilla::psm::OCSPCache::Get.

David, this is happening pretty frequently and we're hurting for answers here. Any chance you could take another look? :)
Flags: needinfo?(dkeeler)
I'm starting by trying to reproduce locally, at least. So far no luck, but I'll let you know how it goes.
Flags: needinfo?(dkeeler)
I've been unable to reproduce this on 10.7, so I ordered 10.8 for my Mac Mini. I'll try again when I've got that set up.
Well, good news/bad news: I got 10.8 set up on my machine and managed to reproduce the problem twice, but got little useful information out of it. I'll keep at it, though.