Closed Bug 1351354 Opened 5 years ago Closed 4 years ago

Intermittent Assertion failure: IsIdle(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:132

Categories

(Toolkit :: Crash Reporting, defect, P1)

Product:
Toolkit
Component:
Crash Reporting
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla62
Tracking Flags:
Tracking Status
firefox-esr52 --- wontfix
firefox-esr60 --- wontfix
firefox60 --- wontfix
firefox61 --- wontfix
firefox62 --- fixed

People

(Reporter: aryx, Assigned: haik)

References

Details

(Keywords: assertion, intermittent-failure, regression)

Attachments

(1 file, 1 obsolete file)

Bug 1351354 - Add missing lock protection to crashReporterAPIData_Hash
59 bytes, text/x-review-board-request
gsvelto
: review+
Details 
https://treeherder.mozilla.org/logviewer.html#?job_id=87007910&repo=mozilla-inbound

This in a tab crash submission test, so don't get confused.

[task 2017-03-28T07:28:17.399046Z] 07:28:17     INFO - GECKO(1728) | XPCOM_MEM_BLOAT_LOG: /tmp/tmppqEaCG.mozrunner/runtests_leaks.log
[task 2017-03-28T07:28:17.400792Z] 07:28:17     INFO - GECKO(1728) | Writing to log: /tmp/tmppqEaCG.mozrunner/runtests_leaks_tab_pid1879.log
[task 2017-03-28T07:28:17.601478Z] 07:28:17     INFO - GECKO(1728) | --DOCSHELL 0xe94d9000 == 0 [pid = 1857] [id = {f91683fa-31f5-4f48-b96c-1b12f825b26f}]
[task 2017-03-28T07:28:18.271318Z] 07:28:18     INFO - GECKO(1728) | Assertion failure: IsIdle(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:132
[task 2017-03-28T07:28:18.273078Z] 07:28:18     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b59d4]
[task 2017-03-28T07:28:18.274488Z] 07:28:18     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b5a0b]
[task 2017-03-28T07:28:18.275875Z] 07:28:18     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8ac5d0]
[task 2017-03-28T07:28:18.279579Z] 07:28:18     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e30177]
[task 2017-03-28T07:28:18.280990Z] 07:28:18     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x509d22]
[task 2017-03-28T07:28:18.282390Z] 07:28:18     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc0c31a]
[task 2017-03-28T07:28:18.284898Z] 07:28:18     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc0c4c7]
[task 2017-03-28T07:28:18.289992Z] 07:28:18     INFO - GECKO(1728) | #08: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc12f94]
[task 2017-03-28T07:28:18.292871Z] 07:28:18     INFO - GECKO(1728) | #09: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.295946Z] 07:28:18     INFO - GECKO(1728) | #10: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92a86]
[task 2017-03-28T07:28:18.299202Z] 07:28:18     INFO - GECKO(1728) | #11: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.309171Z] 07:28:18     INFO - GECKO(1728) | #12: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.311292Z] 07:28:18     INFO - GECKO(1728) | #13: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.314501Z] 07:28:18     INFO - GECKO(1728) | Assertion failure: IsRead(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:125
[task 2017-03-28T07:28:18.316706Z] 07:28:18     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b1280]
[task 2017-03-28T07:28:18.328256Z] 07:28:18     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2c6b8]
[task 2017-03-28T07:28:18.331443Z] 07:28:18     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2e4ec]
[task 2017-03-28T07:28:18.335736Z] 07:28:18     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2f5be]
[task 2017-03-28T07:28:18.337831Z] 07:28:18     INFO - GECKO(1728) | #14: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.340059Z] 07:28:18     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37a7b]
[task 2017-03-28T07:28:18.342856Z] 07:28:18     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37b30]
[task 2017-03-28T07:28:18.353488Z] 07:28:18     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37b66]
[task 2017-03-28T07:28:18.355402Z] 07:28:18     INFO - GECKO(1728) | #08: ???[/lib/i386-linux-gnu/libpthread.so.0 +0x6d4c]
[task 2017-03-28T07:28:18.359190Z] 07:28:18     INFO - GECKO(1728) | #15: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.364151Z] 07:28:18     INFO - GECKO(1728) | #16: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.367500Z] 07:28:18     INFO - GECKO(1728) | #17: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34a85ce]
[task 2017-03-28T07:28:18.369527Z] 07:28:18     INFO - GECKO(1728) | #18: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34981f4]
[task 2017-03-28T07:28:18.371539Z] 07:28:18     INFO - GECKO(1728) | #19: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e5b5]
[task 2017-03-28T07:28:18.373787Z] 07:28:18     INFO - GECKO(1728) | #20: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e674]
[task 2017-03-28T07:28:18.378023Z] 07:28:18     INFO - GECKO(1728) | #21: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.379954Z] 07:28:18     INFO - GECKO(1728) | #22: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92cc9]
[task 2017-03-28T07:28:18.383590Z] 07:28:18     INFO - GECKO(1728) | #23: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.385515Z] 07:28:18     INFO - GECKO(1728) | #24: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.388451Z] 07:28:18     INFO - GECKO(1728) | #25: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.390455Z] 07:28:18     INFO - GECKO(1728) | #26: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.400129Z] 07:28:18     INFO - GECKO(1728) | #27: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.402034Z] 07:28:18     INFO - GECKO(1728) | #28: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.403906Z] 07:28:18     INFO - GECKO(1728) | #29: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.405770Z] 07:28:18     INFO - GECKO(1728) | #30: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.408883Z] 07:28:18     INFO - GECKO(1728) | #31: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.410784Z] 07:28:18     INFO - GECKO(1728) | #32: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.412690Z] 07:28:18     INFO - GECKO(1728) | #33: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x3419fea]
[task 2017-03-28T07:28:18.414768Z] 07:28:18     INFO - GECKO(1728) | #34: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.418241Z] 07:28:18     INFO - GECKO(1728) | #35: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92a86]
[task 2017-03-28T07:28:18.420146Z] 07:28:18     INFO - GECKO(1728) | #36: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.422088Z] 07:28:18     INFO - GECKO(1728) | #37: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.423997Z] 07:28:18     INFO - GECKO(1728) | #38: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34a85ce]
[task 2017-03-28T07:28:18.426000Z] 07:28:18     INFO - GECKO(1728) | #39: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34981f4]
[task 2017-03-28T07:28:18.429374Z] 07:28:18     INFO - GECKO(1728) | #40: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e5b5]
[task 2017-03-28T07:28:18.431333Z] 07:28:18     INFO - GECKO(1728) | #41: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e674]
[task 2017-03-28T07:28:18.435510Z] 07:28:18     INFO - GECKO(1728) | #42: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.437711Z] 07:28:18     INFO - GECKO(1728) | #43: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92cc9]
[task 2017-03-28T07:28:18.439696Z] 07:28:18     INFO - GECKO(1728) | #44: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.441608Z] 07:28:18     INFO - GECKO(1728) | #45: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.443500Z] 07:28:18     INFO - GECKO(1728) | #46: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.445875Z] 07:28:18     INFO - GECKO(1728) | #47: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.448262Z] 07:28:18     INFO - GECKO(1728) | #48: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.452084Z] 07:28:18     INFO - GECKO(1728) | #49: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.453998Z] 07:28:18     INFO - GECKO(1728) | #50: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x33c6b29]
[task 2017-03-28T07:28:18.455871Z] 07:28:18     INFO - GECKO(1728) | #51: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc116b7]
[task 2017-03-28T07:28:18.457780Z] 07:28:18     INFO - GECKO(1728) | #52: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc121fa]
[task 2017-03-28T07:28:18.463595Z] 07:28:18     INFO - GECKO(1728) | #53: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x50a899]
[task 2017-03-28T07:28:18.466140Z] 07:28:18     INFO - GECKO(1728) | #54: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x5234b5]
[task 2017-03-28T07:28:18.468557Z] 07:28:18     INFO - GECKO(1728) | #55: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d65a]
[task 2017-03-28T07:28:18.470898Z] 07:28:18     INFO - GECKO(1728) | #56: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d92f]
[task 2017-03-28T07:28:18.474279Z] 07:28:18     INFO - GECKO(1728) | #57: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d98e]
[task 2017-03-28T07:28:18.476215Z] 07:28:18     INFO - GECKO(1728) | #58: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x52038b]
[task 2017-03-28T07:28:18.478544Z] 07:28:18     INFO - GECKO(1728) | #59: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x70a2d7]
[task 2017-03-28T07:28:18.480568Z] 07:28:18     INFO - GECKO(1728) | #60: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x70187b]
[task 2017-03-28T07:28:18.482643Z] 07:28:18     INFO - GECKO(1728) | #61: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x501a80]
[task 2017-03-28T07:28:18.485029Z] 07:28:18     INFO - GECKO(1728) | #62: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x503d44]
[task 2017-03-28T07:28:18.487135Z] 07:28:18     INFO - GECKO(1728) | #63: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8c1928]
[task 2017-03-28T07:28:18.490241Z] 07:28:18     INFO - GECKO(1728) | #64: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eed6]
[task 2017-03-28T07:28:18.492203Z] 07:28:18     INFO - GECKO(1728) | #65: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eefb]
[task 2017-03-28T07:28:18.494856Z] 07:28:18     INFO - GECKO(1728) | #66: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x1fd30bf]
[task 2017-03-28T07:28:18.498774Z] 07:28:18     INFO - GECKO(1728) | #67: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2d9e054]
[task 2017-03-28T07:28:18.500731Z] 07:28:18     INFO - GECKO(1728) | #68: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1d72b]
[task 2017-03-28T07:28:18.502630Z] 07:28:18     INFO - GECKO(1728) | #69: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1de85]
[task 2017-03-28T07:28:18.504502Z] 07:28:18     INFO - GECKO(1728) | #70: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1e131]
[task 2017-03-28T07:28:18.506452Z] 07:28:18     INFO - GECKO(1728) | #71: ???[/home/worker/workspace/build/application/firefox/firefox +0x551d]
[task 2017-03-28T07:28:18.508543Z] 07:28:18     INFO - GECKO(1728) | #72: ???[/home/worker/workspace/build/application/firefox/firefox +0x4d3d]
[task 2017-03-28T07:28:18.561829Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::GenerateDump cloned child 1901
[task 2017-03-28T07:28:18.566934Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2017-03-28T07:28:18.571547Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2017-03-28T07:28:19.673233Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2017-03-28T07:28:19.681169Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2017-03-28T07:28:19.684580Z] 07:28:19     INFO - GECKO(1728) | [Child 1857] WARNING: pipe error (3): Connection reset by peer: file /home/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 346
[task 2017-03-28T07:28:19.694202Z] 07:28:19     INFO - GECKO(1728) | Hit MOZ_CRASH(Aborting on channel error.) at /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2277
[task 2017-03-28T07:28:19.694600Z] 07:28:19     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8bf513]
[task 2017-03-28T07:28:19.701854Z] 07:28:19     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8bf671]
[task 2017-03-28T07:28:19.707602Z] 07:28:19     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x89c533]
[task 2017-03-28T07:28:19.713404Z] 07:28:19     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88b10c]
[task 2017-03-28T07:28:19.714804Z] 07:28:19     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8a1776]
[task 2017-03-28T07:28:19.717233Z] 07:28:19     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88da79]
[task 2017-03-28T07:28:19.718648Z] 07:28:19     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eed6]
[task 2017-03-28T07:28:19.720576Z] 07:28:19     INFO - GECKO(1728) | #08: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eefb]
[task 2017-03-28T07:28:19.784575Z] 07:28:19     INFO - GECKO(1728) | #09: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8997f7]
[task 2017-03-28T07:28:19.799125Z] 07:28:19     INFO - GECKO(1728) | #10: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88ae3d]
[task 2017-03-28T07:28:19.802593Z] 07:28:19     INFO - GECKO(1728) | #11: ???[/lib/i386-linux-gnu/libpthread.so.0 +0x6d4c]
[task 2017-03-28T07:28:19.823803Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown
This is reproducible in bughunter on Windows.
It's too bad the original stacks don't have any useful information in them...
Priority: -- → P3
Bughunter reproduces this assertion on Windows and Linux Beta/57, Nightly/58 on attachment 8909139 [details]. If you need, I can attach the crash reports and logs if reproducing on that attachment isn't sufficient.
Though, I think this was fixed in bug 1400701 where the test case is attached. This matches the last time orange factor flagged this bug and when bughunter last reproduced this as well. I think you might mark this fixed.
Duplicate of this bug: 1444736
Duplicate of this bug: 1447860
It looks like these assertions are hitting inside CrashReporter::AnnotateCrashReport(). Maybe another thread is also trying to write to the hashtable? But there's a lock so I'm not sure how that would happen.


00:02:09     INFO -  GECKO(5596) | Assertion failure: IsIdle(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:132
00:02:09     INFO -  GECKO(5596) | Assertion failure: IsRead(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:125
00:02:09     INFO -  GECKO(5596) | #01: Checker::StartWriteOp() [xpcom/ds/PLDHashTable.h:130]
00:02:09     INFO -  GECKO(5596) | #02: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &,std::nothrow_t const &) [xpcom/ds/nsBaseHashtable.h:146]
00:02:09     INFO -  GECKO(5596) | #03: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &) [xpcom/ds/nsBaseHashtable.h:137]
00:02:09     INFO -  GECKO(5596) | #04: CrashReporter::AnnotateCrashReport(nsTSubstring<char> const &,nsTSubstring<char> const &) [toolkit/crashreporter/nsExceptionHandler.cpp:2148]
00:02:09     INFO -  GECKO(5596) | #05: nsXULAppInfo::AnnotateCrashReport(nsTSubstring<char> const &,nsTSubstring<char> const &) [toolkit/xre/nsAppRunner.cpp:1340]
Component: XPCOM → Crash Reporting
Product: Core → Toolkit
For Marionette tests those assertions happen for the test test_crash.py, and especially when we trigger a crash of the content process. Haik, given that you fixed some issues lately maybe you are interested. Please see comment 15.
Flags: needinfo?(haftandilian)
Taking what mccr8 said in comment 15 a little further, the two asserts hit here

https://treeherder.mozilla.org/logviewer.html#?job_id=169571172&repo=autoland&lineNumber=24779
00:02:09 INFO - GECKO(5596) | Assertion failure: IsIdle(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:132
00:02:09 INFO - GECKO(5596) | Assertion failure: IsRead(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:125
00:02:09     INFO -  GECKO(5596) | #01: Checker::StartWriteOp() [xpcom/ds/PLDHashTable.h:130]
00:02:09     INFO -  GECKO(5596) | #02: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &,std::nothrow_t const &) [xpcom/ds/nsBaseHashtable.h:146]

are from

   122   void EndReadOp()
   123   {
   124     uint32_t oldState = mState--;     // this is an atomic decrement
   125     MOZ_ASSERT(IsRead(oldState));
   126   }
   127 
   128   void StartWriteOp()
   129   {
   130     MOZ_ASSERT(IsWritable());
   131     uint32_t oldState = mState.exchange(kWrite);
   132     MOZ_ASSERT(IsIdle(oldState));
   133   }

which implies (assuming this isn't memory corruption) we have a read/write race where another thread is doing a read of the hash table |crashReporterAPIData_Hash| without acquiring the lock |crashReporterAPILock| first.

In AnnotateCrashReport(), where we hit the crash, we are holding crashReporterAPILock while we do a Put()/write, but there are other uses of crashReporterAPIData_Hash that don't acquire the lock.

In GetAnnotation() we call crashReporterAPIData_Hash->Get() without the lock which looks like a bug.

And in nsExceptionHandler.cpp - WriteExtraData() we iterate over the table (passed in via the |data| argument) without acquring the lock, but that needs more debugging to determine if the lock is needed. That looks like its called from breakpad in exception context where threads can be suspended and hence locks are problematic (see bug 1457501 for an example.)

I'll take some time to look at the uses of crashReporterAPIData_Hash to try to determine if we're missing locking. Leaving ni for myself.
Assignee: nobody → haftandilian
Priority: P3 → P1
I haven't reproduced the crash yet, but I think this patch is likely to address the problem. It addresses two cases where we access crashReporterAPIData_Hash without acquiring the required lock. Given that this is a crash test, it's probably the use of the hash table from WriteExtraForMinidump() that is causing the problem. I'll work on reproducing this crash and then trying to validate the fix.
Flags: needinfo?(haftandilian)
I've run toolkit/content/tests/browser/browser_crash_previous_frameloader.js for ~500 iterations on OS X and haven't been able to reproduce the crash so far. I'm going to move forward with posted fix which I've run for ~100 iterations of browser_crash_previous_frameloader.js and got a clean run on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7af65fdaa584666364ffe9b1747af3e1e05ac6ed
Attachment #8985387 - Attachment is obsolete: true
Comment on attachment 8985684 [details]
Bug 1351354 - Add missing lock protection to crashReporterAPIData_Hash

https://reviewboard.mozilla.org/r/251216/#review257570

Nice find. Though every time we run into these kind of issues I can't help but think we should mercilessly refactor this code.
Attachment #8985684 - Flags: review?(gsvelto) → review+
Pushed by haftandilian@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/578ef56130c9
Add missing lock protection to crashReporterAPIData_Hash r=gsvelto
https://hg.mozilla.org/mozilla-central/rev/578ef56130c9
Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox62: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
You need to log in before you can comment on or make changes to this bug.