1351354 - Intermittent Assertion failure: IsIdle(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:132

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=87007910&repo=mozilla-inbound

This in a tab crash submission test, so don't get confused.

[task 2017-03-28T07:28:17.399046Z] 07:28:17     INFO - GECKO(1728) | XPCOM_MEM_BLOAT_LOG: /tmp/tmppqEaCG.mozrunner/runtests_leaks.log
[task 2017-03-28T07:28:17.400792Z] 07:28:17     INFO - GECKO(1728) | Writing to log: /tmp/tmppqEaCG.mozrunner/runtests_leaks_tab_pid1879.log
[task 2017-03-28T07:28:17.601478Z] 07:28:17     INFO - GECKO(1728) | --DOCSHELL 0xe94d9000 == 0 [pid = 1857] [id = {f91683fa-31f5-4f48-b96c-1b12f825b26f}]
[task 2017-03-28T07:28:18.271318Z] 07:28:18     INFO - GECKO(1728) | Assertion failure: IsIdle(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:132
[task 2017-03-28T07:28:18.273078Z] 07:28:18     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b59d4]
[task 2017-03-28T07:28:18.274488Z] 07:28:18     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b5a0b]
[task 2017-03-28T07:28:18.275875Z] 07:28:18     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8ac5d0]
[task 2017-03-28T07:28:18.279579Z] 07:28:18     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e30177]
[task 2017-03-28T07:28:18.280990Z] 07:28:18     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x509d22]
[task 2017-03-28T07:28:18.282390Z] 07:28:18     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc0c31a]
[task 2017-03-28T07:28:18.284898Z] 07:28:18     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc0c4c7]
[task 2017-03-28T07:28:18.289992Z] 07:28:18     INFO - GECKO(1728) | #08: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc12f94]
[task 2017-03-28T07:28:18.292871Z] 07:28:18     INFO - GECKO(1728) | #09: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.295946Z] 07:28:18     INFO - GECKO(1728) | #10: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92a86]
[task 2017-03-28T07:28:18.299202Z] 07:28:18     INFO - GECKO(1728) | #11: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.309171Z] 07:28:18     INFO - GECKO(1728) | #12: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.311292Z] 07:28:18     INFO - GECKO(1728) | #13: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.314501Z] 07:28:18     INFO - GECKO(1728) | Assertion failure: IsRead(oldState), at /home/worker/workspace/build/src/xpcom/ds/PLDHashTable.h:125
[task 2017-03-28T07:28:18.316706Z] 07:28:18     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x4b1280]
[task 2017-03-28T07:28:18.328256Z] 07:28:18     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2c6b8]
[task 2017-03-28T07:28:18.331443Z] 07:28:18     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2e4ec]
[task 2017-03-28T07:28:18.335736Z] 07:28:18     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e2f5be]
[task 2017-03-28T07:28:18.337831Z] 07:28:18     INFO - GECKO(1728) | #14: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.340059Z] 07:28:18     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37a7b]
[task 2017-03-28T07:28:18.342856Z] 07:28:18     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37b30]
[task 2017-03-28T07:28:18.353488Z] 07:28:18     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e37b66]
[task 2017-03-28T07:28:18.355402Z] 07:28:18     INFO - GECKO(1728) | #08: ???[/lib/i386-linux-gnu/libpthread.so.0 +0x6d4c]
[task 2017-03-28T07:28:18.359190Z] 07:28:18     INFO - GECKO(1728) | #15: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.364151Z] 07:28:18     INFO - GECKO(1728) | #16: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.367500Z] 07:28:18     INFO - GECKO(1728) | #17: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34a85ce]
[task 2017-03-28T07:28:18.369527Z] 07:28:18     INFO - GECKO(1728) | #18: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34981f4]
[task 2017-03-28T07:28:18.371539Z] 07:28:18     INFO - GECKO(1728) | #19: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e5b5]
[task 2017-03-28T07:28:18.373787Z] 07:28:18     INFO - GECKO(1728) | #20: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e674]
[task 2017-03-28T07:28:18.378023Z] 07:28:18     INFO - GECKO(1728) | #21: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.379954Z] 07:28:18     INFO - GECKO(1728) | #22: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92cc9]
[task 2017-03-28T07:28:18.383590Z] 07:28:18     INFO - GECKO(1728) | #23: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.385515Z] 07:28:18     INFO - GECKO(1728) | #24: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.388451Z] 07:28:18     INFO - GECKO(1728) | #25: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.390455Z] 07:28:18     INFO - GECKO(1728) | #26: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.400129Z] 07:28:18     INFO - GECKO(1728) | #27: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.402034Z] 07:28:18     INFO - GECKO(1728) | #28: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.403906Z] 07:28:18     INFO - GECKO(1728) | #29: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.405770Z] 07:28:18     INFO - GECKO(1728) | #30: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.408883Z] 07:28:18     INFO - GECKO(1728) | #31: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.410784Z] 07:28:18     INFO - GECKO(1728) | #32: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.412690Z] 07:28:18     INFO - GECKO(1728) | #33: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x3419fea]
[task 2017-03-28T07:28:18.414768Z] 07:28:18     INFO - GECKO(1728) | #34: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.418241Z] 07:28:18     INFO - GECKO(1728) | #35: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92a86]
[task 2017-03-28T07:28:18.420146Z] 07:28:18     INFO - GECKO(1728) | #36: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.422088Z] 07:28:18     INFO - GECKO(1728) | #37: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.423997Z] 07:28:18     INFO - GECKO(1728) | #38: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34a85ce]
[task 2017-03-28T07:28:18.426000Z] 07:28:18     INFO - GECKO(1728) | #39: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x34981f4]
[task 2017-03-28T07:28:18.429374Z] 07:28:18     INFO - GECKO(1728) | #40: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e5b5]
[task 2017-03-28T07:28:18.431333Z] 07:28:18     INFO - GECKO(1728) | #41: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x349e674]
[task 2017-03-28T07:28:18.435510Z] 07:28:18     INFO - GECKO(1728) | #42: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f714fa]
[task 2017-03-28T07:28:18.437711Z] 07:28:18     INFO - GECKO(1728) | #43: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92cc9]
[task 2017-03-28T07:28:18.439696Z] 07:28:18     INFO - GECKO(1728) | #44: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.441608Z] 07:28:18     INFO - GECKO(1728) | #45: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f852bd]
[task 2017-03-28T07:28:18.443500Z] 07:28:18     INFO - GECKO(1728) | #46: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92680]
[task 2017-03-28T07:28:18.445875Z] 07:28:18     INFO - GECKO(1728) | #47: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92c2b]
[task 2017-03-28T07:28:18.448262Z] 07:28:18     INFO - GECKO(1728) | #48: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92edd]
[task 2017-03-28T07:28:18.452084Z] 07:28:18     INFO - GECKO(1728) | #49: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2f92fa7]
[task 2017-03-28T07:28:18.453998Z] 07:28:18     INFO - GECKO(1728) | #50: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x33c6b29]
[task 2017-03-28T07:28:18.455871Z] 07:28:18     INFO - GECKO(1728) | #51: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc116b7]
[task 2017-03-28T07:28:18.457780Z] 07:28:18     INFO - GECKO(1728) | #52: ???[/home/worker/workspace/build/application/firefox/libxul.so +0xc121fa]
[task 2017-03-28T07:28:18.463595Z] 07:28:18     INFO - GECKO(1728) | #53: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x50a899]
[task 2017-03-28T07:28:18.466140Z] 07:28:18     INFO - GECKO(1728) | #54: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x5234b5]
[task 2017-03-28T07:28:18.468557Z] 07:28:18     INFO - GECKO(1728) | #55: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d65a]
[task 2017-03-28T07:28:18.470898Z] 07:28:18     INFO - GECKO(1728) | #56: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d92f]
[task 2017-03-28T07:28:18.474279Z] 07:28:18     INFO - GECKO(1728) | #57: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x51d98e]
[task 2017-03-28T07:28:18.476215Z] 07:28:18     INFO - GECKO(1728) | #58: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x52038b]
[task 2017-03-28T07:28:18.478544Z] 07:28:18     INFO - GECKO(1728) | #59: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x70a2d7]
[task 2017-03-28T07:28:18.480568Z] 07:28:18     INFO - GECKO(1728) | #60: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x70187b]
[task 2017-03-28T07:28:18.482643Z] 07:28:18     INFO - GECKO(1728) | #61: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x501a80]
[task 2017-03-28T07:28:18.485029Z] 07:28:18     INFO - GECKO(1728) | #62: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x503d44]
[task 2017-03-28T07:28:18.487135Z] 07:28:18     INFO - GECKO(1728) | #63: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8c1928]
[task 2017-03-28T07:28:18.490241Z] 07:28:18     INFO - GECKO(1728) | #64: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eed6]
[task 2017-03-28T07:28:18.492203Z] 07:28:18     INFO - GECKO(1728) | #65: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eefb]
[task 2017-03-28T07:28:18.494856Z] 07:28:18     INFO - GECKO(1728) | #66: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x1fd30bf]
[task 2017-03-28T07:28:18.498774Z] 07:28:18     INFO - GECKO(1728) | #67: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2d9e054]
[task 2017-03-28T07:28:18.500731Z] 07:28:18     INFO - GECKO(1728) | #68: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1d72b]
[task 2017-03-28T07:28:18.502630Z] 07:28:18     INFO - GECKO(1728) | #69: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1de85]
[task 2017-03-28T07:28:18.504502Z] 07:28:18     INFO - GECKO(1728) | #70: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x2e1e131]
[task 2017-03-28T07:28:18.506452Z] 07:28:18     INFO - GECKO(1728) | #71: ???[/home/worker/workspace/build/application/firefox/firefox +0x551d]
[task 2017-03-28T07:28:18.508543Z] 07:28:18     INFO - GECKO(1728) | #72: ???[/home/worker/workspace/build/application/firefox/firefox +0x4d3d]
[task 2017-03-28T07:28:18.561829Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::GenerateDump cloned child 1901
[task 2017-03-28T07:28:18.566934Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2017-03-28T07:28:18.571547Z] 07:28:18     INFO - GECKO(1728) | ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2017-03-28T07:28:19.673233Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2017-03-28T07:28:19.681169Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2017-03-28T07:28:19.684580Z] 07:28:19     INFO - GECKO(1728) | [Child 1857] WARNING: pipe error (3): Connection reset by peer: file /home/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 346
[task 2017-03-28T07:28:19.694202Z] 07:28:19     INFO - GECKO(1728) | Hit MOZ_CRASH(Aborting on channel error.) at /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2277
[task 2017-03-28T07:28:19.694600Z] 07:28:19     INFO - GECKO(1728) | #01: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8bf513]
[task 2017-03-28T07:28:19.701854Z] 07:28:19     INFO - GECKO(1728) | #02: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8bf671]
[task 2017-03-28T07:28:19.707602Z] 07:28:19     INFO - GECKO(1728) | #03: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x89c533]
[task 2017-03-28T07:28:19.713404Z] 07:28:19     INFO - GECKO(1728) | #04: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88b10c]
[task 2017-03-28T07:28:19.714804Z] 07:28:19     INFO - GECKO(1728) | #05: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8a1776]
[task 2017-03-28T07:28:19.717233Z] 07:28:19     INFO - GECKO(1728) | #06: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88da79]
[task 2017-03-28T07:28:19.718648Z] 07:28:19     INFO - GECKO(1728) | #07: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eed6]
[task 2017-03-28T07:28:19.720576Z] 07:28:19     INFO - GECKO(1728) | #08: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88eefb]
[task 2017-03-28T07:28:19.784575Z] 07:28:19     INFO - GECKO(1728) | #09: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x8997f7]
[task 2017-03-28T07:28:19.799125Z] 07:28:19     INFO - GECKO(1728) | #10: ???[/home/worker/workspace/build/application/firefox/libxul.so +0x88ae3d]
[task 2017-03-28T07:28:19.802593Z] 07:28:19     INFO - GECKO(1728) | #11: ???[/lib/i386-linux-gnu/libpthread.so.0 +0x6d4c]
[task 2017-03-28T07:28:19.823803Z] 07:28:19     INFO - GECKO(1728) | [GFX1-]: Receive IPC close with reason=AbnormalShutdown

Comment 1

[Bob Clary [:bc]]

This is reproducible in bughunter on Windows.

Comment 2

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 1032 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 4

[Nathan Froyd [:froydnj]]

It's too bad the original stacks don't have any useful information in them...

Comment 5

[Bob Clary [:bc]]

Bughunter reproduces this assertion on Windows and Linux Beta/57, Nightly/58 on attachment 8909139 [details]. If you need, I can attach the crash reports and logs if reproducing on that attachment isn't sufficient.

Comment 6

[Bob Clary [:bc]]

Though, I think this was fixed in bug 1400701 where the test case is attached. This matches the last time orange factor flagged this bug and when bughunter last reproduced this as well. I think you might mark this fixed.

Comment 7

[Intermittent Failures Robot]

2 failures in 864 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 1
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 8

[Intermittent Failures Robot]

5 failures in 912 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 2
* windows10-64: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 9

[Intermittent Failures Robot]

3 failures in 857 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* linux32: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 792 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 11

[Intermittent Failures Robot]

3 failures in 590 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 2
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 12

[Intermittent Failures Robot]

1 failures in 747 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1

Platform breakdown:
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1351354&startday=2018-03-05&endday=2018-03-11&tree=all

Comment 15

[Andrew McCreight [:mccr8]]

It looks like these assertions are hitting inside CrashReporter::AnnotateCrashReport(). Maybe another thread is also trying to write to the hashtable? But there's a lock so I'm not sure how that would happen.


00:02:09     INFO -  GECKO(5596) | Assertion failure: IsIdle(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:132
00:02:09     INFO -  GECKO(5596) | Assertion failure: IsRead(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:125
00:02:09     INFO -  GECKO(5596) | #01: Checker::StartWriteOp() [xpcom/ds/PLDHashTable.h:130]
00:02:09     INFO -  GECKO(5596) | #02: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &,std::nothrow_t const &) [xpcom/ds/nsBaseHashtable.h:146]
00:02:09     INFO -  GECKO(5596) | #03: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &) [xpcom/ds/nsBaseHashtable.h:137]
00:02:09     INFO -  GECKO(5596) | #04: CrashReporter::AnnotateCrashReport(nsTSubstring<char> const &,nsTSubstring<char> const &) [toolkit/crashreporter/nsExceptionHandler.cpp:2148]
00:02:09     INFO -  GECKO(5596) | #05: nsXULAppInfo::AnnotateCrashReport(nsTSubstring<char> const &,nsTSubstring<char> const &) [toolkit/xre/nsAppRunner.cpp:1340]

Comment 16

[Intermittent Failures Robot]

2 failures in 639 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 1
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1351354&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 17

[Intermittent Failures Robot]

2 failures in 655 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2

Platform breakdown:
* osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1351354&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 18

[Henrik Skupin (:whimboo) [⌚️UTC+2] (away 07/24 - 08/10)]

For Marionette tests those assertions happen for the test test_crash.py, and especially when we trigger a crash of the content process. Haik, given that you fixed some issues lately maybe you are interested. Please see comment 15.

Comment 19

[Haik Aftandilian [:haik]]

Taking what mccr8 said in comment 15 a little further, the two asserts hit here

https://treeherder.mozilla.org/logviewer.html#?job_id=169571172&repo=autoland&lineNumber=24779
00:02:09 INFO - GECKO(5596) | Assertion failure: IsIdle(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:132
00:02:09 INFO - GECKO(5596) | Assertion failure: IsRead(oldState), at z:\build\build\src\xpcom\ds\PLDHashTable.h:125
00:02:09     INFO -  GECKO(5596) | #01: Checker::StartWriteOp() [xpcom/ds/PLDHashTable.h:130]
00:02:09     INFO -  GECKO(5596) | #02: nsBaseHashtable<nsCStringHashKey,nsTString<char>,nsTString<char> >::Put(nsTSubstring<char> const &,nsTString<char> const &,std::nothrow_t const &) [xpcom/ds/nsBaseHashtable.h:146]

are from

   122   void EndReadOp()
   123   {
   124     uint32_t oldState = mState--;     // this is an atomic decrement
   125     MOZ_ASSERT(IsRead(oldState));
   126   }
   127 
   128   void StartWriteOp()
   129   {
   130     MOZ_ASSERT(IsWritable());
   131     uint32_t oldState = mState.exchange(kWrite);
   132     MOZ_ASSERT(IsIdle(oldState));
   133   }

which implies (assuming this isn't memory corruption) we have a read/write race where another thread is doing a read of the hash table |crashReporterAPIData_Hash| without acquiring the lock |crashReporterAPILock| first.

In AnnotateCrashReport(), where we hit the crash, we are holding crashReporterAPILock while we do a Put()/write, but there are other uses of crashReporterAPIData_Hash that don't acquire the lock.

In GetAnnotation() we call crashReporterAPIData_Hash->Get() without the lock which looks like a bug.

And in nsExceptionHandler.cpp - WriteExtraData() we iterate over the table (passed in via the |data| argument) without acquring the lock, but that needs more debugging to determine if the lock is needed. That looks like its called from breakpad in exception context where threads can be suspended and hence locks are problematic (see bug 1457501 for an example.)

I'll take some time to look at the uses of crashReporterAPIData_Hash to try to determine if we're missing locking. Leaving ni for myself.

Comment 20

[Intermittent Failures Robot]

2 failures in 669 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows10-64-ccov: 1
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1351354&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 21

[Haik Aftandilian [:haik]]

Attachment: WIP Patch to add missing locking protection for crashReporterAPIData_Hash

I haven't reproduced the crash yet, but I think this patch is likely to address the problem. It addresses two cases where we access crashReporterAPIData_Hash without acquiring the required lock. Given that this is a crash test, it's probably the use of the hash table from WriteExtraForMinidump() that is causing the problem. I'll work on reproducing this crash and then trying to validate the fix.

Comment 22

[Haik Aftandilian [:haik]]

I've run toolkit/content/tests/browser/browser_crash_previous_frameloader.js for ~500 iterations on OS X and haven't been able to reproduce the crash so far. I'm going to move forward with posted fix which I've run for ~100 iterations of browser_crash_previous_frameloader.js and got a clean run on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7af65fdaa584666364ffe9b1747af3e1e05ac6ed

Comment 23

[Haik Aftandilian [:haik]]

Attachment: Bug 1351354 - Add missing lock protection to crashReporterAPIData_Hash

Add missing mutex acquisition calls to protect crashReporterAPIData_Hash and avoid
races with CrashReporter::AnnotateCrashReport() that cause assertion failures.

Review commit: https://reviewboard.mozilla.org/r/251216/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/251216/

Comment 24

[Gabriele Svelto [:gsvelto]]

Comment on attachment 8985684 [details]
Bug 1351354 - Add missing lock protection to crashReporterAPIData_Hash

https://reviewboard.mozilla.org/r/251216/#review257570

Nice find. Though every time we run into these kind of issues I can't help but think we should mercilessly refactor this code.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/578ef56130c9
Add missing lock protection to crashReporterAPIData_Hash r=gsvelto

Comment 26

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/578ef56130c9

Comment 27

[Intermittent Failures Robot]

1 failures in 2383 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1351354&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 2561 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1351354&startday=2019-03-18&endday=2019-03-24&tree=all