Closed Bug 706751 Opened 14 years ago Closed 12 years ago

Intermittent test_content_annotation.js | command timed out: 1200 seconds without output, attempting to kill

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Platform:
x86_64
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla22
Tracking Flags:
Tracking Status
firefox19 --- wontfix
firefox20 --- affected
firefox21 --- fixed
firefox22 --- fixed

People

(Reporter: philor, Assigned: ted)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

Possible fix v2
1.92 KB, patch
Details | Diff | Splinter Review
Make OOP mac crashreporting exit after writing dump.
1.64 KB, patch
bajaj
: approval-mozilla-aurora+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=7675796&tree=Mozilla-Inbound Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound opt test xpcshell on 2011-11-30 21:31:57 PST for push 43e18ca6cc8c TEST-INFO | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | running test ... command timed out: 1200 seconds without output, attempting to kill process killed by signal 9 program finished with exit code -1
http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTrunk/1329833280.1329837791.26796.gz#err0 TEST-INFO | /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | running test ... command timed out: 1200 seconds without output, killing pid 200 process killed by signal 9 program finished with exit code -1 elapsedTime=4414.753371 TinderboxPrint: xpcshell<br/><em class="testfail">T-FAIL</em> buildbot.slave.commands.TimeoutError: command timed out: 1200 seconds without output, killing pid 200
At the moment this seems to be perma-orange on the hidden 10.7 64 bit debug builders: https://tbpl.mozilla.org/php/getParsedLog.php?id=11419543&tree=Thunderbird-Try
The 10.7 opt (universal) log has more info - could 10.7 builds be missing the .extra file? https://tbpl.mozilla.org/php/getParsedLog.php?id=11419737&tree=Thunderbird-Try TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | test failed (with xpcshell return code: 0), see following log: >>>>>>> TEST-INFO | (xpcshell/head.js) | test 1 pending parent: TEST-INFO | (xpcshell/head.js) | test 2 pending parent: TEST-INFO | (xpcshell/head.js) | test 2 finished parent: TEST-INFO | (xpcshell/head.js) | running event loop [Child 3421] ###!!! ABORT: Intentional crash: file ../../../../../../mozilla/toolkit/crashreporter/test/nsTestCrasher.cpp, line 62 [Child 3421] ###!!! ABORT: Intentional crash: file ../../../../../../mozilla/toolkit/crashreporter/test/nsTestCrasher.cpp, line 62 parent: TEST-INFO | (xpcshell/head.js) | test 2 pending parent: TEST-UNEXPECTED-FAIL | ../unit/head_crashreporter.js | false == true - See following stack: parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_throw :: line 462 parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: _do_check_eq :: line 556 parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_eq :: line 577 parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_true :: line 591 parent: JS frame :: ../unit/head_crashreporter.js :: handleMinidump :: line 97 parent: JS frame :: ../unit/head_crashreporter.js :: <TOP_LEVEL> :: line 131 parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: <TOP_LEVEL> :: line 429 parent: TEST-INFO | (xpcshell/head.js) | exiting test parent: TEST-UNEXPECTED-FAIL | ../unit/head_crashreporter.js | Unexpected exception 2147500036, see following stack: undefined parent: TEST-INFO | (xpcshell/head.js) | exiting test parent: TEST-INFO | (xpcshell/head.js) | test 2 finished
That seems likely given that log. I wonder if we're writing a minidump but failing, so that the code errors and doesn't get to write the .extra?
I just pushed a patch to try and debug this and happened on the fact that it seems like the .extra file is still being written and adding a delay before checking for the file seems to make the test pass. I'm now trying a better patch that should be a bit more acceptable for checkin and trying it on all platforms.
Assignee: nobody → mbanner
Attached patch Possible fix (obsolete) — — Splinter Review
Somehow this got lost before, hopefully not this time. This is the patch I'd pushed to try, and I must have forgot to check the outcome, so I'm pushing it again. https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Attached patch Possible fix v2 — — Splinter Review
This time with an added return statement, to make the timeout loop actually take effect. https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Attachment #654140 - Attachment is obsolete: true
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363 Results (out of 19 total builds): exception: 3 success: 9 warnings: 3 failure: 4 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Depends on: 784688
Sorry about releng bot going wild earlier, I couldn't do much about that ;-) As to this bug, I've confirmed that the .extra file does not exist for at least 30 seconds. I can therefore only conclude it isn't being written. I tried debug output but it didn't get picked up in xpcshell. So I think this really does need someone with 10.7 to debug it - I suspect this is an issue with the crashreporter in some form.
Assignee: mbanner → nobody
https://tbpl.mozilla.org/php/getParsedLog.php?id=15169939&tree=Firefox - because if you wait long enough, WinXP can fail *any* test.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15144703&tree=Ionmonkey - twice, if somebody maybe broke something.
Whiteboard: [orange]
Ted, can you look into this? This is one of our longest-running failures on OSX.
Flags: needinfo?(ted)
I was able to reproduce this hang locally, and I think I've tracked down the root cause: bug 848075.
Depends on: 848075
Flags: needinfo?(ted)
Should be fixed by way of bug 848880.
Assignee: nobody → ted
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla22
(In reply to TinderboxPushlog Robot from comment #275) > edmorley > https://tbpl.mozilla.org/php/getParsedLog.php?id=20454426&tree=Mozilla- > Inbound > Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test mochitest-4 on > 2013-03-08 03:11:15 > slave: talos-r4-snow-084 This was clearly a mis-star, since this bug is about an xpcshell test and this was a mochitest-4 run.
Yeah my bad, sorry
We could backport just the patch that fixed this if this occurs frequently enough on aurora: http://code.google.com/p/google-breakpad/source/detail?r=1127 It's a pretty small change.
Absolutely, please request approval. Heck, I'd love to see this on beta too if we can swing it.
This is that patch split out on its own so I can request approval on it.
Comment on attachment 723539 [details] [diff] [review] Make OOP mac crashreporting exit after writing dump. [Approval Request Comment] Bug caused by (feature/regressing bug #): long-standing issue. User impact if declined: Intermittent orange on tinderbox Testing completed (on m-c, etc.): landed on m-c, has eliminated this intermittent orange. Risk to taking this patch (and alternatives if risky): This patch affects the shutdown of the plugin process after a crash has happened. Since we already kill the plugin process in this situation it's unlikely to have any negative impact. String or UUID changes made by this patch: None This is a fairly minor change, it simply causes the plugin process to exit immediately after generating a minidump when handling a crash instead of waiting for the browser process to kill it. It fixes a potential deadlock that manifests as an intermittent orange on tinderbox.
Attachment #723539 - Flags: approval-mozilla-aurora?
Comment on attachment 723539 [details] [diff] [review] Make OOP mac crashreporting exit after writing dump. helps avoid intermittent orange, tested on m-c.
Attachment #723539 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(In reply to TinderboxPushlog Robot from comment #286) > RyanVM > https://tbpl.mozilla.org/php/getParsedLog.php?id=20536249&tree=Mozilla-Beta > Rev5 MacOSX Mountain Lion 10.8 mozilla-beta opt test xpcshell on 2013-03-11 > 13:53:20 > slave: talos-mtnlion-r5-060 > > command timed out: 1200 seconds without output, attempting to kill Ted, any reason to not land this on beta also?
It's a fairly safe patch, but nothing is zero risk, and we haven't had it baking that long, so it's probably better to wait and make sure there's no fallout.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: