Closed Bug 706751 Opened 8 years ago Closed 7 years ago

Intermittent test_content_annotation.js | command timed out: 1200 seconds without output, attempting to kill

Categories

(Toolkit :: Crash Reporting, defect)

x86_64
macOS
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla22
Tracking Status
firefox19 --- wontfix
firefox20 --- affected
firefox21 --- fixed
firefox22 --- fixed

People

(Reporter: philor, Assigned: ted)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

https://tbpl.mozilla.org/php/getParsedLog.php?id=7675796&tree=Mozilla-Inbound
Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound opt test xpcshell on 2011-11-30 21:31:57 PST for push 43e18ca6cc8c

TEST-INFO | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | running test ...

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTrunk/1329833280.1329837791.26796.gz#err0

TEST-INFO | /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | running test ...

command timed out: 1200 seconds without output, killing pid 200
process killed by signal 9
program finished with exit code -1
elapsedTime=4414.753371
TinderboxPrint: xpcshell<br/><em class="testfail">T-FAIL</em>
buildbot.slave.commands.TimeoutError: command timed out: 1200 seconds without output, killing pid 200
At the moment this seems to be perma-orange on the hidden 10.7 64 bit debug builders:

https://tbpl.mozilla.org/php/getParsedLog.php?id=11419543&tree=Thunderbird-Try
The 10.7 opt (universal) log has more info - could 10.7 builds be missing the .extra file?

https://tbpl.mozilla.org/php/getParsedLog.php?id=11419737&tree=Thunderbird-Try

TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>

TEST-INFO | (xpcshell/head.js) | test 1 pending

parent: TEST-INFO | (xpcshell/head.js) | test 2 pending

parent: TEST-INFO | (xpcshell/head.js) | test 2 finished

parent: TEST-INFO | (xpcshell/head.js) | running event loop
[Child 3421] ###!!! ABORT: Intentional crash: file ../../../../../../mozilla/toolkit/crashreporter/test/nsTestCrasher.cpp, line 62
[Child 3421] ###!!! ABORT: Intentional crash: file ../../../../../../mozilla/toolkit/crashreporter/test/nsTestCrasher.cpp, line 62

parent: TEST-INFO | (xpcshell/head.js) | test 2 pending

parent: TEST-UNEXPECTED-FAIL | ../unit/head_crashreporter.js | false == true - See following stack:
parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_throw :: line 462
parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: _do_check_eq :: line 556
parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_eq :: line 577
parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_true :: line 591
parent: JS frame :: ../unit/head_crashreporter.js :: handleMinidump :: line 97
parent: JS frame :: ../unit/head_crashreporter.js :: <TOP_LEVEL> :: line 131
parent: JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: <TOP_LEVEL> :: line 429

parent: TEST-INFO | (xpcshell/head.js) | exiting test

parent: TEST-UNEXPECTED-FAIL | ../unit/head_crashreporter.js | Unexpected exception 2147500036, see following stack:
undefined

parent: TEST-INFO | (xpcshell/head.js) | exiting test

parent: TEST-INFO | (xpcshell/head.js) | test 2 finished
That seems likely given that log. I wonder if we're writing a minidump but failing, so that the code errors and doesn't get to write the .extra?
I just pushed a patch to try and debug this and happened on the fact that it seems like the .extra file is still being written and adding a delay before checking for the file seems to make the test pass.

I'm now trying a better patch that should be a bit more acceptable for checkin and trying it on all platforms.
Assignee: nobody → mbanner
Attached patch Possible fix (obsolete) — Splinter Review
Somehow this got lost before, hopefully not this time. This is the patch I'd pushed to try, and I must have forgot to check the outcome, so I'm pushing it again.

https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Attached patch Possible fix v2Splinter Review
This time with an added return statement, to make the timeout loop actually take effect.

https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Attachment #654140 - Attachment is obsolete: true
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for 1050fe6f355e is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1050fe6f355e
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-1050fe6f355e
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Try run for b613c8c86363 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b613c8c86363
Results (out of 19 total builds):
    exception: 3
    success: 9
    warnings: 3
    failure: 4
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-b613c8c86363
Depends on: 784688
Sorry about releng bot going wild earlier, I couldn't do much about that ;-)

As to this bug, I've confirmed that the .extra file does not exist for at least 30 seconds. I can therefore only conclude it isn't being written. I tried debug output but it didn't get picked up in xpcshell.

So I think this really does need someone with 10.7 to debug it - I suspect this is an issue with the crashreporter in some form.
Assignee: mbanner → nobody
https://tbpl.mozilla.org/php/getParsedLog.php?id=15169939&tree=Firefox - because if you wait long enough, WinXP can fail *any* test.
Whiteboard: [orange]
Ted, can you look into this? This is one of our longest-running failures on OSX.
Flags: needinfo?(ted)
I was able to reproduce this hang locally, and I think I've tracked down the root cause: bug 848075.
Depends on: 848075
Flags: needinfo?(ted)
Should be fixed by way of bug 848880.
Assignee: nobody → ted
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla22
(In reply to TinderboxPushlog Robot from comment #275)
> edmorley
> https://tbpl.mozilla.org/php/getParsedLog.php?id=20454426&tree=Mozilla-
> Inbound
> Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test mochitest-4 on
> 2013-03-08 03:11:15
> slave: talos-r4-snow-084

This was clearly a mis-star, since this bug is about an xpcshell test and this was a mochitest-4 run.
Yeah my bad, sorry
We could backport just the patch that fixed this if this occurs frequently enough on aurora:
http://code.google.com/p/google-breakpad/source/detail?r=1127

It's a pretty small change.
Absolutely, please request approval. Heck, I'd love to see this on beta too if we can swing it.
This is that patch split out on its own so I can request approval on it.
Comment on attachment 723539 [details] [diff] [review]
Make OOP mac crashreporting exit after writing dump.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): long-standing issue.
User impact if declined: Intermittent orange on tinderbox
Testing completed (on m-c, etc.): landed on m-c, has eliminated this intermittent orange.
Risk to taking this patch (and alternatives if risky): This patch affects the shutdown of the plugin process after a crash has happened. Since we already kill the plugin process in this situation it's unlikely to have any negative impact.
String or UUID changes made by this patch: None

This is a fairly minor change, it simply causes the plugin process to exit immediately after generating a minidump when handling a crash instead of waiting for the browser process to kill it. It fixes a potential deadlock that manifests as an intermittent orange on tinderbox.
Attachment #723539 - Flags: approval-mozilla-aurora?
Comment on attachment 723539 [details] [diff] [review]
Make OOP mac crashreporting exit after writing dump.

 helps avoid intermittent orange, tested on m-c.
Attachment #723539 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(In reply to TinderboxPushlog Robot from comment #286)
> RyanVM
> https://tbpl.mozilla.org/php/getParsedLog.php?id=20536249&tree=Mozilla-Beta
> Rev5 MacOSX Mountain Lion 10.8 mozilla-beta opt test xpcshell on 2013-03-11
> 13:53:20
> slave: talos-mtnlion-r5-060
> 
> command timed out: 1200 seconds without output, attempting to kill

Ted, any reason to not land this on beta also?
It's a fairly safe patch, but nothing is zero risk, and we haven't had it baking that long, so it's probably better to wait and make sure there's no fallout.
You need to log in before you can comment on or make changes to this bug.