1560598 - Intermittent Automation Error: mozprocess timed out after 1000 seconds, after "Killing GPU process due to IPC reply timeout"

Status: RESOLVED INCOMPLETE

(Core :: Graphics, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=252835185&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/KdYtp-arQgaqvNYw_WoFhA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/KdYtp-arQgaqvNYw_WoFhA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2019-06-21T18:24:33.992Z] 18:24:33 INFO - 'WORKING_DIR': '/builds/worker'}
[task 2019-06-21T18:24:33.992Z] 18:24:33 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/reftest/runreftest.py', '--total-chunks', '8', '--this-chunk', '8', '--appname=/builds/worker/workspace/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', u'--symbols-path=https://queue.taskcluster.net/v1/task/b0_BHxv7SXOD8URFtoiHLw/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/reftest_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=/builds/worker/workspace/build', '--log-raw=-', '--suite=reftest', '--setpref=layers.acceleration.force-enabled=true', '--', 'tests/reftest/tests/layout/reftests/reftest.list'] with output_timeout 1000
[task 2019-06-21T18:24:39.470Z] 18:24:39 INFO - REFTEST INFO | INFO | runtests.py | ASan using symbolizer at /builds/worker/workspace/build/application/firefox/llvm-symbolizer
[task 2019-06-21T18:24:39.479Z] 18:24:39 INFO - REFTEST INFO | INFO | runtests.py | ASan running in default memory configuration
[task 2019-06-21T18:24:39.479Z] 18:24:39 INFO - REFTEST INFO | Running with e10s: True
[task 2019-06-21T18:24:39.479Z] 18:24:39 INFO - REFTEST INFO | Application command: /builds/worker/workspace/build/application/firefox/firefox -marionette -profile /tmp/tmpG1BSGU.mozrunner
[task 2019-06-21T18:25:00.409Z] 18:25:00 INFO - 1561141500402 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
[task 2019-06-21T18:25:00.411Z] 18:25:00 INFO - 1561141500403 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: telemetry
[task 2019-06-21T18:25:00.413Z] 18:25:00 INFO - 1561141500404 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
[task 2019-06-21T18:25:00.415Z] 18:25:00 INFO - 1561141500404 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
[task 2019-06-21T18:25:01.223Z] 18:25:01 INFO - 1561141501216 Marionette TRACE Received observer notification profile-after-change
[task 2019-06-21T18:25:01.482Z] 18:25:01 INFO - 1561141501471 Marionette TRACE Received observer notification command-line-startup
[task 2019-06-21T18:25:01.482Z] 18:25:01 INFO - 1561141501472 Marionette TRACE Received observer notification nsPref:changed
[task 2019-06-21T18:25:01.482Z] 18:25:01 INFO - 1561141501473 Marionette DEBUG Init aborted (running=false, enabled=true, finalUIStartup=false)
[task 2019-06-21T18:25:07.846Z] 18:25:07 INFO - 1561141507837 Marionette TRACE Received observer notification toplevel-window-ready
[task 2019-06-21T18:25:22.822Z] 18:25:22 INFO - Crash Annotation GraphicsCriticalError: |[0][GFX1-]: Killing GPU process due to IPC reply timeout (t=43.3404) [GFX1-]: Killing GPU process due to IPC reply timeout
[task 2019-06-21T18:25:23.003Z] 18:25:23 INFO - Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=13.0208) [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2019-06-21T18:25:23.004Z] 18:25:23 INFO - Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=13.9336) [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2019-06-21T18:25:23.040Z] 18:25:23 INFO - Crash Annotation GraphicsCriticalError: |[0][GFX1-]: Killing GPU process due to IPC reply timeout (t=43.3404) |[1]CP+[GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=43.5528) |[2][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=43.553) [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2019-06-21T18:25:23.040Z] 18:25:23 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x4D001A,name=PGPU::Msg_ShutdownVR) Channel error: cannot send/recv
[task 2019-06-21T18:42:03.066Z] 18:42:03 INFO - Automation Error: mozprocess timed out after 1000 seconds running ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/reftest/runreftest.py', '--total-chunks', '8', '--this-chunk', '8', '--appname=/builds/worker/workspace/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', u'--symbols-path=https://queue.taskcluster.net/v1/task/b0_BHxv7SXOD8URFtoiHLw/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/reftest_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=/builds/worker/workspace/build', '--log-raw=-', '--suite=reftest', '--setpref=layers.acceleration.force-enabled=true', '--', 'tests/reftest/tests/layout/reftests/reftest.list']
[task 2019-06-21T18:42:03.103Z] 18:42:03 ERROR - timed out after 1000 seconds of no output
[task 2019-06-21T18:42:03.104Z] 18:42:03 ERROR - Return code: -15
[task 2019-06-21T18:42:03.119Z] 18:42:03 ERROR - No checks run.
[task 2019-06-21T18:42:03.120Z] 18:42:03 ERROR - No suite end message was emitted by this harness.
[task 2019-06-21T18:42:03.121Z] 18:42:03 INFO - TinderboxPrint: reftest-reftest<br/><em class="testfail">T-FAIL</em>
[task 2019-06-21T18:42:03.121Z] 18:42:03 ERROR - # TBPL FAILURE #
[task 2019-06-21T18:42:03.122Z] 18:42:03 WARNING - setting return code to 2
[task 2019-06-21T18:42:03.122Z] 18:42:03 ERROR - The reftest suite: reftest ran with return status: FAILURE

Comment 1

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

The full error message is Killing GPU process due to IPC reply timeout, which is here.

Comment 2

[Intermittent Failures Robot]

1 failures in 2581 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1560598&startday=2019-06-17&endday=2019-06-23&tree=all

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:jld, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Intermittent Failures Robot]

1 failures in 4710 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• linux64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1560598&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

I think the question we should be asking is: what is the GPU process doing instead of answering its messages?

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.