Closed Bug 1714137 Opened 6 months ago Closed 4 months ago

Intermittent The mochitest suite: mochitest-chrome ran with return status: FAILURE after Return code: 1

Categories

(Testing :: Mochitest, defect, P5)

defect

Tracking

(firefox93 fixed)

RESOLVED FIXED
93 Branch
Tracking Status
firefox93 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: ahal)

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(1 file, 1 obsolete file)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=341552745&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BnjA5OfrRv6fhXFOWv7EtA/runs/0/artifacts/public/logs/live_backing.log


[task 2021-06-02T16:31:04.046Z] 16:31:04     INFO - TEST-START | dom/ipc/tests/test_process_error_oom.xhtml
[task 2021-06-02T16:31:04.046Z] 16:31:04     INFO - TEST-SKIP | dom/ipc/tests/test_process_error_oom.xhtml | took 0ms
[task 2021-06-02T16:31:04.046Z] 16:31:04     INFO -  0 INFO TEST-START | Shutdown
[task 2021-06-02T16:31:04.047Z] 16:31:04     INFO -  1 INFO Passed:  0
[task 2021-06-02T16:31:04.047Z] 16:31:04     INFO -  2 INFO Failed:  0
[task 2021-06-02T16:31:04.047Z] 16:31:04     INFO -  3 INFO Todo:    0
[task 2021-06-02T16:31:04.048Z] 16:31:04     INFO -  4 INFO Mode:    non-e10s
[task 2021-06-02T16:31:04.048Z] 16:31:04     INFO -  5 INFO SimpleTest FINISHED
[task 2021-06-02T16:31:04.048Z] 16:31:04     INFO - Buffered messages finished
[task 2021-06-02T16:31:04.048Z] 16:31:04     INFO - SUITE-END | took 0s
[task 2021-06-02T16:31:04.086Z] 16:31:04    ERROR - Return code: 1
[task 2021-06-02T16:31:04.087Z] 16:31:04     INFO - TinderboxPrint: mochitest-mochitest-chrome<br/>2/0/0
[task 2021-06-02T16:31:04.088Z] 16:31:04    ERROR - # TBPL FAILURE #
[task 2021-06-02T16:31:04.088Z] 16:31:04  WARNING - setting return code to 2
[task 2021-06-02T16:31:04.088Z] 16:31:04    ERROR - The mochitest suite: mochitest-chrome ran with return status: FAILURE
Summary: Intermittent The mochitest suite: mochitest-chrome ran with return status: FAILURE → Intermittent The mochitest suite: mochitest-chrome ran with return status: FAILURE after Return code: 1

Update:

There have been 58 failures within the last 7 days:

  • 7 failures on Linux 18.04 x64 WebRender asan opt
  • 51 failures on Linux 18.04 x64 asan opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=345468828&repo=autoland&lineNumber=2259

[task 2021-07-16T18:35:54.572Z] 18:35:54     INFO -  0 INFO TEST-START | Shutdown
[task 2021-07-16T18:35:54.573Z] 18:35:54     INFO -  1 INFO Passed:  0
[task 2021-07-16T18:35:54.573Z] 18:35:54     INFO -  2 INFO Failed:  0
[task 2021-07-16T18:35:54.574Z] 18:35:54     INFO -  3 INFO Todo:    0
[task 2021-07-16T18:35:54.575Z] 18:35:54     INFO -  4 INFO Mode:    non-e10s
[task 2021-07-16T18:35:54.575Z] 18:35:54     INFO -  5 INFO SimpleTest FINISHED
[task 2021-07-16T18:35:54.576Z] 18:35:54     INFO - Buffered messages finished
[task 2021-07-16T18:35:54.577Z] 18:35:54     INFO - SUITE-END | took 0s
[task 2021-07-16T18:35:54.597Z] 18:35:54    ERROR - Return code: 1
[task 2021-07-16T18:35:54.598Z] 18:35:54     INFO - TinderboxPrint: mochitest-mochitest-chrome<br/>2/0/0
[task 2021-07-16T18:35:54.599Z] 18:35:54    ERROR - # TBPL FAILURE #
[task 2021-07-16T18:35:54.600Z] 18:35:54  WARNING - setting return code to 2
[task 2021-07-16T18:35:54.600Z] 18:35:54    ERROR - The mochitest suite: mochitest-chrome ran with return status: FAILURE
[task 2021-07-16T18:35:54.600Z] 18:35:54     INFO - Running post-action listener: _package_coverage_data
[task 2021-07-16T18:35:54.601Z] 18:35:54     INFO - Running post-action listener: _resource_record_post_action
[task 2021-07-16T18:35:54.601Z] 18:35:54     INFO - Running post-action listener: process_java_coverage_data
[task 2021-07-16T18:35:54.601Z] 18:35:54     INFO - [mozharness: 2021-07-16 18:35:54.599245Z] Finished run-tests step (success)
[task 2021-07-16T18:35:54.601Z] 18:35:54     INFO - Running post-run listener: _resource_record_post_run
[task 2021-07-16T18:35:54.651Z] 18:35:54     INFO - Validating Perfherder data against /builds/worker/workspace/mozharness/external_tools/performance-artifact-schema.json
[task 2021-07-16T18:35:54.654Z] 18:35:54     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"name": "mochitest.mochitest-chrome.overall", "extraOptions": ["taskcluster-m5.large"], "subtests": [{"name": "cpu_percent", "value": 50.641999999999996}, {"name": "io_write_bytes", "value": 1931165696}, {"name": "io.read_bytes", "value": 25354240}, {"name": "io_write_time", "value": 233388}, {"name": "io_read_time", "value": 264}]}, {"name": "mochitest.mochitest-chrome.start-pulseaudio", "subtests": [{"name": "time", "value": 0.03464007377624512}]}, {"name": "mochitest.mochitest-chrome.install", "subtests": [{"name": "time", "value": 49.81815528869629}, {"name": "cpu_percent", "value": 50.57395833333334}]}, {"name": "mochitest.mochitest-chrome.stage-files", "subtests": [{"name": "time", "value": 0.0010485649108886719}]}, {"name": "mochitest.mochitest-chrome.run-tests", "subtests": [{"name": "time", "value": 0.44002318382263184}]}]}
[task 2021-07-16T18:35:54.655Z] 18:35:54     INFO - Total resource usage - Wall time: 50s; CPU: Can't collect data; Read bytes: 25354240; Write bytes: 1931165696; Read time: 264; Write time: 233388
[task 2021-07-16T18:35:54.656Z] 18:35:54     INFO - TinderboxPrint: I/O read bytes / time<br/>25,354,240 / 264
[task 2021-07-16T18:35:54.658Z] 18:35:54     INFO - TinderboxPrint: I/O write bytes / time<br/>1,931,165,696 / 233,388
[task 2021-07-16T18:35:54.659Z] 18:35:54     INFO - TinderboxPrint: CPU idle<br/>43.8 (43.8%)
[task 2021-07-16T18:35:54.660Z] 18:35:54     INFO - TinderboxPrint: CPU iowait<br/>5.5 (5.5%)
[task 2021-07-16T18:35:54.661Z] 18:35:54     INFO - TinderboxPrint: CPU system<br/>1.5 (1.5%)
[task 2021-07-16T18:35:54.662Z] 18:35:54     INFO - TinderboxPrint: CPU user<br/>49.2 (49.2%)
[task 2021-07-16T18:35:54.663Z] 18:35:54     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2021-07-16T18:35:54.664Z] 18:35:54     INFO - start-pulseaudio - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2021-07-16T18:35:54.665Z] 18:35:54     INFO - install - Wall time: 50s; CPU: 51%; Read bytes: 25354240; Write bytes: 1931165696; Read time: 264; Write time: 233388
[task 2021-07-16T18:35:54.666Z] 18:35:54     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2021-07-16T18:35:54.667Z] 18:35:54     INFO - run-tests - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2021-07-16T18:35:54.668Z] 18:35:54  WARNING - returning nonzero exit status 2
[task 2021-07-16T18:35:54.689Z] cleanup
[task 2021-07-16T18:35:54.689Z] + cleanup
[task 2021-07-16T18:35:54.689Z] + local rv=2
[task 2021-07-16T18:35:54.690Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2021-07-16T18:35:54.690Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2021-07-16T18:35:54.692Z] + '[' ']'
[task 2021-07-16T18:35:54.692Z] + true
[task 2021-07-16T18:35:54.692Z] + cleanup_xvfb
[task 2021-07-16T18:35:54.692Z] ++ pidof Xvfb
[task 2021-07-16T18:35:54.695Z] + local xvfb_pid=49
[task 2021-07-16T18:35:54.696Z] + local vnc=false
[task 2021-07-16T18:35:54.696Z] + local interactive=false
[task 2021-07-16T18:35:54.696Z] + '[' -n 49 ']'
[task 2021-07-16T18:35:54.697Z] + [[ false == false ]]
[task 2021-07-16T18:35:54.698Z] + [[ false == false ]]
[task 2021-07-16T18:35:54.699Z] + kill 49
[task 2021-07-16T18:35:54.699Z] + screen -XS xvfb quit
[task 2021-07-16T18:35:54.704Z] + exit 2
[taskcluster 2021-07-16 18:35:55.046Z] === Task Finished ===
[taskcluster 2021-07-16 18:35:57.499Z] Unsuccessful task run with exit code: 2 completed in 165.222 seconds
Whiteboard: [stockwell needswork:owner]

Andrew, could you help us assign this to someone?
Thank you.

Flags: needinfo?(ahal)

This is on my radar, haven't had a chance to investigate yet.

This bug happens when dom/ipc/tests/chrome.ini is scheduled on its own. The reason is that it contains only two tests, and both have skip-if = !crashreporter. In our mozinfo guess in the taskgraph, we hardcode crashreporter=True, but I talked to decoder and it's disabled for asan and tsan builds. So rectifying this should make it go away.

Flags: needinfo?(ahal)

I'll also note that since this only fails given a specific manifest running on its own, retriggers and backfills will be perma-fail. Indeed, looking at orangefactor, almost all instances there are either a retrigger or backfill :p

In otherwords, this is much less frequent than orangefactor indicates. Though I'll put up the fix regardless.

Assignee: nobody → ahal
Status: NEW → ASSIGNED
Attachment #9232885 - Attachment is obsolete: true
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d6f3465132d3
[taskgraph] Unset 'crashreporter' for asan/tsan tests in mozinfo guess, r=jmaher
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 93 Branch
You need to log in before you can comment on or make changes to this bug.