Closed Bug 1883809 Opened 3 months ago Closed 20 days ago

Perma [tier 2] remote [taskcluster:error] exit status 2

Categories

(Remote Protocol :: CDP, defect, P5)

Product:
Remote Protocol
Component:
CDP
Type:
defect

Tracking

(firefox125 affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox125 --- affected

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, regression, regressionwindow-wanted)

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=449691658&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/COEQFYplSaqPghCgFzaqZg/runs/0/artifacts/public/logs/live_backing.log

[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO - TEST-START | remote/cdp/test/browser/emulation/browser_setUserAgentOverride.js
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO - TEST-SKIP | remote/cdp/test/browser/emulation/browser_setUserAgentOverride.js | took 0ms
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  TEST-INFO | checking window state
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  Browser Chrome Test Summary
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  	Passed: 0
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  	Failed: 0
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  	Todo: 0
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  	Mode: e10s
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO -  *** End BrowserChrome Test Results ***
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO - Buffered messages finished
[task 2024-03-06T01:07:34.123Z] 01:07:34     INFO - SUITE-END | took 0s
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Return code: 1
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - TinderboxPrint: mochitest-mochitest-remote<br/>3/0/0
[task 2024-03-06T01:07:34.197Z] 01:07:34  WARNING - setting return code to 2
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - The mochitest suite: mochitest-remote ran with return status: FAILURE
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running post-action listener: _package_coverage_data
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running post-action listener: _resource_record_post_action
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running post-action listener: process_java_coverage_data
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - [mozharness: 2024-03-06 01:07:34.197007Z] Finished run-tests step (success)
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - [mozharness: 2024-03-06 01:07:34.197046Z] Running uninstall step.
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running pre-action listener: _resource_record_pre_action
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running main action method: uninstall
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Skipping uninstall for non-MSIX test
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running post-action listener: _resource_record_post_action
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - [mozharness: 2024-03-06 01:07:34.197190Z] Finished uninstall step (success)
[task 2024-03-06T01:07:34.197Z] 01:07:34     INFO - Running post-run listener: _resource_record_post_run
[task 2024-03-06T01:07:34.994Z] 01:07:34     INFO - instance_metadata.json not found; unable to determine instance type
[task 2024-03-06T01:07:34.995Z] 01:07:34     INFO - Validating Perfherder data against /home/ubuntu/tasks/task_170968663499466/mozharness/external_tools/performance-artifact-schema.json
[task 2024-03-06T01:07:34.998Z] 01:07:34     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"name": "mochitest.mochitest-remote.overall", "extraOptions": ["e10s", "buildbot-unknown"], "subtests": [{"name": "cpu_percent", "value": 52.09357976653696}, {"name": "io_write_bytes", "value": 12378112}, {"name": "io.read_bytes", "value": 1744896}, {"name": "io_write_time", "value": 2696}, {"name": "io_read_time", "value": 929}]}, {"name": "mochitest.mochitest-remote.start-pulseaudio", "subtests": [{"name": "time", "value": 0.0018122129999937897}, {"name": "cpu_percent", "value": 0}]}, {"name": "mochitest.mochitest-remote.install", "subtests": [{"name": "time", "value": 23.95084980999991}, {"name": "cpu_percent", "value": 52.63389121338912}]}, {"name": "mochitest.mochitest-remote.stage-files", "subtests": [{"name": "time", "value": 0.00026879800009282917}, {"name": "cpu_percent", "value": 0}]}, {"name": "mochitest.mochitest-remote.run-tests", "subtests": [{"name": "time", "value": 1.8438413090000267}, {"name": "cpu_percent", "value": 46.161764705882355}]}, {"name": "mochitest.mochitest-remote.uninstall", "subtests": [{"name": "time", "value": 8.855800001583702e-05}, {"name": "cpu_percent", "value": 0}]}]}
[task 2024-03-06T01:07:34.998Z] 01:07:34     INFO - Total resource usage - Wall time: 26s; CPU: Can't collect data; Read bytes: 1744896; Write bytes: 12378112; Read time: 929; Write time: 2696
[task 2024-03-06T01:07:34.998Z] 01:07:34     INFO - TinderboxPrint: I/O read bytes / time<br/>1,744,896 / 929
[task 2024-03-06T01:07:34.998Z] 01:07:34     INFO - TinderboxPrint: I/O write bytes / time<br/>12,378,112 / 2,696
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: CPU idle<br/>23.4 (60.3%)
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: CPU iowait<br/>0.5 (1.2%)
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: CPU softirq<br/>0.7 (1.7%)
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: CPU system<br/>1.3 (3.4%)
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: CPU user<br/>13.0 (33.4%)
[task 2024-03-06T01:07:34.999Z] 01:07:34     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2024-03-06T01:07:35.000Z] 01:07:35     INFO - start-pulseaudio - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-03-06T01:07:35.001Z] 01:07:35     INFO - install - Wall time: 24s; CPU: 53%; Read bytes: 1212416; Write bytes: 12378112; Read time: 430; Write time: 2696
[task 2024-03-06T01:07:35.002Z] 01:07:35     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-03-06T01:07:35.002Z] 01:07:35     INFO - run-tests - Wall time: 2s; CPU: 46%; Read bytes: 409600; Write bytes: 0; Read time: 446; Write time: 0
[task 2024-03-06T01:07:35.003Z] 01:07:35     INFO - uninstall - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-03-06T01:07:35.060Z] 01:07:35  WARNING - returning nonzero exit status 2
[taskcluster 2024-03-06T01:07:35.134Z]    Exit Code: 2
[taskcluster 2024-03-06T01:07:35.134Z]    User Time: 2m53.492331s
[taskcluster 2024-03-06T01:07:35.134Z]  Kernel Time: 36.54969s
[taskcluster 2024-03-06T01:07:35.134Z]    Wall Time: 3m51.005322491s
[taskcluster 2024-03-06T01:07:35.134Z]       Result: FAILED
[taskcluster 2024-03-06T01:07:35.134Z] === Task Finished ===
[taskcluster 2024-03-06T01:07:35.134Z] Task Duration: 3m51.015494244s
[taskcluster 2024-03-06T01:07:35.285Z] Uploading artifact public/logs/localconfig.json from file /home/ubuntu/tasks/task_170968663499466/logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:35.509Z] Uploading artifact public/test_info/manifests.list from file /home/ubuntu/tasks/task_170968663499466/build/blobber_upload_dir/manifests.list with content encoding "gzip", mime type "application/octet-stream" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:35.747Z] Uploading artifact public/test_info/mochitest-remote_errorsummary.log from file /home/ubuntu/tasks/task_170968663499466/build/blobber_upload_dir/mochitest-remote_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:35.964Z] Uploading artifact public/test_info/profile_resource-usage.json from file /home/ubuntu/tasks/task_170968663499466/build/blobber_upload_dir/profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:36.193Z] Uploading artifact public/test_info/resource-usage.json from file /home/ubuntu/tasks/task_170968663499466/build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:36.424Z] Uploading artifact public/test_info/system-info.log from file /home/ubuntu/tasks/task_170968663499466/build/blobber_upload_dir/system-info.log with content encoding "gzip", mime type "text/plain" and expiry 2025-03-06T00:37:47.628Z
[taskcluster 2024-03-06T01:07:36.624Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-03-06T00:37:47.628Z
[taskcluster:error] exit status 2
See Also: → 1538599

:edgar, since you are the author of the regressor, bug 1858627, could you take a look?

For more information, please visit BugBot documentation.

Flags: needinfo?(echen)
Summary: Perma [tier 2] remote [taskcluster:error] exit status 2 | single tracking bug → Perma [tier 2] remote [taskcluster:error] exit status 2

I don't think that this is a regression from bug 1858627 given that backfill jobs fail already for earlier jobs. I've triggered even more:

https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=remote%2Cwayland&tochange=ad694926aa451a86fcdc3a83a7bc49da3f66caa0&fromchange=c13ff57ddd8d6b9f971a631b45c6da97b3c14fb9

What I noticed is that for those jobs that are failing we do not run any test! That means we have 0 passing, 0 failing, and 0 skipped tests. Other remote jobs that are passing for Wayland run tests. Given that we do not use chunking I wonder how tests are collected for Mochitests.

Geoff, do you understand what's going on here? Was there maybe a change recently which could have triggered this behavior?

Flags: needinfo?(echen)
Keywords: intermittent-testcase → regressionwindow-wanted
No longer regressed by: 1858627
Flags: needinfo?(gbrown)

In a failing case I can see SUITE-START | Running 3 tests, which all are skipped, while in a passing case we have SUITE-START | Running 125 tests.

That means that something is wrong with the test collection here.

Sorry, no idea.

Flags: needinfo?(gbrown)

the failing tasks are running a subset of tests via bugbug test selection. You can see in the log:

[task 2024-03-06T01:07:32.487Z] 01:07:32     INFO -  'MOZHARNESS_TEST_PATHS': '{"mochitest-remote": '
[task 2024-03-06T01:07:32.488Z] 01:07:32     INFO -                           '["remote/cdp/test/browser/emulation/browser.toml"]}',

I assume in the changes pointed out as a regression there were files touched that have caused a regression in the remote/cdp/test/browser/emulation/browser.toml tests sometime in the recent past. This is a problem once in a while in general.

Thanks Joel! Does it mean that we can basically close this bug, or should bugbug be updated / fixed to not run when there are basically no tests selected?

Flags: needinfo?(jmaher)

This bug has been marked as a regression. Setting status flag for Nightly to affected.

status-firefox125: --- → affected

I have a patch I am working on that should help out. Inside of taskgraph (decision task) we choose the number of chunks and this means we need to figure out mozinfo variables. As this is skip-if = wayland, we need to add display: wayland to our mozinfo data:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/util/chunking.py#37

Flags: needinfo?(jmaher)

Last time this failed was on 2024-03-13: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2024-01-19&endday=2024-05-18&tree=all&failurehash=all&bug=1883809
Let's deem this as fixed to prevent misclasifications.

Status: NEW → RESOLVED
Closed: 20 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.