Closed Bug 1552812 Opened 2 years ago Closed 2 years ago

Very high frequency raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-tumblr-chromium

Categories

(Testing :: Raptor, defect, P1)

Version 3
defect

Tracking

(firefox-esr60 unaffected, firefox67 unaffected, firefox68 unaffected, firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox-esr60 --- unaffected
firefox67 --- unaffected
firefox68 --- unaffected
firefox69 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: igoldan)

References

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell disable-recommended])

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=247335120&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/BXL7gFwQRvyzWxeH6b440g/runs/0/artifacts/public/logs/live_backing.log


12:27:27 INFO - results-handler summarizing raptor test results
12:27:27 INFO - raptor-output error: no raptor test results found for raptor-tp6-tumblr-chromium
12:27:27 INFO - raptor-output error: no raptor test results found, so no need to combine browser cycles
12:27:27 INFO - raptor-output error: no summarized raptor results found for raptor-tp6-tumblr-chromium
12:27:27 INFO - raptor-output screen captures can be found locally at: C:\Users\task_1558352325\build\screenshots.html
12:27:27 INFO - raptor-control-server shutting down control server
12:27:27 INFO - raptor-main finished
12:27:27 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-tumblr-chromium
12:27:28 ERROR - Return code: 1
12:27:28 WARNING - setting return code to 1
12:27:28 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
12:27:28 INFO - copying raptor results to upload dir:
12:27:28 INFO - C:\Users\task_1558352325\build\blobber_upload_dir\perfherder-data.json
12:27:28 INFO - copying raptor results from C:\Users\task_1558352325\build\raptor.json to C:\Users\task_1558352325\build\blobber_upload_dir\perfherder-data.json
12:27:28 CRITICAL - Error copying results C:\Users\task_1558352325\build\raptor.json to upload dir C:\Users\task_1558352325\build\blobber_upload_dir\perfherder-data.json
12:27:28 INFO - [Errno 2] No such file or directory: u'C:\Users\task_1558352325\build\raptor.json'
12:27:28 INFO - C:\Users\task_1558352325\build\blobber_upload_dir\screenshots.html
12:27:28 INFO - copying raptor results from C:\Users\task_1558352325\build\screenshots.html to C:\Users\task_1558352325\build\blobber_upload_dir\screenshots.html
12:27:28 INFO - Running post-action listener: _package_coverage_data
12:27:28 INFO - Running post-action listener: _resource_record_post_action
12:27:28 INFO - Running post-action listener: process_java_coverage_data
12:27:28 INFO - Running post-action listener: stop_device
12:27:28 INFO - [mozharness: 2019-05-20 12:27:28.072000Z] Finished run-tests step (success)
12:27:28 INFO - Running post-run listener: _resource_record_post_run
12:27:28 INFO - Total resource usage - Wall time: 257s; CPU: 61.0%; Read bytes: 293505024; Write bytes: 867442688; Read time: 6; Write time: 15
12:27:28 INFO - TinderboxPrint: CPU usage<br/>61.0%
12:27:28 INFO - TinderboxPrint: I/O read bytes / time<br/>293,505,024 / 6
12:27:28 INFO - TinderboxPrint: I/O write bytes / time<br/>867,442,688 / 15
12:27:28 INFO - TinderboxPrint: CPU idle<br/>804.0 (38.9%)
12:27:28 INFO - TinderboxPrint: CPU system<br/>436.1 (21.1%)
12:27:28 INFO - TinderboxPrint: CPU user<br/>817.0 (39.6%)
12:27:28 INFO - install - Wall time: 2s; CPU: 28.0%; Read bytes: 65536; Write bytes: 1811968; Read time: 0; Write time: 0
12:27:28 INFO - run-tests - Wall time: 256s; CPU: 61.0%; Read bytes: 291179520; Write bytes: 849492480; Read time: 6; Write time: 14
12:27:28 WARNING - returning nonzero exit status 1
[taskcluster 2019-05-20T12:27:28.172Z] Exit Code: 1
[taskcluster 2019-05-20T12:27:28.172Z] User Time: 0s
[taskcluster 2019-05-20T12:27:28.172Z] Kernel Time: 15.625ms
[taskcluster 2019-05-20T12:27:28.172Z] Wall Time: 5m32.1239009s
[taskcluster 2019-05-20T12:27:28.172Z] Result: FAILED
[taskcluster 2019-05-20T12:27:28.172Z] === Task Finished ===
[taskcluster 2019-05-20T12:27:28.172Z] Task Duration: 5m32.1249038s
[taskcluster 2019-05-20T12:27:28.481Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-05-19T12:11:58.704Z
[taskcluster 2019-05-20T12:27:29.177Z] Uploading artifact public/test_info/mitmproxy.log from file build\blobber_upload_dir\mitmproxy.log with content encoding "gzip", mime type "text/plain" and expiry 2020-05-19T12:11:58.704Z
[taskcluster 2019-05-20T12:27:29.580Z] Uploading artifact public/test_info/resource-usage.json from file build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-05-19T12:11:58.704Z
[taskcluster 2019-05-20T12:27:30.047Z] Uploading artifact public/test_info/screenshots.html from file build\blobber_upload_dir\screenshots.html with content encoding "gzip", mime type "text/html; charset=utf-8" and expiry 2020-05-19T12:11:58.704Z
[taskcluster 2019-05-20T12:27:30.614Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/BXL7gFwQRvyzWxeH6b440g/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-05-19T12:11:58.704Z
[taskcluster:error] exit status 1

Flags: needinfo?(igoldan)
Summary: Intermittent raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-tumblr-chromium → Very high frequency raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-tumblr-chromium

It's possible one of the latest Chromium revisions we're using made some breaking changes for Raptor. Generally, we're updating Chromium daily, via a cron job.

Flags: needinfo?(igoldan)
Duplicate of this bug: 1552876
Duplicate of this bug: 1552821
Duplicate of this bug: 1552823
Duplicate of this bug: 1552824
Duplicate of this bug: 1552825
Duplicate of this bug: 1552826
Duplicate of this bug: 1552827
Duplicate of this bug: 1552831

if you see:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=chromium&fromchange=82437bfbc17b5ea4f90839b3c1b67b92e5fee23d&tochange=17d1c1e26e31551fd7baf9b3ea80f8f2382385a0

we updated chromium on rev 9b2f851979cb, but the problems started on the rev immediately afterwards. I did some retriggers on rev 9b2f851979cb and it is looking good overall.

Duplicate of this bug: 1552669

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #10)

if you see:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=chromium&fromchange=82437bfbc17b5ea4f90839b3c1b67b92e5fee23d&tochange=17d1c1e26e31551fd7baf9b3ea80f8f2382385a0

we updated chromium on rev 9b2f851979cb, but the problems started on the rev immediately afterwards. I did some retriggers on rev 9b2f851979cb and it is looking good overall.

So it's possible the Chromium revision is ok, but the culprit is in the first patch which brought the Chromium intermittents?
I don't really think that's the case. For 9b2f851979cb I see we used Chromium 660782, while for the e013f1f17109 which followed, we used Chromium 661171.

Flags: needinfo?(jmaher)

yeah, there is bisection of that on autoland with no success:
https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=e013f1f17109&fromchange=8e8ea33ecb3da138dbd0af56c2e5550902e7b05a&searchStr=chromium&group_state=expanded

I think it is most likely the updated chromium version

Flags: needinfo?(jmaher)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #13)

yeah, there is bisection of that on autoland with no success:
https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=e013f1f17109&fromchange=8e8ea33ecb3da138dbd0af56c2e5550902e7b05a&searchStr=chromium&group_state=expanded

I think it is most likely the updated chromium version

We had the same explanation yesterday. Thanks for your confirmation, Joel!

Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Priority: P5 → P1
Assignee: nobody → igoldan
Duplicate of this bug: 1552889
Duplicate of this bug: 1552864
Duplicate of this bug: 1552849
Depends on: 1553137
Duplicate of this bug: 1552859

This got fixed by bug 1553137.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
You need to log in before you can comment on or make changes to this bug.