Closed Bug 1524545 Opened 5 years ago Closed 3 years ago

Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-<product>' timed out loading test page: waiting for pending metrics

Categories

(Testing :: Raptor, defect, P5)

Product:
Testing
Component:
Raptor
Version:
Version 3
Platform:
x86_64
Linux
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Depends on 2 open bugs)

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(3 files, 1 obsolete file)

Bug 1524545 - disabled raptor-unity-webgl-firefox r=jmaher
47 bytes, text/x-phabricator-request
Details | Review
Bug 1524545 Move raptor-unity-webgl-firefox to tier 3 and stop running on autoland r?davehunt,bebe,#perftest
47 bytes, text/x-phabricator-request
Details | Review
Bug 1524545 Run raptor-unity-webgl only on autoland
47 bytes, text/x-phabricator-request
Details | Review

#[markdown(off)]
Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=225430143&repo=autoland

https://queue.taskcluster.net/v1/task/TnqTSFbOS5OF2Dgygfp4iA/runs/0/artifacts/public/logs/live_backing.log

03:27:09 INFO - 127.0.0.1 - - [01/Feb/2019 03:27:09] "POST / HTTP/1.1" 200 -
03:27:09 INFO - raptor-control-server received webext_raptor-page-timeout: [u'raptor-unity-webgl-firefox', u'http://127.0.0.1:47368/unity-webgl/index.html?raptor']
03:27:09 INFO - PID 3476 | If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.JavaScript error: , line 0: uncaught exception: abort() at jsStackTrace@http://127.0.0.1:47368/unity-webgl/Data/WebGLBenchmarks.js:1:18533
03:27:09 INFO - PID 3476 | JavaScript error: , line 0: InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable
03:27:09 INFO - 127.0.0.1 - - [01/Feb/2019 03:27:09] "POST / HTTP/1.1" 200 -
03:27:09 INFO - raptor-control-server received webext_status: __raptor_shutdownBrowser
03:27:09 INFO - raptor-control-server shutting down browser (pid: 3476)
03:27:09 INFO - 127.0.0.1 - - [01/Feb/2019 03:27:09] "POST / HTTP/1.1" 200 -
03:27:09 INFO - raptor-control-server received webext_status: Removed tab 2
03:27:25 INFO - raptor-main removing webext /home/cltbld/tasks/task_1549019208/build/tests/raptor/webext/raptor
03:27:25 INFO - results-handler summarizing raptor test results
03:27:25 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-firefox
03:27:25 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-firefox
03:27:25 INFO - raptor-control-server shutting down control server
03:27:26 INFO - raptor-main finished
03:27:26 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
03:27:26 ERROR - Return code: 1
03:27:26 WARNING - setting return code to 1
03:27:26 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
03:27:26 INFO - copying raptor results to upload dir:
03:27:26 INFO - /home/cltbld/tasks/task_1549019208/build/blobber_upload_dir/perfherder-data.json
03:27:26 INFO - copying raptor results from /home/cltbld/tasks/task_1549019208/build/raptor.json to /home/cltbld/tasks/task_1549019208/build/blobber_upload_dir/perfherder-data.json
03:27:26 CRITICAL - Error copying results /home/cltbld/tasks/task_1549019208/build/raptor.json to upload dir /home/cltbld/tasks/task_1549019208/build/blobber_upload_dir/perfherder-data.json
03:27:26 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1549019208/build/raptor.json'
03:27:26 INFO - Running post-action listener: _package_coverage_data
03:27:26 INFO - Running post-action listener: _resource_record_post_action
03:27:26 INFO - Running post-action listener: process_java_coverage_data
03:27:26 INFO - Running post-action listener: stop_device
03:27:26 INFO - [mozharness: 2019-02-01 11:27:26.203424Z] Finished run-tests step (success)
03:27:26 INFO - Running post-run listener: _resource_record_post_run
03:27:26 INFO - Total resource usage - Wall time: 1165s; CPU: 11.0%; Read bytes: 100302848; Write bytes: 1285218304; Read time: 3424; Write time: 177616

There are 56 total failures in the last 7 days on linux64 opt and pgo, linux64-qr, linux64-pgo-qr opt and pgo, linux64-shippable opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237271725&repo=mozilla-central&lineNumber=592

08:22:11 INFO - raptor-main starting firefox
08:22:11 INFO - Application command: /home/cltbld/tasks/task_1554104700/build/application/firefox/firefox -profile /tmp/tmpIvcD3I.mozrunner
08:22:12 INFO - 127.0.0.1 - - [01/Apr/2019 08:22:12] "POST / HTTP/1.1" 200 -
08:22:12 INFO - raptor-control-server received webext_status: raptor runner.js is loaded!

08:41:03 INFO - raptor-control-server received webext_status: Removed tab 2
08:41:20 INFO - raptor-main removing webext /home/cltbld/tasks/task_1554104700/build/tests/raptor/raptor/../webext/raptor
08:41:20 INFO - results-handler summarizing raptor test results
08:41:20 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-firefox
08:41:20 INFO - raptor-output error: no raptor test results found, so no need to combine browser cycles
08:41:20 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-firefox
08:41:20 INFO - raptor-control-server shutting down control server
08:41:20 INFO - raptor-main finished
08:41:20 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
08:41:20 ERROR - Return code: 1
08:41:20 WARNING - setting return code to 1
08:41:20 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
08:41:20 INFO - copying raptor results to upload dir:
08:41:20 INFO - /home/cltbld/tasks/task_1554104700/build/blobber_upload_dir/perfherder-data.json
08:41:20 INFO - copying raptor results from /home/cltbld/tasks/task_1554104700/build/raptor.json to /home/cltbld/tasks/task_1554104700/build/blobber_upload_dir/perfherder-data.json
08:41:20 CRITICAL - Error copying results /home/cltbld/tasks/task_1554104700/build/raptor.json to upload dir /home/cltbld/tasks/task_1554104700/build/blobber_upload_dir/perfherder-data.json
08:41:20 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1554104700/build/raptor.json'

Robert can you take a look or assign someone?

Flags: needinfo?(rwood)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

Thanks Andreea. Alex is currently working on upgrading the unity webgl benchmark source inBug 1506865, which hopefully will solve this.

Depends on: 1506865
Flags: needinfo?(rwood)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

Over the lat 7 days there have been 83 failures on this bug. These happened on linux64, linux64-qr, linux64-shippable, linux64-shippable-qr.

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239182092&repo=autoland&lineNumber=601

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There are 81 total failures in the last 7 days on linux64, linux64-qr opt builds and the majority are on linux64-shippable opt.

Recent failure log:

00:23:41 INFO - raptor-main starting firefox
00:23:41 INFO - Application command: /home/cltbld/tasks/task_1555201386/build/application/firefox/firefox -profile /tmp/tmp0eN4aW.mozrunner
00:23:42 INFO - 127.0.0.1 - - [14/Apr/2019 00:23:42] "POST / HTTP/1.1" 200 -
00:23:42 INFO - raptor-control-server received webext_status: raptor runner.js is loaded!
00:23:42 INFO - 127.0.0.1 - - [14/Apr/2019 00:23:42] "GET /raptor-unity-webgl-firefox.json HTTP/1.1" 200 -

00:42:55 INFO - results-handler summarizing raptor test results
00:42:55 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-firefox
00:42:55 INFO - raptor-output error: no raptor test results found, so no need to combine browser cycles
00:42:55 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-firefox
00:42:55 INFO - raptor-control-server shutting down control server
00:42:55 INFO - raptor-main finished
00:42:55 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
00:42:55 ERROR - Return code: 1
00:42:55 WARNING - setting return code to 1
00:42:55 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
00:42:55 INFO - copying raptor results to upload dir:
00:42:55 INFO - /home/cltbld/tasks/task_1555201386/build/blobber_upload_dir/perfherder-data.json
00:42:55 INFO - copying raptor results from /home/cltbld/tasks/task_1555201386/build/raptor.json to /home/cltbld/tasks/task_1555201386/build/blobber_upload_dir/perfherder-data.json
00:42:55 CRITICAL - Error copying results /home/cltbld/tasks/task_1555201386/build/raptor.json to upload dir /home/cltbld/tasks/task_1555201386/build/blobber_upload_dir/perfherder-data.json
00:42:55 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1555201386/build/raptor.json'

Waiting for a resolution on bug 1506865.

Whiteboard: [stockwell disable-recommended] → [stockwell needswork][waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [stockwell needswork][waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [stockwell needswork][waiting for 1506865]

Over the last 7 days there are 77 failures present on this bug. These happen on linux32-shippable, linux64, linux64-shippable, linux64-shippable-qr

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=241766557&repo=mozilla-inbound&lineNumber=579

11:21:55 INFO - raptor-control-server received webext_status: update tab: 2
11:21:55 INFO - raptor-control-server received webext_status: test tab updated: 2
11:25:09 INFO - raptor-control-server received webext_status: results received
11:25:10 INFO - raptor-control-server received webext_status: begin pagecycle 2
11:25:11 INFO - raptor-control-server received webext_status: update tab: 2
11:25:11 INFO - raptor-control-server received webext_status: test tab updated: 2
11:40:10 INFO - raptor-control-server received webext_raptor-page-timeout: [u'raptor-unity-webgl-firefox', u'http://127.0.0.1:53214/unity-webgl/index.html?raptor']
11:40:10 INFO - raptor-control-server received webext_status: __raptor_shutdownBrowser
11:40:10 INFO - raptor-control-server shutting down browser (pid: 3586)
11:40:10 INFO - raptor-control-server received webext_status: Removed tab: 2
11:40:26 INFO - raptor-main removing webext /home/cltbld/tasks/task_1555930616/build/tests/raptor/raptor/../webext/raptor
11:40:26 INFO - results-handler summarizing raptor test results
11:40:26 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-firefox
11:40:26 INFO - raptor-output error: no raptor test results found, so no need to combine browser cycles
11:40:26 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-firefox
11:40:26 INFO - raptor-control-server shutting down control server
11:40:27 INFO - raptor-main finished
11:40:27 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
11:40:27 ERROR - Return code: 1
11:40:27 WARNING - setting return code to 1
11:40:27 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
11:40:27 INFO - copying raptor results to upload dir:
11:40:27 INFO - /home/cltbld/tasks/task_1555930616/build/blobber_upload_dir/perfherder-data.json
11:40:27 INFO - copying raptor results from /home/cltbld/tasks/task_1555930616/build/raptor.json to /home/cltbld/tasks/task_1555930616/build/blobber_upload_dir/perfherder-data.json
11:40:27 CRITICAL - Error copying results /home/cltbld/tasks/task_1555930616/build/raptor.json to upload dir /home/cltbld/tasks/task_1555930616/build/blobber_upload_dir/perfherder-data.json
11:40:27 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1555930616/build/raptor.json'
11:40:27 INFO - Running post-action listener: _package_coverage_data
11:40:27 INFO - Running post-action listener: _resource_record_post_action
11:40:27 INFO - Running post-action listener: process_java_coverage_data
11:40:27 INFO - Running post-action listener: stop_device
11:40:27 INFO - [mozharness: 2019-04-22 11:40:27.229823Z] Finished run-tests step (success)
11:40:27 INFO - Running post-run listener: _resource_record_post_run
11:40:27 INFO - Total resource usage - Wall time: 1152s; CPU: 7.0%; Read bytes: 99401728; Write bytes: 2174976000; Read time: 1584; Write time: 175276
11:40:27 INFO - TinderboxPrint: CPU usage<br/>7.0%
11:40:27 INFO - TinderboxPrint: I/O read bytes / time<br/>99,401,728 / 1,584
11:40:27 INFO - TinderboxPrint: I/O write bytes / time<br/>2,174,976,000 / 175,276
11:40:27 INFO - TinderboxPrint: CPU idle<br/>8,498.0 (93.1%)
11:40:27 INFO - TinderboxPrint: CPU user<br/>573.9 (6.3%)
11:40:27 INFO - TinderboxPrint: Swap in / out<br/>0 / 0
11:40:27 INFO - install - Wall time: 7s; CPU: 13.0%; Read bytes: 73728; Write bytes: 836337664; Read time: 40; Write time: 65024
11:40:27 INFO - run-tests - Wall time: 1146s; CPU: 7.0%; Read bytes: 97288192; Write bytes: 1146527744; Read time: 1472; Write time: 93532
11:40:27 WARNING - returning nonzero exit status 1
[taskcluster 2019-04-22T11:40:27.592Z] Exit Code: 1
[taskcluster 2019-04-22T11:40:27.592Z] User Time: 9m33.024s
[taskcluster 2019-04-22T11:40:27.592Z] Kernel Time: 43.82s
[taskcluster 2019-04-22T11:40:27.592Z] Wall Time: 19m48.237614532s
[taskcluster 2019-04-22T11:40:27.592Z] Result: FAILED
[taskcluster 2019-04-22T11:40:27.592Z] === Task Finished ===
[taskcluster 2019-04-22T11:40:27.592Z] Task Duration: 19m48.237771128s
[taskcluster 2019-04-22T11:40:28.027Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2020-04-21T09:55:57.211Z

Flags: needinfo?(rwood)

Thanks, yes this depends on the upgrade to the unity-webgl benchmark source, in Bug 1506865.

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

This bug failed 81 times in the last 7 days. Occurs on Linux64 platforms on otp build types.

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242935741&repo=mozilla-inbound&lineNumber=586

Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865][stockwell needswork:owner]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

This bug failed 92 times in the past week on Linux platforms.

Recent log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245732792&repo=mozilla-central&lineNumber=585

12:03:01 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-firefox
12:03:01 INFO - raptor-output screen captures can be found locally at: /home/cltbld/tasks/task_1557483873/build/screenshots.html
12:03:01 INFO - raptor-control-server shutting down control server
12:03:01 INFO - raptor-main finished
12:03:01 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
12:03:01 ERROR - Return code: 1
12:03:01 WARNING - setting return code to 1
12:03:01 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
12:03:01 INFO - copying raptor results to upload dir:
12:03:01 INFO - /home/cltbld/tasks/task_1557483873/build/blobber_upload_dir/perfherder-data.json
12:03:01 INFO - copying raptor results from /home/cltbld/tasks/task_1557483873/build/raptor.json to /home/cltbld/tasks/task_1557483873/build/blobber_upload_dir/perfherder-data.json
12:03:01 CRITICAL - Error copying results /home/cltbld/tasks/task_1557483873/build/raptor.json to upload dir /home/cltbld/tasks/task_1557483873/build/blobber_upload_dir/perfherder-data.json
12:03:01 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1557483873/build/raptor.json'
12:03:01 INFO - /home/cltbld/tasks/task_1557483873/build/blobber_upload_dir/screenshots.html

Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

Robert would a general bug help here https://bugzilla.mozilla.org/buglist.cgi?quicksearch=no%20raptor%20test%20results%20were%20found&list_id=14720600? or does each test need to be looked at separately?

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [stockwell needswork:owner][waiting for 1506865]

(In reply to Andreea Pavel [:apavel] from comment #40)

Robert would a general bug help here https://bugzilla.mozilla.org/buglist.cgi?quicksearch=no%20raptor%20test%20results%20were%20found&list_id=14720600? or does each test need to be looked at separately?

It's best to keep them separate thanks. This particular bug will be resolved once we upgrade the benchmark (Bug 1506865).

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

There are 56 failures on this bug over the last 7 days. These happen on linux64, linux64-qr, linux64-shippable, linux64-shippable-qr.

Here is the most recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249072399&repo=mozilla-inbound&lineNumber=593

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

(In reply to Stefan Hindli [:stefan_hindli] from comment #46)

There are 56 failures on this bug over the last 7 days. These happen on linux64, linux64-qr, linux64-shippable, linux64-shippable-qr.

Here is the most recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249072399&repo=mozilla-inbound&lineNumber=593

Yep I'm aware of this thanks, it will be resolved by Bug 1506865, apologies for the hassle of having to flag this all the time. We will get to it soon - it keeps being put back due to higher priorities.

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1506865]

This bug has failed 64 times in the last 7 days. Occurs on Linux platforms on opt build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250622859&repo=autoland&lineNumber=576

Joel, taking into consideration comment 26, is there anything else we can do here?

There are 59 total failures in the last 7 days on linux64, linux64-qr and linux64-shippable all opt builds.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250897074&repo=mozilla-central&lineNumber=576

Flags: needinfo?(jmaher)

:apavel, I would disable this, we are almost at 2 weeks with no progress here.

you can comment out the test in the test-sets.yml file:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml

Flags: needinfo?(jmaher)

Hi Dave, i've disabled the test, can I land the patch?

Flags: needinfo?(dave.hunt)

Does disabling the fetch task also disable the test? Rob, are you okay with this being disabled, or should we remove this in favour of bug 1506865, which is being worked on by Alexandru?

Flags: needinfo?(dave.hunt) → needinfo?(rwood)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #59)

Does disabling the fetch task also disable the test? Rob, are you okay with this being disabled, or should we remove this in favour of bug 1506865, which is being worked on by Alexandru?

This test is still reporting data to the dashboard [0]. I suggest we leave it running but make it tier 3 instead (like we did for UGL on android).

https://arewefastyet.com/linux64/unity-webgl?numDays=60

Flags: needinfo?(rwood)

Sounds good. Let's make this tier 3 and once we have addressed the more pressing intermittents we can come back to this one.

Depends on: 1558625
Whiteboard: [waiting for 1506865][stockwell disable-recommended] → [waiting for 1558625][comment 61]
Whiteboard: [waiting for 1558625][comment 61][stockwell disable-recommended] → [waiting for 1558625][comment 61]

There are 22 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-11-28&endday=2019-12-05&tree=trunk&bug=1524545

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=279673717&repo=autoland&lineNumber=4340

Robert, could you take a look?

Flags: needinfo?(rwood)
Whiteboard: [waiting for 1558625][comment 61] → [waiting for 1558625][comment 61][stockwell needswork:owner]

:bebe please have a look (or someone on the SV team) thank you!

Flags: needinfo?(rwood) → needinfo?(fstrugariu)

Robert, this is related to the webgl Bug 1506865 and there are few chances to be fixed soon. Not sure what can we do about other than deactivating it for the moment. Dave?

Flags: needinfo?(fstrugariu) → needinfo?(dave.hunt)

(In reply to Alexandru Ionescu :alexandrui from comment #86)

Robert, this is related to the webgl Bug 1506865 and there are few chances to be fixed soon. Not sure what can we do about other than deactivating it for the moment. Dave?

There's been a recent increase in the number of failures. We should investigate and see if there is a new cause for this.

Flags: needinfo?(dave.hunt) → needinfo?(fstrugariu)

this might be caused by Bug 1593598 - Investigate failures of browsertime speedometer test on Windows

But I'm not sure about it. Anyway bug Bug 1593598 was fixed so let's keep an eye on this bug for the next days

Flags: needinfo?(fstrugariu)
Priority: P5 → P1

There are 29 total failures in the last 7 days on linux64-shippable, linux64-shippable-qr opt builds

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283353114&repo=autoland&lineNumber=6049

[task 2020-01-03T14:46:05.640Z] 14:46:05 INFO - raptor-control-server Info: received webext_raptor-page-timeout: [u'raptor-unity-webgl-firefox', u'http://127.0.0.1:50062/unity-webgl/index.html?raptor']
[task 2020-01-03T14:46:05.643Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] post success"
[task 2020-01-03T14:46:05.747Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] checking results..."
[task 2020-01-03T14:46:05.828Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] posting to control server"
[task 2020-01-03T14:46:05.832Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] closed tab 2"
[task 2020-01-03T14:46:05.832Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] benchmark test finished"
[task 2020-01-03T14:46:05.832Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] posting to control server"
[task 2020-01-03T14:46:05.832Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] "
[task 2020-01-03T14:46:05.848Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] Removed tab: 2"
[task 2020-01-03T14:46:05.848Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] posting to control server"
[task 2020-01-03T14:46:05.848Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] Removed tab: 2"
[task 2020-01-03T14:46:05.865Z] 14:46:05 INFO - raptor-control-server Info: received webext_screenshot
[task 2020-01-03T14:46:05.865Z] 14:46:05 INFO - perftest-results-handler Info: received screenshot
[task 2020-01-03T14:46:05.865Z] 14:46:05 INFO - raptor-control-server Info: received request to shutdown the browser
[task 2020-01-03T14:46:05.865Z] 14:46:05 INFO - raptor-control-server Info: shutting down browser (pid: 3480)
[task 2020-01-03T14:46:05.865Z] 14:46:05 INFO - raptor-control-server Info: received webext_status: Removed tab: 2
[task 2020-01-03T14:46:05.866Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] post success"
[task 2020-01-03T14:46:05.866Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] post success"
[task 2020-01-03T14:46:05.866Z] 14:46:05 INFO - PID 3480 | console.log: "[raptor-runnerjs] post success"
[task 2020-01-03T14:46:06.004Z] 14:46:06 INFO - PID 3480 | console.log: "[raptor-runnerjs] checking results..."
[task 2020-01-03T14:46:06.262Z] 14:46:06 INFO - PID 3480 | console.log: "[raptor-runnerjs] checking results..."
[task 2020-01-03T14:46:06.500Z] 14:46:06 INFO - PID 3480 | console.log: "[raptor-runnerjs] checking results..."
[task 2020-01-03T14:46:06.759Z] 14:46:06 INFO - PID 3480 | console.log: "[raptor-runnerjs] checking results..."
[task 2020-01-03T14:46:07.805Z] 14:46:07 INFO - raptor-main Info: removing webext /home/cltbld/tasks/task_1578061403/build/tests/raptor/raptor/../webext/raptor
[task 2020-01-03T14:46:07.806Z] 14:46:07 INFO - perftest-results-handler Info: summarizing raptor test results
[task 2020-01-03T14:46:07.806Z] 14:46:07 INFO - perftest-output Error: no raptor test results found for raptor-unity-webgl-firefox
[task 2020-01-03T14:46:07.806Z] 14:46:07 INFO - perftest-output Info: error: no raptor test results found, so no need to combine browser cycles
[task 2020-01-03T14:46:07.806Z] 14:46:07 INFO - perftest-output Error: no summarized raptor results found for raptor-unity-webgl-firefox
[task 2020-01-03T14:46:07.806Z] 14:46:07 INFO - perftest-output Info: screen captures can be found locally at: /home/cltbld/tasks/task_1578061403/build/screenshots.html
[task 2020-01-03T14:46:07.822Z] 14:46:07 INFO - perftest-results-handler Critical: PERFHERDER_DATA was seen 0 times, expected 1.
[task 2020-01-03T14:46:07.822Z] 14:46:07 INFO - raptor-control-server Info: shutting down control server
[task 2020-01-03T14:46:07.863Z] 14:46:07 INFO - raptor-main Info: finished
[task 2020-01-03T14:46:07.863Z] 14:46:07 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox
[task 2020-01-03T14:46:07.921Z] 14:46:07 ERROR - Return code: 1
[task 2020-01-03T14:46:07.921Z] 14:46:07 WARNING - setting return code to 1
[task 2020-01-03T14:46:07.921Z] 14:46:07 INFO - Copying Raptor results to upload dir:
[task 2020-01-03T14:46:07.921Z] 14:46:07 INFO - /home/cltbld/tasks/task_1578061403/build/blobber_upload_dir/perfherder-data.json
[task 2020-01-03T14:46:07.921Z] 14:46:07 INFO - Copying raptor results from /home/cltbld/tasks/task_1578061403/build/raptor.json to /home/cltbld/tasks/task_1578061403/build/blobber_upload_dir/perfherder-data.json
[task 2020-01-03T14:46:07.922Z] 14:46:07 CRITICAL - Error copying results /home/cltbld/tasks/task_1578061403/build/raptor.json to upload dir /home/cltbld/tasks/task_1578061403/build/blobber_upload_dir/perfherder-data.json
[task 2020-01-03T14:46:07.922Z] 14:46:07 INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1578061403/build/raptor.json'
[task 2020-01-03T14:46:07.923Z] 14:46:07 INFO - /home/cltbld/tasks/task_1578061403/build/blobber_upload_dir/screenshots.html

Florin are there updates here? Bug 1593598 was fixed one month ago and since then there have been 117 total failures here.

Flags: needinfo?(fstrugariu)

There are 39 total failures in the last 7 days on linux64-shippable, linux64-shippable-qr opt builds

Whiteboard: [waiting for 1558625][comment 61][stockwell unknown] → [stockwell needswork:owner]

tests run fine but sometimes it takes a looong time to finish and we reach the timeout...

From the logs in the cycle that times out we always see this message:

[task 2020-01-11T11:28:14.363Z] 11:28:14     INFO -  PID 3657 | If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.console.log: "[raptor-runnerjs] checking results..."

:rwood does this looks familar to you? Do you know if this is from the web-extension or from the unity webgl ?

Flags: needinfo?(fstrugariu) → needinfo?(rwood)

also there is this task: Bug 1506865 - Convert raptor-unity-webgl to a pageload test

should we disable this test? or move it to Tire 3

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #99)

also there is this task: Bug 1506865 - Convert raptor-unity-webgl to a pageload test

should we disable this test? or move it to Tire 3

Yes this has been around for ages and * might * be fixed by Bug 1506865 (as noted in Comment 12). I suggest either we make it a priority to convert the test (Bug 1506865) or else disable it. Dave's call!

Flags: needinfo?(rwood) → needinfo?(dave.hunt)
Assignee: nobody → fstrugariu

Update:
There have been 34 failures within the last 7 days:

  • 5 failures on Linux x64 opt
  • 3 failures on Linux x64 QuantumRender opt
  • 19 failures on Linux x64 shippable opt
  • 7 failures on Linux x64 QuantumRender Shippable opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285871267&repo=autoland&lineNumber=5194

We should determine what value this test is providing. How many alerts were raised by this test in the last 6 months? Of those, how many were fixed/backedout and how many were invalid/wontfix? How noisy is the data?

Bebe: Could you coordinate answering the above? Ionut and Kyle should be able to help with gathering this data. I'm also interested to know what effort would be needed to migrate this test to Browsertime to see if that helps with the stability.

Flags: needinfo?(dave.hunt) → needinfo?(fstrugariu)

@igoldan can you help provide a list of alerts generated by raptor-unity-webgl on all platforms in the last 6 months.

I will make the rest of the documentation here.

Flags: needinfo?(fstrugariu) → needinfo?(igoldan)

I prepared this Redash query to answer that.
Seems like we've only generated one alert out of raptor-unity-webgl. And that one's invalid too.
And yet, this test has around 1700 signatures.

Looks like the test constantly experiences big outliers, which affect our student t-test-based alert generation algorithm.

Flags: needinfo?(igoldan)
Attachment #9120728 - Attachment is obsolete: true

I suggest we remove the raptor-unity-webgl test entirely.

Flags: needinfo?(dave.hunt)

Based on this comment of Dave we should probably wait on Dave to decide what to do with this intermittent.
IMO, we should move this to tier 3 as there doesn't seem to be any intent to handle this soon.

See Also: → 1600487

It doesn't appear that this test is providing much value. Let's drop it down to tier 3 due to the intermittents, and stop running it on autoland.

Flags: needinfo?(dave.hunt)
Assignee: fstrugariu → aionescu
Attachment #9124049 - Attachment description: Bug 1524545 Move raptor-unity-webgl-firefox to tier 3 and stop running on autoland → Bug 1524545 Move raptor-unity-webgl-firefox to tier 3 and stop running on autoland r?davehunt,bebe,#perftest
Pushed by aionescu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d7d2319910a1
Move raptor-unity-webgl-firefox to tier 3 and stop running on autoland r=Bebe,perftest-reviewers

https://hg.mozilla.org/mozilla-central/rev/d7d2319910a1

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox74: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla74

The underlying problem here hasn't been fixed yet. Instead it's just less visible now due to Tier3.

Assignee: aionescu → nobody
Status: RESOLVED → REOPENED
status-firefox74: fixed → ---
Priority: P1 → P5
Resolution: FIXED → ---
Target Milestone: mozilla74 → ---

Please note that there is this error visible while the test is running:

[task 2020-02-28T10:27:08.400Z] 10:27:08    ERROR -  PID 3507 | console.error: "exception thrown: TypeError: text.split is not a function,processText@http://127.0.0.1:41235/unity-webgl/Data/mozbench.js:11:19\nModule.print@http://127.0.0.1:41235/unity-webgl/index.html?raptor:60:24\nabort@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:98:69204\nTKn@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:89:1\nuRb@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:41:1\ntRb@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:41:1\nzjc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:21:1\nGjc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:21:1\nYRc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:9:1\nRRc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:9:1\nunc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:21:1\nLRc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:9:1\njgc@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:21:1\nbdd@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:33:1\nx8f@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:45:1\nfGn@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:89:1\ndynCall@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:1:4741\nBrowser_mainLoop_runner/<@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:1:243649\nrunIter@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:1:132109\nBrowser_mainLoop_runner@http://127.0.0.1:41235/unity-webgl/Data/WebGLBenchmarks.js:1:243554\n"
[task 2020-02-28T10:27:08.400Z] 10:27:08    ERROR -  PID 3507 | TypeError: text.split is not a functionconsole.info: "[raptor-runnerjs] checking results..."

This might be the problem why the test times out.

Status: REOPENED → NEW
Summary: Intermittent raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox → Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-firefox' timed out loading test page: http://127.0.0.1:<random>/unity-webgl/index.html?raptor
Summary: Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-firefox' timed out loading test page: http://127.0.0.1:<random>/unity-webgl/index.html?raptor → Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-firefox' timed out loading test page
Summary: Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-firefox' timed out loading test page → Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-<product>' timed out loading test page
OS: Unspecified → Linux
Hardware: Unspecified → x86_64

No more failures since Saturday when my patch on bug 1625892 landed. Lets observe the bug for the next week.

Depends on: 1625892

If this is resolved we should promote the test back to the default tier for the group.

Summary: Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-<product>' timed out loading test page → Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-unity-webgl-<product>' timed out loading test page: waiting for pending metrics
Whiteboard: [stockwell unknown] → [stockwell unknown][perftest:triage]
Depends on: 1657359
Whiteboard: [stockwell unknown][perftest:triage] → [stockwell unknown]
See Also: → 1666915
See Also: → 1666942

:bebe could we experiment with enabling this on autoland again, perhaps at a lower frequency, and keeping it as tier 3?

Flags: needinfo?(fstrugariu)
Assignee: nobody → fstrugariu
Status: NEW → ASSIGNED
Flags: needinfo?(fstrugariu)
Pushed by fstrugariu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ca76c94d3703
Run raptor-unity-webgl only on autoland r=davehunt,perftest-reviewers

https://hg.mozilla.org/mozilla-central/rev/ca76c94d3703

Status: ASSIGNED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox84: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 84 Branch

The patch only enabled the test on autoland, we should check to see if the test is still failing before closing this bug.

Status: RESOLVED → REOPENED
status-firefox84: fixed → ---
Resolution: FIXED → ---
Target Milestone: 84 Branch → ---
Assignee: fstrugariu → nobody

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 4 years ago → 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: