1614275 - Raptor test results aren't always written to disk (no raptor test results were found)

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There are lots of different tests which fail in CI because no raptor tests can be found:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288091565&repo=mozilla-central&lineNumber=1488-1514

[task 2020-02-09T12:18:16.565Z] 11:23:54     INFO -  raptor-control-server Info: received webext_status: test tab updated: 0
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received webext_raptor-page-timeout: [u'raptor-youtube-playback-geckoview-live', u'http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true']
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received webext_screenshot
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  perftest-results-handler Info: received screenshot
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received request to shutdown the browser
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: shutting down android app org.mozilla.geckoview_example
[task 2020-02-09T12:18:16.565Z] 12:17:55     INFO -  adb shell_output: adb -s ZY322LHD2L wait-for-device shell am force-stop org.mozilla.geckoview_example, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  raptor-webext-android Info: removing reverse socket connections
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  adb command_output: adb -s ZY322LHD2L wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  adb shell_bool: adb -s ZY322LHD2L wait-for-device shell test -d /sdcard/raptor/profile/minidumps, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  adb shell_output: adb -s ZY322LHD2L wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  adb shell_bool: adb -s ZY322LHD2L wait-for-device shell test -d /sdcard/raptor/profile/minidumps, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  adb command_output: adb -s ZY322LHD2L wait-for-device pull /sdcard/raptor/profile/minidumps /tmp/tmp6UbYCY/minidumps, timeout: None, timedout: None, exitcode: 0, output: /sdcard/raptor/profile/minidumps/: 0 files pulled.
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  raptor-webext Info: removing webext /builds/task_1581247258/workspace/build/tests/raptor/raptor/webextension/../../webext/raptor
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-results-handler Info: summarizing raptor test results
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-output Error: no raptor test results found for raptor-youtube-playback-geckoview-live
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-output Info: error: no raptor test results found, so no need to combine browser cycles
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-output Error: no summarized raptor results found for raptor-youtube-playback-geckoview-live
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-output Info: screen captures can be found locally at: /builds/task_1581247258/workspace/build/screenshots.html
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  perftest-results-handler Critical: PERFHERDER_DATA was seen 0 times, expected 1.
[task 2020-02-09T12:18:16.565Z] 12:17:56     INFO -  raptor-webext-android Info: removing test folder for raptor: /sdcard/raptor
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  adb shell_output: adb -s ZY322LHD2L wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  adb shell_output: adb -s ZY322LHD2L wait-for-device shell rm -r /sdcard/raptor, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  adb shell_output: adb -s ZY322LHD2L wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  adb shell_bool: adb -s ZY322LHD2L wait-for-device shell test -e /sdcard/raptor, timeout: None, root: False, timedout: None, exitcode: 1, output:
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  raptor-control-server Info: shutting down control server
[task 2020-02-09T12:18:16.565Z] 12:17:57     INFO -  raptor-webext Info: finished

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Lets have a look at the ytp tests...

Here a case for passing tests and the control server received the results:
https://firefoxci.taskcluster-artifacts.net/KEDcSTCSRtekuIhTSZ2OiA/0/public/logs/live_backing.log

[task 2020-02-09T12:16:51.684Z] 11:53:35     INFO -  raptor-control-server Info: received webext_status: running 1 pagecycles of http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true
[task 2020-02-09T12:16:51.684Z] 11:53:36     INFO -  raptor-control-server Info: received webext_status: begin pagecycle 1
[task 2020-02-09T12:16:51.684Z] 11:53:37     INFO -  raptor-control-server Info: received webext_status: update tab: 0
[task 2020-02-09T12:16:51.684Z] 11:53:37     INFO -  raptor-control-server Info: received webext_status: test tab updated: 0
[task 2020-02-09T12:16:51.684Z] 12:16:51     INFO -  raptor-control-server Info: received webext_status: results received
[task 2020-02-09T12:16:51.684Z] 12:16:51     INFO -  raptor-control-server Info: received webext_results: {u'expected_browser_cycles': 1,  ... }

And here a case with failing test results:
https://firefoxci.taskcluster-artifacts.net/VAJkESXPRs2eFyTsY-6I7w/0/public/logs/live_backing.log

[task 2020-02-09T12:18:16.565Z] 11:23:52     INFO -  raptor-control-server Info: received webext_status: running 1 pagecycles of http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true
[task 2020-02-09T12:18:16.565Z] 11:23:53     INFO -  raptor-control-server Info: received webext_status: begin pagecycle 1
[task 2020-02-09T12:18:16.565Z] 11:23:54     INFO -  raptor-control-server Info: received webext_status: update tab: 0
[task 2020-02-09T12:18:16.565Z] 11:23:54     INFO -  raptor-control-server Info: received webext_status: test tab updated: 0
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received webext_raptor-page-timeout: [u'raptor-youtube-playback-geckoview-live', u'http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true']
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received webext_screenshot
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  perftest-results-handler Info: received screenshot
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: received request to shutdown the browser
[task 2020-02-09T12:18:16.565Z] 12:17:54     INFO -  raptor-control-server Info: shutting down android app org.mozilla.geckoview_example

So this is always the same across all test types. Whenever a page-timeout happened we do not have test results, which actually makes sense. But given that the control server is already aware of that, no error log line should be logged for the test results not been found, but instead for the page timeout. That's what important here, and which needs to be fixed.

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

I'll look into this

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Unfortunately I can't investigate this now. If you still need someone to look into it in a week, feel free to n-i me.

Comment 4

[Marian Raiciof [:marauder]]

Attachment: Bug 1614275 - Don't report "no raptor test results were found" if page load tests time out

Comment 5

[Marian Raiciof [:marauder]]

Push to try:
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=54c5417ec3eaf083648b20a818a6a7b56a6e378b&group_state=expanded

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/84964e5cb506
Don't report "no raptor test results were found" if page load tests time out r=perftest-reviewers,whimboo

Comment 7

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/84964e5cb506

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dave, how are we handling uplifts? I wonder if we should get this landed on beta to help sheriffs with the classification. Not sure about esr68 yet.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

[Tracking Requested - why for this release]: Harness fix that improved the error message for failures, to help with classification.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note that this patch is test-only (part of the Raptor harness), and is not affecting users of Firefox. As such it is enough to request the uplift.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Beta got already merged to release, so it's too late for this uplift.

Also don't we run Raptor jobs for Fennec on ESR68? Why can't I see any of those?

https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&tier=1%2C2%2C3&searchStr=raptor&fromchange=e1268329b7d6f2a687737a7d0a0bd03bfb3a207f

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #12)

Beta got already merged to release, so it's too late for this uplift.

Also don't we run Raptor jobs for Fennec on ESR68? Why can't I see any of those?

https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&tier=1%2C2%2C3&searchStr=raptor&fromchange=e1268329b7d6f2a687737a7d0a0bd03bfb3a207f

We only do this for mobile to compare against current Fennec release. When we do this we use mozilla-central but test against an esr68 build artifact, so there's no need to uplift.