Closed Bug 1524495 Opened 9 months ago Closed 4 months ago

Intermittent raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-geckoview

Categories

(Testing :: Raptor, defect, P5)

Version 3
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: NarcisB)

References

Details

(Keywords: intermittent-failure, Whiteboard: [retriggered][stockwell disabled])

Attachments

(2 files)

#[markdown(off)]
Filed by: archaeopteryx [at] coole-files.de

http://localhost:5000/logviewer.html#?job_id=225400069&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/SpuRw0gCT_u0_VGg6J_fDQ/runs/0/artifacts/public/logs/live_backing.log

08:09:05 INFO - Activity: org.mozilla.geckoview_example/.GeckoViewActivity
08:09:05 INFO - ThisTime: 448
08:09:05 INFO - TotalTime: 448
08:09:05 INFO - WaitTime: 462
08:09:05 INFO - Complete
08:09:08 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:08] "POST / HTTP/1.1" 200 -
08:09:08 INFO - raptor-control-server received webext_status: raptor runner.js is loaded!
08:09:08 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:08] "GET /raptor-unity-webgl-geckoview.json HTTP/1.1" 200 -
08:09:08 INFO - raptor-control-server reading test settings from raptor-unity-webgl-geckoview.json
08:09:08 INFO - raptor-control-server sent test settings to web ext runner
08:09:08 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:08] "POST / HTTP/1.1" 200 -
08:09:08 INFO - raptor-control-server received webext_status: * pausing 30 seconds to let browser settle... *
08:09:38 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:38] "POST / HTTP/1.1" 200 -
08:09:38 INFO - raptor-control-server received webext_status: running 1 pagecycles of http://127.0.0.1:60169/unity-webgl/index.html?raptor
08:09:39 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:39] "POST / HTTP/1.1" 200 -
08:09:39 INFO - raptor-control-server received webext_status: begin pagecycle 1
08:09:40 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:40] "POST / HTTP/1.1" 200 -
08:09:40 INFO - raptor-control-server received webext_status: update tab 0
08:09:40 INFO - 127.0.0.1 - - [01/Feb/2019 08:09:40] "POST / HTTP/1.1" 200 -
08:09:40 INFO - raptor-control-server received webext_status: test tab updated 0
08:24:32 INFO - raptor-main application timed out after 930 seconds
08:24:33 INFO - adb shell_output: adb -s ZY322LHDDG wait-for-device shell am force-stop org.mozilla.geckoview_example; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
08:24:34 INFO - adb command_output: adb -s ZY322LHDDG wait-for-device pull /sdcard/raptor/profile/minidumps /tmp/tmpFx84NN/minidumps, timeout: None, timedout: None, exitcode: 0, output: pull: building file list...
08:24:34 INFO - /sdcard/raptor/profile/minidumps/: 0 files pulled.
08:24:34 INFO - raptor-main removing webext /builds/worker/workspace/build/tests/raptor/webext/raptor
08:24:34 INFO - results-handler summarizing raptor test results
08:24:34 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-geckoview
08:24:34 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-geckoview
08:24:34 INFO - raptor-control-server shutting down control server
08:24:34 INFO - raptor-main removing reverse socket connections
08:24:34 INFO - adb command_output: adb -s ZY322LHDDG wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
08:24:34 INFO - raptor-main finished
08:24:34 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-geckoview

Yes. This is the same error we have seen for all raptor tests but now we know which test failed and can track them separately.

Flags: needinfo?(bob)

(In reply to Bob Clary [:bc:] from comment #3)

Yes. This is the same error we have seen for all raptor tests but now we know which test failed and can track them separately.

Thank you for clarifying Bob.

Blocks: 1520130

Update: there have been 88 failures within the last 7 days on Android 8.0 Pixel2 opt and Android 7.0 MotoG5 opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227762796&repo=mozilla-inbound&lineNumber=1064

22:41:47 INFO - 127.0.0.1 - - [11/Feb/2019 22:41:47] "POST / HTTP/1.1" 200 -
22:41:47 INFO - raptor-control-server received webext_status: raptor runner.js is loaded!
22:41:47 INFO - 127.0.0.1 - - [11/Feb/2019 22:41:47] "GET /raptor-unity-webgl-geckoview.json HTTP/1.1" 200 -
22:41:47 INFO - raptor-control-server reading test settings from raptor-unity-webgl-geckoview.json
22:41:47 INFO - raptor-control-server sent test settings to web ext runner
22:41:47 INFO - 127.0.0.1 - - [11/Feb/2019 22:41:47] "POST / HTTP/1.1" 200 -
22:41:47 INFO - raptor-control-server received webext_status: * pausing 30 seconds to let browser settle... *
22:42:17 INFO - 127.0.0.1 - - [11/Feb/2019 22:42:17] "POST / HTTP/1.1" 200 -
22:42:17 INFO - raptor-control-server received webext_status: running 1 pagecycles of http://127.0.0.1:34196/unity-webgl/index.html?raptor
22:42:18 INFO - 127.0.0.1 - - [11/Feb/2019 22:42:18] "POST / HTTP/1.1" 200 -
22:42:18 INFO - raptor-control-server received webext_status: begin pagecycle 1
22:42:19 INFO - 127.0.0.1 - - [11/Feb/2019 22:42:19] "POST / HTTP/1.1" 200 -
22:42:19 INFO - raptor-control-server received webext_status: update tab 0
22:42:19 INFO - 127.0.0.1 - - [11/Feb/2019 22:42:19] "POST / HTTP/1.1" 200 -
22:42:19 INFO - raptor-control-server received webext_status: test tab updated 0
22:57:13 INFO - raptor-main application timed out after 930 seconds
22:57:13 INFO - adb shell_output: adb -s HT85S1A02559 wait-for-device shell am force-stop org.mozilla.geckoview_example; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
22:57:14 INFO - adb command_output: adb -s HT85S1A02559 wait-for-device pull /sdcard/raptor/profile/minidumps /tmp/tmp4rIl6J/minidumps, timeout: None, timedout: None, exitcode: 0, output: pull: building file list...
22:57:14 INFO - /sdcard/raptor/profile/minidumps/: 0 files pulled.
22:57:14 INFO - raptor-main removing webext /builds/worker/workspace/build/tests/raptor/webext/raptor
22:57:14 INFO - results-handler summarizing raptor test results
22:57:14 INFO - raptor-output error: no raptor test results found for raptor-unity-webgl-geckoview
22:57:14 INFO - raptor-output error: no summarized raptor results found for raptor-unity-webgl-geckoview
22:57:14 INFO - raptor-control-server shutting down control server
22:57:14 INFO - raptor-main removing reverse socket connections
22:57:14 INFO - adb command_output: adb -s HT85S1A02559 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
22:57:14 INFO - raptor-main finished
22:57:14 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-geckoview
22:57:15 ERROR - Return code: 1
22:57:15 WARNING - setting return code to 1
22:57:15 INFO - Killing logcat pid 431.
22:57:15 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
22:57:15 INFO - copying raptor results to upload dir:
22:57:15 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
22:57:15 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
22:57:15 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
22:57:15 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'

Whiteboard: [retriggered] → [retriggered][stockwell needswork]
Whiteboard: [retriggered][stockwell disable-recommended] → [retriggered][stockwell needswork]

This bug failed 71 times in the last 7 days. Occurs on Android 8.0 Pixel2 opt and Android 7.0 MotoG5 opt.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=229357007&repo=autoland&lineNumber=1069

rwood: Can you please take a look at this bug?

Flags: needinfo?(rwood)
Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9b617c97285a
Temporarily disable raptor-unity-webgl-geckoview due to frequent failures r=jmaher
Backout by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/984c1d9f8a44
Backed out changeset 9b617c97285a for Py2 failure
Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e5fc352e6d65
Temporarily disable raptor-unity-webgl-geckoview due to frequent failures r=jmaher
Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4f37d111e4a9
Backed out changeset e5fc352e6d65 for Raptor performance tests failures.
https://hg.mozilla.org/integration/autoland/rev/2e473aa11c19
Temporarily disable raptor-unity-webgl-geckoview due to frequent failures. r=jmaher
Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Assignee: nobody → nbeleuzu

Last disable push did his job but triggered a perma Android 7.0 MotoG5 opt - ugl failure.
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=230581981&repo=autoland&lineNumber=894

I`m not sure how we can properly disable it.
:rwood , can you please take a look?

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

If we still want to disable this, I believe doing so in raptor.yml is the right place.
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml

Status: REOPENED → RESOLVED
Closed: 8 months ago8 months ago
Resolution: --- → FIXED

Had two tries on fixing this on try but to no avail:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5249fc7eb727e0a2461171eecb3268c95a3e8871&selectedJob=230963976

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6e899fa6dc062670f62d12eef5021cd0471e4c48&selectedJob=230959091

The actual fail I think complains about not finding the test anymore since has been disabled.

07:37:52 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', u'/builds/worker/workspace/build/tests/raptor/raptor/raptor.py', u'--test', 'raptor-unity-webgl', u'--binary', 'org.mozilla.geckoview_example', u'--app', 'geckoview', u'--symbolsPath', 'https://queue.taskcluster.net/v1/task/Fvq-V7AKT5OvonpL8SLjdA/artifacts/public/build/target.crashreporter-symbols.zip', u'--log-tbpl-level=debug'] with output_timeout 3600
07:37:53 INFO - raptor-main raptor-start
07:37:53 INFO - raptor-main received command line arguments: Namespace(activity='GeckoViewActivity', app='geckoview', binary='org.mozilla.geckoview_example', debug_mode=False, gecko_profile=False, gecko_profile_entries=None, gecko_profile_interval=None, host='127.0.0.1', is_release_build=False, log_errorsummary=None, log_grouped=None, log_html=None, log_mach=None, log_mach_buffer=None, log_mach_level=None, log_mach_screenshot=None, log_mach_verbose=None, log_raw=None, log_raw_level=None, log_tbpl=None, log_tbpl_buffer=None, log_tbpl_compact=None, log_tbpl_level='debug', log_unittest=None, log_xunit=None, obj_path=None, page_cycles=None, page_timeout=None, power_test=False, run_local=False, symbols_path='https://queue.taskcluster.net/v1/task/Fvq-V7AKT5OvonpL8SLjdA/artifacts/public/build/target.crashreporter-symbols.zip', test='raptor-unity-webgl')
07:37:53 INFO - raptor-manifest /builds/worker/workspace/build/tests/raptor/raptor/raptor.ini
07:37:53 INFO - raptor-manifest abort: specified test name doesn't exist
07:37:53 INFO - raptor-main abort: no tests found

raptor.ini contains the whole chunk of those test, so no possibility of disabling this from there: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.ini#26

Looking at https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#656 disabling this directly from that file seems a bit too much for me. Hopefully Robert could take a look at this.

Flags: needinfo?(rwood)
Whiteboard: [retriggered][stockwell disable-recommended] → [retriggered][stockwell disabled]
Target Milestone: mozilla67 → ---
Flags: needinfo?(rwood)

(In reply to Cosmin Sabou [:CosminS] from comment #32)

Looking at https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#656 disabling this directly from that file seems a bit too much for me. Hopefully Robert could take a look at this.

Thanks Cosmin yes using 'diabled =' in a test INI in production is only good for subtests (i.e. tp6* suites) as there are other subtests in the suite that are still found/run. Disabling an entire test at the test level / just one test has to be done at the taskcluster config level as you noted raptor.yml. I'll make a patch now.

Flags: needinfo?(rwood)
Attachment #9047458 - Flags: review?(dave.hunt)
See Also: → 1531441
Attachment #9047458 - Flags: review?(dave.hunt) → review+
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1a272d807e84
Disble Raptor ugl geckoview job b/c of permafail; r=davehunt
Status: REOPENED → RESOLVED
Closed: 8 months ago8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67

This was not actually fixed but disabled @comment 36 so setting the flags accordingly.

Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Target Milestone: mozilla67 → ---
Status: REOPENED → RESOLVED
Closed: 8 months ago4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.