Closed Bug 1423771 Opened 7 years ago Closed 6 years ago

Intermittent REFTEST ERROR | EXCEPTION: No tests to run : TV - test verify

Categories

(Testing :: General, defect, P5)

Product:
Testing
Component:
General
Version:
Version 3
Type:
defect

Tracking

(firefox59 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla59
Tracking Flags:
Tracking Status
firefox59 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell fixed:other])

Attachments

(1 file)

suppress "No tests run" error in verify mode
1.44 KB, patch
jmaher
: review+
Details | Diff | Splinter Review
Assignee: nobody → gbrown
See Also: → 1406407
This has been filed 2 days ago.
In this time frame it has registered 32 failures.

Occurrences per platform:
8 - Linux x64
6 - Windows 7
6 - windows10-64
4 - OS X 10.10
2 - android-4-3-armv7-api16
2 - android-api-16-gradle
1 - macosx64-nightly
1 - linux64-nightly
1 - windows7-32-nightly
1 - windows10-64-nightly

Occurrences per build type:
16 - opt
8  - debug
6  - pgo
2  - asan

Here is a relevant log file: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=150342395
And a snippet from it: 
[task 2017-12-06T23:37:43.921Z] 23:37:43     INFO - REFTEST INFO | Dumping JSON representation of sandbox
60192
[task 2017-12-06T23:37:43.923Z] 23:37:43     INFO - REFTEST INFO | {"isDebugBuild":false,"xulRuntime":{"widgetToolkit":"gtk3","OS":"Linux","XPCOMABI":"x86_64-gcc3"},"smallScreen":false,"d2d":false,"dwrite":false,"gpuProcess":false,"azureCairo":false,"azureSkia":true,"skiaContent":true,"azureSkiaGL":0,"contentSameGfxBackendAsCanvas":true,"layersGPUAccelerated":true,"d3d11":false,"d3d9":false,"layersOpenGL":true,"webrender":false,"layersOMTC":true,"advancedLayers":false,"layerChecksEnabled":true,"retainedDisplayList":true,"Android":false,"cocoaWidget":false,"gtkWidget":true,"qtWidget":false,"winWidget":false,"transparentScrollbars":true,"AddressSanitizer":false,"webrtc":true,"retainedDisplayLists":true,"compareRetainedDisplayLists":false,"stylo":true,"styloVsGecko":false,"skiaPdf":false,"release_or_beta":false,"http":{"userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0","appName":"Mozilla","appVersion":"5.0","platform":"X11","oscpu":"Linux x86_64","misc":"rv:59.0"},"haveTestPlugin":true,"windowsDefaultTheme":false,"nativeThemePref":true,"gpuProcessForceEnabled":false,"prefs":{},"browserIsRemote":true,"asyncPan":true,"usesRepeatResampling":false}
60193
[task 2017-12-06T23:37:43.943Z] 23:37:43     INFO - REFTEST SUITE-START | Running 0 tests
60194
[task 2017-12-06T23:37:43.944Z] 23:37:43    ERROR - REFTEST ERROR | EXCEPTION: No tests to run
60195
[task 2017-12-06T23:37:43.944Z] 23:37:43     INFO - REFTEST INFO | Result summary:
60196
[task 2017-12-06T23:37:43.944Z] 23:37:43     INFO - REFTEST INFO | Successful: 0 (0 pass, 0 load only)
60197
[task 2017-12-06T23:37:43.945Z] 23:37:43     INFO - REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 1 exception)
60198
[task 2017-12-06T23:37:43.945Z] 23:37:43     INFO - REFTEST INFO | Known problems: 0 (0 known fail, 0 known asserts, 0 random, 0 skipped, 0 slow)
60199
[task 2017-12-06T23:37:43.946Z] 23:37:43     INFO - REFTEST SUITE-END | Shutdown
60200
[task 2017-12-06T23:37:43.947Z] 23:37:43     INFO - REFTEST INFO | Slowest test took 0ms (undefined)
60201
[task 2017-12-06T23:37:43.948Z] 23:37:43     INFO - REFTEST INFO | Total canvas count = 0
Flags: needinfo?(gbrown)
Whiteboard: [stockwell needswork]
This was triggered by bug 1422979, but seems to be a fault of test-verify. 

All failures were on 1 day. Failures on inbound remain un-starred. Other than those, I do not expect this to recur.
Blocks: 1406407
Flags: needinfo?(gbrown)
(In reply to Geoff Brown [:gbrown] (away Dec 22 - Jan 3) from comment #3)
> This was triggered by bug 1422979, but seems to be a fault of test-verify.

For example, consider

== pal8offs.bmp pal8.png

If pal8.png is modified, TV thinks pal8.png is a test, and there's no way to determine it is a reference file from the name or the manifest interface.
Comment on attachment 8939109 [details] [diff] [review]
suppress "No tests run" error in verify mode

It's hard to catch all the edge cases in mozharness to reliably determine which changed files are real reftests. To allow for occasional errors, we can simply suppress the "No tests to run" exception when in verify mode.
Attachment #8939109 - Flags: review?(jmaher)
Comment on attachment 8939109 [details] [diff] [review]
suppress "No tests run" error in verify mode

Review of attachment 8939109 [details] [diff] [review]:
-----------------------------------------------------------------

this should work!
Attachment #8939109 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1771c4a5c270
Suppress error when no reftests verified; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/1771c4a5c270
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox59: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
Whiteboard: [stockwell needswork] → [stockwell fixed:other]
This helped, but there's another hoop to jump through: Structured logging may complain "No checks run."

https://treeherder.mozilla.org/logviewer.html#?job_id=165592716&repo=try&lineNumber=5464
Blocks: 1442790
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: