Closed
Bug 1477976
Opened 6 years ago
Closed 6 years ago
Windows TVg failed with "No tests run or test summary not found" on a reftest reference file
Categories
(Testing :: General, defect, P5)
Testing
General
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 1478801
People
(Reporter: intermittent-bug-filer, Assigned: gbrown)
References
(Blocks 1 open bug)
Details
(Keywords: intermittent-failure)
Filed by: ncsoregi [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=189722575&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/IjwtHSU1RBS3goLKvWiM0Q/runs/0/artifacts/public/logs/live_backing.log 09:04:36 INFO - REFTEST SUITE-START | Running 0 tests 09:04:36 INFO - REFTEST INFO | Result summary: 09:04:36 INFO - REFTEST INFO | Successful: 0 (0 pass, 0 load only) 09:04:36 INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) 09:04:36 INFO - REFTEST INFO | Known problems: 0 (0 known fail, 0 known asserts, 0 random, 0 skipped, 0 slow) 09:04:36 INFO - REFTEST SUITE-END | Shutdown 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - REFTEST INFO | ::: Test verification summary for: 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - REFTEST INFO | ::: Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/greenbox.html 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - REFTEST INFO | ::: 1. Run each test 10 times in one browser. : Pass 09:04:36 INFO - REFTEST INFO | ::: 2. Run each test 5 times in a new browser each time. : Pass 09:04:36 INFO - REFTEST INFO | ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass 09:04:36 INFO - REFTEST INFO | ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - REFTEST INFO | ::: Test verification PASSED 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - Return code: 0 09:04:36 ERROR - No tests run or test summary not found 09:04:36 INFO - TinderboxPrint: reftest-reftest<br/><em class="testfail">T-FAIL</em> 09:04:36 WARNING - # TBPL WARNING # 09:04:36 WARNING - setting return code to 1 09:04:36 WARNING - TinderboxPrint: Per-test run of ...\css-mediaqueries\greenbox.html<br/>: WARNING 09:04:36 INFO - Running command: ['Z:\\task_1532422227\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1532422227\\build\\tests\\reftest\\runreftest.py', '--appname=Z:\\task_1532422227\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_raw.log', '--log-errorsummary=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=Z:\\task_1532422227\\build', '--suite=reftest', '--verify-max-time=900', '--verify', u'Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html'] in Z:\task_1532422227\build 09:04:36 INFO - Copy/paste: Z:\task_1532422227\build\venv\Scripts\python -u Z:\task_1532422227\build\tests\reftest\runreftest.py --appname=Z:\task_1532422227\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip --log-raw=Z:\task_1532422227\build\blobber_upload_dir\reftest_raw.log --log-errorsummary=Z:\task_1532422227\build\blobber_upload_dir\reftest_errorsummary.log --cleanup-crashes --marionette-startup-timeout=180 --sandbox-read-whitelist=Z:\task_1532422227\build --suite=reftest --verify-max-time=900 --verify Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html 09:04:36 INFO - Using env: (same as previous command) 09:04:36 INFO - Calling ['Z:\\task_1532422227\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1532422227\\build\\tests\\reftest\\runreftest.py', '--appname=Z:\\task_1532422227\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_raw.log', '--log-errorsummary=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=Z:\\task_1532422227\\build', '--suite=reftest', '--verify-max-time=900', '--verify', u'Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html'] with output_timeout 1000 09:04:36 INFO - REFTEST INFO | ::: 09:04:36 INFO - REFTEST INFO | ::: Running test verification step "1. Run each test 10 times in one browser."... 09:04:36 INFO - REFTEST INFO | ::: 09:04:37 INFO - REFTEST INFO | Running with e10s: True
Updated•6 years ago
|
Flags: needinfo?(hikezoe)
Comment 1•6 years ago
|
||
This looks like a test harness bug. Not sure why we think it's useful for this Test-Verify job to be "Running 0 tests"...
Comment 2•6 years ago
|
||
Hmm, it looks like Test-Verify was thinking "greenbox.html" was a new testcase, and was attempting to run all of the reftest comparisons where it is the testcase -- and there are 0 of those, because it's not a testcase -- it's a reference case. Maybe the test-verify logic expects reference cases to end in "-ref" and assumes that anything else is a testcase, perhaps? Not sure. In any case, this won't cause lasting problems because I think test-verify job only cares about tests that were added in the commit in question. (So this isn't a sign of perma-orange or anything like that.)
Comment 3•6 years ago
|
||
Thank you, Daniel. When I did wake up, you already figure out what the problem is. :) You are absolutely right. https://searchfox.org/mozilla-central/rev/bdfd20ef30d521b57d5b6feeda71325e8b4cad66/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#260 I'd want the harness just to pick up new files which are only at the left side on the comparison in reftest.list.
Component: Layout → General
Flags: needinfo?(hikezoe)
Product: Core → Testing
Summary: Intermittent test-verify No tests run or test summary not found with WARNING - TinderboxPrint: Per-test run of ...\css-mediaqueries\greenbox.html<br/>: WARNING → Test harness should pick up new files which are at the left side in comparison in reftest.list for test-verify
Comment 4•6 years ago
|
||
Ah OK, we can't just pick up the left side ones. I noticed a comment there; # Special handling for modified reftest reference files: https://searchfox.org/mozilla-central/rev/bdfd20ef30d521b57d5b6feeda71325e8b4cad66/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#270 So, we should skip targeting the reference file somewhere?
Assignee | ||
Comment 6•6 years ago
|
||
This is a tricky "feature" of test-verify that has given us some trouble. There are special checks to try to avoid verification on reference (left side) files, but cases like this greenbox.html are expected to evade those checks. Still, test-verify should not be failing, and you can see most of the same behavior in the osx and linux TVg jobs on this push -- without the error. Something special happened here on Windows only and I don't quite understand what it was. Will try to have a look soon...
Blocks: test-verify
Flags: needinfo?(gbrown)
Assignee | ||
Comment 7•6 years ago
|
||
"No tests run or test summary not found" comes from DesktopUnittestOutputParser.evaluate_parser(); that's called in the Windows TVg task, but not in the corresponding Linux/osx TVg tasks.
Assignee | ||
Comment 8•6 years ago
|
||
...that's because the StructuredOutputParser is not used on Windows. https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/scripts/desktop_unittest.py#515-520 ...and that's because of mozharness configuration differences. The linux, osx, and old (buildbot) windows configuration files have unstructured_flavors defined, https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/configs/unittests/win_unittest.py#252 but the taskcluster windows configuration does not. Instead, it has: https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/configs/unittests/win_taskcluster_unittest.py#300 "structured_suites": ["reftest"], ...but structured_suites has no effect in desktop_unittest, as far as I can tell.
Assignee | ||
Comment 9•6 years ago
|
||
There are a few ways of looking at this bug. I think we're here because Windows TVg failed with "No tests run or test summary not found". Bug 1478801 will address the logging configuration anomaly in Windows, which will avoid this failure.
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Summary: Test harness should pick up new files which are at the left side in comparison in reftest.list for test-verify → Windows TVg failed with "No tests run or test summary not found" on a reftest reference file
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Updated•6 years ago
|
You need to log in
before you can comment on or make changes to this bug.
Description
•