Closed Bug 1477976 Opened 6 years ago Closed 6 years ago

Windows TVg failed with "No tests run or test summary not found" on a reftest reference file

Categories

(Testing :: General, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1478801

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure)

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=189722575&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/IjwtHSU1RBS3goLKvWiM0Q/runs/0/artifacts/public/logs/live_backing.log

09:04:36     INFO -  REFTEST SUITE-START | Running 0 tests
09:04:36     INFO -  REFTEST INFO | Result summary:
09:04:36     INFO -  REFTEST INFO | Successful: 0 (0 pass, 0 load only)
09:04:36     INFO -  REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception)
09:04:36     INFO -  REFTEST INFO | Known problems: 0 (0 known fail, 0 known asserts, 0 random, 0 skipped, 0 slow)
09:04:36     INFO -  REFTEST SUITE-END | Shutdown
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO -  REFTEST INFO | ::: Test verification summary for:
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO -  REFTEST INFO | ::: Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/greenbox.html
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO -  REFTEST INFO | ::: 1. Run each test 10 times in one browser. : Pass
09:04:36     INFO -  REFTEST INFO | ::: 2. Run each test 5 times in a new browser each time. : Pass
09:04:36     INFO -  REFTEST INFO | ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
09:04:36     INFO -  REFTEST INFO | ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO -  REFTEST INFO | ::: Test verification PASSED
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO - Return code: 0
09:04:36    ERROR - No tests run or test summary not found
09:04:36     INFO - TinderboxPrint: reftest-reftest<br/><em class="testfail">T-FAIL</em>
09:04:36  WARNING - # TBPL WARNING #
09:04:36  WARNING - setting return code to 1
09:04:36  WARNING - TinderboxPrint: Per-test run of ...\css-mediaqueries\greenbox.html<br/>: WARNING
09:04:36     INFO - Running command: ['Z:\\task_1532422227\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1532422227\\build\\tests\\reftest\\runreftest.py', '--appname=Z:\\task_1532422227\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_raw.log', '--log-errorsummary=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=Z:\\task_1532422227\\build', '--suite=reftest', '--verify-max-time=900', '--verify', u'Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html'] in Z:\task_1532422227\build
09:04:36     INFO - Copy/paste: Z:\task_1532422227\build\venv\Scripts\python -u Z:\task_1532422227\build\tests\reftest\runreftest.py --appname=Z:\task_1532422227\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip --log-raw=Z:\task_1532422227\build\blobber_upload_dir\reftest_raw.log --log-errorsummary=Z:\task_1532422227\build\blobber_upload_dir\reftest_errorsummary.log --cleanup-crashes --marionette-startup-timeout=180 --sandbox-read-whitelist=Z:\task_1532422227\build --suite=reftest --verify-max-time=900 --verify Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html
09:04:36     INFO - Using env: (same as previous command)
09:04:36     INFO - Calling ['Z:\\task_1532422227\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1532422227\\build\\tests\\reftest\\runreftest.py', '--appname=Z:\\task_1532422227\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://queue.taskcluster.net/v1/task/AYvetICkTHeQXmLYmx5q8g/artifacts/public/build/target.crashreporter-symbols.zip', '--log-raw=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_raw.log', '--log-errorsummary=Z:\\task_1532422227\\build\\blobber_upload_dir\\reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=Z:\\task_1532422227\\build', '--suite=reftest', '--verify-max-time=900', '--verify', u'Z:/task_1532422227/build/tests/reftest/tests/layout/reftests/css-mediaqueries/mq_prefers_reduced_motion_no_preference.html'] with output_timeout 1000
09:04:36     INFO -  REFTEST INFO | :::
09:04:36     INFO -  REFTEST INFO | ::: Running test verification step "1. Run each test 10 times in one browser."...
09:04:36     INFO -  REFTEST INFO | :::
09:04:37     INFO -  REFTEST INFO | Running with e10s: True
Flags: needinfo?(hikezoe)
This looks like a test harness bug. Not sure why we think it's useful for this Test-Verify job to be "Running 0 tests"...
Hmm, it looks like Test-Verify was thinking "greenbox.html" was a new testcase, and was attempting to run all of the reftest comparisons where it is the testcase -- and there are 0 of those, because it's not a testcase -- it's a reference case.

Maybe the test-verify logic expects reference cases to end in "-ref" and assumes that anything else is a testcase, perhaps? Not sure.

In any case, this won't cause lasting problems because I think test-verify job only cares about tests that were added in the commit in question. (So this isn't a sign of perma-orange or anything like that.)
Thank you, Daniel.  When I did wake up, you already figure out what the problem is. :)

You are absolutely right.
https://searchfox.org/mozilla-central/rev/bdfd20ef30d521b57d5b6feeda71325e8b4cad66/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#260

I'd want the harness just to pick up new files which are only at the left side on the comparison in reftest.list.
Component: Layout → General
Flags: needinfo?(hikezoe)
Product: Core → Testing
Summary: Intermittent test-verify No tests run or test summary not found with WARNING - TinderboxPrint: Per-test run of ...\css-mediaqueries\greenbox.html<br/>: WARNING → Test harness should pick up new files which are at the left side in comparison in reftest.list for test-verify
Ah OK, we can't just pick up the left side ones.  I noticed a comment there;

 # Special handling for modified reftest reference files:

https://searchfox.org/mozilla-central/rev/bdfd20ef30d521b57d5b6feeda71325e8b4cad66/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#270

So, we should skip targeting the reference file somewhere?
This is a tricky "feature" of test-verify that has given us some trouble. There are special checks to try to avoid verification on reference (left side) files, but cases like this greenbox.html are expected to evade those checks. Still, test-verify should not be failing, and you can see most of the same behavior in the osx and linux TVg jobs on this push -- without the error.

Something special happened here on Windows only and I don't quite understand what it was. Will try to have a look soon...
Blocks: test-verify
Flags: needinfo?(gbrown)
"No tests run or test summary not found" comes from DesktopUnittestOutputParser.evaluate_parser(); that's called in the Windows TVg task, but not in the corresponding Linux/osx TVg tasks.
...that's because the StructuredOutputParser is not used on Windows.

https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/scripts/desktop_unittest.py#515-520

...and that's because of mozharness configuration differences. The linux, osx, and old (buildbot) windows configuration files have unstructured_flavors defined, 

https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/configs/unittests/win_unittest.py#252

but the taskcluster windows configuration does not. Instead, it has:

https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/configs/unittests/win_taskcluster_unittest.py#300

 "structured_suites": ["reftest"],

...but structured_suites has no effect in desktop_unittest, as far as I can tell.
See Also: → 1478801
There are a few ways of looking at this bug. I think we're here because Windows TVg failed with "No tests run or test summary not found". Bug 1478801 will address the logging configuration anomaly in Windows, which will avoid this failure.
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Summary: Test harness should pick up new files which are at the left side in comparison in reftest.list for test-verify → Windows TVg failed with "No tests run or test summary not found" on a reftest reference file
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
See Also: 1478801
You need to log in before you can comment on or make changes to this bug.