1403565 - test-verify incorrectly handles reftest verification when reference file is modified: "No tests to run"

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Many reftests compare a "test" file to a "reference" file, as captured in a manifest with something like:

== test.html test-ref.html

'mach reftest test.html' runs the test.

'mach reftest test-ref.html' does not run the test and complains 

  REFTEST ERROR | EXCEPTION: No tests to run


The issue is the same when the reference file is requested directly on the runreftest.py command line.

Currently, when a reftest reference file is modified in a changeset, test-verify tries to run reftests for that reference file, resulting in this error. If the corresponding test file is also modified, that test is run for verification, and that succeeds.

I think the desired behavior is:
 - when the test and reference file are both modified, the test is verified
 - when either the test or the reference are modified, the test is verified

Comment 1

[Geoff Brown [:gbrown]]

An example:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=be79515b9f2afc703b5bcfc14546d9dad7692437&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

Comment 2

[Geoff Brown [:gbrown]]

Attachment: improve reftest reference file verification

The python reftest manifest does not provide enough information to differentiate the test from the reference...and even if it did, it would be awkward to use that in test verification. Instead, I make use of the convention that reference files are usually named "<test>-ref.html" to detect and differentiate reference files and handle them correctly.

Works great when reference file is modified alone or when both test and reference are modified:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8976b22e219fe0d7b5ea8806a6e8b3cfa2a2c284&filter-tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3760d172819c0795113c420c25ab7894758bf2e7&filter-tier=1&filter-tier=2&filter-tier=3

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8912693 [details] [diff] [review]
improve reftest reference file verification

Review of attachment 8912693 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/mozharness/mozilla/testing/verify_tools.py
@@ +147,5 @@
>              for file in files:
>                  if suite in ['reftest', 'crashtest']:
>                      file = os.path.join(self.reftest_test_dir, file)
> +                if suite == 'reftest':
> +                    nonref = file.replace('-ref.', '.')

about about the reference of about:blank or other oddities?  we should find a way to write unittests for these edge cases.

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3)
> about about the reference of about:blank or other oddities?  we should find
> a way to write unittests for these edge cases.

I think about:blank is okay: Usually the test is '== <test>.html about:blank'. If <test>.html changes, we request that test be run from the harness, and all's well. (If about:blank changes, we have bigger problems!!)

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/45f0f5d939ee
Improve test-verify behavior when reftest reference file is modified; r=jmaher

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/45f0f5d939ee