1426120 - test-verify sometimes verifies a test twice

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

A curious example: 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e892d02a792258e04bd18dad2b0d2d79f56d5f12&selectedJob=152290685

It looks like both 

layout/reftests/display-list/retained-dl-zindex-1.html
layout/reftests/display-list/retained-dl-zindex-2.html

were verified twice. I guess that isn't a big problem, but it is unexpected.

Comment 1

[Geoff Brown [:gbrown]]

This issue arises for plain reftests when both the test file and the associated reference file are modified on the same push. There is existing code which addresses this exact issue by eliminating the reference file from the verification list when the associated test file is already due for verification. There is a bug in that code: The file path is adjusted at

https://dxr.mozilla.org/mozilla-central/rev/1624b88874765bf57e9feba176d30149c748d9d2/testing/mozharness/mozharness/mozilla/testing/verify_tools.py#199

which results in a consistent failure in the test at

https://dxr.mozilla.org/mozilla-central/rev/1624b88874765bf57e9feba176d30149c748d9d2/testing/mozharness/mozharness/mozilla/testing/verify_tools.py#207

(at this point, "nonref" contains the full path, while "files" contains relative paths).

Comment 2

[Geoff Brown [:gbrown]]

Attachment: updates files list with full paths before searching

This works fine now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=65156d8f9b1a76062e59bf4a44c6229d8dc0d7f5

(failure is expected / I verified that only the test files are verified and each one is only verified once).

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8938229 [details] [diff] [review]
updates files list with full paths before searching

Review of attachment 8938229 [details] [diff] [review]:
-----------------------------------------------------------------

good catch

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/70c6fd6643cf
Update test-verify so that -ref files do not trigger re-verification of non-ref file; r=jmaher

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/70c6fd6643cf