1841353 - `./mach try fuzzy [path to reftest or manifest]` runs an entirely different set of reftests, for the linux debug reftest-snapshot task

Status: NEW

(Developer Infrastructure :: Try, defect)

Description

[Daniel Holbert [:dholbert]]

STR:

1. Run ./mach try fuzzy layout/reftests/bugs/reftest.list
2. Accept all the suggested tasks.
3. When your try run completes, look at the full log for your reftest-snapshot reftest run, to see the tests that were actually run. (You can search for the first & last instance of REFTEST TEST in the log to see what the first/last tests were.)

EXPECTED RESULTS:
The tests in the layout/reftests/bugs/reftest.list manifest should get run. For current mozilla-central, that means the first test should be layout/reftests/bugs/105-1.html, and the last should be layout/reftests/bugs/1836024-1.html (or whatever tests get added after that).

ACTUAL RESULTS:
An entirely different set of reftests get run for some reason.

Here's my Try run where I hit this:
https://treeherder.mozilla.org/jobs?repo=try&revision=1629611241557a0ac47c52cb35adfa16c17c6c39&searchStr=linux%2Cdebug%2Creftest

Here's my test-linux1804-64-qr/debug-reftest-1 log from that run, which shows EXPECTED RESULTS (it starts with 105-1.html):
https://firefoxci.taskcluster-artifacts.net/AZxurMcLT6G6-I9VL_V4sQ/0/public/logs/live_backing.log

And here's my test-linux1804-64-qr/debug-reftest-snapshot-1 log from that run, which shows ACTUAL RESULTS:
https://firefoxci.taskcluster-artifacts.net/OiQnmTrBR0G-IuRITjDfzg/0/public/logs/live_backing.log
In particular, this^ log unexpected runs this range of reftests from completely different directories (and zero tests from the manifest that I asked for, i.e. no tests that live in layout/reftests/bugs):

REFTEST TEST-START | layout/reftests/css-grid/grid-whitespace-handling-1a.xhtml == layout/reftests/css-grid/grid-whitespace-handling-1-ref.xhtml
[...]
REFTEST TEST-START | image/test/reftest/webp/icc-bit-no-icc-chunk.webp == image/test/reftest/webp/blue.png

Comment 1

[Daniel Holbert [:dholbert]]

(This led to me being backed out for landing what I had thought was a correctly-annotated reftest, in bug 1824242 comment 18. I was adding several new tests, and I'd run them through Try with the command from comment 0; but in fact, my Try run's reftest-snapshot task didn't run my new test at all; it ran an entirely different set of tests.)

Comment 2

[Daniel Holbert [:dholbert]]

This affects single-test runs, too (e.g. ./mach try fuzzy [path to some specific reftest])

I just kicked off a try run with

./mach try fuzzy layout/reftests/bugs/skew-blank-test.html

...where skew-blank-test.html is a test I created in a local commit. That got me this green reftest-snapshot task:
https://firefoxci.taskcluster-artifacts.net/b4CnzShuTCSbbtCuBMzp5w/0/public/logs/live_backing.log

As in comment 0, though: the test run didn't run the thing that I asked for. Instead, it starts with layout/reftests/css-grid/grid-whitespace-handling-1a.xhtml for some reason, and it runs ~1600 tests, up through editor/reftests/xul/emptytextbox-4.xhtml (rather than the one test that I asked it to run).

Comment 3

[Daniel Holbert [:dholbert]]

Also for reference: here's a try run based off of unmodified mozilla-central, just to prove that this isn't something broken from one of the patches that I'm pushing to Try. :)
https://treeherder.mozilla.org/jobs?repo=try&revision=1fccb0f27368ded209fd18ebdbf85654e03d928d&selectedTaskRun=BlNCNzbERYqPQIieBrFH6w.0

The reftest-snapshot task there (the selectedTaskRun) ends up running the entirely wrong set of tests, just as discussed above.