1162954 - REFTEST_MANIFESTS moz.build property does nothing in automation and is a footgun

Status: NEW

(Firefox Build System :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

STR:
- Create some new reftests, list them in a reftest.list, and add that list file to REFTEST_MANIFESTS in a moz.build file (see e.g. https://hg.mozilla.org/try/rev/bf9ec4319db1)
- Run reftests locally. This works fine
- Push to try.

Expected: reftests run.
Actual: Reftests don't get run.

It looks like the reftest jobs that run in automation completely ignore REFTEST_MANIFESTS and just hard-code layout/reftests/reftest.list as the one and only reftest manifest. This is bad because you could write a bunch of reftests, get them passing locally, do a push to try which is green and land without realizing that your fancy new reftests are not getting run in automation at all.