1162955 - REFTEST_MANIFESTS moz.build property does nothing locally

Status: NEW

(Firefox Build System :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

+++ This bug was initially created as a clone of Bug #1162954 +++

STR:
- Create some new reftests, list them in a reftest.list, and add that list file to REFTEST_MANIFESTS in a moz.build file (see e.g. https://hg.mozilla.org/try/rev/bf9ec4319db1)
- Run reftests locally. This works fine
- Now, remove the reftest.list from the moz.build file (i.e. don't add it to REFTEST_MANIFESTS) and try to run the reftests

Expected: reftests don't run
Actual: Reftests DO get run.

Now, rename your reftest.list file to reftest.foo or something else, and put it back in your moz.build file in REFTEST_MANIFESTS. Try to run reftests.

Expected: reftests run
Actual: reftests do not run, because mach cannot find the "reftest.list" file.

Apparently it doesn't bother using REFTEST_MANIFESTS at all, and just has reftest.list hard-coded somewhere. It dumps something like this out:

Exception: No manifest file was found at gfx/layers/apz/reftests/reftest.list.

  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/mach_commands.py", line 422, in run_reftest
    return self._run_reftest(test_file, suite='reftest', **kwargs)
  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/mach_commands.py", line 450, in _run_reftest
    return reftest.run_desktop_test(test_file, suite=suite, **kwargs)
  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/mach_commands.py", line 265, in run_desktop_test
    raise Exception('No manifest file was found at %s.' % path)

Comment 1

[Gregory Szorc [:gps]]

The goal was always to follow up and do something useful with the metadata. When you don't have full-time staffing on the build system, things like this fall through the cracks.

It's been a year and AFAIK you are the only one to fall through this crack. So this bug will likely linger.