REFTEST_MANIFESTS property does nothing locally




Build Config
3 years ago
3 years ago


(Reporter: kats, Unassigned)


Firefox Tracking Flags

(Not tracked)


+++ This bug was initially created as a clone of Bug #1162954 +++

- Create some new reftests, list them in a reftest.list, and add that list file to REFTEST_MANIFESTS in a file (see e.g.
- Run reftests locally. This works fine
- Now, remove the reftest.list from the file (i.e. don't add it to REFTEST_MANIFESTS) and try to run the reftests

Expected: reftests don't run
Actual: Reftests DO get run.

Now, rename your reftest.list file to or something else, and put it back in your file in REFTEST_MANIFESTS. Try to run reftests.

Expected: reftests run
Actual: reftests do not run, because mach cannot find the "reftest.list" file.

Apparently it doesn't bother using REFTEST_MANIFESTS at all, and just has reftest.list hard-coded somewhere. It dumps something like this out:

Exception: No manifest file was found at gfx/layers/apz/reftests/reftest.list.

  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/", line 422, in run_reftest
    return self._run_reftest(test_file, suite='reftest', **kwargs)
  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/", line 450, in _run_reftest
    return reftest.run_desktop_test(test_file, suite=suite, **kwargs)
  File "/home/kats/zspace/B2G-flame-kk/gecko/layout/tools/reftest/", line 265, in run_desktop_test
    raise Exception('No manifest file was found at %s.' % path)
Blocks: 1027215

Comment 1

3 years ago
The goal was always to follow up and do something useful with the metadata. When you don't have full-time staffing on the build system, things like this fall through the cracks.

It's been a year and AFAIK you are the only one to fall through this crack. So this bug will likely linger.
You need to log in before you can comment on or make changes to this bug.