563722 - Implement universal manifest format for reftest

Status: NEW

(Testing :: Reftest, defect)

Description

[Jonathan Griffin (:jgriffin)]

The manifest format should be JSON, per auto-tools consensus, and might look something like this:

{"tests:
[ "== test.html reference.html",
  "!= test.html about:blank",
  {"test": "!= other.html about:blank",
   "skip-if": "!haveTestPlugin"}
]
}

For the most basic case, the test is expressed as the same string currently used in reftest.list manifests.  For cases containing additional conditions, these can be expressed in a more verbose format like the third test in the example above.

The manifest should allow tests to be collected in groups, with conditions applied to the entire group, e.g.:

{"tests:
[ "== test.html reference.html",
  "!= test.html about:blank",
  {"group": [
      "!= other.html about:blank",
      "== this.html that.html",
      "== this.html other.html",
    ],
   "skip-if": "!haveTestPlugin" }
]
}

After initial implementation, we can look at adding other manifest features, like setting per-test prefs.

We can possibly use http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/print-manifest-dirs.py as a starting point for creating an automatic reftest.list-to-univeral.manifest conversion tool.

Comment 1

[alice nodelman [:alice] [:anode]]

Taking.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Is this intended to replace the existing reftest manifests in the tree?  If so, why?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

dbaron: yes, this is intended to replace the existing reftest manifests.  Why: basically we have run into scenarios with reftests where we would like more control (such as setting a pref for a single test) and the manifests are starting to become more hacked up.  

As derived from the initial description above, we found the reftest manifests to be a role model for developing a manifest solution for mochitest and xpcshell.  Looking at the picture from a whole it seems logical to have a shared format and parsing code for all tests.  

Now have we figured out the best solution, not yet.  One thing we are very specific about is continuing to have the ability to easily edit a text file without adding a bunch of extra confusion in there.  Also having the ability to extend the metadata and conditions around a specific test case or set of tests holds value.

Why would we do this to reftest first?  There is already a manifest solution and we don't have to change as much code.

Please voice any concerns or future plans to extend/enhance the manifest that might be related.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'd like to keep our reftests as something that we can realistically share with other browser makers and standards organizations.  And we have in fact done this this to varying degrees with the CSS and SVG working groups.

I've argued in the past that reftests that need specific preferences set or things like that should be tested using mochitest; we have the ability to do reftest-ish tests in mochitests quite easily.

I think setting preferences falls into the category of things that we should let people write script to do rather than coming up with a declarative format to do that (and then the next thing, and then the next thing).

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

A little followup here:  after my previous comment I met with a bunch of people in the test tools group, and they explained some of the use cases for this project.  I agreed that some of the use cases (particularly setting a preference for a large group of tests) seem valuable, so that the plan sounded acceptable, but that Alice should try to get additional feedback from people who use the reftest manifests regularly.

Most of that additional feedback on her blog seems to be negative, which has led me to conclude that using JSON isn't the right thing here.  We want a format that's easy for humans to read and write, and lots of brace and bracket matching doesn't fit that bill.  I think we can stick to the current line-based format and still add the features that are needed.  (And I'd like to note that of all our test suites, reftests are likely to have the most complicated lines, since they're the only ones where each file is a single test and therefore the only ones where known-failure conditions can live outside the test.)

Comment 6

[alice nodelman [:alice] [:anode]]

Moving over the jhammel - this hasn't been my project in some time.

Comment 7

[Jeff Hammel]

Not working on this at the moment, though we should revisit