Talos tests suites are defined in buildbot which then invokes talos with various options per suite to run any number of talos tests. These are reported to TBPL which consumes the buildbot names and the talos test names. Graphserver also has names not only for the talos tests, but all the counters we define as well. I have written about this in more detail here: http://k0s.org/mozilla/blog/20120724135349 The names of the various pieces sans the counters are illustrated here: http://k0s.org:8080/ I would like to move to a central location for the naming of talos tests and suites. Currently talos tests live in http://hg.mozilla.org/build/talos/file/tip/talos/test.py . The main reason that tests are defined as python objects is for inheritence. This tells something about a lack of rigor in how talos tests are defined. To begin to unravel how things look now, there are (basically) two kinds of talos tests: pageload tests and startup tests. startup tests are defined by: - a URL of the page to load - a URL timestamp (eww!) pageloader tests are defined by: - a manifest of pages to load - whether to run with chrome - whether to measure at mozafterpaint or on page load There is actually a further division in pageloader tests: some tests record their own data and just use the pageloader extension. In this case, even though we record it in the configuration. The rest of these "test definitions" are really how the tests are run: - cycles, tpcycles, tppagecycles - timeout - profile_path - pretty much everything else concerns what counters (auxiliary data) to gather IMHO, these should not be in the test definitions at all. These should live in how the test is invoked, which is currently in buildbot-configs (etc?) though I would recommend putting this in m-c, in testing/talos/talos.json . This will allow the actual parameterization of talos testing to follow the tree. Mozharness may provide for this. The situation in graphserver is much much worse: https://hg.mozilla.org/graphs/file/tip/sql/data.sql If we had a central place where graphserver could look for what could be run, this could mostly amount to generated SQL. Longer term, I would propose the following: Let's make all talos tests the same. For pageloader tests with a manifest, this essentially amounts to passing the manifest URL as a URL parameter. Pageloader should probably also be broken up into 2 extensions: one is pageloader as a framework that uses no/chrome and mozafterpaint and which builds upon a more minimal pageloader for tests that take their own measurements. See also: https://bugzilla.mozilla.org/show_bug.cgi?id=770460
we have found that the wiki and talos.json are good enough. with perfherder making test selection easy, I don't think there is a big need for something else.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.