797186 - Have a central consumable location for what tests are run on talos

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Jeff Hammel]

Talos tests suites are defined in buildbot which then invokes talos
with various options per suite to run any number of talos tests.
These are reported to TBPL which consumes the buildbot names and the
talos test names.  Graphserver also has names not only for the talos
tests, but all the counters we define as well.  I have written about
this in more detail here: http://k0s.org/mozilla/blog/20120724135349

The names of the various pieces sans the counters are illustrated
here:  http://k0s.org:8080/

I would like to move to a central location for the naming of talos
tests and suites.  Currently talos tests live in
http://hg.mozilla.org/build/talos/file/tip/talos/test.py . The main
reason that tests are defined as python objects is for inheritence.
This tells something about a lack of rigor in how talos tests are defined.

To begin to unravel how things look now, there are (basically) two kinds of talos
tests: pageload tests and startup tests.

startup tests are defined by:
- a URL of the page to load
- a URL timestamp (eww!)

pageloader tests are defined by:
- a manifest of pages to load
- whether to run with chrome
- whether to measure at mozafterpaint or on page load

There is actually a further division in pageloader tests: some tests
record their own data and just use the pageloader extension.  In this
case, even though we record it in the configuration.

The rest of these "test definitions" are really how the tests are run:

- cycles, tpcycles, tppagecycles
- timeout
- profile_path
- pretty much everything else concerns what counters (auxiliary data)
  to gather

IMHO, these should not be in the test definitions at all.  These
should live in how the test is invoked, which is currently in
buildbot-configs (etc?) though I would recommend putting this in m-c,
in testing/talos/talos.json . This will allow the actual
parameterization of talos testing to follow the tree.
Mozharness may provide for this.

The situation in graphserver is much much worse:
https://hg.mozilla.org/graphs/file/tip/sql/data.sql
If we had a central place where graphserver could look for what could
be run, this could mostly amount to generated SQL.

Longer term, I would propose the following:  Let's make all talos
tests the same.  For pageloader tests with a manifest, this
essentially amounts to passing the manifest URL as a URL parameter.
Pageloader should probably also be broken up into 2 extensions: one is
pageloader as a framework that uses no/chrome and mozafterpaint and
which builds upon a more minimal pageloader for tests that take their
own measurements.

See also: https://bugzilla.mozilla.org/show_bug.cgi?id=770460

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we have found that the wiki and talos.json are good enough.  with perfherder making test selection easy, I don't think there is a big need for something else.