814228 - unification of talos tests

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect)

Description

[Jeff Hammel]

There are several different variations on how Talos runs tests:

- startup tests: these use a URL that actually corresponds to the test page; e.g. http://hg.mozilla.org/build/talos/file/75f0eacc80f9/talos/startup_test/startup_test.html .

- page load tests: the "URL" these use is actually the command-line invokation: http://hg.mozilla.org/build/talos/file/75f0eacc80f9/talos/run_tests.py#l162 . parameters, such as tpchrome and tpmanifest are passed on the command line.

- variation on the above: page load tests that accumulate their own data; a variation on the above; instead of using the pageloader infrastructure, these tests just gather their own damn data

- (and something about robocop tests)

It would be nice if we didn't have multiple cases.  Instead I would suggest the following:

- a test is just a URL, as startup tests are now

- for pageload tests, this URL can be an e.g. resource:// or chrome:// URL that takes parameters e.g. resource://pageloader/pageloader.js?manifest=file:///path/to/manifest.txt&tpchrome

- pageloader should be split into two extensions:
  1. the part of pageloader that just does generic stuff
  2. the part of pageloader that specifically deals with the typical "time until an event" stuff that builds off of 1.
  This let's 1. be a pretty lean extension to build off of for tests that calculate their own metrics

- for this system you'll want to pass in e.g. whatever extensions are used verbosely -- that is, do not include pageloader magically

This architecture significantly decouples the python runner from whatever test is being run.  Effectively, you have a contract: Talos will setup and start Firefox given a URL and it will expect the test to print its performance data to stdout in a specified form, namely this one: https://wiki.mozilla.org/Buildbot/Talos/DataFormat

Comment 1

[Jeff Hammel]

See also: bug 789488 , bug 795172

Comment 2

[Julien Pagès (:parkouss)]

This is an interesting point, and that goes in the way of the discussion we had yesterday on irc :jmaher. I really think that some standardization around the tests would be a hudge win, allowing us to "decouples the python runner from whatever test is being run". A nice approach for extensibility and maintainability.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:parkouss, we have done a lot to the test definitions over the last 6 months, is there any ideas you have to make it even better?  I think we have come far enough to close this bug out.

Comment 4

[Julien Pagès (:parkouss)]

Yeah, agreed let's close this bug.