770744 - talos test definitions should be used more liberally and advantageously in talos runtime

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Jeff Hammel]

http://hg.mozilla.org/build/talos/file/195293d7404c/talos/test.py has
a lot of test definition.  However, these are currently used only as a
basis for configuration.  When it comes to actually running and
reporting the test results, we pass around a test_config dictionary
that is read (basically) from the configuration.

However, it may be to our advantage to offload more of this to the
test (and here I'm loosening the word a bit) "definition".  This could
make for more flexibility in how tests are done if we give more
flexibility into allowing tests to do different things.

Overall, I'm a bit skeptical of this as long-term architecture.
Configuration should be configuration and logic should be logic. While
this could (and IMHO should) be done in such a way as to separate the
configuration definition layer from any sort of "logic".  However, if
it helps us solve problems today, it could be a good exercise in
bootstrapping.  In any case, it feels like passing around test_config
in one place and using test (definition) classes in another place is a
bit wonky.

I realize this is a fairly vaguely written bug as I think we'll have
to find a real problem in order to make this practicable, but noting
for tracking.

Comment 1

[Jeff Hammel]

As an example of something where this does make a practical difference, take bug 767226 .

If we passed around full test objects vs a test config dictionary, we could do something like

test.getResults() 

to use the special v8 filter and get AVERAGE vs values from the test definition as well

Instead, we have to resort to the following magic:
* we report the scores for each benchmark
* we put special logic in filter.py that *assumes* that the order observed == the order in the manifest
* we add another check in reporting for the v8 test to report the average

All and all, this is fragile and will make the code kinda awful.  I thought about fixing this first, but really that will delay landing.

That said, we will have to do things like this again.  We already do enough awful; let's not make it worse

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

adding :parkouss on here as this might be a useful enhancement to the talos code base.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I think this has been around for many years and is probably out of scope given the current talos.