740512 - Simplify modifying talos suites

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Armen [:armenzg]]

I would like to manage through talos.json (or similar) which parameters get passed for each talos suite.

In other words match a test builder to an entry with info on talos.json.

This will allow us to:
* test on the try server new test suites or different parameters
** hands-off for the a-team
* no special casing for esr10/m-a/m-b/m-r (remove comments on merge days)

We can have additional "builder buckets" for try/mozilla-central when we need to test side by side.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is a great idea!

Comment 2

[Jeff Hammel]

Any idea how the resulting talos.json would look? (rough sketch)

Comment 3

[Armen [:armenzg]]

{
    "talos.zip": {
        "url": "http://build.mozilla.org/talos/zips/talos.bug739768.eb99eb78e42c.zip",
        "path": ""
    }
    "suites": {
         "tprow": "['--activeTests', 'tp5row', '--mozAfterPaint', '--responsiveness', '--filter', 'ignore_first:5', '--filter', 'median', '--sampleConfig', 'sample.2.config']",
         ...
         "remote-ts": "['--activeTests', 'ts', '--mozAfterPaint', '--noChrome']",
         ...
    }
}

With the current buildbot setup we could have suites in talos.json not defined or obsoleted in talos.json.
We could add a test script to verify that things are not falling apart.

What do you think?

Comment 4

[Jeff Hammel]

This looks good to me.  I assume the keys in the suites hash is the buildbot name of things?

Comment 5

[Jeff Hammel]

Assuming this applies to all platforms

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I would say we should document the possible suites: tp5row, remote-ts, chrome, nochrome, etc...

Here is what I see needing to do:
* adjust an existing test by adding a parameter
** tp5row would be adding ['--filter', 'ignore_first:5', '--filter', 'median'] to tp5r
* add a test for side by side testing
** here I would like to add tsvgrow 
** I don't think this will work
* turn on/off a test
** I don't think this will work
* create a new test altogether
** I need to currently add 'tprovider' for a remote test.
** I don't think this will work


Overall the idea is great, I just want to solve some problems.  I could fiddle with stuff and cheat:

* add a new test 'tprovider':
** trobopan would change ['--activeTests', 'trobopan'] to ['--activeTests', 'tprovider']

This would allow me to run tests on try, but not a solution to get into production

Another way is to:
** trobopan would change ['--activeTests', 'trobopan'] to ['--activeTests', 'trobopan:tprovider']

This gives us a working solution out of the box, but probably a really long run time, not ideal.


In terms of solving on/off for branches, I don't think the proposed solution works.  I would like something like:
    "suites": {
         "tprow": {"enabled": true, "tp_options": ['--activeTests', 'tp5row', '--mozAfterPaint', '--responsiveness', '--filter', 'ignore_first:5', '--filter', 'median', '--sampleConfig', 'sample.2.config']},
         ...
         "remote-ts": {"enabled": false, "tp_options": ['--activeTests', 'ts', '--mozAfterPaint', '--noChrome']}
    ...
    }

Comment 7

[[e-mail] (:bhearsum)]

Moving to Taskcluster makes this pretty easy to do in tree. Not going to fix it for Buildbot.

Comment 8

[Armen [:armenzg]]

This already got fixed somewhere.

Talos.json contains all of this information:
https://hg.mozilla.org/integration/mozilla-inbound/file/default/testing/talos/talos.json#l13