Simplify modifying talos suites

RESOLVED FIXED

Status

Release Engineering
General Automation
P4
normal
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: armenzg, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

I would like to manage through talos.json (or similar) which parameters get passed for each talos suite.

In other words match a test builder to an entry with info on talos.json.

This will allow us to:
* test on the try server new test suites or different parameters
** hands-off for the a-team
* no special casing for esr10/m-a/m-b/m-r (remove comments on merge days)

We can have additional "builder buckets" for try/mozilla-central when we need to test side by side.

Comment 2

6 years ago
Any idea how the resulting talos.json would look? (rough sketch)
{
    "talos.zip": {
        "url": "http://build.mozilla.org/talos/zips/talos.bug739768.eb99eb78e42c.zip",
        "path": ""
    }
    "suites": {
         "tprow": "['--activeTests', 'tp5row', '--mozAfterPaint', '--responsiveness', '--filter', 'ignore_first:5', '--filter', 'median', '--sampleConfig', 'sample.2.config']",
         ...
         "remote-ts": "['--activeTests', 'ts', '--mozAfterPaint', '--noChrome']",
         ...
    }
}

With the current buildbot setup we could have suites in talos.json not defined or obsoleted in talos.json.
We could add a test script to verify that things are not falling apart.

What do you think?

Comment 4

6 years ago
This looks good to me.  I assume the keys in the suites hash is the buildbot name of things?

Comment 5

6 years ago
Assuming this applies to all platforms
OS: Mac OS X → All
Hardware: x86 → All
I would say we should document the possible suites: tp5row, remote-ts, chrome, nochrome, etc...

Here is what I see needing to do:
* adjust an existing test by adding a parameter
** tp5row would be adding ['--filter', 'ignore_first:5', '--filter', 'median'] to tp5r
* add a test for side by side testing
** here I would like to add tsvgrow 
** I don't think this will work
* turn on/off a test
** I don't think this will work
* create a new test altogether
** I need to currently add 'tprovider' for a remote test.
** I don't think this will work


Overall the idea is great, I just want to solve some problems.  I could fiddle with stuff and cheat:

* add a new test 'tprovider':
** trobopan would change ['--activeTests', 'trobopan'] to ['--activeTests', 'tprovider']

This would allow me to run tests on try, but not a solution to get into production

Another way is to:
** trobopan would change ['--activeTests', 'trobopan'] to ['--activeTests', 'trobopan:tprovider']

This gives us a working solution out of the box, but probably a really long run time, not ideal.


In terms of solving on/off for branches, I don't think the proposed solution works.  I would like something like:
    "suites": {
         "tprow": {"enabled": true, "tp_options": ['--activeTests', 'tp5row', '--mozAfterPaint', '--responsiveness', '--filter', 'ignore_first:5', '--filter', 'median', '--sampleConfig', 'sample.2.config']},
         ...
         "remote-ts": {"enabled": false, "tp_options": ['--activeTests', 'ts', '--mozAfterPaint', '--noChrome']}
    ...
    }
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
Moving to Taskcluster makes this pretty easy to do in tree. Not going to fix it for Buildbot.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
This already got fixed somewhere.

Talos.json contains all of this information:
https://hg.mozilla.org/integration/mozilla-inbound/file/default/testing/talos/talos.json#l13
Resolution: WONTFIX → FIXED
You need to log in before you can comment on or make changes to this bug.