Closed Bug 1354296 Opened 3 years ago Closed 3 years ago

Integrate the new talos perf-reftest suite into taskcluster/buildbot CI

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla55

People

(Reporter: rwood, Assigned: rwood)

References

Details

Attachments

(2 files, 1 obsolete file)

We are working on a new Talos test suite called 'perf-reftest', a part of the efforts for stylo. Once the tests are ready (Bug 1353900) we need to integrate the new test suite into taskcluster/buildbot/treeherder.

The new 'perf-reftest' suite needs to run on all platforms along with the other Talos tests, and a new test job symbol created in the treeherder Talos groups ("T", and "T-e10s").

From :jmaher:

for taskcluster, add the definition here:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/tests.yml#1274

and the test set:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#66


This doesn't handle the buildbotbridge work, but thatis sort of automatic.

for buildbot-configs, add regular and e10s entries here:
https://hg.mozilla.org/build/buildbot-configs/file/c6ef67e55f4c/mozilla-tests/config.py#l197
Assignee: nobody → rwood
Status: NEW → ASSIGNED
we should ensure the buildbot stuff is landed prior to landing the tc stuff, otherwise we will have zombie entries in the BBB queue.
(In reply to Joel Maher ( :jmaher) from comment #2)
> we should ensure the buildbot stuff is landed prior to landing the tc stuff,
> otherwise we will have zombie entries in the BBB queue.

Yeah good call, working on the buildbot patch now
Comment on attachment 8857941 [details]
Bug 1354296 - Add TC configs for new talos perf-reftest;

https://reviewboard.mozilla.org/r/129968/#review132636
Attachment #8857941 - Flags: review?(wcosta) → review+
Attached patch bug1354296.patch (obsolete) — Splinter Review
I couldn't get the list_builder_differences.sh script to work on my mac osx, so am unable to upload the list of builder changes that this patch creates.
Attachment #8858083 - Flags: review?(kmoir)
Comment on attachment 8858083 [details] [diff] [review]
bug1354296.patch

So I did the builder diff on my master and noticed it enables the tests on beta. Is that okay or is there something that needs to ride the trains to enable this new suite?
(In reply to Kim Moir [:kmoir] from comment #6)

> So I did the builder diff on my master and noticed it enables the tests on
> beta. Is that okay or is there something that needs to ride the trains to
> enable this new suite?

Good point, thanks Kim. The new test itself just landed in central last week in Bug 1353900:

https://hg.mozilla.org/mozilla-central/rev/ad206b7a2644

Is there any way that it can just be enabled on try/inbound/central?
Flags: needinfo?(kmoir)
You need to do something like this and set 'enable_by_default': False, for this new test in buildbot

 'perf-reftest': {       
        'enable_by_default': False,
        'suites': GRAPH_CONFIG + ['--activeTests', 'bloom_basic:bloom_basic_ref', '--filter', 'ignore_first:5', '--filter', 'median'],
        'options': ({}, ALL_TALOS_PLATFORMS),
    },
    'perf-reftest-e10s': {      
        'enable_by_default': False,
        'suites': GRAPH_CONFIG + ['--activeTests', 'bloom_basic:bloom_basic_ref', '--filter', 'ignore_first:5', '--filter', 'median'],
        'options': ({}, ALL_TALOS_PLATFORMS),
    },

And then enable it on the branches running the version of gecko you specify, similar to this
https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l2698

In your taskcluster patch, it looks like you have enabled them to run on try 

 run-on-projects:
        by-test-platform:
            linux64-stylo/.*: ['try']
Flags: needinfo?(kmoir)
Attachment #8858083 - Attachment is obsolete: true
Attachment #8858083 - Flags: review?(kmoir)
Thanks Kim!
Attachment #8858892 - Flags: review?(kmoir)
Attachment #8858892 - Flags: review?(kmoir) → review+
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a649627e9ad1
Add TC configs for new talos perf-reftest; r=wcosta
Looks like the buildbot config changes are causing some kind of confusion on central, inbound, and autoland treeherder - on Win7/8 and OSX. There is a new "Reftest" section on treeherder (symbol "R" and "R-e10s" with the new talos bloom-basic tests: 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=bb38d935d699e0529f9e0bb35578d381026415c4&filter-searchStr=talos

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a649627e9ad1363e444d80279426f31dc4a0971a&filter-searchStr=talos

:kmoir, any idea what is happening? Maybe the name 'perf-reftest' is being confused for 'reftest' somewhere? Looks like this started happening *before* I landed the TC config part.
Flags: needinfo?(kmoir)
I suspect this is a treeherder magic parsing issue.

I see the buildername:
 Rev7 MacOSX Yosemite 10.10.5 autoland talos perf-reftest

and a talos buildername looks like:
 Rev7 MacOSX Yosemite 10.10.5 autoland talos chromez

:emorley, any thoughts on name matching here?
Flags: needinfo?(emorley)
Treeherder doesn't parse Taskcluster jobs submitted via Pulse, they set their own names/groups.
Flags: needinfo?(emorley)
For the buildbot jobs, it is probably parsing the name and defaulting to reftest see

https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py#L543

You will need a patch to fix this and update the tests, this is an example I did recently

https://github.com/mozilla/treeherder/commit/2f7d8cda765889a31e0e05eab5415f80aaf8cca7
Flags: needinfo?(kmoir)
https://hg.mozilla.org/mozilla-central/rev/a649627e9ad1
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Depends on: 1357790
Reopen - the perf-refest is not running on try on linux at all. Get it running on try, inbound, autoland, and central. NOT beta and release yet as the test addition hasn't ridden the trains yet.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Correction - it is running on Linux x64 Stylo opt as expected. On regular Linux x64 opt it doesn't run by default for some reason but it can be added afterwards via 'add new jobs'.
MozReview needs a new bug instead of reopening this one, opened Bug 1359145 for this follow-up work.
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.