1354296 - Integrate the new talos perf-reftest suite into taskcluster/buildbot CI

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Robert Wood [:rwood]]

We are working on a new Talos test suite called 'perf-reftest', a part of the efforts for stylo. Once the tests are ready (Bug 1353900) we need to integrate the new test suite into taskcluster/buildbot/treeherder.

The new 'perf-reftest' suite needs to run on all platforms along with the other Talos tests, and a new test job symbol created in the treeherder Talos groups ("T", and "T-e10s").

From :jmaher:

for taskcluster, add the definition here:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/tests.yml#1274

and the test set:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#66


This doesn't handle the buildbotbridge work, but thatis sort of automatic.

for buildbot-configs, add regular and e10s entries here:
https://hg.mozilla.org/build/buildbot-configs/file/c6ef67e55f4c/mozilla-tests/config.py#l197

Comment 1

[Robert Wood [:rwood]]

Attachment: Bug 1354296 - Add TC configs for new talos perf-reftest;

Review commit: https://reviewboard.mozilla.org/r/129968/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/129968/

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we should ensure the buildbot stuff is landed prior to landing the tc stuff, otherwise we will have zombie entries in the BBB queue.

Comment 3

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) from comment #2)
> we should ensure the buildbot stuff is landed prior to landing the tc stuff,
> otherwise we will have zombie entries in the BBB queue.

Yeah good call, working on the buildbot patch now

Comment 4

[Wander Lairson Costa]

Comment on attachment 8857941 [details]
Bug 1354296 - Add TC configs for new talos perf-reftest;

https://reviewboard.mozilla.org/r/129968/#review132636

Comment 5

[Robert Wood [:rwood]]

Attachment: bug1354296.patch

I couldn't get the list_builder_differences.sh script to work on my mac osx, so am unable to upload the list of builder changes that this patch creates.

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8858083 [details] [diff] [review]
bug1354296.patch

So I did the builder diff on my master and noticed it enables the tests on beta. Is that okay or is there something that needs to ride the trains to enable this new suite?

Comment 7

[Robert Wood [:rwood]]

(In reply to Kim Moir [:kmoir] from comment #6)

> So I did the builder diff on my master and noticed it enables the tests on
> beta. Is that okay or is there something that needs to ride the trains to
> enable this new suite?

Good point, thanks Kim. The new test itself just landed in central last week in Bug 1353900:

https://hg.mozilla.org/mozilla-central/rev/ad206b7a2644

Is there any way that it can just be enabled on try/inbound/central?

Comment 8

[Kim Moir [:kmoir] ET]

You need to do something like this and set 'enable_by_default': False, for this new test in buildbot

 'perf-reftest': {       
        'enable_by_default': False,
        'suites': GRAPH_CONFIG + ['--activeTests', 'bloom_basic:bloom_basic_ref', '--filter', 'ignore_first:5', '--filter', 'median'],
        'options': ({}, ALL_TALOS_PLATFORMS),
    },
    'perf-reftest-e10s': {      
        'enable_by_default': False,
        'suites': GRAPH_CONFIG + ['--activeTests', 'bloom_basic:bloom_basic_ref', '--filter', 'ignore_first:5', '--filter', 'median'],
        'options': ({}, ALL_TALOS_PLATFORMS),
    },

And then enable it on the branches running the version of gecko you specify, similar to this
https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l2698

In your taskcluster patch, it looks like you have enabled them to run on try 

 run-on-projects:
        by-test-platform:
            linux64-stylo/.*: ['try']

Comment 9

[Robert Wood [:rwood]]

Attachment: bug1354296-two.patch

Thanks Kim!

Comment 10

[Robert Wood [:rwood]]

Landed the buildbot configs:

https://hg.mozilla.org/build/buildbot-configs/rev/7d082a3222c2492cd90f763dcb17187f4c409655

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a649627e9ad1
Add TC configs for new talos perf-reftest; r=wcosta

Comment 12

[Robert Wood [:rwood]]

Looks like the buildbot config changes are causing some kind of confusion on central, inbound, and autoland treeherder - on Win7/8 and OSX. There is a new "Reftest" section on treeherder (symbol "R" and "R-e10s" with the new talos bloom-basic tests: 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=bb38d935d699e0529f9e0bb35578d381026415c4&filter-searchStr=talos

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a649627e9ad1363e444d80279426f31dc4a0971a&filter-searchStr=talos

:kmoir, any idea what is happening? Maybe the name 'perf-reftest' is being confused for 'reftest' somewhere? Looks like this started happening *before* I landed the TC config part.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I suspect this is a treeherder magic parsing issue.

I see the buildername:
 Rev7 MacOSX Yosemite 10.10.5 autoland talos perf-reftest

and a talos buildername looks like:
 Rev7 MacOSX Yosemite 10.10.5 autoland talos chromez

:emorley, any thoughts on name matching here?

Comment 14

[Ed Morley [:emorley]]

Treeherder doesn't parse Taskcluster jobs submitted via Pulse, they set their own names/groups.

Comment 15

[Kim Moir [:kmoir] ET]

For the buildbot jobs, it is probably parsing the name and defaulting to reftest see

https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py#L543

You will need a patch to fix this and update the tests, this is an example I did recently

https://github.com/mozilla/treeherder/commit/2f7d8cda765889a31e0e05eab5415f80aaf8cca7

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/a649627e9ad1

Comment 17

[Robert Wood [:rwood]]

Reopen - the perf-refest is not running on try on linux at all. Get it running on try, inbound, autoland, and central. NOT beta and release yet as the test addition hasn't ridden the trains yet.

Comment 18

[Robert Wood [:rwood]]

Correction - it is running on Linux x64 Stylo opt as expected. On regular Linux x64 opt it doesn't run by default for some reason but it can be added afterwards via 'add new jobs'.

Comment 19

[Robert Wood [:rwood]]

MozReview needs a new bug instead of reopening this one, opened Bug 1359145 for this follow-up work.