737581 - run peptest on m-c

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mark Côté [:mcote]]

We're collecting peptest results from try (currently in a temporary location at http://mrcote.info/peptest/), and we need to add mozilla-central for "real" results.

Given that peptest is a performance analyzer with quite a variance in results, we've changed our idea from a pass/fail scenario to something more like talos. We want the test to run on every push but go green regardless of results. We will collect results and do analyses over time, reporting big jumps to dev.tree-management or elsewhere. But while we're working on that, we want to collect results so we have data to work with.

Eventually we'd like the tests themselves to report to a central server, but for now we have a pulse listener set up that is gathering the results.

Comment 1

[Aki Sasaki (not active)]

Attachment: add peptest to m-c

As requested, this will turn on peptest for m-c.
I'm not sure if we want it on project branches, inbound, etc.; if this will go orange or red and will be a tree-closing metric we should probably turn it on everywhere applicable.  If not, this may be fine; we may want to hide it or not depending on signal/noise.

Comment 2

[Mark Côté [:mcote]]

Yeah inbound is probably a good idea too. I'll have to see if we want other branches as well.

Comment 3

[Aki Sasaki (not active)]

Attachment: add peptest to m-c, m-i

Comment 4

[Armen [:armenzg]]

Comment on attachment 608442 [details] [diff] [review]
add peptest to m-c, m-i

If ./tests-masters.sh does not complain about the # of builders fill free to land it.
Otherwise, you might want to wait a day or two for me to finish bug 737632.

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 608442 [details] [diff] [review]
add peptest to m-c, m-i

http://hg.mozilla.org/build/buildbot-configs/rev/f3d8f3976eec

Passes test-masters; running setup-master.py -t and watching preprod to be sure.

Comment 6

[Mark Côté [:mcote]]

Also, can you force this test to go green, regardless of results? This is, as I understand, how talos works, since the results will be analyzed externally. I can try to add an option to peptest to mangle logs so it doesn't appear to fail, but it's probably easier to do it on buildbot's side...

Comment 7

[Aki Sasaki (not active)]

Attachment: peptest perma-green

Handling in mozharness.
Rather than putting the perma-green hack at the end, I'm still going to detect harness/infrastructure failures.  But peptest test failures should go green with this patch.

Comment 8

[Mark Côté [:mcote]]

Comment on attachment 608473 [details] [diff] [review]
peptest perma-green

Perfect, thanks.

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 608473 [details] [diff] [review]
peptest perma-green

http://hg.mozilla.org/build/mozharness/rev/93c2b774fd38

Comment 10

[Armen [:armenzg]]

Live in production as of 7AM PDT.

Comment 11

[Aki Sasaki (not active)]

Attachment: hg cp win7_config.py w764_config.py

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 608743 [details] [diff] [review]
hg cp win7_config.py w764_config.py

http://hg.mozilla.org/build/mozharness/rev/5f44ba08f4be

Comment 13

[Aki Sasaki (not active)]

Looks to be fixed.