458540 - Create a script to find performance regressions through bisection

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Mauricio Collares [:collares]]

Bug 458065 shows that even having extensive performance tests, some checkins will cause regressions and lots of hassle in the future sometimes. An automated script which automatically binary searches for regressions above a certain (absolute or relative) threshold could avoid lots of needless backouts.

(This doesn't really block 458065, I'm just adding it because it is related to it.)

Comment 1

[Chris Cooper [:coop] (he/him)]

I might be misremembering, but doesn't Jesse already have a script for this? Not for performance per se, but a script that performs the binary search part at least.

Comment 2

[Jesse Ruderman]

My script (Lithium) is for reducing testcases, not narrowing regression ranges.

It should be easy to build a script around |hg bisect| to do this.  The toughest part will probably be dealing with noise in performance data.

Comment 3

[(not currently active) Ted Mielczarek]

db48x has a regression-search script here: http://db48x.net/regression-search/

It knows how to binary search over nightlies and hourlies, but it doesn't know how to hg bisect. (That would be a bit harder, only because it doesn't currently know how to build the source, just run binaries.)

I had a buildbot setup to run regression-search with a reftest or mochitest, but it was running on a VM and I never kept it up long enough to be useful. I think teaching regression-search how to bisect, and teaching it how to run perf tests would be awesome.

Comment 4

[(not currently active) Ted Mielczarek]

This is a good idea, but I don't think RelEng should be on the hook for doing it. I don't think there's any good component for this, in fact, so I'm just going to WONTFIX it. I'd love to see someone actually come up with a working prototype though.