785936 - Find an owner and maintain compare-talos

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Jeff Hammel]

As far as everyone knows, compare-talos is unmaintained:

http://perf.snarkfest.net/compare-talos/

The code should be moved to the talos repo or consume the talos repo
as that is the canonical and up-to-date source of the tests.

(This could conceivably also be mangled with talos-names, as there is
overlap here in demangling the names).

Alternatively, if
http://hg.mozilla.org/build/talos/file/6d79047595a4/compare.py is
sufficient, we could deprecate compare-talos and/or make a web client
around this.

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I am seeing as similar issue with compare.py. Looks like the tests we run have changed and it needs to be updated.

Given that it is a local script, updating it is probably easier that updating compare talus, so my vote would go for deprecating compare talus.

compare.py is also missing a feature: It should be possible to explicitly specify a old revision, right now one can only specify the branch.

Comment 2

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Sorry, there is one more feature missing from compare.py: doing statistics to find out if a change is significant or not.

Compare talos can do that when there are multiple talos runs in a single revision or when it is given multilpe revisions.

Comment 3

[(no longer active)]

(In reply to comment #2)
> Sorry, there is one more feature missing from compare.py: doing statistics to
> find out if a change is significant or not.
> 
> Compare talos can do that when there are multiple talos runs in a single
> revision or when it is given multilpe revisions.

Yes.  In fact, that is the only reason why compare-talos exists and is useful!  If compare.py doesn't do that, then it doesn't fit the bill as a replacement.

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I am marking this as blocking 784691 since we need to know if it causes performance regressions or not.

Comment 5

[Jeff Hammel]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #1)
> I am seeing as similar issue with compare.py. Looks like the tests we run
> have changed and it needs to be updated.

IMHO, compare.py/compare-talos should use the canonical definition of the tests in talos/tests.py . We should always be up to date with what is available.  Obviously graphserver and buildbot (and to a lesser degree TBPL) make this even more fun, but I don't think we can punt on this.

Comment 6

[Phil Ringnalda (:philor)]

It's really not that horribly abandoned or blocky, just annoying. Ask in the right places, and you'll hear that mconnor reluctantly owns it despite wishing someone would take it away (tbpl was his previous hope, and it remains a reasonable one since even though the talos owners never do tell tbpl when they add a new test, they really really very much really need to), and that it's hosted on https://bitbucket.org/mconnor/compare-talos and that a simple clone will give you a perfectly workable local copy you can update with whatever suites you are missing while you wait for him to deal with your pull request.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

compare-talos is deprecated, we have a new version inside of perfherder.