1260926 - Perfherder mixes in talos-aws results

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[Benoit Girard (:BenWa)]

Example:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=7bcc12d3791c&newProject=try&newRevision=88619ff4054e&framework=1&filter=TART&showOnlyImportant=0

The linux results include the native and the AWS results. The AWS are significantly higher and probably shouldn't be averaged together.

Comment 1

[William Lachance (:wlach) (use needinfo!)]

This is due to the test in bug 1253341. Not sure why compare view is mashing stuff together, I will investigate. Meanwhile, I've requested that :catlee turn off the AWS jobs for now.

Comment 2

[Joel Maher ( :jmaher ) (UTC-4)]

right now these are different frameworks- we could consider going with a different platform name- that is a bit more work on the treeherder side.  I imagine a few more experiments we will be running over time, so supporting this functionality seems like the right way to go.

Comment 3

[William Lachance (:wlach) (use needinfo!)]

Ok, did some digging and figured out that perfherder doesn't handle the case where the same signature occurs with different frameworks. I'll fix this. :)

Comment 4

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1260926 > mozilla:master

Comment 6

[Joel Maher ( :jmaher ) (UTC-4)]

thanks for looking at this Will.  Let me know if you need more information, testing, reviews.

Comment 7

[William Lachance (:wlach) (use needinfo!)]

Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master

Actually, yes, could you review this? :P

Comment 8

[Joel Maher ( :jmaher ) (UTC-4)]

Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master

overall this is nice.  I assume that we won't have confusing double test name/signatures in the graph selection?

Comment 9

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/085a71ea0395d9fa2660ef0c4f19d856509f57a1
Bug 1260926 - Properly differentiate performance framework for signatures

* Handle case where we have two signatures with the same hash, same project,
  but different framework
* Allow filtering on framework in the signatures endpoint

https://github.com/mozilla/treeherder/commit/880896b9aad4fb0861663feac2850b7a40d0676c
Bug 1260926 - Don't mix results from different frameworks in perf comparisons

We usually handled this correctly, but there was an edge case that didn't
work properly (same signature hash, different frameworks)

Comment 10

[William Lachance (:wlach) (use needinfo!)]

(In reply to Joel Maher (:jmaher) from comment #8)
> overall this is nice.  I assume that we won't have confusing double test
> name/signatures in the graph selection?

Not quite, doing proper filtering in the test chooser is bug 1230652.

Comment 11

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1260926-2 > mozilla:master

Comment 12

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/899ad743ef4b8db1c94421c20fddb187ca1c6b28
Bug 1260926 - Fix etl code for perf signatures to handle framework properly

There was one more case where we didn't handle multiple signatures correctly

https://github.com/mozilla/treeherder/commit/9411b39ff3b48d38aec70e23a1f4fe69e16374d3
Merge pull request #1397 from wlach/1260926-2

Bug 1260926 - Fix etl code for perf signatures to handle framework properly