Closed Bug 1260926 Opened 4 years ago Closed 4 years ago

Perfherder mixes in talos-aws results

Categories

(Tree Management :: Perfherder, defect)

Product:
Tree Management
Component:
Perfherder
Type:
defect
Priority:
Not set

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: BenWa, Assigned: wlach)

References

Details

Attachments

(2 files)

[treeherder] wlach:1260926 > mozilla:master
47 bytes, text/x-github-pull-request
jmaher
: review+
Details | Review
[treeherder] wlach:1260926-2 > mozilla:master
47 bytes, text/x-github-pull-request
Details | Review 
Example:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=7bcc12d3791c&newProject=try&newRevision=88619ff4054e&framework=1&filter=TART&showOnlyImportant=0

The linux results include the native and the AWS results. The AWS are significantly higher and probably shouldn't be averaged together.
This is due to the test in bug 1253341. Not sure why compare view is mashing stuff together, I will investigate. Meanwhile, I've requested that :catlee turn off the AWS jobs for now.
right now these are different frameworks- we could consider going with a different platform name- that is a bit more work on the treeherder side.  I imagine a few more experiments we will be running over time, so supporting this functionality seems like the right way to go.
Ok, did some digging and figured out that perfherder doesn't handle the case where the same signature occurs with different frameworks. I'll fix this. :)
Assignee: nobody → wlachance
Component: Talos → Perfherder
Product: Testing → Tree Management
Version: Version 3 → ---
Duplicate of this bug: 1258403
thanks for looking at this Will.  Let me know if you need more information, testing, reviews.
Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master

Actually, yes, could you review this? :P
Attachment #8737329 - Flags: review?(jmaher)
Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master

overall this is nice.  I assume that we won't have confusing double test name/signatures in the graph selection?
Attachment #8737329 - Flags: review?(jmaher) → review+
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/085a71ea0395d9fa2660ef0c4f19d856509f57a1
Bug 1260926 - Properly differentiate performance framework for signatures

* Handle case where we have two signatures with the same hash, same project,
  but different framework
* Allow filtering on framework in the signatures endpoint

https://github.com/mozilla/treeherder/commit/880896b9aad4fb0861663feac2850b7a40d0676c
Bug 1260926 - Don't mix results from different frameworks in perf comparisons

We usually handled this correctly, but there was an edge case that didn't
work properly (same signature hash, different frameworks)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
(In reply to Joel Maher (:jmaher) from comment #8)
> overall this is nice.  I assume that we won't have confusing double test
> name/signatures in the graph selection?

Not quite, doing proper filtering in the test chooser is bug 1230652.
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/899ad743ef4b8db1c94421c20fddb187ca1c6b28
Bug 1260926 - Fix etl code for perf signatures to handle framework properly

There was one more case where we didn't handle multiple signatures correctly

https://github.com/mozilla/treeherder/commit/9411b39ff3b48d38aec70e23a1f4fe69e16374d3
Merge pull request #1397 from wlach/1260926-2

Bug 1260926 - Fix etl code for perf signatures to handle framework properly
You need to log in before you can comment on or make changes to this bug.