Closed
Bug 1260926
Opened 9 years ago
Closed 9 years ago
Perfherder mixes in talos-aws results
Categories
(Tree Management :: Perfherder, defect)
Tree Management
Perfherder
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: BenWa, Assigned: wlach)
References
Details
Attachments
(2 files)
Example:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=7bcc12d3791c&newProject=try&newRevision=88619ff4054e&framework=1&filter=TART&showOnlyImportant=0
The linux results include the native and the AWS results. The AWS are significantly higher and probably shouldn't be averaged together.
Assignee | ||
Comment 1•9 years ago
|
||
This is due to the test in bug 1253341. Not sure why compare view is mashing stuff together, I will investigate. Meanwhile, I've requested that :catlee turn off the AWS jobs for now.
Comment 2•9 years ago
|
||
right now these are different frameworks- we could consider going with a different platform name- that is a bit more work on the treeherder side. I imagine a few more experiments we will be running over time, so supporting this functionality seems like the right way to go.
Assignee | ||
Comment 3•9 years ago
|
||
Ok, did some digging and figured out that perfherder doesn't handle the case where the same signature occurs with different frameworks. I'll fix this. :)
Assignee: nobody → wlachance
Component: Talos → Perfherder
Product: Testing → Tree Management
Version: Version 3 → ---
Comment 4•9 years ago
|
||
Comment 6•9 years ago
|
||
thanks for looking at this Will. Let me know if you need more information, testing, reviews.
Assignee | ||
Comment 7•9 years ago
|
||
Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master
Actually, yes, could you review this? :P
Attachment #8737329 -
Flags: review?(jmaher)
Comment 8•9 years ago
|
||
Comment on attachment 8737329 [details] [review]
[treeherder] wlach:1260926 > mozilla:master
overall this is nice. I assume that we won't have confusing double test name/signatures in the graph selection?
Attachment #8737329 -
Flags: review?(jmaher) → review+
Comment 9•9 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/085a71ea0395d9fa2660ef0c4f19d856509f57a1
Bug 1260926 - Properly differentiate performance framework for signatures
* Handle case where we have two signatures with the same hash, same project,
but different framework
* Allow filtering on framework in the signatures endpoint
https://github.com/mozilla/treeherder/commit/880896b9aad4fb0861663feac2850b7a40d0676c
Bug 1260926 - Don't mix results from different frameworks in perf comparisons
We usually handled this correctly, but there was an edge case that didn't
work properly (same signature hash, different frameworks)
Assignee | ||
Updated•9 years ago
|
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 10•9 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #8)
> overall this is nice. I assume that we won't have confusing double test
> name/signatures in the graph selection?
Not quite, doing proper filtering in the test chooser is bug 1230652.
Comment 11•9 years ago
|
||
Comment 12•9 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/899ad743ef4b8db1c94421c20fddb187ca1c6b28
Bug 1260926 - Fix etl code for perf signatures to handle framework properly
There was one more case where we didn't handle multiple signatures correctly
https://github.com/mozilla/treeherder/commit/9411b39ff3b48d38aec70e23a1f4fe69e16374d3
Merge pull request #1397 from wlach/1260926-2
Bug 1260926 - Fix etl code for perf signatures to handle framework properly
You need to log in
before you can comment on or make changes to this bug.
Description
•