Closed
Bug 1466525
Opened 7 years ago
Closed 3 years ago
Can't compare try pushes using `mozharness: --geckoProfile`
Categories
(Testing :: Talos, defect, P3)
Testing
Talos
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: ochameau, Unassigned)
References
Details
(Whiteboard: [PI:June])
When pushing to try with `mozharness: --geckoProfile`, DAMP no longer uploads `perfherder-data.json` artifact, nor generates PERFHERDER_DATA in its output. Thus we can no longer compare builds that are profiled.
This isn't handy as it doesn't allow to know if a regression that we are tracking without the profiler, also reproduces with the profiler.
Here is two example try pushes:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b5b8629a272b5b763f5a9c8b7cafacc5bf4c841&selectedJob=181693848
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8e543e21997254f9cedf6f86145a403c39839dc
Comment 1•7 years ago
|
||
this is by design- a profiled build will produce misleading data and can skew our results for alerts and trends. Why are you trying to compare results with --geckoProfile?
Reporter | ||
Comment 2•7 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
> this is by design- a profiled build will produce misleading data and can
> skew our results for alerts and trends. Why are you trying to compare
> results with --geckoProfile?
I totally agree that the data would be misleading, especially when comparing with a non-profiled one.
But it could also help ensuring that a particular regression also reproduces with the profiler turned on.
Honza originaly reported that to me when looking at a hard to debug regression on DAMP, on a "settle" subtest regression that involved spurious asynchronous code overlapping between two subtests.
Comment 3•7 years ago
|
||
maybe for try runs we could allow it to upload data with profile on
Whiteboard: [PI:June]
Comment 4•7 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> maybe for try runs we could allow it to upload data with profile on
Yes, this would help.
The perf/damp issue I am debugging is only reproducible on Try and I wanted to
also get profile data to analyze it using perf.html
Honza
Comment 5•6 years ago
|
||
Here's the logic that prevents the Perfherder data from being logged when --geckoProfile
is specified: https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/testing/talos/talos/output.py#210-214
We would likely need to move that logic into the mozharness code to allow us to output the results but prevent us from submitting the results to Treeherder.
Priority: -- → P3
Comment 6•3 years ago
|
||
:ochameau, this should be resolved now: https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#221-229
Feel free to re-open if this is still an issue.
Updated•3 years ago
|
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•