Open Bug 1466525 Opened 2 years ago Updated 1 year ago

Can't compare try pushes using `mozharness: --geckoProfile`

Categories

(Testing :: Talos, defect, P3)

defect

Tracking

(Not tracked)

People

(Reporter: ochameau, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [PI:June])

When pushing to try with `mozharness: --geckoProfile`, DAMP no longer uploads `perfherder-data.json` artifact, nor generates PERFHERDER_DATA in its output. Thus we can no longer compare builds that are profiled.
This isn't handy as it doesn't allow to know if a regression that we are tracking without the profiler, also reproduces with the profiler.

Here is two example try pushes:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b5b8629a272b5b763f5a9c8b7cafacc5bf4c841&selectedJob=181693848

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8e543e21997254f9cedf6f86145a403c39839dc
this is by design- a profiled build will produce misleading data and can skew our results for alerts and trends.  Why are you trying to compare results with --geckoProfile?
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
> this is by design- a profiled build will produce misleading data and can
> skew our results for alerts and trends.  Why are you trying to compare
> results with --geckoProfile?

I totally agree that the data would be misleading, especially when comparing with a non-profiled one.
But it could also help ensuring that a particular regression also reproduces with the profiler turned on.

Honza originaly reported that to me when looking at a hard to debug regression on DAMP, on a "settle" subtest regression that involved spurious asynchronous code overlapping between two subtests.
maybe for try runs we could allow it to upload data with profile on
Whiteboard: [PI:June]
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> maybe for try runs we could allow it to upload data with profile on
Yes, this would help.

The perf/damp issue I am debugging is only reproducible on Try and I wanted to
also get profile data to analyze it using perf.html

Honza

Here's the logic that prevents the Perfherder data from being logged when --geckoProfile is specified: https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/testing/talos/talos/output.py#210-214

We would likely need to move that logic into the mozharness code to allow us to output the results but prevent us from submitting the results to Treeherder.

Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.