1911742 - 22.47% decision () regression on Mon July 29 2024

Status: NEW

(Testing :: Performance, defect, P5)

Description

[Treeherder Bug Filer]

Perfherder has detected a build_metrics performance regression from push e17d8ab7474d5f73f30c33d6e2cb822e49a6d853. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
22%	decision	gecko-decision		153.05 -> 187.44

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

You can run these tests on try with ./mach try perf --alert 1583

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Bilt]

Hello taskcluster team
Just wanted to check if this regression alerted on the decision test is of concern for the taskcluster team.

Comment 2

[Andrew Halberstadt [:ahal]]

How long has this alert been live and how many data points were used to determine it's a regression? I'd expect Decision tasks to take substantially longer on merge pushes with lots of commits like this one. If the alert is comparing against a push with very few commits, I'd say this is expected and we should somehow update the alerts to account for this.

If we already are accounting for this, the regression is possibly related to processing reftest manifests as part of test loading now? Though I didn't notice that in the changelog. There are some Taskgraph updates and other things that could be the culprit.. But I'm not really sure.

Comment 3

[Greg Mierzwinski [:sparky]]

It uses 12 data points before/after a push to see if there was a regression/improvement. Looking at the graph for mozilla-release, I'm seeing a sustained regression in the data: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=mozilla-release,2780728,1,2&timerange=5184000&zoom=1721141510248,1723022587516,111.02438006867727,256.8783877731165

If you compare it to autoland data, I see that there was a jump on around July 16th - not sure which commit caused it though, but it may be related to this mozilla-release alert: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=mozilla-release,2780728,1,2&series=autoland,2680290,1,2&timerange=5184000&zoom=1720766733309,1723058440393,86.37045146460405,260.31925849127094

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

This bug has been marked as a regression. Setting status flag for Nightly to affected.