4.61 - 6.32% raptor-tp6-bing-firefox-cold / raptor-tp6-bing-firefox-cold loadtime (macosx1014-64-shippable) regression on push 0f875399e986934acd11a4418a5b8270060325d3 (Tue November 5 2019)
Categories
(Testing :: General, defect, P1)
Tracking
(Not tracked)
People
(Reporter: marauder, Assigned: gbrown)
References
Details
(Keywords: perf, perf-alert, regression)
Raptor has detected a Firefox performance regression from push:
As author of one of the patches included in that push, we need your help to address this regression.
Regressions:
6% raptor-tp6-bing-firefox-cold loadtime macosx1014-64-shippable opt 577.50 -> 614.00
5% raptor-tp6-bing-firefox-cold macosx1014-64-shippable opt 406.57 -> 425.29
You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23723
On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.
To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor
*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***
Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling
Reporter | ||
Comment 1•5 years ago
|
||
Hi Jon,
The recent Taskcluster deployment has had the impact of preventing triggering jobs on any commits prior to November 11th.
And because of that i can't perform retriggers to have a better picture.
Do you think this regression might be expected ?
Thanks!
Comment 2•5 years ago
|
||
It's nothing to do with bug 1594061 as this code is not reachable from the browser at the moment.
Updated•5 years ago
|
Updated•4 years ago
|
Comment 3•4 years ago
|
||
The linked regression from the push is a python 3 compat fix in the vendoring tools, so it seems completely unrelated? Not sure how bug 1594061 was brought into this?
Reporter | ||
Comment 4•4 years ago
|
||
Looking at it now, it seems that https://bugzilla.mozilla.org/show_bug.cgi?id=1592754 caused the regression
The issue is that i can't retrigger the jobs to be sure what is going on.
"Error message: Wrong version of actions.json, unable to continue"
But looking at the graph for a period of 2 months we can see that on ~ Dec 2nd the graph returned to the previous state.
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=4db479&highlightedRevisions=144a647f&series=autoland,2134254,1,10&series=mozilla-inbound,2136919,1,10&timerange=5184000&zoom=1571926588000,1577096685000,450.22222222222223,745.7777777777778
What should we do next ?
Thanks!
Comment 5•4 years ago
|
||
Hi Geoff, can you confirm this perf change was due to testing infra changes?
Assignee | ||
Comment 6•4 years ago
|
||
No, sorry, I cannot. There should be no relation to bug 1592754: It might affect test-verify or test-coverage tasks, but not raptor, and makes no changes to Firefox.
Comment 7•4 years ago
|
||
The priority flag is not set for this bug.
:gbrown, could you have a look please?
For more information, please visit auto_nag documentation.
Assignee | ||
Comment 8•4 years ago
|
||
:marauder -- What's the next step here?
Reporter | ||
Comment 9•4 years ago
|
||
Because i am unable to retrigger or backfill, i can't do much here.
Looking at the graph for the last 2 - 3 months, it returned to previous state around Dec 4th.
On Jan 6th there are signs of improvements (although the graph looks bimodal) and it's similar to what we had before October 16th,
before the big changes on cold tests from https://bugzilla.mozilla.org/show_bug.cgi?id=1589070
where the big regression was expected.
I think we can mark it as won't fix since i can't retrigger/backfill and the suspect datapoints are not related to the regression.
What 's your opinion, Geoff ?
Thanks!
Assignee | ||
Comment 10•4 years ago
|
||
It sounds like won't fix is the best we can do.
Description
•