1596808 - 4.61 - 6.32% raptor-tp6-bing-firefox-cold / raptor-tp6-bing-firefox-cold loadtime (macosx1014-64-shippable) regression on push 0f875399e986934acd11a4418a5b8270060325d3 (Tue November 5 2019)

Status: RESOLVED WONTFIX

(Testing :: General, defect, P1)

Description

[Marian Raiciof [:marauder]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=0f875399e986934acd11a4418a5b8270060325d3

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

6% raptor-tp6-bing-firefox-cold loadtime macosx1014-64-shippable opt 577.50 -> 614.00
5% raptor-tp6-bing-firefox-cold macosx1014-64-shippable opt 406.57 -> 425.29

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23723

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Marian Raiciof [:marauder]]

Hi Jon,

The recent Taskcluster deployment has had the impact of preventing triggering jobs on any commits prior to November 11th.
And because of that i can't perform retriggers to have a better picture.

Do you think this regression might be expected ?
Thanks!

Comment 2

[Jon Coppeard (:jonco)]

It's nothing to do with bug 1594061 as this code is not reachable from the browser at the moment.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

The linked regression from the push is a python 3 compat fix in the vendoring tools, so it seems completely unrelated? Not sure how bug 1594061 was brought into this?

Comment 4

[Marian Raiciof [:marauder]]

Looking at it now, it seems that https://bugzilla.mozilla.org/show_bug.cgi?id=1592754 caused the regression

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=4db479&highlightedRevisions=144a647f&selected=2134254,585672,103.53396907663293,81.31554242424038,966614148&series=autoland,2134254,1,10&series=mozilla-inbound,2136919,1,10&timerange=5184000&zoom=1572933815988,1573656231886,520.4666630309306,647.6840498417984

After 2 backouts it landed again:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=4db479&highlightedRevisions=144a647f&selected=2134254,587214,383.9445532855646,97.35108639460067,966912457&series=autoland,2134254,1,10&series=mozilla-inbound,2136919,1,10&timerange=5184000&zoom=1572933815988,1573656231886,520.4666630309306,647.6840498417984

The issue is that i can't retrigger the jobs to be sure what is going on.
"Error message: Wrong version of actions.json, unable to continue"

But looking at the graph for a period of 2 months we can see that on ~ Dec 2nd the graph returned to the previous state.
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=4db479&highlightedRevisions=144a647f&series=autoland,2134254,1,10&series=mozilla-inbound,2136919,1,10&timerange=5184000&zoom=1571926588000,1577096685000,450.22222222222223,745.7777777777778

What should we do next ?
Thanks!

Comment 5

[Gian-Carlo Pascutto [:gcp]]

Hi Geoff, can you confirm this perf change was due to testing infra changes?

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

No, sorry, I cannot. There should be no relation to bug 1592754: It might affect test-verify or test-coverage tasks, but not raptor, and makes no changes to Firefox.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:gbrown, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 8

[Geoff Brown [:gbrown] (pto Apr 11-18)]

:marauder -- What's the next step here?

Comment 9

[Marian Raiciof [:marauder]]

Because i am unable to retrigger or backfill, i can't do much here.

Looking at the graph for the last 2 - 3 months, it returned to previous state around Dec 4th.
On Jan 6th there are signs of improvements (although the graph looks bimodal) and it's similar to what we had before October 16th,
before the big changes on cold tests from https://bugzilla.mozilla.org/show_bug.cgi?id=1589070
where the big regression was expected.

I think we can mark it as won't fix since i can't retrigger/backfill and the suspect datapoints are not related to the regression.
What 's your opinion, Geoff ?
Thanks!

Comment 10

[Geoff Brown [:gbrown] (pto Apr 11-18)]

It sounds like won't fix is the best we can do.