1446336 - 4.51% tsvg_static (windows7-32) regression on push 2979f3dd583a (Thu Mar 15 2018)

Status: RESOLVED WONTFIX

(WebExtensions :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=7585e7acb34b63d21504aefb24a4ba43f90fa169&tochange=2979f3dd583aa9891d4e2db396647e0520bbe5e2

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  5%  tsvg_static windows7-32 pgo e10s stylo     51.62 -> 53.95

Improvements:

 12%  tp5o responsiveness windows10-64 opt e10s stylo     2.04 -> 1.78
 11%  tsvg_static windows10-64 opt e10s stylo             57.09 -> 50.83


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=12171

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:kmag Bug 1445780 produce a perf regression on Windows PGO builds.
The OPT builds don't seem to be affected, but there's some strange & high noise on them.

Please look over addressing this issue.

Comment 2

[Kris Maglione [:kmag]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> :kmag Bug 1445780 produce a perf regression on Windows PGO builds.

That's not possible. Those were test-only changes that aren't even included in talos runs.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Ionut, any chance you can take another look at the regression range?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)
> Ionut, any chance you can take another look at the regression range?

Yes, I retriggered the tests on Windows 7. Other explanation would be infra changes, if the new results are higher.
But if the trend holds, then I don't have any other explanations.

:jmaher Thougts?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

the regressions seem to be pgo only, but look to be valid for what Ionut identified- maybe this is a build specific issue or a random artifact of how the PGO process works.  maybe worksforme/wontfix this?

Comment 6

[Kris Maglione [:kmag]]

The pushlog in the alert also contains bug 1445551, which makes more sense. In which case, I'd say that this is either noise or an artifact of the PGO build process. That bug sped a lot of things up, and may have just changed exactly which tasks run between the points where we collect the timing for that test.

Comment 7

[Kris Maglione [:kmag]]

Pretty convinced this is just noise, and shows up next to more significant improvements, so I don't think we should be concerned.