1562135 - 4.9% raptor-tp6-google-firefox loadtime (linux64-shippable) regression on push 0649547f4b2947c188067f0502733100b6d7f92d

Status: RESOLVED WONTFIX

(Core :: Graphics: ImageLib, defect, P3)

Description

[Ionuț Goldan [:igoldan]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=197df45a8076f497c59951d3ab818af9ba7eedcc&tochange=0649547f4b2947c188067f0502733100b6d7f92d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% raptor-tp6-google-firefox loadtime linux64-shippable opt 135.52 -> 142.17

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=20998

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:tnikkel, it seems bug 1552077 caused a pretty big pageload regression + a lot of noise for this particular perf test. Please look over this issue.

Comment 2

[Timothy Nikkel (:tnikkel)]

From bug 1552077, comment 4 we got pretty big wins from this patch elsewhere. Since this regression is only linux and only on one site it seems like just taking this regression would be an overall win. Making more image runnables medium high means other things might run later, and this is a tradeoff if those other things are actually more important to getting the page loaded.

Is there someone who makes the call in these "two steps forward one step back" perf situations?

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

I believe overall the visual metrics wins overrule the regressions here.
Denis may have an opinion.

Comment 4

[Denis Palmeiro [:denispal]]

I strongly recommend we do not back this change out. It was a very significant improvement in page load when I measured it on real live pages. I'll dig a bit deeper into this specific case and try to collect some visual metrics for it to see if it shows up there as well, but I think the 7ms regression here is diminished by the real world gains we saw on the reference hardware.

Comment 5

[Denis Palmeiro [:denispal]]

I ran this on the live website, and got the following results:
Nightly-2016-05-17:
[2019-06-28 09:59:14] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 258ms (±6.37ms), firstPaint: 346ms (±5.65ms), firstVisualChange: 60ms (±12.96ms), DOMContentLoaded: 1.08s (±57.66ms), Load: 1.58s (±92.40ms), speedIndex: 815 (±58.35), perceptualSpeedIndex: 361 (±23.07), contentfulSpeedIndex: 687 (±58.92), visualComplete85: 5ms (±0.14ms), lastVisualChange: 1.16s (±62.74ms), rumSpeedIndex: 260 (±6.39) (10 runs)

Nightly-2016-05-19:
[2019-06-28 10:01:45] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 253ms (±3.79ms), firstPaint: 336ms (±3.93ms), firstVisualChange: 40ms (±0.00ms), DOMContentLoaded: 998ms (±36.48ms), Load: 1.56s (±43.73ms), speedIndex: 741 (±34.04), perceptualSpeedIndex: 319 (±11.80), contentfulSpeedIndex: 622 (±27.15), visualComplete85: 5ms (±0.15ms), lastVisualChange: 1.09s (±33.49ms), rumSpeedIndex: 256 (±3.79) (10 runs)

With some network throttling, I still don't see much of a difference:
Nightly-2016-05-17:
[2019-06-28 10:22:36] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 241ms (±3.74ms), firstPaint: 497ms (±7.53ms), firstVisualChange: 208ms (±12.39ms), DOMContentLoaded: 1.60s (±25.30ms), Load: 5.11s (±65.64ms), speedIndex: 862 (±13.63), perceptualSpeedIndex: 467 (±9.62), contentfulSpeedIndex: 483 (±14.60), visualComplete85: 5ms (±0.09ms), lastVisualChange: 3.92s (±112.71ms), rumSpeedIndex: 277 (±3.37) (10 runs)

Nightly-2016-05-19:
[2019-06-28 10:27:12] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 248ms (±15.83ms), firstPaint: 500ms (±19.84ms), firstVisualChange: 216ms (±18.07ms), DOMContentLoaded: 1.69s (±46.42ms), Load: 5.07s (±100.42ms), speedIndex: 926 (±27.59), perceptualSpeedIndex: 496 (±18.12), contentfulSpeedIndex: 485 (±18.34), visualComplete85: 5ms (±0.00ms), lastVisualChange: 4.12s (±133.27ms), rumSpeedIndex: 287 (±15.59) (10 runs)

Running visual metrics on the recordings wasn't very useful, it just loads too fast. But from the results above, I don't think this change affects the google search url too much.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:aosmond, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

Sounds like we suspect this to be a false positive, and we have measured substantial improvements against the real world. As such, marking as WONTFIX.

Comment 8

[Ionuț Goldan [:igoldan]]

Thanks for looking over this!