Closed Bug 1562135 Opened 3 years ago Closed 2 years ago

4.9% raptor-tp6-google-firefox loadtime (linux64-shippable) regression on push 0649547f4b2947c188067f0502733100b6d7f92d

Categories

(Core :: ImageLib, defect, P3)

x86_64
Linux
defect

Tracking

()

RESOLVED WONTFIX

People

(Reporter: igoldan, Unassigned)

References

(Regression)

Details

(Keywords: perf, regression)

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=197df45a8076f497c59951d3ab818af9ba7eedcc&tochange=0649547f4b2947c188067f0502733100b6d7f92d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% raptor-tp6-google-firefox loadtime linux64-shippable opt 135.52 -> 142.17

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=20998

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Component: General → ImageLib
Product: Testing → Core
Flags: needinfo?(tnikkel)

:tnikkel, it seems bug 1552077 caused a pretty big pageload regression + a lot of noise for this particular perf test. Please look over this issue.

From bug 1552077, comment 4 we got pretty big wins from this patch elsewhere. Since this regression is only linux and only on one site it seems like just taking this regression would be an overall win. Making more image runnables medium high means other things might run later, and this is a tradeoff if those other things are actually more important to getting the page loaded.

Is there someone who makes the call in these "two steps forward one step back" perf situations?

Flags: needinfo?(tnikkel)

I believe overall the visual metrics wins overrule the regressions here.
Denis may have an opinion.

Flags: needinfo?(dpalmeiro)

I strongly recommend we do not back this change out. It was a very significant improvement in page load when I measured it on real live pages. I'll dig a bit deeper into this specific case and try to collect some visual metrics for it to see if it shows up there as well, but I think the 7ms regression here is diminished by the real world gains we saw on the reference hardware.

Flags: needinfo?(dpalmeiro)

I ran this on the live website, and got the following results:
Nightly-2016-05-17:
[2019-06-28 09:59:14] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 258ms (±6.37ms), firstPaint: 346ms (±5.65ms), firstVisualChange: 60ms (±12.96ms), DOMContentLoaded: 1.08s (±57.66ms), Load: 1.58s (±92.40ms), speedIndex: 815 (±58.35), perceptualSpeedIndex: 361 (±23.07), contentfulSpeedIndex: 687 (±58.92), visualComplete85: 5ms (±0.14ms), lastVisualChange: 1.16s (±62.74ms), rumSpeedIndex: 260 (±6.39) (10 runs)

Nightly-2016-05-19:
[2019-06-28 10:01:45] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 253ms (±3.79ms), firstPaint: 336ms (±3.93ms), firstVisualChange: 40ms (±0.00ms), DOMContentLoaded: 998ms (±36.48ms), Load: 1.56s (±43.73ms), speedIndex: 741 (±34.04), perceptualSpeedIndex: 319 (±11.80), contentfulSpeedIndex: 622 (±27.15), visualComplete85: 5ms (±0.15ms), lastVisualChange: 1.09s (±33.49ms), rumSpeedIndex: 256 (±3.79) (10 runs)

With some network throttling, I still don't see much of a difference:
Nightly-2016-05-17:
[2019-06-28 10:22:36] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 241ms (±3.74ms), firstPaint: 497ms (±7.53ms), firstVisualChange: 208ms (±12.39ms), DOMContentLoaded: 1.60s (±25.30ms), Load: 5.11s (±65.64ms), speedIndex: 862 (±13.63), perceptualSpeedIndex: 467 (±9.62), contentfulSpeedIndex: 483 (±14.60), visualComplete85: 5ms (±0.09ms), lastVisualChange: 3.92s (±112.71ms), rumSpeedIndex: 277 (±3.37) (10 runs)

Nightly-2016-05-19:
[2019-06-28 10:27:12] INFO: https://www.google.com/search?hl=en&q=barack+obama&cad=h 0, backEndTime: 248ms (±15.83ms), firstPaint: 500ms (±19.84ms), firstVisualChange: 216ms (±18.07ms), DOMContentLoaded: 1.69s (±46.42ms), Load: 5.07s (±100.42ms), speedIndex: 926 (±27.59), perceptualSpeedIndex: 496 (±18.12), contentfulSpeedIndex: 485 (±18.34), visualComplete85: 5ms (±0.00ms), lastVisualChange: 4.12s (±133.27ms), rumSpeedIndex: 287 (±15.59) (10 runs)

Running visual metrics on the recordings wasn't very useful, it just loads too fast. But from the results above, I don't think this change affects the google search url too much.

The priority flag is not set for this bug.
:aosmond, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(aosmond)

Sounds like we suspect this to be a false positive, and we have measured substantial improvements against the real world. As such, marking as WONTFIX.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(aosmond)
Priority: -- → P3
Resolution: --- → WONTFIX

Thanks for looking over this!

You need to log in before you can comment on or make changes to this bug.