1540691 - 7.84 - 51.77% raptor-tp6 (osx-10-10-shippable) regression on push 5bdb280c1f4eea14d9c303d8eff13bc59c1c2e4c (Thu Mar 28 2019)

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect, P2)

Description

[Ionuț Goldan [:igoldan]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=5bdb280c1f4eea14d9c303d8eff13bc59c1c2e4c

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

52% raptor-tp6-yahoo-news-firefox fcp osx-10-10-shippable opt 698.02 -> 1,059.38
45% raptor-tp6-yahoo-news-firefox loadtime osx-10-10-shippable opt 1,381.54 -> 1,996.50
26% raptor-tp6-reddit-firefox loadtime osx-10-10-shippable opt 6,697.62 -> 8,452.00
24% raptor-tp6-reddit-firefox fcp osx-10-10-shippable opt 859.88 -> 1,064.96
13% raptor-tp6-microsoft-firefox fcp osx-10-10-shippable opt 1,207.90 -> 1,364.21
8% raptor-tp6-amazon-firefox fcp osx-10-10-shippable opt 628.12 -> 677.38

Improvements:

49% raptor-tp6-instagram-firefox osx-10-10-shippable opt 478.93 -> 244.18
34% raptor-tp6-sheets-firefox osx-10-10-shippable opt 1,050.29 -> 691.17
33% raptor-tp6-imgur-firefox osx-10-10-shippable opt 997.54 -> 670.64
30% raptor-tp6-microsoft-firefox osx-10-10-shippable opt 2,655.80 -> 1,852.45
29% raptor-tp6-docs-firefox osx-10-10-shippable opt 1,717.52 -> 1,221.74
28% raptor-tp6-yahoo-mail-firefox osx-10-10-shippable opt 657.29 -> 476.24
24% raptor-tp6-google-firefox osx-10-10-shippable opt 292.62 -> 221.84
20% raptor-tp6-twitter-firefox osx-10-10-shippable opt 589.26 -> 468.51
20% raptor-tp6-imdb-firefox osx-10-10-shippable opt 420.60 -> 335.15
19% raptor-tp6-yandex-firefox osx-10-10-shippable opt 324.23 -> 263.01
18% raptor-tp6-slides-firefox osx-10-10-shippable opt 2,758.57 -> 2,261.32
18% raptor-tp6-facebook-firefox osx-10-10-shippable opt 877.86 -> 719.67
16% raptor-tp6-youtube-firefox osx-10-10-shippable opt 914.65 -> 764.73
16% raptor-tp6-amazon-firefox osx-10-10-shippable opt 911.18 -> 766.89
14% raptor-tp6-wikia-firefox osx-10-10-shippable opt 416.34 -> 357.54
7% raptor-tp6m-facebook-geckoview fcp android-hw-g5-7-0-arm7-api-16-pgo opt 735.73 -> 687.29

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=20168

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Are these new baselines looking ok? Could we close this as WONTFIX?

Comment 2

[Robert Wood [:rwood]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)

Are these new baselines looking ok? Could we close this as WONTFIX?

I can understand why we would see improvements, as the overall geomean will be reduced when not measuring TTFI and hero. The regressions on fcp and loadtime don't make sense though. Is there any chance that the regressions are actually from the landing of Bug 1536758? I could see perhaps that bug creating a regression as that injects Raptor at a different place in the test page content - potentially allowing more page source to run/load.

:acreskey, any ideas here? Thanks!

Comment 3

[Andrew Creskey [:acreskey]]

Yes, that is one large and unexpected change in loadtime that I feel we need to get to the bottom of:
raptor-tp6-yahoo-news-firefox loadtime osx-10-10-shippable opt 1,381.54 < 1,996.5 44.51%

I'll second :rwood's question -- are we sure that this is the commit that introduced it and not Bug 1536758?

A change that big should be reproduceable locally so I'll do a test run on my own OSX device and see if re-introducing the changes impacts loadtime.

Comment 4

[Andrew Creskey [:acreskey]]

I did a local rollback of https://hg.mozilla.org/integration/autoland/rev/5bdb280c1f4e and ran the test that I thought regressed the worst - raptor-tp6-yahoo-news-firefox

dcf:        281.0
fnbp:       294.0
fcp:        321.0
loadtime:   595.5

ttfi re-added (rollback):
dcf:        287.5
fcp:        336.5
fnbp:       303.0
ttfi:      3281.5
loadtime:   591.0

Within the error bars I don't see a change to loadtime.
This was OSX, although on high-end OSX instead of the MacMini that runs in the lab.

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Robert Wood [:rwood] from comment #2)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)

Are these new baselines looking ok? Could we close this as WONTFIX?

Is there any chance that the regressions are actually from the landing of Bug 1536758? I could see perhaps that bug creating a regression as that injects Raptor at a different place in the test page content - potentially allowing more page source to run/load.

I don't think there's any change for that. Bug 1536758 landed one day later and I didn't notice it behaving as a regression; on a few tests, quite the opposite. I think bug 1536874 is the origin of these regressions.

Comment 6

[Robert Wood [:rwood]]

This has sat for quite some time now; I'm assuming we've just accepted this as the new baseline. Will close as wont-fix. If anyone objects and still wants to look into this then feel free to re-open.