1555967 - 4.08 - 23.25% raptor-tp6-slides-firefox / raptor-tp6-slides-firefox loadtime / raptor-tp6-yahoo-news-firefox loadtime regression on push 5585ac78378a9ad68dc1cf3ae64ce27d7a8da83c (Tue May 28 2019)

Status: VERIFIED FIXED

(Core :: JavaScript: GC, defect, P1)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=5585ac78378a9ad68dc1cf3ae64ce27d7a8da83c

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

23% raptor-tp6-slides-firefox loadtime windows10-64-shippable-qr opt 1,014.46 -> 1,250.33
23% raptor-tp6-yahoo-news-firefox loadtime linux64-shippable-qr opt 380.58 -> 468.00
22% raptor-tp6-yahoo-news-firefox loadtime linux64-shippable opt 363.69 -> 442.42
16% raptor-tp6-yahoo-news-firefox linux64-shippable-qr opt 290.66 -> 336.09
14% raptor-tp6-yahoo-news-firefox linux64-shippable opt 279.00 -> 317.11
14% raptor-tp6-yahoo-news-firefox fcp linux64-shippable opt 272.10 -> 309.00
12% raptor-tp6-yahoo-news-firefox fcp linux64-shippable-qr opt 284.98 -> 320.29
12% raptor-tp6-yahoo-news-firefox fcp linux64-shippable opt 269.65 -> 301.42
4% raptor-tp6-slides-firefox windows10-64-shippable-qr opt 852.03 -> 886.82

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21190

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Jon Coppeard (:jonco)]

This push changed the memory accounting which is used to trigger GCs. I expect we are now triggering GC more often during these tests. I'll look into adjusting the GC parameters to take account of this tomorrow.

Comment 2

[Ionuț Goldan [:igoldan]]

:Bebe please also provide the Gecko profiles for one of the alerts. They'll be of use when working on the fix.

Comment 3

[Jon Coppeard (:jonco)]

I can't reproduce this locally. I can't see any differences looking at the Rap-prof builds from around the time this landed.

Can you back out and we'll see if this returns to normal?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Jon Coppeard (:jonco) from comment #3)

I can't reproduce this locally. I can't see any differences looking at the Rap-prof builds from around the time this landed.

Can you back out and we'll see if this returns to normal?

You mean backout from mozilla-central?

Comment 5

[Florin Strugariu [:Bebe]]

Gecko profiles:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&group_state=expanded&searchStr=linux%2Cx64%2Cshippable%2Copt%2Craptor%2Cperformance%2Ctests%2Con%2Cfirefox%2Ctest-linux64-shippable%2Fopt-raptor-tp6-6-firefox-e10s%2Crap%28tp6-6%2Cp%29&tochange=5585ac78378a9ad68dc1cf3ae64ce27d7a8da83c&fromchange=900ff508219cddc29c6f6307c65f0e90c3422c24

Comment 6

[Jon Coppeard (:jonco)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #4)

You mean backout from mozilla-central?

I'm not sure where else to go from here.

I've tried again and can't reproduce any change in this benchmark result locally (I'm looking at raptor-tp6-yahoo-news-firefox at this change and +/- 50 changesets). I don't think my change is responsible, but it's possible that it changed GC timing and we're now triggering collections in places we're measuring where we weren't before.

Any ideas on how else to progress? It would be good to find whether this change was responsible or not.

Comment 7

[Ionuț Goldan [:igoldan]]

Florin, let's simulate the backout on Try 1st.

Comment 8

[Ionuț Goldan [:igoldan]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #7)

Florin, let's simulate the backout on Try 1st.

You'll be able to see the performance comparison here (right side includes bug 1395509, left side doesn't). Still waiting for some jobs to finish.

Comment 9

[Jon Coppeard (:jonco)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #8)
The results seem to be mixed although there are some improvements. What do you think of them?

Comment 10

[Ionuț Goldan [:igoldan]]

(In reply to Jon Coppeard (:jonco) from comment #9)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #8)
The results seem to be mixed although there are some improvements. What do you think of them?

Raptor Yahoo-News test requires more retriggers on the right side.

Comment 11

[Jon Coppeard (:jonco)]

I'm still not sure whether it was my changes to GC memory accounting that caused this, but my current plan is to land a patch (in bug 1395509 comment 33) that will make this much closer to the original scheme. Hopefully this will fix these benchmark issues.

Comment 12

[Florin Strugariu [:Bebe]]

== Change summary for alert #21414 (as of Wed, 12 Jun 2019 09:14:35 GMT) ==

Improvements:

21% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 opt 596.50 -> 473.58
20% raptor-tp6-yahoo-news-firefox loadtime linux64-shippable-qr opt 472.38 -> 376.79
18% raptor-tp6-yahoo-news-firefox loadtime linux64-shippable opt 453.08 -> 370.33
18% raptor-tp6-slides-firefox loadtime linux64-shippable-qr opt 1,119.35 -> 918.38
17% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 opt 1,168.58 -> 975.75
16% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 pgo 570.88 -> 478.67
15% raptor-tp6-yahoo-news-firefox linux64-shippable-qr opt 341.93 -> 290.08
14% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 1,123.58 -> 966.62
14% raptor-tp6-yahoo-news-firefox fcp linux64-shippable-qr opt 324.96 -> 280.25
13% raptor-tp6-slides-firefox loadtime linux64-shippable opt 980.08 -> 849.92
13% raptor-tp6-yahoo-news-firefox linux64-shippable opt 325.34 -> 282.29
13% raptor-tp6-yahoo-news-firefox fcp linux64-shippable opt 307.96 -> 268.71
12% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-p2-8-0-arm7-api-16 opt 555.38 -> 488.92
7% raptor-tp6-docs-firefox loadtime linux64-shippable opt 1,021.81 -> 952.21
7% raptor-tp6-docs-firefox linux64-shippable opt 846.12 -> 790.40
5% raptor-tp6-docs-firefox fcp linux64-shippable opt 961.42 -> 910.42

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21414