1647646 - Performance regression when we unset EARLY_BETA_OR_EARLIER

Status: RESOLVED WONTFIX

(Testing :: Performance, defect, P1)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a raptor performance regression from push 293fc8f2ed921584a6e413aca2099acfa955e75d. As author of one of the patches included in that push, we need your help to address this regression.

== Change summary for alert #26302 (as of Tue, 23 Jun 2020 05:59:07 GMT) ==

Regressions:

44% raptor-tp6-microsoft-firefox-cold not-replayed linux64-shippable-qr opt 128.08 -> 184.17
43% raptor-tp6-microsoft-firefox-cold not-replayed linux64-shippable opt 124.23 -> 177.67
32% raptor-tp6-microsoft-firefox-cold loadtime macosx1014-64-shippable opt 918.46 -> 1,212.17
28% raptor-tp6-instagram-firefox-cold loadtime macosx1014-64-shippable opt 1,640.92 -> 2,097.58
23% raptor-tp6-microsoft-firefox-cold loadtime linux64-shippable opt 668.42 -> 820.42
19% raptor-tp6-facebook-redesign-firefox-cold not-replayed macosx1014-64-shippable opt 248.50 -> 295.92
18% raptor-tp6-microsoft-firefox-cold loadtime linux64-shippable-qr opt 677.75 -> 798.58
16% raptor-tp6-instagram-firefox-cold loadtime linux64-shippable-qr opt 1,169.77 -> 1,355.00
13% raptor-tp6-microsoft-firefox-cold loadtime windows7-32-shippable opt 580.42 -> 653.08
12% raptor-tp6-microsoft-firefox-cold loadtime windows10-64-shippable opt 588.50 -> 656.17
7% raptor-tp6-instagram-firefox-cold replayed windows10-64-shippable-qr opt 1,229.08 -> 1,145.08
6% raptor-tp6-instagram-firefox-cold replayed windows10-64-shippable opt 1,228.67 -> 1,150.42
6% raptor-tp6-microsoft-firefox-cold confidence linux64-shippable-qr opt 84.42 -> 79.25
6% raptor-tp6-microsoft-firefox-cold confidence linux64-shippable opt 84.92 -> 79.75
6% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable opt 1,366.08 -> 1,443.58
6% raptor-tp6-twitter-firefox-cold loadtime linux64-shippable opt 1,445.00 -> 1,524.92
5% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable-qr opt 1,357.85 -> 1,432.17
5% raptor-tp6-twitter-firefox-cold loadtime windows7-32-shippable opt 1,363.83 -> 1,433.67
5% raptor-tp6-instagram-firefox-cold replayed windows7-32-shippable opt 1,220.85 -> 1,159.08
4% raptor-tp6-instagram-firefox-cold replayed linux64-shippable opt 1,258.54 -> 1,202.08
4% raptor-motionmark-animometer-firefox windows10-64-shippable-qr opt 45.85 -> 44.08

Improvements:

19% raptor-tp6-facebook-redesign-firefox-cold windows7-32-shippable opt 960.52 -> 781.18
18% raptor-tp6-yahoo-mail-firefox-cold not-replayed macosx1014-64-shippable opt 290.17 -> 237.92
18% raptor-tp6-facebook-redesign-firefox-cold windows10-64-shippable opt 968.64 -> 795.86
18% raptor-tp6-yahoo-mail-firefox-cold not-replayed linux64-shippable-qr opt 300.83 -> 247.75
17% raptor-tp6-yahoo-mail-firefox-cold not-replayed windows10-64-shippable-qr opt 292.33 -> 243.00
17% raptor-tp6-facebook-redesign-firefox-cold fcp windows10-64-shippable-qr opt 944.75 -> 787.08
17% raptor-tp6-yahoo-mail-firefox-cold not-replayed linux64-shippable opt 299.92 -> 250.00
16% raptor-tp6-facebook-redesign-firefox-cold windows10-64-shippable-qr opt 908.43 -> 758.67
16% raptor-tp6-facebook-redesign-firefox-cold fcp windows7-32-shippable opt 1,053.25 -> 880.33
16% raptor-tp6-facebook-redesign-firefox-cold fcp linux64-shippable opt 1,002.17 -> 841.25
16% raptor-tp6-facebook-redesign-firefox-cold fcp windows10-64-shippable opt 1,054.92 -> 888.42
15% raptor-tp6-facebook-redesign-firefox-cold fcp linux64-shippable-qr opt 1,064.25 -> 901.08
15% raptor-tp6-google-mail-firefox-cold replayed linux64-shippable-qr opt 1,358.92 -> 1,565.42
14% raptor-tp6-yahoo-mail-firefox-cold not-replayed windows7-32-shippable opt 292.38 -> 250.08
13% raptor-tp6-facebook-redesign-firefox-cold linux64-shippable-qr opt 941.59 -> 815.43
13% raptor-tp6-facebook-redesign-firefox-cold linux64-shippable opt 902.88 -> 784.32
13% raptor-tp6-facebook-redesign-firefox-cold fcp macosx1014-64-shippable opt 1,471.85 -> 1,282.17
12% raptor-tp6-facebook-redesign-firefox-cold macosx1014-64-shippable opt 1,369.50 -> 1,208.48
10% raptor-tp6-facebook-redesign-firefox-cold not-replayed linux64-shippable opt 349.83 -> 314.42
8% raptor-tp6-facebook-redesign-firefox-cold not-replayed linux64-shippable-qr opt 353.67 -> 326.58
7% raptor-tp6-twitter-firefox-cold fcp linux64-shippable-qr opt 631.50 -> 586.50
7% raptor-tp6-twitter-firefox-cold fcp linux64-shippable opt 611.42 -> 571.25
6% raptor-tp6-microsoft-firefox-cold fcp linux64-shippable opt 504.17 -> 473.75
6% raptor-tp6-ebay-firefox-cold fcp windows10-64-shippable-qr opt 580.92 -> 546.83
6% raptor-tp6-microsoft-firefox-cold fcp linux64-shippable-qr opt 551.15 -> 520.08
5% raptor-tp6-twitter-firefox-cold fcp windows10-64-shippable opt 590.75 -> 558.92
5% raptor-webaudio-firefox linux64-shippable-qr opt 143.25 -> 135.83
5% raptor-tp6-twitter-firefox-cold fcp windows10-64-shippable-qr opt 559.58 -> 532.75
5% raptor-tp6-twitch-firefox-cold fcp linux64-shippable-qr opt 104.67 -> 99.83
5% raptor-webaudio-firefox linux64-shippable opt 137.67 -> 131.33
4% raptor-tp6-facebook-redesign-firefox-cold not-replayed windows10-64-shippable-qr opt 356.15 -> 340.58
4% raptor-tp6-facebook-firefox-cold fcp linux64-shippable opt 701.33 -> 671.00
4% raptor-tp6-yahoo-mail-firefox-cold confidence macosx1014-64-shippable opt 76.54 -> 79.75
4% raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt 76.00 -> 79.17
4% raptor-tp6-twitter-firefox-cold fcp windows7-32-shippable opt 585.21 -> 561.17
4% raptor-tp6-twitter-firefox-cold linux64-shippable opt 737.71 -> 707.82
4% raptor-tp6-twitter-firefox-cold linux64-shippable-qr opt 747.11 -> 717.19
4% raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable opt 75.00 -> 78.00
4% raptor-tp6-reddit-firefox-cold loadtime linux64-shippable opt 2,805.08 -> 2,696.17
4% raptor-motionmark-htmlsuite-firefox linux64-shippable opt 31.21 -> 32.42
4% raptor-tp6-facebook-firefox-cold linux64-shippable opt 963.40 -> 926.84
4% raptor-tp6-reddit-firefox-cold linux64-shippable-qr opt 746.37 -> 718.76
4% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable opt 466.75 -> 449.58
4% raptor-tp6-facebook-firefox-cold fcp windows10-64-shippable opt 692.00 -> 667.50
3% raptor-tp6-facebook-firefox-cold fcp linux64-shippable-qr opt 718.83 -> 694.25
3% raptor-tp6-facebook-firefox-cold fcp windows10-64-shippable-qr opt 665.15 -> 643.75
3% raptor-tp6-yahoo-mail-firefox-cold confidence windows7-32-shippable opt 76.46 -> 78.92
3% raptor-tp6-microsoft-firefox-cold fcp windows7-32-shippable opt 464.50 -> 449.83
3% raptor-tp6-facebook-firefox-cold linux64-shippable-qr opt 981.29 -> 951.21
3% raptor-tp6-twitter-firefox-cold windows10-64-shippable opt 690.42 -> 669.28
3% raptor-tp6-facebook-firefox-cold windows10-64-shippable-qr opt 841.14 -> 818.27
3% raptor-tp6-twitter-firefox-cold windows10-64-shippable-qr opt 677.14 -> 659.45
3% raptor-tp6-reddit-firefox-cold linux64-shippable opt 719.54 -> 700.85
2% raptor-tp6-microsoft-firefox-cold windows10-64-shippable-qr opt 473.71 -> 461.90
2% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable-qr opt 509.08 -> 496.83
2% raptor-tp6-facebook-firefox-cold loadtime linux64-shippable opt 2,685.77 -> 2,626.00
2% raptor-stylebench-firefox linux64-shippable-qr opt 44.24 -> 45.20
2% raptor-tp6-yahoo-mail-firefox-cold loadtime linux64-shippable-qr opt 952.54 -> 932.83
2% raptor-tp6-twitch-firefox-cold linux64-shippable-qr opt 261.22 -> 255.91

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26302

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Strugariu [:Bebe]]

Any ideas why do we see this regression from unseting EARLY_BETA_OR_EARLIER

Comment 2

[Julien Cristau [:jcristau]]

No, you'd have to break down all uses of it to figure out which one causes this.

Comment 3

[Florin Strugariu [:Bebe]]

same regression we are seeing in different frameworks:

Browsertime:
== Change summary for alert #26276 (as of Fri, 19 Jun 2020 03:37:33 GMT) ==

Regressions:

9% facebook-cold Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt 0.77 -> 0.70
7% facebook-cold Similarity android-hw-p2-8-0-android-aarch64-shippable opt 0.78 -> 0.72

Improvements:

52% booking-cold loadtime android-hw-p2-8-0-android-aarch64-shippable opt 3,066.18 -> 1,486.71
42% instagram-cold ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt 1,440.33 -> 836.25
24% booking-cold loadtime android-hw-g5-7-0-arm7-api-16-shippable opt 2,747.25 -> 2,076.38
21% booking-cold android-hw-p2-8-0-android-aarch64-shippable opt 725.46 -> 573.46
19% instagram-cold LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt 3,786.67 -> 3,056.67
15% booking-cold android-hw-g5-7-0-arm7-api-16-shippable opt 1,274.30 -> 1,086.43
12% booking-cold LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt 3,183.33 -> 2,813.33
9% instagram-cold PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt 1,838.86 -> 1,677.58
8% cnn-cold FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt 3,514.67 -> 3,220.00
8% cnn-cold fcp android-hw-g5-7-0-arm7-api-16-shippable opt 3,533.33 -> 3,237.08
6% booking-cold ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt 1,051.75 -> 985.50
6% booking-cold SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt 1,214.60 -> 1,146.50

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26276

AWSY:
== Change summary for alert #26274 (as of Fri, 19 Jun 2020 02:59:39 GMT) ==

Improvements:

5% Base Content Explicit linux1804-64-shippable opt 13,533,696.00 -> 12,845,909.33
5% Base Content Explicit windows10-64-shippable opt 10,511,189.33 -> 10,014,208.00
4% Base Content Explicit linux1804-64-shippable-qr opt 13,809,191.38 -> 13,261,482.67
4% Images linux1804-64-shippable-qr opt tp6 8,754,271.64 -> 8,429,171.01

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26274

Comment 4

[Florin Strugariu [:Bebe]]

It looks like the primarily regressions are microsoft, instagram, twitter, and motionmark.

Julien can you help/offer advice on how to identify the relevant uses of EARLY_BETA_OR_EARLIER and narrow down the cause.

Comment 5

[Julien Cristau [:jcristau]]

Patricia, can you find someone to take a look?

Comment 6

[Sean Feng [:sefeng]]

I think the correct approach here is figuring out the newly added pref that uses EARLY_BETA_OR_EARLIER since we flip it last time, and push to try to compare. I'll do the work.

Comment 7

[Sean Feng [:sefeng]]

I tried to flip the prefs that used EARLY_BETA_OR_EARLIER and were added between May and June to see if I can reproduce the regression. And I managed to see some regressions in the result, however I don't know if they are just some noises, because if I bisect the prefs even further, the regressions are gone.

Then I check the graphs, I noticed some of the regressions occurred on June 12, so I am confused. Does it mean we even have these regressions before we flip EARLY_BETA_OR_EARLIER?

Bebe, would you be able to provide more information related to the regression? And I'd also like to know what the baseline is here, which commit is it?

Thanks

Comment 8

[Florin Strugariu [:Bebe]]

all regresssions logged are from the June 21 on commit: 293fc8f2ed921584a6e413aca2099acfa955e75d
your result look with 8 re-triggers so they have a fair amount of confidence.

from the graph:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&selected=2141079,1164744659&series=mozilla-beta,2141079,1,10&series=autoland,2134884,1,10&timerange=7776000&zoom=1591797707648,1592162770940,1214.922920058784,1496.9325003580782

you can see the base is commit: 99d567559f934fe02e31bec9f470146dc80908c3
and the next commit with build is: 293fc8f2ed921584a6e413aca2099acfa955e75d

Between them are some tag commits and a regression of 5.47% on raptor-tp6-twitter-firefox-cold test for example.

Comment 9

[Sean Feng [:sefeng]]

For some reason, I keep getting raptor-perftest Critical: Connection to Raptor webextension failed error when I push the corresponding commits to try.

The good news is that I am able to reproduce the regression locally and it turned out the network.preload flag is the one that creates the regression.

bebe, how do we want to proceed? I am not worried about the regression because it's actually good news, as the preload pref improves performance.

Comment 10

[Florin Strugariu [:Bebe]]

If you are OK with this regression and it's accepted then we can close the bug and alert with WONTFIX

Comment 11

[Sean Feng [:sefeng]]

Yup, I am OK with this regression and I dropped a comment in the original bug to let devs aware this.

Comment 12

[Sean Feng [:sefeng]]

Well, actually, there are improvements in this bug too which means real regressions by flipping the pref. So I think I'd like to NI Honza to make sure he's aware.

Honza, just making sure you are aware of these regressions/improvements. I think we can leave the bug open if you want.

Comment 13

[Honza Bambas (:mayhemer)]

Preload definitely has a positive effect. The improvements (hence, regressions when the pref is turned back on) seems related to bug 1641532, specifically the two that fixes the regressions (added them in the see also list). Both landed on 79 only because on beta preload was about to be disabled soon at the time of landing those two bugs. So, on 78 there is this behavior.

WONTFIX is the right resolution.