1677475 - 1.51% Base Content JS (macosx1014-64-shippable-qr) regression on push bbf90b97018afa7599012fb67787c0814cc2091b (Tue November 10 2020)

Status: RESOLVED INVALID

(Remote Protocol :: Marionette, defect)

Description

[Andra Esanu (needinfo me)]

Perfherder has detected a awsy performance regression from push bbf90b97018afa7599012fb67787c0814cc2091b. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
2%	Base Content JS		macosx1014-64-shippable-qr		2,798,629.33 -> 2,841,002.67

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andra Esanu from comment #0)

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
2%	Base Content JS		macosx1014-64-shippable-qr		2,798,629.33 -> 2,841,002.67

We are aware that this might have caused a small regression across platforms because both the framescript and actor implementations are running at the same time, but with bug 1669169 all commands and events are getting routed through actors now. Next step for us is to not even register the framescript when actors are enabled. This work should be done end of this week or early next week (see bug 1669174).

What I don't understand is why the base content JS test only regresses on Mac but not on other platforms (Linux, Windows). Andra, do you have some links to these?

Nevertheless I would suggest that we ignore this warning, and wontfix this bug.

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Henrik, the other platforms' Base Content JS tests went pretty noisy (linux1804-64-shippable, windows10-64-shippable-qr) and perfherder didn't detected them as regressions. Graph here. I wouldn't rush on WONTFIXing this bug regression given the noise it caused.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, thanks! So lets keep this bug open and revisit it once bug 1669174 has been landed. It should happen later this week or early next week.

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #27686 (as of Mon, 16 Nov 2020 14:18:36 GMT) ==

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
452%	reddit	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	283.75 -> 1,565.33
323%	reddit	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	394.92 -> 1,669.58
249%	reddit	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	206.21 -> 719.58
217%	reddit	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	240.42 -> 761.08
119%	reddit		android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	764.20 -> 1,672.30
97%	allrecipes	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	399.04 -> 785.29
96%	allrecipes	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm	398.29 -> 781.92
83%	reddit		android-hw-p2-8-0-android-aarch64-shippable	warm webrender	424.32 -> 775.13
78%	allrecipes	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	496.92 -> 886.50
77%	allrecipes	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm	497.79 -> 880.75
60%	allrecipes	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	195.12 -> 312.50
57%	allrecipes	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm	1,470.04 -> 2,303.50
54%	allrecipes	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	1,472.92 -> 2,263.92
50%	allrecipes	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	240.33 -> 359.58
47%	web-de	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	114.58 -> 168.29
46%	allrecipes	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm	247.42 -> 360.50
42%	web-de	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	269.75 -> 382.46
40%	web-de	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	115.75 -> 162.08
37%	allrecipes	ContentfulSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	446.00 -> 611.67
34%	reddit	PerceptualSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	2,076.25 -> 2,783.67
33%	allrecipes		android-hw-p2-8-0-android-aarch64-shippable	warm webrender	429.14 -> 572.83
33%	reddit	PerceptualSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	953.25 -> 1,269.00
32%	allrecipes		android-hw-p2-8-0-android-aarch64-shippable	warm	441.63 -> 581.01
30%	allrecipes		android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	953.39 -> 1,236.23
30%	allrecipes		android-hw-g5-7-0-arm7-api-16-shippable	warm	946.59 -> 1,226.49
26%	web-de	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	163.58 -> 205.67
25%	web-de	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	377.62 -> 473.42
19%	web-de		android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	634.51 -> 756.17
18%	web-de	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	977.58 -> 1,157.17
18%	stackoverflow	ContentfulSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	334.17 -> 395.33
18%	stackoverflow	LastVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	334.17 -> 395.33
18%	stackoverflow	PerceptualSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	334.17 -> 394.50
18%	web-de	PerceptualSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	1,007.50 -> 1,188.67
18%	stackoverflow	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	334.17 -> 393.42
17%	stackoverflow	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	334.17 -> 392.58
17%	stackoverflow	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	272.33 -> 319.33
16%	stackoverflow		android-hw-p2-8-0-android-aarch64-shippable	warm webrender	308.07 -> 357.50
16%	reddit	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	2,667.21 -> 3,086.17
16%	web-de		android-hw-p2-8-0-android-aarch64-shippable	warm webrender	278.78 -> 322.15
15%	web-de	PerceptualSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	429.08 -> 494.42
13%	web-de	SpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	1,272.21 -> 1,442.58
13%	stackoverflow	loadtime	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	1,018.94 -> 1,153.96
12%	reddit	ContentfulSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	1,226.08 -> 1,373.33
12%	web-de	ContentfulSpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	454.09 -> 507.58
11%	ebay-kleinanzeigen-search	LastVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	952.09 -> 1,060.33
11%	stackoverflow	loadtime	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	448.42 -> 499.08
11%	stackoverflow		android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	609.06 -> 677.00
11%	stackoverflow	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	517.25 -> 572.75
10%	ebay-kleinanzeigen-search	LastVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	2,489.75 -> 2,740.67
9%	web-de	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	569.73 -> 623.42
9%	stackoverflow	PerceptualSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	651.08 -> 707.08
9%	stackoverflow	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	651.08 -> 707.00
8%	stackoverflow	SpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	651.08 -> 702.83
8%	ebay-kleinanzeigen-search	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	767.04 -> 825.33
7%	allrecipes	loadtime	android-hw-p2-8-0-android-aarch64-shippable	warm	1,480.27 -> 1,590.46
7%	stackoverflow	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	651.08 -> 698.92
7%	ebay-kleinanzeigen-search	SpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	1,786.75 -> 1,910.92
4%	reddit	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	1,784.61 -> 1,851.42

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=27686

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #4)

== Change summary for alert #27686 (as of Mon, 16 Nov 2020 14:18:36 GMT) ==

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
452%	reddit	fcp	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	283.75 -> 1,565.33
323%	reddit	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	394.92 -> 1,669.58
249%	reddit	fcp	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	206.21 -> 719.58
217%	reddit	FirstVisualChange	android-hw-p2-8-0-android-aarch64-shippable	warm webrender	240.42 -> 761.08
119%	reddit		android-hw-g5-7-0-arm7-api-16-shippable	warm webrender	764.20 -> 1,672.30

All those so called regressions in Browsertime jobs aren't tied to Marionette and the enabling of actors. When you check the graphs for each of those individual regressions you can see that numbers dropped again on Nov 11th:

https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,2857959,1,13

I haven't checked them all, but I highly suspect that all these regressions are unrelated here.

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Henrik, seems like a bunch of alert items in the summary were not caused by your patch. Thus, this alert summary contains only 2 items that are improvements. Closing this as invalid. Thanks for your input.