1674317 - 94.91 - 140.43% about_newtab_with_snippets (linux64-shippable, macosx1014-64-shippable, windows10-64-shippable, windows7-32-shippable) regression on push 9dfa75f6ec73f10849e81c214271f8dd4a20ad45 (Sat October 24 2020)

Status: RESOLVED FIXED

(Firefox :: New Tab Page, defect, P3)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a talos performance regression from push 9dfa75f6ec73f10849e81c214271f8dd4a20ad45. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
140%	about_newtab_with_snippets		windows10-64-shippable	e10s stylo	29.11 -> 70.00
140%	about_newtab_with_snippets		windows7-32-shippable	e10s stylo	28.08 -> 67.33
116%	about_newtab_with_snippets		linux64-shippable	e10s stylo	30.72 -> 66.29
95%	about_newtab_with_snippets		macosx1014-64-shippable	e10s stylo	36.37 -> 70.90

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
26%	startup_about_home_paint_realworld_webextensions	startup_about_home_paint_realworld_webextensions	windows7-32-shippable	e10s stylo	718.17 -> 529.58
25%	startup_about_home_paint_realworld_webextensions	startup_about_home_paint_realworld_webextensions	windows10-64-shippable	e10s stylo	712.42 -> 533.25
11%	startup_about_home_paint_realworld_webextensions	startup_about_home_paint_realworld_webextensions	linux64-shippable	e10s stylo	685.08 -> 610.75
11%	startup_about_home_paint_realworld_webextensions	startup_about_home_paint_realworld_webextensions	linux64-shippable	e10s stylo	685.17 -> 611.33

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Because 2cd3d250088 broke the builds for 3 revisions, I had to make a local backout and push to try to be able to see the results.

• this try push corresponds to da718fa047b which is the last good which mesaures about 33 for about_newtab_with_snippets opt e10s stylo
• this try push corresponds to 9dfa75f6ec which is the revision corresponding to Bug 1672517 that seems to be introducing the regression. It measures about 72 for about_newtab_with_snippets opt e10s stylo

The comparison is here. I'm not sure if the link contains the filtering, but if you search for about_newtab_with_snippets opt e10s stylo you'll see a regression of over 110%.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1672517

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Looking.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

This is a real regression. The problem here appears to be that when we load the first about:home from the about:home startup cache, the scripts that are loaded for it aren't inserted into the ScriptPreloader cache. This means that subsequent loads of about:home / about:newtab don't hit the cache, which results in slower times to load the page.

I'm discussing solutions with the JS team right now.

Comment 5

[Mike Conley (:mconley) (:⚙️)]

The JS team is working on a big sweeping change to how JS caching works in the browser, so what tcampbell recommended I do is workaround this issue by using the subscript loader again for the about:home startup cache, and file a bug to remove that hackery and use the shiny new goodness when it's ready (~6 months).

Comment 6

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1674317 - Load about:home scripts via the subscript loader when hitting the about:home startup cache. r?k88hudson!

Using the subscript loader is currently our best way of getting the loaded scripts recorded for entry
in the ScriptPreloader which is used to cache startup-related JavaScript within the startup window.

Originally, we were using <script> tags to load those scripts, but unfortunately, that loading mechanism
does not qualify for being noted by the ScriptPreloader.

This is a workaround until the Stencil project by the SpiderMonkey team Phase 2 is complete (see bug 1663956),
which should allow us to more easily note scripts to cache at runtime.

Comment 7

[Mike Conley (:mconley) (:⚙️)]

This also means we weren't hitting the ScriptPreloader cache when we were loading the scripts from the about:home startup cache, nor for new tabs / new windows. I'm glad our internal benchmark caught this!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cf8e07fcfaa7
Load about:home scripts via the subscript loader when hitting the about:home startup cache. r=k88hudson

Comment 9

[Razvan Maries]

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=QSt9vX-bTUqzJpPzqofPGQ.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=cf8e07fcfaa7e53593baecaaa3e8b47209ff94a3&searchStr=xpcshell

Log: https://treeherder.mozilla.org/logviewer?job_id=320624506&repo=autoland&lineNumber=2947

Backout: https://hg.mozilla.org/integration/autoland/rev/1743a09c38a8dc7aa11ba39037eb48c34d8a79af

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d345090ddec
Load about:home scripts via the subscript loader when hitting the about:home startup cache. r=k88hudson

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/3d345090ddec

Comment 12

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #27535 (as of Fri, 06 Nov 2020 05:33:41 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
56%	about_newtab_with_snippets		windows7-32-shippable	e10s stylo	65.94 -> 29.17
54%	about_newtab_with_snippets		linux64-shippable-qr	e10s stylo webrender-sw	66.56 -> 30.39
54%	about_newtab_with_snippets		linux64-shippable-qr	e10s stylo webrender-sw	66.23 -> 30.30
54%	about_newtab_with_snippets		windows10-64-shippable-qr	e10s stylo webrender-sw	67.53 -> 31.31
53%	about_newtab_with_snippets		windows10-64-shippable	e10s stylo	66.73 -> 31.13
52%	about_newtab_with_snippets		linux64-shippable	e10s stylo	65.56 -> 31.21
9%	rasterflood_svg		windows10-64-shippable-qr	e10s stylo webrender-sw	10,794.98 -> 9,820.63
8%	rasterflood_svg		linux64-shippable-qr	e10s stylo webrender-sw	10,540.22 -> 9,737.55
2%	startup_about_home_paint_realworld_webextensions	startup_about_home_paint_realworld_webextensions	linux64-shippable-qr	e10s stylo webrender	872.75 -> 853.83

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=27535