1725384 - 16.89% wikia SpeedIndex (Linux) regression on Mon August 2 2021

Status: RESOLVED FIXED

(Core :: DOM: Navigation, defect, P1)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Perfherder has detected a browsertime performance regression from push afcdd8e3c67656f49be753e0cf04f18838639543. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
17%	wikia SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	418.50 -> 489.17

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:emilio could one of your patches have caused this regression?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

There were other regressions as well:
warm: 25% for youtube, also bing, fandom, gsearch, gslides, imdb, linkedin, microsoft. A few possible cold-load regressions as well - buzzfeed, microsoft, netflix, wikia, youtube

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Netflix cold definitely regressed, and the apparent regression point strongly implies Emelio's patch

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&selected=3426576,1394312938&series=autoland,3381559,1,13&series=mozilla-central,3745694,1,13&series=mozilla-central,3745746,1,13&series=mozilla-central,3426576,1,13&timerange=5184000&zoom=1627695300110,1628220257673,603.3230135609122,772.5848045629614

Comment 5

[Chris Peterson [:cpeterson]]

Tentatively assigning this bug to Emilio, since this bug looks like a regression from his grey flash fix (bug 1677324).

On AWFY, I see a warm load regression blip on Aug 3 for some sites (e.g. Amazon and Instagram), but the only persistent regressions I see since Aug 3 are Google Slides, Wikia, and YouTube:

https://arewefastyet.com/linux64/warm-page-load/overview?numDays=30&series=Firefox,Firefox-Fission

Comment 6

[Emilio Cobos Álvarez (:emilio)]

Yeah, I'm actively looking into this.

Comment 7

[Emilio Cobos Álvarez (:emilio)]

So here's a try run as a baseline ("bad"):

• https://treeherder.mozilla.org/jobs?repo=try&revision=a65a1a3eae58f7f3e77732fa35aa8bb2a9fe20ee

And here's one with my patch effectively reverted ("good"):

• https://treeherder.mozilla.org/jobs?repo=try&revision=f95a8c2918a3b18222e845bb8801e95f1885b2a3

Here's the perherder comparison ("good" vs. "bad"):

• https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=f95a8c2918a3b18222e845bb8801e95f1885b2a3&newProject=try&newRevision=a65a1a3eae58f7f3e77732fa35aa8bb2a9fe20ee&framework=13&page=1&showOnlyImportant=1

Which does show at least a huge regression in imgur FirstVisualChange warm, and the smaller Wikia SpeedIndex warm regression in comment 0.

I'm waiting for profiles for the wikia thing (because I accidentally requested them on windows, but the regression seems Linux-specific for some reason), so I decided to look at the imgur regression. But I'm confused, because if I download the Linux imgur results, I expect the "good" results to have a FirstVisualChange of ~80, but I don't see any video with that, all the videos under both results here have a "First Visual Change" of ~440. I've been running this:

for video in imgur-good/browsertime-results/imgur/pages/imgur_com/gallery/m5tYJL6/data/video/*.mp4; do echo $video; ~/src/moz/gecko-4/tools/browsertime/node_modules/browsertime/browsertime/visualmetrics.py -i $video 2>/dev/null; echo ""; done

Are those videos somehow only for cold loads? Is there an error in that comparison I linked above somehow?

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1725384 - Cleanup a bit ChangeRemotenessCommon. r=smaug

This shouldn't change behavior, but cleans up the code (some of the conditions
were from the original patch from Nika).

Comment 9

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1725384 - Raise the priority of DidUnsuppressPainting. r=smaug

This seems to help a bit:

https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=a8b34f39bb33c4b03b3a0e4f28f504c1829a68f9&newProject=try&newRevision=38c35d2f1c39d7d6c680258cae6fc1c96f3c6b27&framework=13&page=1

Depends on D123364

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Answered in matrix; we really need dave hunt or sparky to answer that. You can look at some site that shows a major difference in warmload from cold to verify

Comment 11

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9dd6beb8b65b
https://hg.mozilla.org/mozilla-central/rev/69bdc7bedcfb

Comment 12

[Emilio Cobos Álvarez (:emilio)]

Hmm, the graph didn't move as much on Linux. It did improve on macOS though, lol.

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&selected=3422675,1405391574&series=autoland,3383022,1,13&series=mozilla-central,3745676,1,13&series=mozilla-central,3422675,1,13&series=mozilla-central,3747146,1,13&timerange=5184000

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Too late for Fx92. Given that bug 1677324 is a Fission M8 bug, I guess we're going to have to live with it for a cycle.

Comment 14

[Emilio Cobos Álvarez (:emilio)]

Bug 1731132 should've helped here. Other than this kind of tweak I'm out of ideas.

Comment 15

[Olli Pettay [:smaug][[e-mail]]]

Wikia numbers look now rather good
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=mozilla-central,3422675,1,13&timerange=7776000

I think we could close this now. We'll keep looking at the performance numbers and optimized whatever we can.