1838177 - 28.55 - 3.22% cnn-nav.landing LastVisualChange / ebay loadtime + 1 more (Linux) regression on Tue June 6 2023

Status: RESOLVED FIXED

(Core :: Web Speech, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 9458874d5cf930c2e5d987062ef101ea45290d18. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
29%	cnn-nav.landing LastVisualChange	linux1804-64-shippable-qr	cold fission webrender	4,074.17 -> 5,237.38
26%	ebay PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,337.58 -> 1,679.42	Before/After
3%	ebay loadtime	linux1804-64-shippable-qr	fission warm webrender	286.47 -> 295.70	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
94%	ebay LastVisualChange	macosx1015-64-shippable-qr	fission warm webrender	4,301.18 -> 255.26
92%	ebay LastVisualChange	linux1804-64-shippable-qr	fission warm webrender	4,321.07 -> 361.72	Before/After
89%	ebay SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	1,462.02 -> 159.25
88%	ebay SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	1,494.92 -> 173.52	Before/After
86%	ebay PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	935.35 -> 126.53
...	...	...	...	...	...
19%	ebay ContentfulSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	534.79 -> 431.83

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1432719

Comment 2

[:Gijs (he/him)]

:lissyx, are there profiler markers for speechd / webspeech at all that we can use to check if these webpages are deliberately invoking web speech APIs in some way? It seems surprising that the patch in question would have had any impact on performance...

Comment 3

[:gerard-majax]

no there are none i think, but we've already have had reports on this where pages are abusing speech synthesis and it triggers the notification. looking at the screenshots in the profile we do see them, so I guess we can just run the raptor tests without the notification enabled

Comment 4

[:gerard-majax]

https://treeherder.mozilla.org/jobs?repo=try&revision=2b27ebb8009f0b980f6ca562b065b6578205d333

• -33% on cnn-nav.landing LastVisualChange reported +29% above: https://treeherder.mozilla.org/perfherder/compare?originalProject=autoland&originalRevision=9458874d5cf930c2e5d987062ef101ea45290d18&newProject=try&newRevision=2b27ebb8009f0b980f6ca562b065b6578205d333&framework=13&page=1&filter=linux+cnn-nav.landing+LastVisualChange
• -70% on ebay PerceptualSpeedIndex reported +28% above: https://treeherder.mozilla.org/perfherder/compare?originalProject=autoland&originalRevision=9458874d5cf930c2e5d987062ef101ea45290d18&newProject=try&newRevision=2b27ebb8009f0b980f6ca562b065b6578205d333&framework=13&page=1&filter=linux+ebay+PerceptualSpeedIndex+opt+cold
• -2.5% on ebay loadtime reported +3% above: https://treeherder.mozilla.org/perfherder/compare?originalProject=autoland&originalRevision=9458874d5cf930c2e5d987062ef101ea45290d18&newProject=try&newRevision=2b27ebb8009f0b980f6ca562b065b6578205d333&framework=13&page=1&filter=linux+ebay+loadtime+opt+cold

Comment 5

[:Gijs (he/him)]

(In reply to Alexandre LISSY :gerard-majax from comment #3)

no there are none i think, but we've already have had reports on this where pages are abusing speech synthesis and it triggers the notification. looking at the screenshots in the profile we do see them,

Ah, OK, that's good confirmation then.

so I guess we can just run the raptor tests without the notification enabled

Well, that would fix the automated regressions reported, but also it makes the test not resemble what happens in the "real world", right? So real world pageload would still be slowed down.

Do we know how much of the overhead is to do with the visual notification and how much is the rest of the code? And/or would flipping the pref just turn off the visual part, but still run the rest of the web speech code?

Also, the regressions also affect macOS - is that expected? I was under the impression the primary usecase was on Linux where web speech can be unavailable for various reasons...

Comment 6

[:gerard-majax]

Attachment: Bug 1838177 - Never display speech dispatcher missing notification during raptor tests r?#perftest-reviewers!

Comment 7

[:gerard-majax]

(In reply to :Gijs (he/him) from comment #5)

(In reply to Alexandre LISSY :gerard-majax from comment #3)

no there are none i think, but we've already have had reports on this where pages are abusing speech synthesis and it triggers the notification. looking at the screenshots in the profile we do see them,

Ah, OK, that's good confirmation then.

so I guess we can just run the raptor tests without the notification enabled

Well, that would fix the automated regressions reported, but also it makes the test not resemble what happens in the "real world", right? So real world pageload would still be slowed down.

I have been told the slowdown is more related in how browsertime interprets data when we display a notification than anything else. the notification is not shown in content itself but chrome

you can see the perf compare links above we get back to normal values

Do we know how much of the overhead is to do with the visual notification and how much is the rest of the code? And/or would flipping the pref just turn off the visual part, but still run the rest of the web speech code?

yes, we still run everything we just dont display the notification

Also, the regressions also affect macOS - is that expected? I was under the impression the primary usecase was on Linux where web speech can be unavailable for various reasons...

this was confirmed as unrelated

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1f345f63ab5a
Never display speech dispatcher missing notification during raptor tests r=perftest-reviewers,AlexandruIonescu,eeejay

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/1f345f63ab5a

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #38691 (as of Thu, 15 Jun 2023 00:38:44 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
49%	cnn-nav.world SpeedIndex	linux1804-64-shippable-qr	cold fission webrender	410.52 -> 612.24
9%	cnn-nav.world PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	776.25 -> 848.66

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
49%	cnn-nav.landing PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	4,057.97 -> 2,072.75
30%	cnn-nav.landing LastVisualChange	linux1804-64-shippable-qr	cold fission webrender	5,336.54 -> 3,719.86
21%	buzzfeed PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,299.99 -> 1,028.02	Before/After
19%	buzzfeed LastVisualChange	linux1804-64-shippable-qr	cold fission webrender	2,258.51 -> 1,830.67	Before/After
16%	expedia PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,561.68 -> 1,305.79	Before/After
...	...	...	...	...	...
11%	cnn-nav.landing SpeedIndex	linux1804-64-shippable-qr	cold fission webrender	2,138.36 -> 1,897.20

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38691