1803500 - 30% regression on AWFY-Speedometer-Inferno around Nov-30

Status: RESOLVED FIXED

(Core :: Performance, defect)

Description

[Mayank Bansal]

https://arewefastyet.com/win10/benchmarks/raptor-desktop-speedometer?numDays=60

Inferno-TodoMVC: (https://treeherder.mozilla.org/perfherder/graphs?series=mozilla-central,263f17287307703ea55e2c8b3d2205c5f99036f7,1,13&timerange=5184000&series=mozilla-central,263f17287307703ea55e2c8b3d2205c5f99036f7,1,13&series=autoland,1cb6b97c615b2503a679487eb869d8f79ac18f5b,1,13)

Inferno-TodoMVC/DeletingItems: (https://treeherder.mozilla.org/perfherder/graphs?series=mozilla-central,a44649f51f610581a64733f462b40878bc1e97c2,1,13&timerange=5184000&series=mozilla-central,a44649f51f610581a64733f462b40878bc1e97c2,1,13&series=autoland,e4730f5098f85c84323fa78f46a391d53bd30b9e,1,13)

Inferno-TodoMVC/DeletingItems/Sync: (https://treeherder.mozilla.org/perfherder/graphs?series=mozilla-central,4cdff4455bae6e1585f0a0fa717a0ae7225d3190,1,13&timerange=5184000&series=mozilla-central,4cdff4455bae6e1585f0a0fa717a0ae7225d3190,1,13&series=autoland,74f2bb3486345372a6fb3267c7ae2d452b5e8c93,1,13)

Comment 1

[Tom S]

[Tracking Requested - why for this release]:

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Interestingly, I see an earlier spike on autoland to around the same level, and in both pushlog ranges I see bug 1790872 in there. Given that there's already another known perf regression from that bug (bug 1803798), it seems possible?

Comment 3

[Nika Layzell [:nika] (ni? for response)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #2)

Interestingly, I see an earlier spike on autoland to around the same level, and in both pushlog ranges I see bug 1790872 in there. Given that there's already another known perf regression from that bug (bug 1803798), it seems possible?

Given that I previously noticed a regression in Inferno-TodoMVC when there was another minor performance regression in IPC message sending performance due to changes to locking scheduling, unfortunately I can definitely believe that adding glean telemetry accumulation to IPC message sending could've caused this regression. I'll try to think of a way that we can reduce this overhead and keep the telemetry around, but there's definitely a chance we'll need to back it out.

On the bright side, we might not need to worry about 109, as the telemetry should be disabled outside of Nightly, meaning that beta's performance should be unimpacted.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

I'd already started the Try push before comment 3, but it at least does confirm that backing out bug 1790872 resolves the regression.
https://treeherder.mozilla.org/perfherder/comparesubtest?originalProject=mozilla-central&newProject=try&newRevision=0caff50b6cb6420c6e889f71b499917d3b6a449c&originalSignature=3412459&newSignature=3451139&framework=13&originalRevision=ced0b84fe0229e3a8a034466b7e836cf2dbec3bf&page=1&showOnlyComparable=1&showOnlyImportant=1

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox109 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:plawless, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 6

[Nika Layzell [:nika] (ni? for response)]

(In reply to Release mgmt bot [:suhaib / :marco/ :calixte] from comment #5)

The bug is marked as tracked for firefox109 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:plawless, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

As I mention in comment 3, this shouldn't impact beta/release as the new telemetry is nightly-only, however if we still need to back it out, we can. It'd be nice to get a bit of telemetry first from the nightly population before we back it out though.

I tried looking more at the telemetry profiles, and couldn't find anything obvious causing the slowdown, though likely the only possible cause is increased IPC message sending overhead due to the new telemetry.

I'll mark this as assigned to me for now, but if we want to do the backout, then feel free to take it.

Comment 7

[Florian Quèze [:florian]]

(In reply to Nika Layzell [:nika] (ni? for response) from comment #6)

I tried looking more at the telemetry profiles, and couldn't find anything obvious causing the slowdown, though likely the only possible cause is increased IPC message sending overhead due to the new telemetry.

Out of curiosity, how were the profiles generated? Were they from try, or from running things locally? Can you share the links?

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

Because this is Nightly only, the configuration is rare, so impact low.
But need to fix this anyhow, since this makes other performance work hard.

Comment 9

[Nika Layzell [:nika] (ni? for response)]

I'm getting bug 1790872 backed out, as some of the issues in other regressions will impact beta, and can't be easily fixed.

(In reply to Florian Quèze [:florian] from comment #7)

Out of curiosity, how were the profiles generated? Were they from try, or from running things locally? Can you share the links?

I clicked on the regression, found a run (e.g. before: https://treeherder.mozilla.org/jobs?repo=autoland&revision=ee38c5741bfe66889c3801d9daa5eb5e999de418&group_state=expanded&selectedTaskRun=PQLi3wG7ReCl2_8kheqTyA.0 and after: https://treeherder.mozilla.org/jobs?repo=autoland&revision=e4ac0dccafdddfa13e91dec0e852d001ca7c528e&group_state=expanded&selectedTaskRun=Id_FL64KQEuKcPXr5coqdQ.0), then clicked on the "Open in Firefox Profiler" buttons which were in the bottom section (before: https://share.firefox.dev/3HiYKnR, after: https://share.firefox.dev/3YbfB1S), and narrowed it down to the Inferno test. These aren't the specific ones which I used (not sure which ones I used), but are just two random ones I clicked on.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Fixed by backout.