1923443 - Bimodal behaviour in cold startup tests around sept 22/sept 23

Status: RESOLVED INVALID

(Firefox for Android :: Performance, defect)

Description

[Kash Shampur [:kshampur] ⌚EST]

Steps to reproduce

1. looking at cold startup performance tests in CI since sept 22 onward

Expected behavior

unimodal

Actual behavior

bimodal

Device information

A55 devices in perf test (bitbar)

• Firefox version: Fenix nightly
• Android device model: A55
• Android OS version: 14

Any additional information?

perfherder graphs: (occurs in both mean and median metrics)

• cvne
• cmff
• cvns (less pronounced as cvne/cmff)

I believe this is the related revision range

• https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f47709f0fa48f6c03b361bebe4b2b334053e80ad&tochange=f4f1159d250a769d457738a3c8a63fe9922714ea
• https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=79fb6e8946e71ed155b771893d5ee9ca9c34e503&tochange=f47709f0fa48f6c03b361bebe4b2b334053e80ad

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

I'll attach some screenshots as well since the graph ranges shift over time and starting point may not be easy to find depending on when this gets investigated

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

Attachment: CVNS

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

Attachment: CMFF

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

Attachment: CVNE

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

Comment on attachment 9429694 [details]
CVNS

as mentioned, less pronounced in cvns test compared to cmff and cvne

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

from :mstange in the android-perf meeting:
"Random bug I found in the first range, probably unrelated: https://bugzilla.mozilla.org/show_bug.cgi?id=1919668"

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

I don't think I see it in Chrome startup tests, could it potentially be an experiment flag on Fenix?

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

Roger pointed out in this perfcompare, which is based off of code from sept 7, is still bimodal.

So likely some infra related changes and not code changes on sept 22/23?

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:kaya, could you have a look please?

For more information, please visit BugBot documentation.

Comment 10

[Kash Shampur [:kshampur] ⌚EST]

for this CMFF graph https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=mozilla-central,182793,1,15&timerange=2592000

I went through each mode for the past month-ish (sept 25th - oct 24th) and made note of the device number eg:

higher (slower) (in order of sept 26 - oct 24)

[9, 9, 1, 3, 3, 4, 28, 9, 23, 16, 25, 5, 16, 28, 24, 8, 3, 3, 17, 16, 32, 18, 51, 24, 5, 51, 26, 40, 40, 7, 49, 23, 13, 52, 13, 13, 20, 24, 23, 1, 49, 31, 58, 53, 28, 19] ->

unique set:
{1, 3, 4, 5, 7, 8, 9, 13, 16, 17, 18, 19, 20, 23, 24, 25, 26, 28, 31, 32, 40, 49, 51, 52, 53, 58}

lower (faster) (sept 25 - oct 24):

[28, 12, 34, 13, 32, 31, 13, 29, 14, 34, 10, 34, 30, 36, 38, 39, 31, 39, 49, 50, 44, 44, 15, 15, 57, 63, 58, 63, 55] ->

unique set:
{10, 12, 13, 14, 15, 28, 29, 30, 31, 32, 34, 36, 38, 39, 44, 49, 50, 55, 57, 58, 63}

---

The overlapping devices between the two populations are {32, 13, 49, 58, 28, 31}

initially I was expecting/hoping there would be no overlap, but if we look at when the overlaps occur... 32 was faster in the beginning of the time period, but became "slower", as did 13, 49, 28, and 31 (not sure about 58). So it could be possible that either Fx on these devices or the devices themselves are becoming "slow" over time (or something like that)?

Comment 11

[Jeff Boek [:boek]]

Hey Kash, are we still experiencing the bimodal behavior?

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

doesn't look bimodal like it did before. but there seems to be a bit of variance (separate issue probably)

Comment 13

[Kaya [:kaya]]

Hi Kash, should we close this ticket, then? What do you think the next steps should be here?

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

Yup I think we can probably close this now unless someone has objections. For tracking variance issue I mentioned in comment 12, we have Bug 1927748.
I'll close as Invalid since the bimodal issue doesn't seem to appear anymore. Feel free to change it to something else you think is more appropriate