1809175 - Determine and fix issue causing Fenix to take over 3 minutes on startup in performance tests

Status: NEW

(Testing :: Raptor, defect, P2)

Description

[Greg Mierzwinski [:sparky]]

The Fenix perftests are taking an incredibly long amount of time to run (up to 60mins for a valid test). This seems to mainly be coming from the browser startup as it takes over 3 minutes at times which is very bad. This is where the majority of test time is being spent as we do 15 iterations where we startup the browser 15 times. At best, we are adding about 15 minutes to the test time assuming 1 minute for startup.

Comment 1

[Greg Mierzwinski [:sparky]]

See any of the raptor performance tests here: https://treeherder.mozilla.org/jobs?repo=fenix&revision=d077210eb810f4db336ffd8025e5c5dd91cbf6e7

Here's a direct link to a task that took nearly 60 minutes: https://treeherder.mozilla.org/jobs?repo=fenix&revision=d077210eb810f4db336ffd8025e5c5dd91cbf6e7&selectedTaskRun=d5HIX_SMRaawwCLe37Z9lQ.0

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

going back to nov 3:
https://treeherder.mozilla.org/jobs?repo=fenix&searchStr=bing&revision=61b7815e08994e2185ddb9225515ee11e5f7bd11

I looked at bing-fission and see pixel 2 @35 minutes and a51 @44 minutes (the 44 minutes seem to be what we run today still)

Comment 3

[Greg Mierzwinski [:sparky]]

The run times have been like this since we've started testing on them unfortunately and I think we can address a good chunk of our device capacity issues if we resolve this. I wonder if there's something we could do at bitbar that might help. I think the issue in bug 1808231 may be related to an increase in test times as well.

Comment 4

[Greg Mierzwinski [:sparky]]

I looked into the times some more on the tests before bug 1808231 and found that Pixel2 would take ~20-25 seconds to start the browser, and the A51 is around double that at 40-50s. At 15 iterations, this means that we are spending an extra 6 minutes on the A51 versus the P2.

Locally, I noticed that using a conditioned profile (not building the profile) caused a noticeable slowdown (it would double the startup time) so I've made a try run here to see if it's the same in CI. The same is happening for GeckoView, and Fenix locally so I'm testing GeckoView since Fenix is currently busted in CI: https://treeherder.mozilla.org/jobs?repo=try&revision=8b182192306246c82af811d87424d2d7d73d756b

Comment 5

[Greg Mierzwinski [:sparky]]

So this is pretty interesting. I've disabled the conditioned profiles on geckoview and the startup is now taking 10-15 seconds rather than 40-50 seconds: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=XyxxE-_pSemFmWH-HUU9Ow.0&revision=8b182192306246c82af811d87424d2d7d73d756b

Looking in browsertime, and the only thing that's different between the two runs is that we add a profile template path for our conditioned profile here: https://github.com/sitespeedio/browsertime/blob/883d73bb3bbc43cefd4ff4f46f90846173361181/lib/firefox/webdriver/builder.js#L64

Comment 6

[Greg Mierzwinski [:sparky]]

Update on this is that we should continue using the conditioned profile, but increase the startup delay 10-30s to allow for the browser to startup a bit.