1537941 - Experiment with “unrelated load before settling”

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, enhancement, P2)

Description

[Robert Wood [:rwood]]

i.e. When Raptor starts up the browser/app, open a new tab, browse to an unrelated URL (like a recording of example.com), close the tab, then do the 'post-startup' settle (currently 30 seconds); and then proceed with the Raptor test as usual.

Comment 1

[Robert Wood [:rwood]]

Setting to P2 as currently blocked on Bug 1539144. If tabs.create won't be supported then another approach needs to be decided on. I tried loading 'example.com' in the same browser tab, settled for 30 seconds, then started the Raptor pageload of the amazon site, but the amazon numbers were even larger than before, guessing because it was using the same tab and had to unload the current page etc.

Comment 2

[Robert Wood [:rwood]]

Attachment: preloadcold.diff

Not for review, just some info:

Made a recording on android of 'www.example.com' using Raptor studio. Hacked Raptor locally (this patch + putting the recording .mp file in obj../testing/mozproxy) to preload 'www.example.com' first, leaves it open during the 30 second browser settle time, then loads the amazon test page and does the regular cold page-load.

Note: Using same browser tab for the preload and main test page as tabs.create is not yet supported in geckoview (Bug 1539144). Preloading this page didn't improve the data noise/stddev as compared to cold page-load without preloading (see emailed google sheet for detailed numbers).

Perhaps the noise wasn't reduced with the preloading because it is using the same tab. If tabs.create support is added to geckoview then I will retry, using a new tab for the preload page, then closing it and opening another new one for the main test page.

Comment 3

[Robert Wood [:rwood]]

Attachment: preload-2.diff

Not for review - just another update.

Applied this patch (and copied android-example.mp recording into obj../testing/mozproxy) and ran the cold page-load android test. This patch modifies Raptor cold page-load to:

• apply the same prefs that are used in browsertime perf

• preload example.com (from mitm recording) on browser / app startup; before the 30 second browser 'settle time'

• clear the js cache on each browser cycle

• re-use the same browser profile on each browser cycle

This test also included the latest changes to Raptor in Bug 1536758 (use add event listener instead of window.onload).

Added numbers to the Raptor Cold Page-Load google sheet.

Comment 4

[Robert Wood [:rwood]]

Looks like these changes (preloading www.example.com, using the browsertime prefs, clearing the js cache, using the same profile for all cold page-load browser cycles) doesn't consistently improve the stdev / data noise (see the 'Raptor Cold Page-Load on Android' google sheet I emailed previously).

If preloading doesn't improve things I wouldn't recommend we land that. Maybe next we could try increasing the browser/app post-startup 'settle time' (currently 30 seconds).

:andrew, :nalexander, any ideas?

Comment 5

[Andrew Creskey [:acreskey]]

Because the set of startup tasks is different on android vs Desktop I think it would be useful to test both.
To be honest, it my tests I only did desktop and compared with perfherder's noise metric:

These were the results from when I added the Browsertime prefs to Windows: the noise metric does look improved (39 jobs before and after)
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=f18b44050f379bd51fd44cd8df1345763ec02c5e&newProject=try&newRevision=4ae89090ee102ef13cb040fc780888f937ffe07c&framework=10

And another one that's recent (March) is the kinto task of downloading and importing intermediate certs (currently desktop only):
This comparison shows the impact of disabling that:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&newProject=try&newRevision=d44e2ef74336492e855df2d7302b71da792920c6&framework=10&selectedTimeRange=604800

This is the bug about experimenting with different settle times:
Bug 1536090

Comment 6

[Nick Alexander :nalexander [he/him]]

Clearing NI: we're using conditioned profiles these days.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Any further experimentation here would likely involve alternate conditioned profiles now. It's quite possible that we'd want to revisit this, but we can open a new bug for it as needed.