1763457 - Add support for profiling full run in CI

Status: NEW

(Testing :: Raptor, enhancement, P3)

Description

[Steve Fink [:sfink] [:s:]]

Something I run into frequently:

I make a change to GC scheduling. It regresses some jobs and improves others, I get a performance alert, my change gets backed out.

I generate profiles for the relevant tests. When I look at the profiles, I can clearly see a difference in the measured values, but the cause of those differences doesn't really show up. I suspect that it's because I've shifted either an entire GC or some GC-induced behavior from within the timed portion to outside of it. These are extra annoying, because improvements can show up as regressions and vice versa.

But I can't see what's happening, because the profiles only show the 'cold' and 'warm' pageloads, and the time in between them (and before 'cold') is unprofiled.

I would like an option to collect a profile for the whole run: ending at the end of 'warm' and preferably starting at startup, but at least when 'cold' starts.

(Optimally, I'd also like markers showing what regions are timed as 'cold' and 'warm', but maybe that'll be obvious with the full profile.)

Examples:

• bug 1661293 caused bug 1758370
• bug 1692308 caused bug 1700313
• bug 1728273 caused bug 1729890

Comment 1

[Greg Mierzwinski [:sparky]]

This needs to be fixed in browsertime. We start and stop the profiler before and after each URL: https://github.com/sitespeedio/browsertime/blob/c13bf8099dd51a3e1b7f8a3629f383866fe95a01/lib/firefox/webdriver/firefox.js#L82-L91

We'd have to make an option that changes that to do full runs here: https://github.com/sitespeedio/browsertime/blob/c13bf8099dd51a3e1b7f8a3629f383866fe95a01/lib/firefox/webdriver/firefox.js#L53

If we could get test start, and end markers into the profiles, then we could make these full profiles the default (assuming they aren't that large).