Align page-load tests under --browsertime with Raptor
Categories
(Testing :: Raptor, defect, P1)
Tracking
(firefox71 fixed)
| Tracking | Status | |
|---|---|---|
| firefox71 | --- | fixed |
People
(Reporter: nalexander, Assigned: rwood)
References
Details
Attachments
(2 files)
Raptor coldload tests sit for 30s (in the absence of Conditioned Profiles). To compare the mean and variance of Browsertime to Raptor, we need to align the harnesses to the best of our ability.
This means at least:
- Turning off video recording in Browsertime.
- Doing the same delays. I have WIP on this, like:
# HG changeset patch
# User Nick Alexander <nalexander@mozilla.com>
# Date 1568409208 25200
# Fri Sep 13 14:13:28 2019 -0700
# Node ID 9c2bffcd9d4dec04ec2e0c2bc56d391c97fe13d8
# Parent 7f1d1203bb86347e68c8ca8e96ede8279127d906
Start to align browsertime behaviour with Raptor behaviour.
diff --git a/testing/raptor/raptor/raptor.py b/testing/raptor/raptor/raptor.py
--- a/testing/raptor/raptor/raptor.py
+++ b/testing/raptor/raptor/raptor.py
@@ -397,6 +397,8 @@ class Browsertime(Perftest):
return ['--browser', 'firefox', '--firefox.binaryPath', binary_path]
def run_test(self, test, timeout):
+ if test.get('cold') is not True:
+ raise ValueError('Browsertime is only supported for cold load tests: {}'.format(test['name']))
self.run_test_setup(test)
@@ -408,6 +410,9 @@ class Browsertime(Perftest):
'--visualMetrics', 'false',
'-vv',
'--resultDir', self.results_handler.result_dir_for_test(test),
+ '--preURL=data:text/html;charset=utf-8,',
+ '--preURLDelay', str(self.post_startup_delay),
+ '--timeouts.pageLoad', str(timeout),
'-n', str(test.get('browser_cycles', 1)), test['test_url']])
# timeout is a single page-load timeout value in ms from the test INI
This may require some bits of Bug 1581560, I just don't know yet.
|
Reporter | |
Comment 1•5 years ago
|
||
Tarek: if warmload (Bug 1577905) isn't fruitful, feel free to start digging into this. I got distracted with other things and couldn't make much progress here today.
|
||
Updated•5 years ago
|
|
Assignee | |
Updated•5 years ago
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 2•5 years ago
|
||
This needs to be applied for both cold page-load and warm-page load (landing in Bug 1577905).
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 3•5 years ago
|
||
Trying out changes noted above, for cold page-load:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdfcd7c5d8e46fd46f053b26dccdc1844e349e63
|
|
Assignee | |
Comment 4•5 years ago
|
||
This patch will add the postStartupDelay (30 seconds) for both cold and warm page-load. For cold page-load it can be added as a preURLDelay as Nick noted above. For warm page-load it will be added to the browsertime_pageload.js which Tarek added in Bug 1577905. The pageCycleDelay was already added by Tarek in his warm-pageload patch in Bug 1577905. I don't see any other inconsistencies when it comes to deliberate delays.
|
|
Assignee | |
Comment 5•5 years ago
|
||
Trying out thus far for cold page-load (will rebase and finish the patch once Bug 1577905 is merged in central):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aef5292f1811f83f8f53ea3af04c303724eaa77d
|
|
Assignee | |
Comment 6•5 years ago
|
||
Another discrepancy I see with raptor-browsertime vs raptor (webext) currently in production, is browser preferences. The current raptor (webext) sets all of the following Firefox browser prefs:
https://searchfox.org/mozilla-central/source/testing/profiles/common/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/raptor/user.js
I don't see any browser preferences being set in our current raptor-browsertime implementation. We are creating the browser profile folder but I don't see the browser profile being passed into browsertime on the command line. From the browsertime documentation I see there is a --firefox.preference argument that takes a single key:value pref, and apparently that argument can be provided multiple times for multiple prefs (but that would be a huge list to implement the above prefs).
Nick, Tarek, Stephen, am I missing it, or is there a way to pass a browser profile into browsertime? I'd like to set the same prefs that the current production Raptor is using, so the comparison of page-load numbers is more fair. Thanks!
|
|
Reporter | |
Comment 7•5 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #6)
Another discrepancy I see with raptor-browsertime vs raptor (webext) currently in production, is browser preferences. The current raptor (webext) sets all of the following Firefox browser prefs:
https://searchfox.org/mozilla-central/source/testing/profiles/common/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/raptor/user.jsI don't see any browser preferences being set in our current raptor-browsertime implementation. We are creating the browser profile folder but I don't see the browser profile being passed into browsertime on the command line. From the browsertime documentation I see there is a
--firefox.preferenceargument that takes a singlekey:valuepref, and apparently that argument can be provided multiple times for multiple prefs (but that would be a huge list to implement the above prefs).Nick, Tarek, Stephen, am I missing it, or is there a way to pass a browser profile into browsertime? I'd like to set the same prefs that the current production Raptor is using, so the comparison of page-load numbers is more fair. Thanks!
Yes, you can provide a "profile template" which will be cloned and extended by browsertime. Note that browsertime also does some pref management so we might need additional work here.
If we're not already doing this, I don't think we can be fetching from mozproxy at all in automation, because the profile was what I expected was telling browsertime what proxy to use. I definitely tested that we were getting canned network results locally, so either I tested WIP that got changed or I'm forgetting the real mechanism in play.
In any case, we want to produce a single profile and clone it in all situations. (And we want to stop monkeying with it: right now we copy things out at various times, which is just a headache. Although now I only see Gecko profiler extraction, which is probably not worth doing differently.)
|
|
Assignee | |
Comment 8•5 years ago
|
||
Interesting - thanks Nick. The numbers on my latest try pushes (adding the 30 second post-startup delay for cold page load etc) of raptor-browsertime vs raptor-webext are significantly different; I was thinking it was the difference in prefs; perhaps raptor-browsertime in production is actually ultimately using live-sites and that's the difference.
|
|
Assignee | |
Comment 9•5 years ago
|
||
First patch will be for the post-startup-delay and timeouts. The prefs/profile work will be an additional patch.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 10•5 years ago
|
||
|
|
Assignee | |
Comment 11•5 years ago
|
||
|
|
Assignee | |
Comment 12•5 years ago
|
||
|
|
Assignee | |
Comment 13•5 years ago
|
||
Updated - fixed page_timeout, was sending it into the browsertime timeouts.pageLoad setting as seconds, corrected to ms.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c641fe94990e908546f2740352730c7e931d7bbd
|
||
Comment 14•5 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/5c30357debb5 Align page-load test delays and timeouts under --browsertime with Raptor r=nalexander,tarek
|
||
Comment 15•5 years ago
|
||
| bugherder | ||
|
|
Assignee | |
Comment 16•5 years ago
|
||
|
|
Assignee | |
Comment 17•5 years ago
|
||
|
|
Assignee | |
Comment 18•5 years ago
|
||
|
|
Assignee | |
Comment 19•5 years ago
|
||
|
|
Assignee | |
Comment 20•5 years ago
|
||
|
|
||
Comment 21•5 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/964c54671103 Have raptor-browsertime tests use the raptor browser profile/prefs r=nalexander
|
|
Assignee | |
Updated•5 years ago
|
|
||
Comment 22•5 years ago
|
||
| bugherder | ||
Description
•