Closed Bug 1581990 Opened 2 months ago Closed 2 months ago

Align page-load tests under --browsertime with Raptor

Categories

(Testing :: Raptor, defect, P1)

Version 3
defect

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: nalexander, Assigned: rwood)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Raptor coldload tests sit for 30s (in the absence of Conditioned Profiles). To compare the mean and variance of Browsertime to Raptor, we need to align the harnesses to the best of our ability.

This means at least:

  1. Turning off video recording in Browsertime.
  2. Doing the same delays. I have WIP on this, like:
# HG changeset patch
# User Nick Alexander <nalexander@mozilla.com>
# Date 1568409208 25200
#      Fri Sep 13 14:13:28 2019 -0700
# Node ID 9c2bffcd9d4dec04ec2e0c2bc56d391c97fe13d8
# Parent  7f1d1203bb86347e68c8ca8e96ede8279127d906
Start to align browsertime behaviour with Raptor behaviour.

diff --git a/testing/raptor/raptor/raptor.py b/testing/raptor/raptor/raptor.py
--- a/testing/raptor/raptor/raptor.py
+++ b/testing/raptor/raptor/raptor.py
@@ -397,6 +397,8 @@ class Browsertime(Perftest):
         return ['--browser', 'firefox', '--firefox.binaryPath', binary_path]
 
     def run_test(self, test, timeout):
+        if test.get('cold') is not True:
+            raise ValueError('Browsertime is only supported for cold load tests: {}'.format(test['name']))
 
         self.run_test_setup(test)
 
@@ -408,6 +410,9 @@ class Browsertime(Perftest):
                 '--visualMetrics', 'false',
                 '-vv',
                 '--resultDir', self.results_handler.result_dir_for_test(test),
+                '--preURL=data:text/html;charset=utf-8,',
+                '--preURLDelay', str(self.post_startup_delay),
+                '--timeouts.pageLoad', str(timeout),
                 '-n', str(test.get('browser_cycles', 1)), test['test_url']])
 
         # timeout is a single page-load timeout value in ms from the test INI

This may require some bits of Bug 1581560, I just don't know yet.

Tarek: if warmload (Bug 1577905) isn't fruitful, feel free to start digging into this. I got distracted with other things and couldn't make much progress here today.

Flags: needinfo?(tarek)
Priority: -- → P1
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Flags: needinfo?(tarek)

This needs to be applied for both cold page-load and warm-page load (landing in Bug 1577905).

Depends on: 1577905
Summary: Align coldload tests under --browsertime with Raptor → Align page-load tests under --browsertime with Raptor
Blocks: 1583284

This patch will add the postStartupDelay (30 seconds) for both cold and warm page-load. For cold page-load it can be added as a preURLDelay as Nick noted above. For warm page-load it will be added to the browsertime_pageload.js which Tarek added in Bug 1577905. The pageCycleDelay was already added by Tarek in his warm-pageload patch in Bug 1577905. I don't see any other inconsistencies when it comes to deliberate delays.

Trying out thus far for cold page-load (will rebase and finish the patch once Bug 1577905 is merged in central):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aef5292f1811f83f8f53ea3af04c303724eaa77d

Another discrepancy I see with raptor-browsertime vs raptor (webext) currently in production, is browser preferences. The current raptor (webext) sets all of the following Firefox browser prefs:

https://searchfox.org/mozilla-central/source/testing/profiles/common/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/raptor/user.js

I don't see any browser preferences being set in our current raptor-browsertime implementation. We are creating the browser profile folder but I don't see the browser profile being passed into browsertime on the command line. From the browsertime documentation I see there is a --firefox.preference argument that takes a single key:value pref, and apparently that argument can be provided multiple times for multiple prefs (but that would be a huge list to implement the above prefs).

Nick, Tarek, Stephen, am I missing it, or is there a way to pass a browser profile into browsertime? I'd like to set the same prefs that the current production Raptor is using, so the comparison of page-load numbers is more fair. Thanks!

Flags: needinfo?(tarek)
Flags: needinfo?(stephen.donner)
Flags: needinfo?(nalexander)

(In reply to Robert Wood [:rwood] from comment #6)

Another discrepancy I see with raptor-browsertime vs raptor (webext) currently in production, is browser preferences. The current raptor (webext) sets all of the following Firefox browser prefs:

https://searchfox.org/mozilla-central/source/testing/profiles/common/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/raptor/user.js

I don't see any browser preferences being set in our current raptor-browsertime implementation. We are creating the browser profile folder but I don't see the browser profile being passed into browsertime on the command line. From the browsertime documentation I see there is a --firefox.preference argument that takes a single key:value pref, and apparently that argument can be provided multiple times for multiple prefs (but that would be a huge list to implement the above prefs).

Nick, Tarek, Stephen, am I missing it, or is there a way to pass a browser profile into browsertime? I'd like to set the same prefs that the current production Raptor is using, so the comparison of page-load numbers is more fair. Thanks!

Yes, you can provide a "profile template" which will be cloned and extended by browsertime. Note that browsertime also does some pref management so we might need additional work here.

If we're not already doing this, I don't think we can be fetching from mozproxy at all in automation, because the profile was what I expected was telling browsertime what proxy to use. I definitely tested that we were getting canned network results locally, so either I tested WIP that got changed or I'm forgetting the real mechanism in play.

In any case, we want to produce a single profile and clone it in all situations. (And we want to stop monkeying with it: right now we copy things out at various times, which is just a headache. Although now I only see Gecko profiler extraction, which is probably not worth doing differently.)

Flags: needinfo?(nalexander)

Interesting - thanks Nick. The numbers on my latest try pushes (adding the 30 second post-startup delay for cold page load etc) of raptor-browsertime vs raptor-webext are significantly different; I was thinking it was the difference in prefs; perhaps raptor-browsertime in production is actually ultimately using live-sites and that's the difference.

https://treeherder.mozilla.org/#/jobs?repo=try&author=rwood%40mozilla.com&fromchange=2e9379498d341118247ac1ea693f674f5cacf11e&tochange=acc9b90434f240526baec893ff695ab6ad14c3ca

Depends on: 1585330

First patch will be for the post-startup-delay and timeouts. The prefs/profile work will be an additional patch.

Keywords: leave-open
Flags: needinfo?(tarek)
Flags: needinfo?(stephen.donner)

Updated - fixed page_timeout, was sending it into the browsertime timeouts.pageLoad setting as seconds, corrected to ms.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c641fe94990e908546f2740352730c7e931d7bbd

Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5c30357debb5
Align page-load test delays and timeouts under --browsertime with Raptor r=nalexander,tarek
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/964c54671103
Have raptor-browsertime tests use the raptor browser profile/prefs r=nalexander
Keywords: leave-open
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
You need to log in before you can comment on or make changes to this bug.