Closed Bug 1575089 Opened Last month Closed 20 days ago

A new Talos test that measures warm startup time with a number of real-world WebExtensions installed

Categories

(Testing :: Talos, task)

Version 3
task
Not set

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: mconley, Assigned: mconley)

Details

Attachments

(2 files)

Just as it says on the tin - during the setup of a warm startup Talos test, we're hoping to have a set of popular real-world add-ons installed and enabled. This is to catch regressions in our WebExtension initialization code.

esmyth is going to check with data science to generate the list of add-ons to use, and is also going to ensure that there aren't any licensing issues to deal with with the set that we use.

Hey rwood, if we were to pursue this, could we use a similar system that tp6 uses to store a snapshot of sites, but instead store snapshots of certain add-ons, which can be updated over time?

Flags: needinfo?(rwood)

(In reply to Mike Conley (:mconley) (:⚙️) (Catching up from PTO) from comment #0)

Just as it says on the tin - during the setup of a warm startup Talos test, we're hoping to have a set of popular real-world add-ons installed and enabled. This is to catch regressions in our WebExtension initialization code.

esmyth is going to check with data science to generate the list of add-ons to use, and is also going to ensure that there aren't any licensing issues to deal with with the set that we use.

Hey rwood, if we were to pursue this, could we use a similar system that tp6 uses to store a snapshot of sites, but instead store snapshots of certain add-ons, which can be updated over time?

Hey Mike, I can't see why not. A couple of options - zip up the web extensions and put them on a tooltool archive. Then provide the tooltool manifest and have talos download them when running the test. Or, use a github repository for the web extensions; do a github release, and use a taskcluster fetch task to grab the extensions in production. This is what we do now for some of our perf benchmarks [0].

FYI you can already tell a talos test to include a web extension in the browser on startup i.e. [1] it may take some talos code tweaking to point to a web extension downloaded instead of one already in central.

[0] https://github.com/mozilla/perf-automation/tree/master/benchmarks
[1] https://searchfox.org/mozilla-central/rev/428cf94f4ddfb80eebc6028023a7d89eb277791b/testing/talos/talos/test.py#155

Flags: needinfo?(rwood)

(In reply to Robert Wood [:rwood] from comment #1)

Or, use a github repository for the web extensions; do a github release, and use a taskcluster fetch task to grab the extensions in production. This is what we do now for some of our perf benchmarks [0].

Thanks, rwood. I'd like to stay as consistent as possible with the other perf tests, so let's try that route to start.

Is there a good example I can work off of to set up the Taskcluster fetch task to download the addons from GitHub? I've been looking at https://hg.mozilla.org/mozilla-central/rev/6aff093ae16410a85cd4ed3a03579ca0a8e6b10e, but it's not clear to me when and where the decompression of the zip occurs... does the fetch task take care of that? Where does it put the files from the perspective of something like Talos?

Flags: needinfo?(rwood)

(In reply to Mike Conley (:mconley) (:⚙️) (Catching up from PTO) from comment #2)

Note that we also use the tooltool option in production too with Raptor - all of our tp6 mitmproxy page recordings are stored on tooltool. I have access to upload to tooltool so if you go that route instead and want an upload, just NI me on a bug and I can do the upload and provide you the manifest.

About fetch tasks, apologies I thought we were using fetch tasks with talos too - however now that I look into it, looks like we aren't - we're only using them with Raptor. With talos we just used tooltool with the benchmarks (which have since been ported to Raptor). In talos the tp5 pageset currently in production is using tooltool, i.e. [0][1], if you want an example there. Hope that helps!

[task 2019-08-21T12:24:49.851Z] 12:24:49 INFO - Downloading {'name': u'tp5n.zip', 'manifest': 'tp5n-pageset.manifest'} with tooltool...

[0] https://searchfox.org/mozilla-central/rev/03853a6e87c4a9405fce1de49e5d03b9e7a7a274/testing/mozharness/mozharness/mozilla/testing/talos.py#254

[1] https://searchfox.org/mozilla-central/source/testing/talos/tp5n-pageset.manifest

Flags: needinfo?(rwood)

Alright, having done some initial poking at this, I agree that tooltool seems like a simpler approach. I've attached a zip of the first batch of add-ons we're hoping to test this way (we might decide to replace this set with a new set later on, but this will do for now).

Thanks!

Flags: needinfo?(rwood)

(In reply to Mike Conley (:mconley) (:⚙️) from comment #5)

Alright, having done some initial poking at this, I agree that tooltool seems like a simpler approach. I've attached a zip of the first batch of add-ons we're hoping to test this way (we might decide to replace this set with a new set later on, but this will do for now).

Thanks!

Uploaded and emailed tooltool manifest to Mike.

Flags: needinfo?(rwood)

Thanks, received.

So I've got a local patch that modifies Talos to download and install the WebExtensions, and is prepared to run the about:home paint tests.

The problem is that at least two of these WebExtensions open up tabs after install to welcome the user, and do so in ways that touch the network - and we can't touch the network in any Talos test; this causes an intentional instant-crash.

My plan for now is to install yet another WebExtension alongside the real-world set to capture and block the outbound network requests before they occur.

Attachment #9087879 - Attachment description: Bug 1575089 - [WIP] A new Talos test that measures warm startup time with a number of real-world WebExtensions installed → Bug 1575089 - Add a new Talos test that measures warm startup time with a number of real-world WebExtensions installed. r?rwood

Hey rwood, it looks like I've got the test running properly on Linux, but the job isn't showing up for macOS or Windows. Where did I go wrong in my Taskcluster talos.yml stuff?

https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=talos&revision=524ae918d5918d2f80c1cb5cc2d6350c6ab81a07

Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/72edc19d4181
Add a new Talos test that measures warm startup time with a number of real-world WebExtensions installed. r=rwood
Status: NEW → RESOLVED
Closed: 20 days ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Assignee: nobody → mconley
You need to log in before you can comment on or make changes to this bug.