1575089 - A new Talos test that measures warm startup time with a number of real-world WebExtensions installed

Status: RESOLVED FIXED

(Testing :: Talos, task)

Description

[Mike Conley (:mconley) (:⚙️)]

Just as it says on the tin - during the setup of a warm startup Talos test, we're hoping to have a set of popular real-world add-ons installed and enabled. This is to catch regressions in our WebExtension initialization code.

esmyth is going to check with data science to generate the list of add-ons to use, and is also going to ensure that there aren't any licensing issues to deal with with the set that we use.

Hey rwood, if we were to pursue this, could we use a similar system that tp6 uses to store a snapshot of sites, but instead store snapshots of certain add-ons, which can be updated over time?

Comment 1

[Robert Wood [:rwood]]

(In reply to Mike Conley (:mconley) (:⚙️) (Catching up from PTO) from comment #0)

Just as it says on the tin - during the setup of a warm startup Talos test, we're hoping to have a set of popular real-world add-ons installed and enabled. This is to catch regressions in our WebExtension initialization code.

esmyth is going to check with data science to generate the list of add-ons to use, and is also going to ensure that there aren't any licensing issues to deal with with the set that we use.

Hey rwood, if we were to pursue this, could we use a similar system that tp6 uses to store a snapshot of sites, but instead store snapshots of certain add-ons, which can be updated over time?

Hey Mike, I can't see why not. A couple of options - zip up the web extensions and put them on a tooltool archive. Then provide the tooltool manifest and have talos download them when running the test. Or, use a github repository for the web extensions; do a github release, and use a taskcluster fetch task to grab the extensions in production. This is what we do now for some of our perf benchmarks [0].

FYI you can already tell a talos test to include a web extension in the browser on startup i.e. [1] it may take some talos code tweaking to point to a web extension downloaded instead of one already in central.

[0] https://github.com/mozilla/perf-automation/tree/master/benchmarks
[1] https://searchfox.org/mozilla-central/rev/428cf94f4ddfb80eebc6028023a7d89eb277791b/testing/talos/talos/test.py#155

Comment 2

[Mike Conley (:mconley) (:⚙️)]

(In reply to Robert Wood [:rwood] from comment #1)

Or, use a github repository for the web extensions; do a github release, and use a taskcluster fetch task to grab the extensions in production. This is what we do now for some of our perf benchmarks [0].

Thanks, rwood. I'd like to stay as consistent as possible with the other perf tests, so let's try that route to start.

Is there a good example I can work off of to set up the Taskcluster fetch task to download the addons from GitHub? I've been looking at https://hg.mozilla.org/mozilla-central/rev/6aff093ae16410a85cd4ed3a03579ca0a8e6b10e, but it's not clear to me when and where the decompression of the zip occurs... does the fetch task take care of that? Where does it put the files from the perspective of something like Talos?

Comment 3

[Robert Wood [:rwood]]

(In reply to Mike Conley (:mconley) (:⚙️) (Catching up from PTO) from comment #2)

Note that we also use the tooltool option in production too with Raptor - all of our tp6 mitmproxy page recordings are stored on tooltool. I have access to upload to tooltool so if you go that route instead and want an upload, just NI me on a bug and I can do the upload and provide you the manifest.

About fetch tasks, apologies I thought we were using fetch tasks with talos too - however now that I look into it, looks like we aren't - we're only using them with Raptor. With talos we just used tooltool with the benchmarks (which have since been ported to Raptor). In talos the tp5 pageset currently in production is using tooltool, i.e. [0][1], if you want an example there. Hope that helps!

[task 2019-08-21T12:24:49.851Z] 12:24:49 INFO - Downloading {'name': u'tp5n.zip', 'manifest': 'tp5n-pageset.manifest'} with tooltool...

[0] https://searchfox.org/mozilla-central/rev/03853a6e87c4a9405fce1de49e5d03b9e7a7a274/testing/mozharness/mozharness/mozilla/testing/talos.py#254

[1] https://searchfox.org/mozilla-central/source/testing/talos/tp5n-pageset.manifest

Comment 4

[Mike Conley (:mconley) (:⚙️)]

Attachment: talos-realworld-webextensions.zip

Comment 5

[Mike Conley (:mconley) (:⚙️)]

Alright, having done some initial poking at this, I agree that tooltool seems like a simpler approach. I've attached a zip of the first batch of add-ons we're hoping to test this way (we might decide to replace this set with a new set later on, but this will do for now).

Thanks!

Comment 6

[Robert Wood [:rwood]]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #5)

Alright, having done some initial poking at this, I agree that tooltool seems like a simpler approach. I've attached a zip of the first batch of add-ons we're hoping to test this way (we might decide to replace this set with a new set later on, but this will do for now).

Thanks!

Uploaded and emailed tooltool manifest to Mike.

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Thanks, received.

So I've got a local patch that modifies Talos to download and install the WebExtensions, and is prepared to run the about:home paint tests.

The problem is that at least two of these WebExtensions open up tabs after install to welcome the user, and do so in ways that touch the network - and we can't touch the network in any Talos test; this causes an intentional instant-crash.

My plan for now is to install yet another WebExtension alongside the real-world set to capture and block the outbound network requests before they occur.

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1575089 - Add a new Talos test that measures warm startup time with a number of real-world WebExtensions installed. r?rwood

Comment 9

[Mike Conley (:mconley) (:⚙️)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36f21e9ccef2493654b232b9bccade131f119beb

Comment 10

[Mike Conley (:mconley) (:⚙️)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=524ae918d5918d2f80c1cb5cc2d6350c6ab81a07

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Hey rwood, it looks like I've got the test running properly on Linux, but the job isn't showing up for macOS or Windows. Where did I go wrong in my Taskcluster talos.yml stuff?

https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=talos&revision=524ae918d5918d2f80c1cb5cc2d6350c6ab81a07

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/72edc19d4181
Add a new Talos test that measures warm startup time with a number of real-world WebExtensions installed. r=rwood

Comment 13

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/72edc19d4181

Comment 14

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/releases/mozilla-beta/rev/72edc19d4181

Comment 15

[Greg Mierzwinski [:sparky]]

:mconley, can you add a section to the wiki about this new test? https://wiki.mozilla.org/TestEngineering/Performance/Talos/Tests

I've created a bug for this: https://bugzilla.mozilla.org/show_bug.cgi?id=1583344
And I've proposed you as the owner, feel free to find someone else though.

Comment 16

[Rob Wu [:robwu]]

I coincidentally came across this bug and noticed that the request from comment 15 has been resolved in bug 1583344. So I'm dropping the needinfo flag to remove it from Mike's queue.