1832059 - Add a layer to mozperftest to run mochitest tests

Status: RESOLVED FIXED

(Testing :: mozperftest, task, P1)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding a mochitest layer to mozperftest so that we can run mochitest tests as performance tests.

Comment 1

[Greg Mierzwinski [:sparky]]

We'll need to integrate this test, and it can be run locally with the following command: https://phabricator.services.mozilla.com/D177427

$ ./mach test --headless dom/serviceworkers/test/test_sw_register_performance.html

Comment 2

[Greg Mierzwinski [:sparky]]

./mach test is awesome :)

I got the logic for it integrated into a utility class for running functional tests, and I'm able to run the dom/serviceworkers/test/test_sw_register_performance.html locally, and capture and report the metrics. Here's a series of patches that does this: https://treeherder.mozilla.org/jobs?repo=try&revision=eea685baa9f90bc0e6519691f30005c3de97f5f9

I'm working on the CI integration now.

Comment 3

[Greg Mierzwinski [:sparky]]

I've managed to get a task running (but failing) for CI tasks here: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=30dca260c0db05477035017ce1bcb5af1e0a0453

The next step is to handle all the plumbing between perftest, and mochitest in CI. I want to try to build a decent adapter between mozharness test suites and our own. With something like that it'll make it easier to integrate other test suites in CI. For all test suites, it looks like we're going to have to implement something on our side to run the tests even though we use ./mach test locally. This is because of the TestResolver here, but I also suspect the we need a full build for the rest of the code to work.

For context, we have xpcshell integrated in CI (it's broken), and the way it's integrated is quite hacky: https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/test/xpcshell.py#80-173
It would be good to take a standardized approach to integrating these functional test suites.

An alternative idea I have for this is to take advantage of Taskcluster. We could use existing suites (e.g. browser-chrome-1) that run the test, and wrap them in mozperftest. This isn't hard to do as it would involve using a transformer to modify the command used, and the platform that the test runs on (from opt to shippable, and on hardware instead of VMs). An issue that comes up here is how to make sure that the mochitest test runs with all the correct settings (passing arguments to it, and ensuring it doesn't override them, or ignore them). For the profile, it's simple, but when it comes to using tools like mitmproxy, we need to be able to keep it running throughout the test. One other thing that I'm thinking is that this might be a band-aid to a larger issue which is that we don't have a standard way of hooking up existing mozharness test suites with newer ones that don't rely on mozharness.

Comment 4

[Greg Mierzwinski [:sparky]]

Update from the last comment. I spent a bit of time looking into getting mach test working in CI but it's a bit too dependent on the build right now. The alternative to building an adapter, which is to take advantage of existing taskcluster tasks, would likely be easier/quicker.

I decided to integrate mochitest directly into mozperftest instead, and I've managed to get a passing test run in CI now. The failure is because it's lacking the metrics parsing atm, but the test did produce results: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=894983957d63c2132619063e83db8b60b2873909

Next steps here are to finish off the metrics parsing, and perfherder data output, add some unit tests, then push some patches for these changes. I'm going to start with landing the linux CI tests, and then work on windows, and finally osx.

Comment 5

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1832059 - Modify ScriptInfo to handle mochitest html files. r?#perftest

This patch adds the ability to parse HTML files (mochitest test files) to ScriptInfo. This makes it possible to parse the perfMetadata from the mochitest tests which will provide mozperftest with default options, and metadata for our performance docs.

Comment 6

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1832059 - Add MachTestRunner in mach test for mozperftest integration. r?jmaher,ahal

This patch adds a MachTestRunner class to make it easier to import externally. It also adds a custom log handler option to mach test so we can parse the logs produced for performance metrics.

Depends on D190990

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1832059 - Add new methods for importing, and installing to mozperftest. r?#perftest

These methods are added as we need to install a requirement file for mochitest (marionette), and we also need to import a class directly from a file. The existing methods we have for importing, and installation don't work for these cases.

Depends on D190991

Comment 8

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1832059 - Add a test runner for functional tests to mozperftest. r?#perftest

This patch adds a FunctionalTestRunner class that can be used by all the layers to run ./mach test which lets us easily run all functional tests locally. It will be used for running mochitest tests, and in the future, it may replace parts of the xpcshell layer.

The runner can only be used locally, and only needs a test path provided. It will then run all the manifests that it finds that test in (but only running that single test). At the same time, it'll parse the logs produced by ./mach test with FunctionalTestProcessor to find all the metrics that were output. However, it does not parse them into a JSON format, that's left for the layers to handle.

Depends on D190992

Comment 9

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1832059 - Add a layer to run mochitest performance tests. r?#perftest

This patch adds a new layer that allows developers to run mochitest tests as performance tests directly. It provides the ability to run them in CI on linux, and locally through ./mach test using the FunctionalTestRunner. Some additional try mapping is also added for future tests. The remote run (CI) method uses the mochitest run_test_harness method directly, along with the mochitest argument parser.

Depends on D190993

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9bbd9c1e8afb
Modify ScriptInfo to handle mochitest html files. r=perftest-reviewers,AlexandruIonescu
https://hg.mozilla.org/integration/autoland/rev/dce1033262df
Add MachTestRunner in mach test for mozperftest integration. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/d5716aa9dd05
Add new methods for importing, and installing to mozperftest. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/eafb5db59a2c
Add a test runner for functional tests to mozperftest. r=perftest-reviewers,kshampur
https://hg.mozilla.org/integration/autoland/rev/f6e5fecee879
Add a layer to run mochitest performance tests. r=perftest-reviewers,kshampur

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/9bbd9c1e8afb
https://hg.mozilla.org/mozilla-central/rev/dce1033262df
https://hg.mozilla.org/mozilla-central/rev/d5716aa9dd05
https://hg.mozilla.org/mozilla-central/rev/eafb5db59a2c
https://hg.mozilla.org/mozilla-central/rev/f6e5fecee879