1427858 - Off-train updates should get automated performance + correctness testing

Status: RESOLVED FIXED

(Toolkit :: General, defect, P3)

Description

[Florian Quèze [:florian]]

Bug 1418556 comment 55 is the second time I stumble on a startup profile showing a system add-on causing a meaningful change to startup order. The first time was bug 1378978 comment 100, and there was a massive performance regression there.

Let's figure out how to avoid doing more of this in the future, by running these system add-ons through automated performance tests.

Currently I mostly have Talos and the browser chrome mochitests in browser/base/content/test/performance/ in mind; there may be other tests worth running.

Comment 1

[Robert Helmer [:rhelmer]]

Clarifying summary that this is for system add-on updates, since the "built-in" system add-ons we ship with Firefox (./browser/extensions/) get all the usual testing of code in m-c.

Comment 2

[Robert Helmer [:rhelmer]]

Bianca and I were just working on this for a Shield study she is working on.

What she did was to add the proposed system add-on to ./browser/extensions in her local mozilla-central repo and hook it up in the moz.build files, then it can be pushed to tryserver and will be present on startup like any other built-in system add-on.

This works great in that xpcshell/mochitest/talos/etc. all run and produced useful results

I think we could follow this pattern in the meantime but it is a bit painful, having some more automated way to do this without requiring a local hg clone and hacking on the build system files would be ideal.

Comment 3

[Robert Helmer [:rhelmer]]

Using the approach from comment 2 will allow us to easily expand this to any sort of off-train updates - shield studies, system add-ons updates, etc. so updating summary to match.

The goal is to provide the same suite of tests that we use for normal features that are landed in mozilla-central to any sort of off-train updates that we ship.

Comment 4

[Mark Banner (:standard8)]

At one time, there was a discussion about providing some system for github based projects to be able to run the same suites/tests with Firefox modified for the project.

I think Dave Hunt may have been working on that, though I've not heard anything recently.

I know activity stream has already got the build to m-c -> run try build process working, and hooked up to their PR system I believe, so if there's nothing else, maybe that can be used as a starting point.

Comment 5

[Robert Helmer [:rhelmer]]

:bdanforth put a bunch of work into this which is documented here: https://docs.telemetry.mozilla.org/cookbooks/client_guidelines.html#testing-extensions