The default bug view has changed. See this FAQ.

faster m-c test feedback loop for github-hosted repos

NEW
Unassigned

Status

Testing
General
23 days ago
5 days ago

People

(Reporter: dmose, Unassigned)

Tracking

Version 3
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

23 days ago
Activity-Stream, like an increasing number of Firefox projects, does primary development from a github repo, and expect to export to mozilla-central regularly.

We’d really love to tighten the feedback loop to quickly see when a change in either Activity Stream code or m-c breaks the m-c tests.

So we’re hoping to find a good solution to automatically run a sizable subset of the m-c tests (mochitests and talos at the very least, ideally more) with specific commits of our add-on installed against the latest nightly builds.

I’ve put together a hacky solution for us temporarily as we green up our code for landing in mozilla-central.  It works by exporting our code to a local copy of mozilla-central and pushing it to the pine project branch each time a PR gets merged to master, which then runs all the tests.

I’ve talked to a few other folks (jlaster from devtools and mythmon from shield) who have different sorts of solutions for somewhat similar problems, and they sounded potentially interested in a clean long-term solution to this as well.  I’ve CCed them and will let them speak for their own interests…

Comment 1

23 days ago
We've looked at this a bit on my team as well, it's something we'd like to get to for numerous projects as well (various system addons, test pilot experiments, etc.)

Obviously the further we get with the whole "experiments -> full features" process we'll need to get this working well, should be something we nail down this year.

The rough idea we had to prototype this was:

1. Add Taskcluster.yml to an experiments repo and connect repo to taskcluster via https://github.com/taskcluster/taskcluster-github
2. Use mozdownload to fetch latest Firefox release as needed
3. Build XPI locally using a standard build script in repo
4. Configure Firefox to run experiment and set any pref’s needed - Mochitest has the --install-extension option, but we also require signed XPIs for release. Marionette can also install some extensions, but we'll need to look into it.
5. Run the mochitest-browser-chrome suite and other suites (try them and see what happens and fix whatever doesn't work).

Steps 2-4 would likely be in a test env set up script to run on the tc infrastructure, and then hopefully Step 5 works as normal. 

I'm hoping we have some time in Q2 to dive into this, but I'm curious, what have jlaster and mythmon have planned/done?
> Marionette can also install some extensions, but we'll need to look into it.

Marionette allows you to install unsigned add-ons at runtime by bypassing the security checks in officially branded builds.  We use this on try for nearly all test harnesses to install relevant test add-ons.
For Normandy's system add-on, we've hacked together a test runner in Taskcluster. It fetches a tarball of moz-central from Github, sets up a build environment, runs mach build, and then mach test on browser/extensions/shield-recipe-client. It doesn't run the full Firefox test suite. This whole process takes a bit more than an hour usually

The test script that runs in Taskcluster is: https://github.com/mozilla/normandy/blob/c2e6d849f3265ce60bdcb9a73bd953d0a6696355/recipe-client-addon/bin/tc/test.sh
You need to log in before you can comment on or make changes to this bug.