Activity-Stream, like an increasing number of Firefox projects, does primary development from a github repo, and expect to export to mozilla-central regularly. We’d really love to tighten the feedback loop to quickly see when a change in either Activity Stream code or m-c breaks the m-c tests. So we’re hoping to find a good solution to automatically run a sizable subset of the m-c tests (mochitests and talos at the very least, ideally more) with specific commits of our add-on installed against the latest nightly builds. I’ve put together a hacky solution for us temporarily as we green up our code for landing in mozilla-central. It works by exporting our code to a local copy of mozilla-central and pushing it to the pine project branch each time a PR gets merged to master, which then runs all the tests. I’ve talked to a few other folks (jlaster from devtools and mythmon from shield) who have different sorts of solutions for somewhat similar problems, and they sounded potentially interested in a clean long-term solution to this as well. I’ve CCed them and will let them speak for their own interests…
We've looked at this a bit on my team as well, it's something we'd like to get to for numerous projects as well (various system addons, test pilot experiments, etc.) Obviously the further we get with the whole "experiments -> full features" process we'll need to get this working well, should be something we nail down this year. The rough idea we had to prototype this was: 1. Add Taskcluster.yml to an experiments repo and connect repo to taskcluster via https://github.com/taskcluster/taskcluster-github 2. Use mozdownload to fetch latest Firefox release as needed 3. Build XPI locally using a standard build script in repo 4. Configure Firefox to run experiment and set any pref’s needed - Mochitest has the --install-extension option, but we also require signed XPIs for release. Marionette can also install some extensions, but we'll need to look into it. 5. Run the mochitest-browser-chrome suite and other suites (try them and see what happens and fix whatever doesn't work). Steps 2-4 would likely be in a test env set up script to run on the tc infrastructure, and then hopefully Step 5 works as normal. I'm hoping we have some time in Q2 to dive into this, but I'm curious, what have jlaster and mythmon have planned/done?
> Marionette can also install some extensions, but we'll need to look into it. Marionette allows you to install unsigned add-ons at runtime by bypassing the security checks in officially branded builds. We use this on try for nearly all test harnesses to install relevant test add-ons.
For Normandy's system add-on, we've hacked together a test runner in Taskcluster. It fetches a tarball of moz-central from Github, sets up a build environment, runs mach build, and then mach test on browser/extensions/shield-recipe-client. It doesn't run the full Firefox test suite. This whole process takes a bit more than an hour usually The test script that runs in Taskcluster is: https://github.com/mozilla/normandy/blob/c2e6d849f3265ce60bdcb9a73bd953d0a6696355/recipe-client-addon/bin/tc/test.sh