1957454 - Graft newtab tests from mozilla-central into mozilla-beta (and eventually mozilla-release) and run them on changes to the newtab code with newtab XPI

Status: RESOLVED FIXED

(Firefox :: New Tab Page, task)

Description

[Mike Conley (:mconley) (:⚙️)]

To ensure that what we land is compatible for train-hopping, I'm hoping to add some special Taskcluster jobs that will run when any code changes under browser/extensions/newtab.

Specifically, for each change that occurs in browser/extensions/newtab, I'm hoping we can:

1. Take a clone of mozilla-beta, overwrite its browser/extensions/newtab/tests directory with the contents from central
2. Take the built newtab XPI from mozilla-central, and run those tests under browser/extensions/newtab on Beta, with the XPI installed.

When bug 1949511 reaches the release channel in May, I'd also like to do (1) and (2) with mozilla-release.

The rationale here is so that when changes land in newtab, we can determine relatively quickly if they'll somehow break if newtab performs a train-hop to beta and/or release. For example, suppose we make a silly mistake and land some code in Nightly's New Tab that relies on a Web API that is not on by default in Beta or Release. Hopefully we have test coverage for that, and hopefully, that test coverage will result in failures with the proposed jobs.

My aim is for these to eventually be Tier 1 jobs that will incur a backout if any of the tests perma-fail on beta or release, even if they're passing with flying colours on Nightly.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Hey ahal, do you have a sense of how difficult such jobs would be? I suspect this isn't something we've done before (run jobs on clones of beta / release during a central push). It should be relatively straight-forward for us to simulate what we want here with some scripts and try pushes, but the platonic ideal here is a Tier 1 job that runs automatically on changes to browser/extensions/newtab.

Comment 2

[Andrew Halberstadt [:ahal]]

Sorry for the late reply. The running on changes to browser/extensions/newtab part is trivial.

You're correct that you're off the beaten path with the whole "running on a mozilla-beta clone and overwriting certain files from central" thing. Still, I don't think it would be too hard to get working. I might suggest not using the built-in run-task checkouts (use run.checkout=false) and then set up the repo as part of the task's command (e.g command: setup-repo.sh && my-command).

I'm not really sure what to recommend vcs wise given the impending switch to Github. Mozilla-unified will still be around for quite awhile though, so if you need something working asap, I think it's fine to still use Mercurial.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1957454 - Add taskcluster / taskgraph configurations for running Nightly newtab tests and XPI on Beta and Release. r?jmaher!,#home-newtab-reviewers!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9aff80d014f7
Add taskcluster / taskgraph configurations for running Nightly newtab tests and XPI on Beta and Release. r=jmaher,pdahiya

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fbe71a25e4ab
Revert "Bug 1957454 - Add taskcluster / taskgraph configurations for running Nightly newtab tests and XPI on Beta and Release. r=jmaher,pdahiya" for causing py3 failures @ test_new_config.py

Comment 6

[amarc]

Backed out for causing py3 failures @ test_new_config.py

• Backout link
• Push with failures
• Failure Log

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/54320d046266
Add taskcluster / taskgraph configurations for running Nightly newtab tests and XPI on Beta and Release. r=jmaher,pdahiya

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1957454 - Add taskcluster / taskgraph configurations for running Nightly newtab tests and XPI on Beta and Release.

Original Revision: https://phabricator.services.mozilla.com/D251386

Comment 9

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/54320d046266

Comment 10

[Mike Conley (:mconley) (:⚙️)]

Mass moving from [hnt-trainhop] to [hnt-trainhop-project] for better JIRA book-keeping.