905837 - Use a Project Branch for Gaia "Try-like" behavior

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[cmtalbert]

We need a mechanism by which Gaia developers can use git to check in potential fixes for failures that occur when running tests in buildbot automation. This mechansim should *NOT* land code on shipping repositories but should run the same automated tests that those shipping repositories run per push.

It's kind of like try for desktop, with some substantial differences (because until bug 868605 is fixed we can't even begin to approximate a more-or-less try scheme for Gaia.  But, Joduinn and I came up with a work around that just might get us 90% of the way there.)

This is what we'd like to do:
1. Create a project branch in hg for this mechanism
2. Create a branch on gaia's github repo that uses the same name as the project branch in hg
3. Mirror that gaia branch into this hg project branch repo
4. Wire up the pushbot so that commits to the gaia branch wind up in the project branch repo
5. Ensure that the same tests we run on b2g-inbound run on the project branch repo.

This way, we can have the following workflow:
1. Developer checks in a test patch to gaia's github repo using this special branch
2 The test patch is mirrored through the layers, winding up invoking the push bot and is committed to the hg project branch
3. The commit from the pushbot runs the same tests that the commit would have ran on b2g-inbound
4. After the test run is complete, (likely on some well-defined schedule) the hg project branch repo re-sets itself to the tip of b2g-inbound and gaia master. This way the branch tracks tip in order to make pushes relevant and to mostly ensure that each push is testing only the pushed code atop tip and that we don't have pushes that are affecting one another.

The ability of pushes to affect one another in this scenario is less than ideal, and that is why this isn't a true "try" system. However, this could serve as a temporary stand-in to give gaia developers try-like abilities, which will be extremely important as they begin to debug test failures that are happening in b2g-inbound. Without a system like this, the developers have no way to replicate the tests that run on b2g-inbound.

If we're going to do this, and make it successful, it needs to happen before the next b2g work week which starts on September 9. So this needs to be in production by then. (We'd like to showcase this at that work week in lieu of a true gaia-try system).

Comment 1

[Jonathan Griffin (:jgriffin)]

There's likely to be a big push at the work week to get gaia-integration-tests in TBPL, and some kind of try support will likely be key.

Comment 2

[Chris AtLee [:catlee]]

Clint and I just chatted about this on IRC. There's one major issue here which needs addressing: how to enable try-like behaviour in the gaia.git repository. It's not clear to me how to make this work with git.

I propose we create a separate repository for gaia-try just to keep the primary repository clean. Developers can add the new remote if they want to push to try.

In addition, we can improve on the proposal in comment 0 but not requiring the pushbot to update gaia.json in the project branch's gaia.json. I'm hoping we can make the workflow look something like this:

1. Developer pushes new code to gaia-try.git
2. vcs2vcs syncs to gaia-try.hg
3. releng infra picks up change from gaia-try.hg's pushlog and triggers test jobs
4. mozharness script downloads an existing b2g desktop build, and swaps out its gaia with the gaia just pushed to gaia-try, and then runs tests

We could use the latest nightly b2g desktop build for tests, or allow the developer to override which build to use with a 'gecko.json' file or similar.

I'm not sure how feasible it is to build gaia on its own, and then swap out the gaia included in the build with the new gaia. jgriffin, any ideas?

Comment 3

[Jonathan Griffin (:jgriffin)]

> I'm not sure how feasible it is to build gaia on its own, and then swap out the > gaia included in the build with the new gaia. jgriffin, any ideas?

It's quite easy to build gaia on its own and use that build in an arbitrary b2g desktop build.  I don't know how to package a new gaia build into the b2g desktop package, however.

For gaia-unit-tests, we already make a custom gaia build on the test slave and use that when running the tests.  We'd have to modify the mozharness script with a flag that tells it not to try and use gaia.json.

For other gaia test jobs (currently, only gaia-ui), we could add logic to the mozharness script that tells it to make its own gaia build rather than use the pre-packaged one.

Comment 4

[Jonathan Griffin (:jgriffin)]

The above requires that the gaia branch being used is mirrored to hg, since we don't have git available yet on the test slaves, AFAIK.

Comment 5

[Chris AtLee [:catlee]]

We can get git installed on the ubuntu VMs easily enough, but we would have no way to report results. TBPL is what's requiring us to use hg AIUI.

Comment 6

[Jonathan Griffin (:jgriffin)]

Ah good point.  Yes, we'll need an hg mirror to use for sendchanges and TBPL reporting, I guess.

Comment 7

[Chris AtLee [:catlee]]

Definitely needed for TBPL reporting. Not so much the sendchanges, and if we go with comment #2 we wouldn't be doing builds -> sendchange -> test anyway.

Comment 8

[cmtalbert]

Talking with James Lal, we discovered that what would really fit the bill here is a kind of pre-testing on a per pull request basis. We'd have the pull request trigger a test run in EC2, once the run completes (should be quick) it then posts results back into the PR. The logs would need to be stored in S3. (Putting that into the PR would be a mess). We'd need some way via the PR or the patch to designate the kind of gecko that we'd want to run on, but that can be figured out.

Catlee and I met to discuss this, and we already have support for this exact tooling (PR based, EC2 powered, responding to PR) in the Rust project. Releng is already maintaining that code. And given that we don't have a pushlog to help us keep track of what's happening on git when we overwrite one another on the master branch (which is how we keep things sane when using try on hg) this should simplify the idea of a gaia try greatly.

In order to fully achieve a gaia-try using this pre-test setup, we'd just need to put up a "gaia-try" github repo that developers could send PR's to, and have it operate the same way as the real one on gaia. (This way we don't fill up gaia with a bunch of trial PR's).

It's not entirely clear if we need a try repo if we have pre-testing either. So, let's keep pushing toward the pre-testing idea.

Comment 9

[james]

"This way we don't fill up gaia with a bunch of trial PR's"

I am confused by this? We basically want to have the same extract travis CI workflow but with the ability to scale out more tests that could run in parallel (so we can keep testing times _very_ low). 

Our current workflow looks like this:

1. make a pr to gaia
2. watch git commit status go green or red (github commit hooks)
3. if red add a new commit or rebase and force push
4. Repeat 2-3 until green then land.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting w/ctalbert:

Similar functionality is already in progress using bugzilla/auto-lander as a Q3 goal. This is making good progress, so can we use this instead? For details see bug#657828 and related bugs.

Comment 11

[cmtalbert]

Ok, after some meetings at b2g work week we have a different direction.

WONTFIXING and will open a new suite of bugs.

Comment 12

[cmtalbert]

See bug 914632 for the new approach