1091066 - No way to test b2g-manifest changes

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

When a dev makes changes to b2g-manifests, they currently have no way to test this change prior to landing it, and such changes have caused tree closures. There is nothing like a b2g-manifests try repo.

Ideally, devs should be able to test such changes in isolation, and be sure that b2g bumper can cope with their changes, and that builds are not negatively impacted, before issuing a pull request.

Catlee pointed out that a possibility might be to have a travis job for b2g-manifests which installs and runs its own b2g bumper, with the final hg push disabled (which can already be disabled as a mozharness command line option).

My understanding is that since the manifests live in-tree, gecko or gaia try pushes can contain hand-crafted manifest files to validate the builds and tests of the resulting frozen manifests, if a dev wants to also test e.g. the device builds of a b2g-manifests change. In other words, the workflow would be something like:

1) hack the manifest files with frozen versions to simulate what the b2g bumper would do, if you had specified a branch name - and push these to gecko/gaia try
2) assuming your hacked manifest files built ok, implement the associated b2g-manifest change in your own fork of b2g-manifest, and push to github, to trigger travis to run. travis will then validate that b2g-manifest will not cause b2g bumper to fall over, and generates a suitable frozen manifest that you have already tested in try in step 1.

With the combination of these steps, b2g-manifest changes can be thoroughly tested by devs in isolation, before submitting a pull request to merge into mainline.

Apologies if I have fundamentally misunderstood any of the concepts above! :p

Comment 1

[Pete Moore [:pmoore][:pete]]

I'm working on a travis job for b2g-manifest that runs b2g bumper.

It's not finished yet, it is an iterative process of checking in changes, watching CI run, and then move on to next part, or fix previous part.

CI results while I am developing it can be monitored here:
https://travis-ci.org/petemoore/b2g-manifest/builds

Code is:
https://github.com/petemoore/build-mozharness/tree/bug1091084
https://github.com/petemoore/b2g-manifest
https://hg.mozilla.org/users/pmoore_mozilla.com/b2g-manifest-test

When I have finished setting the up the tests, I will attach patches to bug. Just providing links above for the curious for anyone that wants to follow progress etc.

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_mozharness_v1.patch

Prerequisite changes in mozharness to be able to run b2g bumper in b2g-manifest travis CI job.

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_b2g-manifest_v1.patch

Probably this should be a pull request, but in order that you can review side-by-side with the mozharness patch, attaching this as a patch too. Then when I roll out, I will first land mozharness changes, and once those have propagated I will issue a pull request for the b2g-manifest changes, and then enable travis on the mozilla-b2g account (assuming I have access to do that).

For your info, see my test runs, e.g.:
https://travis-ci.org/petemoore/b2g-manifest/builds/40332362

Comment 4

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8519161 [details] [diff] [review]
bug1091066_b2g-manifest_v1.patch

Review of attachment 8519161 [details] [diff] [review]:
-----------------------------------------------------------------

::: validate.sh
@@ +40,5 @@
> +cd ..
> +
> +echo "Installing mozharness..."
> +echo "Current directory: '$(pwd)'"
> +git clone -b bug1091084 https://github.com/petemoore/build-mozharness mozharness

whoops - I see already I left this in - will switch this to be https://github.com/mozilla/build-mozharness

Comment 5

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pete][:pmoore] from comment #4)
> > +git clone -b bug1091084 https://github.com/petemoore/build-mozharness mozharness
> 
> whoops - I see already I left this in - will switch this to be
> https://github.com/mozilla/build-mozharness

and of course master branch, not bug1091084 branch... sorry!

Comment 6

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8519159 [details] [diff] [review]
bug1091066_mozharness_v1.patch

Review of attachment 8519159 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_bumper.py
@@ +442,2 @@
>  
> +        self.fatal("Breaking early since treestatus reports tree %s is closed" % tree)

In hindsight, this should sys.exit(0) as before, since otherwise when there is a tree closure, we'll start getting nagios alerts for the stamp files being too old, since a failure will count as a problem.

So I'll also switch this back.

Comment 7

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_mozharness_v2.patch

Tweaked as per correction above (comment 6).

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_b2g-manifest_v2.patch

Tweaked as per correction above (comment 4 and comment 5).

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_b2g-manifest_v3.patch

Same as before, but now call explicit actions, rather than disabling explicit actions:

<    mozharness/scripts/b2g_bumper.py -c "${config_file}" -c "${B2GBUMPER_DIR}/travis-mozharness-config.py" --no-check-treestatus --no-commit-manifests --no-checkout-gecko --no-push

>    mozharness/scripts/b2g_bumper.py -c "${config_file}" -c "${B2GBUMPER_DIR}/travis-mozharness-config.py" --checkout-manifests --massage-manifests

And I removed the comment about the retries.

Comment 10

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_b2g-manifest_v4.patch

Removed some redundant code (e.g. no need to check out b2g-manifest with b2g bumper action, since travis has checked it out already).

Reduced the patch to the .travis.yml and a run_travis_tests.sh script.

Comment 11

[Pete Moore [:pmoore][:pete]]

Attachment: bug1091066_b2g-manifest_v5.patch

Whoooooops (as http://www.mrmen.com/characters/mr-clumsy/index.html would say) ....

I forgot to set the executable bits on the script - fixed.

Comment 12

[Pete Moore [:pmoore][:pete]]

I've landed this now.

Michael, are you able to enable the travis CI via instructions here:

https://github.com/mozilla-b2g/b2g-manifest/pull/246

Comment 13

[Pete Moore [:pmoore][:pete]]

Michael, please be aware this patch from bug 1097784 https://bugzilla.mozilla.org/attachment.cgi?id=8524579&action=edit needs to land before you enabling travis, otherwise the travis tests will fail.

Thanks!
Pete

Comment 14

[Pete Moore [:pmoore][:pete]]

Rail was able to enable the travis, so no action needed Michael. Thanks.

Tests passing!

Email sent to dev-b2g list:
https://groups.google.com/forum/#!topic/mozilla.dev.b2g/gEBcgxMukfI

Closing bug.

Pete

Comment 15

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pete][:pmoore] from comment #14)
> Tests passing!
https://travis-ci.org/mozilla-b2g/b2g-manifest/builds/41381280