1026246 - Cypress-like test branch for Gaia-Try

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

I need to be able to test mozharness patch changes in a real environment.

We have cypress for testing all other mozharness patches, but I need the same for Gaia-Try.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: add staging-gaia-try

This patch adds buildbot branch 'staging-gaia-try'.  It uses my user repo for both gaia-try and mozharness.  The majority of the patch is just copying each line of gaia-try and adding a corresponding one for staging-gaia-try.  This passes ./setup-masters.py -t.

While this branch is named 'staging-', it's not intended to duplicate the load of hg.m.o/integration/gaia-try, merely provide a place for people to test mozharness changes.


$ ./setup-master.py -t
INFO  - creating "bm01-tests1-linux32" master
INFO  - created  "bm01-tests1-linux32" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm01-tests1-linux32
INFO  - creating "bm51-tests1-linux64" master
INFO  - created  "bm51-tests1-linux64" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm51-tests1-linux64
INFO  - creating "bm69-tests1-windows" master
INFO  - created  "bm69-tests1-windows" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm69-tests1-windows
INFO  - creating "bm70-build1" master
INFO  - created  "bm70-build1" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm70-build1
INFO  - creating "bm75-try1" master
INFO  - created  "bm75-try1" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm75-try1
INFO  - creating "bm81-build_scheduler" master
INFO  - created  "bm81-build_scheduler" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm81-build_scheduler
INFO  - creating "bm81-tests_scheduler" master
INFO  - created  "bm81-tests_scheduler" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm81-tests_scheduler
INFO  - creating "bm88-tests1-tegra" master
INFO  - created  "bm88-tests1-tegra" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm88-tests1-tegra
INFO  - creating "bm89-tests1-panda" master
INFO  - created  "bm89-tests1-panda" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm89-tests1-panda
INFO  - creating "bm103-tests1-linux" master
INFO  - created  "bm103-tests1-linux" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm103-tests1-linux
INFO  - creating "bm106-tests1-macosx" master
INFO  - created  "bm106-tests1-macosx" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm106-tests1-macosx
INFO  - creating "bm109-tests1-windows" master
INFO  - created  "bm109-tests1-windows" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm109-tests1-windows
INFO  - creating "bm01-tests1-linux32-universal" master
INFO  - created  "bm01-tests1-linux32-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm01-tests1-linux32-universal
INFO  - creating "bm51-tests1-linux64-universal" master
INFO  - created  "bm51-tests1-linux64-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm51-tests1-linux64-universal
INFO  - creating "bm69-tests1-windows-universal" master
INFO  - created  "bm69-tests1-windows-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm69-tests1-windows-universal
INFO  - creating "bm70-build1-universal" master
INFO  - created  "bm70-build1-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm70-build1-universal
INFO  - creating "bm75-try1-universal" master
INFO  - created  "bm75-try1-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm75-try1-universal
INFO  - creating "bm88-tests1-tegra-universal" master
INFO  - created  "bm88-tests1-tegra-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm88-tests1-tegra-universal
INFO  - creating "bm89-tests1-panda-universal" master
INFO  - created  "bm89-tests1-panda-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm89-tests1-panda-universal
INFO  - creating "bm103-tests1-linux-universal" master
INFO  - created  "bm103-tests1-linux-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm103-tests1-linux-universal
INFO  - creating "bm106-tests1-macosx-universal" master
INFO  - created  "bm106-tests1-macosx-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm106-tests1-macosx-universal
INFO  - creating "bm109-tests1-windows-universal" master
INFO  - created  "bm109-tests1-windows-universal" master, running checkconfig
INFO  - TEST-PASS checkconfig OK for bm109-tests1-windows-universal
INFO  - TEST-SUMMARY: 22 tested, 0 failed

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: With mozharness tag

Sorry, forgot about the tag as well.  I don't want to do any development on production branch, even locally, to avoid the chance where I accidentally push a bunch of stuff to production

diff -u b/mozilla-tests/b2g_config.py b/mozilla-tests/b2g_config.py
--- b/mozilla-tests/b2g_config.py
+++ b/mozilla-tests/b2g_config.py
@@ -1637,6 +1637,7 @@
 BRANCHES['gaia-try']['repo_path'] = "integration/gaia-try"
 BRANCHES['staging-gaia-try']['repo_path'] = "users/jford_mozilla.com/gaia-try"
 BRANCHES['staging-gaia-try']['mozharness_repo'] = "https://hg.mozilla.org/users/jford_mozilla.com/mozharness"
+BRANCHES['staging-gaia-try']['mozharness_tag'] = "default"

 # new linux64_gecko tests as of gecko 32
 for name, branch in items_at_least(BRANCHES, 'gecko_version', 32):

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8441058 [details] [diff] [review]
With mozharness tag

Minusing for incompleteness.
http://mxr.mozilla.org/build/search?string=gaia-try

http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#3263 especially.  This is a hardcoded project scheduler to try to deal with the ugliness that is gaia try without taskcluster: find all the builders with 'gaia-try' in the name, and trigger them all with a scheduler named 'gaia-try'.  This is going to hit serious problems when we add a staging-gaia-try: even if we removed the scheduler name conflict, any change in either gaia-try or staging-gaia-try would try to trigger the builders of each branch.

The only reason this isn't hitting problems in test-masters.sh is you didn't enable it in B2G_PROJECTS:

>     'gaia-try': {
>         'enable_merging': False,
>     },
>+    'staging-gaia-try': {
>+        'enable_merging': False,
>+    },
> }
>
> PLATFORM_VARS = {}
>
> PROJECTS = {}
> B2G_PROJECTS = {
>     'gaia-try': {
>         'scripts_repo': 'https://hg.mozilla.org/build/mozharness',
>     },

It would belong here ^^

I recommend geting the triggering working locally with patched buildbot-configs and buildbotcustom.
Once you're able to get this working:

a) if we put this in production, you would also have to touch

http://mxr.mozilla.org/build/source/puppet/modules/buildmaster/templates/postrun-default.cfg.erb#25
http://mxr.mozilla.org/build/source/tools/buildfarm/maintenance/production-branches.json#199
https://hg.mozilla.org/webtools/tbpl/file/ed14868e13f9/js/Config.js#l59

b) if you're able to trigger jobs locally, I think you're actually no longer blocked by bug 1026246, since you can use that local dev master to test any gaia-try changes before they roll out to production.  That means if you take this approach, not only can you skip the above 3 patches, but you can also re-use the 'gaia-try' branch on your dev master, and just point the mozharness path/tag + repo_path to your user repos.  That means you also don't have to touch the scheduler portion in buildbotcustom.

I think (b) makes a lot more sense.  You're welcome to continue to pursue (a), but I think it's a lot more work.

Comment 4

[Aki Sasaki (not active)]

Is this a WONTFIX in favor of bug 1026721 ?

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Aki Sasaki [:aki] from comment #4)
> Is this a WONTFIX in favor of bug 1026721 ?

Bug 1026721 is a band-aid solution so that we can resolve bug 1025322.  The problem is that once we turn off Travis, we can't really afford to do this kind of testing with Gaia-Try any more.

Resolving 1026721 lowers the priority of this bug for me for the foreseeable future.

Comment 6

[Aki Sasaki (not active)]

Hopefully the foreseeable future lasts until taskcluster.

Comment 7

[Rail Aliiev [:rail]]

I believe we moved to in-tree referenced mozharness and will be moving mozharness to in-tree as well.