985718 - Schedule desktop tests only on Holly

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Billm is using the holly twig for some work, and needs only desktop tests scheduled on it; Android and B2G tests should not be scheduled there.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Thinking a little more about this, I'd like to restrict the tests to mochitests and reftests. No talos, Cpp, XPCShell, Jetpack, or marionette.

Also, it would be nice if I could change this myself. Is there a place I can check code into that will affect the tests being run?

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Right now it looks like no tests are being run at all on holly. Hopefully this bug can also fix that.

Comment 3

[Andrew Halberstadt [:ahal]]

You'll need to add a new entry for holly here:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1724

If you end up doing it yourself, definitely get review from someone in releng though :). Though it's strange that nothing is running at all.. there might be something else that needs to be done.

Comment 4

[Jonathan Griffin (:jgriffin)]

We'll also need to update the tree here:

http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/project_branches.py#214

Comment 5

[Aki Sasaki (not active)]

Ok, first we need to change the builds.
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/project_branches.py#214

Remove the gecko_version override, since you probably want to mimic mozilla-central.
You probably want to keep branch_projects at [] to avoid building spidermonkey.
Set pgo_strategy to None (turn off pgo builds)
Remove the enable_nightly, create_snippet, create_partial lines (updates)
Add a 'lock_platforms': True
And make the 'platforms' dict look like this:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla/config.py#l1562
Of course, if you don't care about all those platforms, remove the ones you don't want to run.
If you don't want talos, add |'enable_talos': False,|

For limiting the tests, you'll do something like we did for Cedar here, except we are adding suites for Cedar:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l1794
You want to remove suites, but instead of a -= or += you can set each
BRANCHES['holly']['platforms'][PLATFORM][SLAVE_PLATFORM]['{opt,debug}_unittest_suites'] to the set of suites you care about.
The defaults are specified here:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l579
The slave platforms are listed in this block: http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l102

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques has hints on how to test your configs once you edit them.  I think you'll have to set up buildbot locally to get this to work, so it won't be doable out of the box, but maybe we can update the docs as we go.

Comment 6

[Aki Sasaki (not active)]

I think this may work for test-masters.sh, but I hit an error during the pip install step, so may need debugging.

mkdir foo
cd foo
virtualenv venv
. venv/bin/activate
hg clone http://hg.mozilla.org/build/buildbot
hg clone http://hg.mozilla.org/build/buildbot-configs
hg clone http://hg.mozilla.org/build/buildbotcustom
hg clone http://hg.mozilla.org/build/tools
cd buildbot-configs
# this next step may need a munged master-pip.txt
pip install -r master-pip.txt
cd ../buildbot/master
hg up -r buildbot-0.8
python setup.py install
cd ../../buildbot-configs
export PYTHONPATH=../..:../../..:../../tools/lib/python:../../../tools/lib/python
./test-masters.sh

This worked for me on osx when I applied the below patch; maybe it'll work out of the box for you:
(venv)deathduck:/src/billm/buildbot-configs [16:27:26] (default)
900$ hg diff master-pip.txt
diff --git a/master-pip.txt b/master-pip.txt
--- a/master-pip.txt
+++ b/master-pip.txt
@@ -1,16 +1,15 @@
 Jinja2==2.5.5
 MySQL-python==1.2.3
 SQLAlchemy==0.6.4
-Twisted==10.1.0
+https://pypi.python.org/packages/source/T/Twisted/Twisted-12.1.0.tar.bz2#md5=f396f1d6f5321e869c2f89b2196a9eb5
 amqplib==0.6.1
 anyjson==0.3
 argparse==1.1
 carrot==0.10.7
-distribute==0.6.14
 pyOpenSSL==0.10
 pyasn1==0.0.11a
 pycrypto==2.3
 pytz==2011d
 wsgiref==0.1.2
 zope.interface==3.6.1
 MozillaPulse==0.80

Comment 7

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: holly-change

Thanks for all the help, Aki. I've tested this by running mozilla-tests/{b2g-,mobile-,}config.py. For some reason, b2g-config.py didn't even print anything about holly--before or after my patch--but I suspect it worked anyway.

I wasn't able to get mozilla/config.py to run properly. There were a couple Python errors. However, from everything you told me, it seems like these changes should be enough. If not, we can try again.

Comment 8

[Aki Sasaki (not active)]

Attachment: holly_builder_list

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8394990 [details] [diff] [review]
holly-change

># HG changeset patch
># Parent a041bcdc32fde1d050238f6a6564e6eacc629945
># User Bill McCloskey <[e-mail]>
>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>--- a/mozilla-tests/config.py
>+++ b/mozilla-tests/config.py
>@@ -1796,16 +1796,26 @@ BRANCHES['cedar']['platforms']['macosx64
> BRANCHES['cedar']['platforms']['macosx64']['mavericks']['debug_unittest_suites'] = UNITTEST_SUITES['debug_unittest_suites'][:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['debug_unittest_suites'] += REFTEST_OMTC[:]
> 
>+# Filter the tests that are enabled on holly.
Could you add "for bug 985718" in the comment for posterity?

Let me know if the builder list I attached looks good.  I think the patch looks good, thanks!

Comment 10

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/build/buildbot-configs/rev/bdb5fc2ab72e

Comment 11

[Aki Sasaki (not active)]

We tend to leave these open til they've been merged+reconfiged.

Comment 12

[Jonathan Griffin (:jgriffin)]

Looks like we're scheduling Android tests on Holly.  I believe we can (and should, to save resources) turn them off.  Can you confirm you don't need Android tests, Bill?

Comment 13

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Looks like we're scheduling Android tests on Holly.  I believe we can (and
> should, to save resources) turn them off.  Can you confirm you don't need
> Android tests, Bill?

That's correct. How do you know we're scheduling them?

Comment 14

[Jonathan Griffin (:jgriffin)]

Because all the Android tests are showing up at https://tbpl.mozilla.org/?tree=Holly

Comment 15

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I've been at a workweek this week, so I haven't done any pushes since this landed. I'll be back tomorrow and do a push. We can see how well it works then :-).

Comment 16

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It looks like Android got disabled, but we're still doing b2g builds.
https://tbpl.mozilla.org/?tree=Holly&rev=48ded9090d38

What do you think is happening, Aki?

Comment 17

[Aki Sasaki (not active)]

(In reply to Bill McCloskey (:billm) from comment #16)
> It looks like Android got disabled, but we're still doing b2g builds.
> https://tbpl.mozilla.org/?tree=Holly&rev=48ded9090d38
> 
> What do you think is happening, Aki?

Looks like the 'holly' line in b2g_project_branches needs commenting out.
http://hg.mozilla.org/build/buildbot-configs/file/7586f9ebe407/mozilla/b2g_project_branches.py#l67

Could you add a "# disabled for bug 985718" comment above it, as well?

Comment 18

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: disable-b2g

Sounds good, thanks.

Comment 19

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/build/buildbot-configs/rev/064ad0c9c07e

Comment 20

[John Hopkins (:jhopkins)]

in production.

Comment 21

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: disable-non-unified

I'd also like to try to disable non-unified builds. It seems like a big waste of resources. (As an aside, is there any reason to do these on any project branches?)

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 8398669 [details] [diff] [review]
disable-non-unified

Ideally we'd have a way to turn this off per-branch without this loop (if http://hg.mozilla.org/build/buildbotcustom/file/2018f46fa730/misc.py#l1055 checked the branch's enable_nonunified_build setting) but without that check I think this is all you can do.  I don't know the reasoning behind this build.

Comment 23

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/build/buildbot-configs/rev/f249c447e1af

Comment 24

[John Hopkins (:jhopkins)]

in production.

Comment 25

[Jonathan Griffin (:jgriffin)]

This looks successful.  Can we close?

Comment 26

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Yes, it looks like this is working. Thanks everyone.