Schedule desktop tests only on Holly

RESOLVED FIXED

Status

defect
RESOLVED FIXED
5 years ago
a year ago

People

(Reporter: jgriffin, Assigned: billm)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

(Reporter)

Description

5 years ago
Billm is using the holly twig for some work, and needs only desktop tests scheduled on it; Android and B2G tests should not be scheduled there.
Thinking a little more about this, I'd like to restrict the tests to mochitests and reftests. No talos, Cpp, XPCShell, Jetpack, or marionette.

Also, it would be nice if I could change this myself. Is there a place I can check code into that will affect the tests being run?
Right now it looks like no tests are being run at all on holly. Hopefully this bug can also fix that.
You'll need to add a new entry for holly here:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1724

If you end up doing it yourself, definitely get review from someone in releng though :). Though it's strange that nothing is running at all.. there might be something else that needs to be done.

Comment 5

5 years ago
Ok, first we need to change the builds.
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/project_branches.py#214

Remove the gecko_version override, since you probably want to mimic mozilla-central.
You probably want to keep branch_projects at [] to avoid building spidermonkey.
Set pgo_strategy to None (turn off pgo builds)
Remove the enable_nightly, create_snippet, create_partial lines (updates)
Add a 'lock_platforms': True
And make the 'platforms' dict look like this:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla/config.py#l1562
Of course, if you don't care about all those platforms, remove the ones you don't want to run.
If you don't want talos, add |'enable_talos': False,|

For limiting the tests, you'll do something like we did for Cedar here, except we are adding suites for Cedar:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l1794
You want to remove suites, but instead of a -= or += you can set each
BRANCHES['holly']['platforms'][PLATFORM][SLAVE_PLATFORM]['{opt,debug}_unittest_suites'] to the set of suites you care about.
The defaults are specified here:
http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l579
The slave platforms are listed in this block: http://hg.mozilla.org/build/buildbot-configs/file/f55936ddc584/mozilla-tests/config.py#l102

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques has hints on how to test your configs once you edit them.  I think you'll have to set up buildbot locally to get this to work, so it won't be doable out of the box, but maybe we can update the docs as we go.

Comment 6

5 years ago
I think this may work for test-masters.sh, but I hit an error during the pip install step, so may need debugging.

mkdir foo
cd foo
virtualenv venv
. venv/bin/activate
hg clone http://hg.mozilla.org/build/buildbot
hg clone http://hg.mozilla.org/build/buildbot-configs
hg clone http://hg.mozilla.org/build/buildbotcustom
hg clone http://hg.mozilla.org/build/tools
cd buildbot-configs
# this next step may need a munged master-pip.txt
pip install -r master-pip.txt
cd ../buildbot/master
hg up -r buildbot-0.8
python setup.py install
cd ../../buildbot-configs
export PYTHONPATH=../..:../../..:../../tools/lib/python:../../../tools/lib/python
./test-masters.sh

This worked for me on osx when I applied the below patch; maybe it'll work out of the box for you:
(venv)deathduck:/src/billm/buildbot-configs [16:27:26] (default)
900$ hg diff master-pip.txt
diff --git a/master-pip.txt b/master-pip.txt
--- a/master-pip.txt
+++ b/master-pip.txt
@@ -1,16 +1,15 @@
 Jinja2==2.5.5
 MySQL-python==1.2.3
 SQLAlchemy==0.6.4
-Twisted==10.1.0
+https://pypi.python.org/packages/source/T/Twisted/Twisted-12.1.0.tar.bz2#md5=f396f1d6f5321e869c2f89b2196a9eb5
 amqplib==0.6.1
 anyjson==0.3
 argparse==1.1
 carrot==0.10.7
-distribute==0.6.14
 pyOpenSSL==0.10
 pyasn1==0.0.11a
 pycrypto==2.3
 pytz==2011d
 wsgiref==0.1.2
 zope.interface==3.6.1
 MozillaPulse==0.80
Posted patch holly-changeSplinter Review
Thanks for all the help, Aki. I've tested this by running mozilla-tests/{b2g-,mobile-,}config.py. For some reason, b2g-config.py didn't even print anything about holly--before or after my patch--but I suspect it worked anyway.

I wasn't able to get mozilla/config.py to run properly. There were a couple Python errors. However, from everything you told me, it seems like these changes should be enough. If not, we can try again.
Assignee: nobody → wmccloskey
Status: NEW → ASSIGNED
Attachment #8394990 - Flags: review?(aki)

Comment 9

5 years ago
Comment on attachment 8394990 [details] [diff] [review]
holly-change

># HG changeset patch
># Parent a041bcdc32fde1d050238f6a6564e6eacc629945
># User Bill McCloskey <wmccloskey@mozilla.com>
>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>--- a/mozilla-tests/config.py
>+++ b/mozilla-tests/config.py
>@@ -1796,16 +1796,26 @@ BRANCHES['cedar']['platforms']['macosx64
> BRANCHES['cedar']['platforms']['macosx64']['mavericks']['debug_unittest_suites'] = UNITTEST_SUITES['debug_unittest_suites'][:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['debug_unittest_suites'] += REFTEST_OMTC[:]
> 
>+# Filter the tests that are enabled on holly.
Could you add "for bug 985718" in the comment for posterity?

Let me know if the builder list I attached looks good.  I think the patch looks good, thanks!
Attachment #8394990 - Flags: review?(aki) → review+
https://hg.mozilla.org/build/buildbot-configs/rev/bdb5fc2ab72e
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
We tend to leave these open til they've been merged+reconfiged.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 12

5 years ago
Looks like we're scheduling Android tests on Holly.  I believe we can (and should, to save resources) turn them off.  Can you confirm you don't need Android tests, Bill?
(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Looks like we're scheduling Android tests on Holly.  I believe we can (and
> should, to save resources) turn them off.  Can you confirm you don't need
> Android tests, Bill?

That's correct. How do you know we're scheduling them?
(Reporter)

Comment 14

5 years ago
Because all the Android tests are showing up at https://tbpl.mozilla.org/?tree=Holly
I've been at a workweek this week, so I haven't done any pushes since this landed. I'll be back tomorrow and do a push. We can see how well it works then :-).
It looks like Android got disabled, but we're still doing b2g builds.
https://tbpl.mozilla.org/?tree=Holly&rev=48ded9090d38

What do you think is happening, Aki?
Flags: needinfo?(aki)
(In reply to Bill McCloskey (:billm) from comment #16)
> It looks like Android got disabled, but we're still doing b2g builds.
> https://tbpl.mozilla.org/?tree=Holly&rev=48ded9090d38
> 
> What do you think is happening, Aki?

Looks like the 'holly' line in b2g_project_branches needs commenting out.
http://hg.mozilla.org/build/buildbot-configs/file/7586f9ebe407/mozilla/b2g_project_branches.py#l67

Could you add a "# disabled for bug 985718" comment above it, as well?
Flags: needinfo?(aki)
Posted patch disable-b2gSplinter Review
Sounds good, thanks.
Attachment #8398211 - Flags: review?(aki)

Updated

5 years ago
Attachment #8398211 - Flags: review?(aki) → review+
in production.
I'd also like to try to disable non-unified builds. It seems like a big waste of resources. (As an aside, is there any reason to do these on any project branches?)
Attachment #8398669 - Flags: review?(aki)
Comment on attachment 8398669 [details] [diff] [review]
disable-non-unified

Ideally we'd have a way to turn this off per-branch without this loop (if http://hg.mozilla.org/build/buildbotcustom/file/2018f46fa730/misc.py#l1055 checked the branch's enable_nonunified_build setting) but without that check I think this is all you can do.  I don't know the reasoning behind this build.
Attachment #8398669 - Flags: review?(aki) → review+
in production.
(Reporter)

Comment 25

5 years ago
This looks successful.  Can we close?
Yes, it looks like this is working. Thanks everyone.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.