Schedule desktop tests only on Holly



5 years ago
a year ago


(Reporter: jgriffin, Assigned: billm)


(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)



(4 attachments)



5 years ago
Billm is using the holly twig for some work, and needs only desktop tests scheduled on it; Android and B2G tests should not be scheduled there.
Thinking a little more about this, I'd like to restrict the tests to mochitests and reftests. No talos, Cpp, XPCShell, Jetpack, or marionette.

Also, it would be nice if I could change this myself. Is there a place I can check code into that will affect the tests being run?
Right now it looks like no tests are being run at all on holly. Hopefully this bug can also fix that.
You'll need to add a new entry for holly here:

If you end up doing it yourself, definitely get review from someone in releng though :). Though it's strange that nothing is running at all.. there might be something else that needs to be done.

Comment 5

5 years ago
Ok, first we need to change the builds.

Remove the gecko_version override, since you probably want to mimic mozilla-central.
You probably want to keep branch_projects at [] to avoid building spidermonkey.
Set pgo_strategy to None (turn off pgo builds)
Remove the enable_nightly, create_snippet, create_partial lines (updates)
Add a 'lock_platforms': True
And make the 'platforms' dict look like this:
Of course, if you don't care about all those platforms, remove the ones you don't want to run.
If you don't want talos, add |'enable_talos': False,|

For limiting the tests, you'll do something like we did for Cedar here, except we are adding suites for Cedar:
You want to remove suites, but instead of a -= or += you can set each
BRANCHES['holly']['platforms'][PLATFORM][SLAVE_PLATFORM]['{opt,debug}_unittest_suites'] to the set of suites you care about.
The defaults are specified here:
The slave platforms are listed in this block: has hints on how to test your configs once you edit them.  I think you'll have to set up buildbot locally to get this to work, so it won't be doable out of the box, but maybe we can update the docs as we go.

Comment 6

5 years ago
I think this may work for, but I hit an error during the pip install step, so may need debugging.

mkdir foo
cd foo
virtualenv venv
. venv/bin/activate
hg clone
hg clone
hg clone
hg clone
cd buildbot-configs
# this next step may need a munged master-pip.txt
pip install -r master-pip.txt
cd ../buildbot/master
hg up -r buildbot-0.8
python install
cd ../../buildbot-configs
export PYTHONPATH=../..:../../..:../../tools/lib/python:../../../tools/lib/python

This worked for me on osx when I applied the below patch; maybe it'll work out of the box for you:
(venv)deathduck:/src/billm/buildbot-configs [16:27:26] (default)
900$ hg diff master-pip.txt
diff --git a/master-pip.txt b/master-pip.txt
--- a/master-pip.txt
+++ b/master-pip.txt
@@ -1,16 +1,15 @@
Posted patch holly-changeSplinter Review
Thanks for all the help, Aki. I've tested this by running mozilla-tests/{b2g-,mobile-,} For some reason, didn't even print anything about holly--before or after my patch--but I suspect it worked anyway.

I wasn't able to get mozilla/ to run properly. There were a couple Python errors. However, from everything you told me, it seems like these changes should be enough. If not, we can try again.
Assignee: nobody → wmccloskey
Attachment #8394990 - Flags: review?(aki)

Comment 9

5 years ago
Comment on attachment 8394990 [details] [diff] [review]

># HG changeset patch
># Parent a041bcdc32fde1d050238f6a6564e6eacc629945
># User Bill McCloskey <>
>diff --git a/mozilla-tests/ b/mozilla-tests/
>--- a/mozilla-tests/
>+++ b/mozilla-tests/
>@@ -1796,16 +1796,26 @@ BRANCHES['cedar']['platforms']['macosx64
> BRANCHES['cedar']['platforms']['macosx64']['mavericks']['debug_unittest_suites'] = UNITTEST_SUITES['debug_unittest_suites'][:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['opt_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['xp-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win7-ix']['debug_unittest_suites'] += REFTEST_OMTC[:]
> BRANCHES['cedar']['platforms']['win32']['win8']['debug_unittest_suites'] += REFTEST_OMTC[:]
>+# Filter the tests that are enabled on holly.
Could you add "for bug 985718" in the comment for posterity?

Let me know if the builder list I attached looks good.  I think the patch looks good, thanks!
Attachment #8394990 - Flags: review?(aki) → review+
Last Resolved: 5 years ago
Resolution: --- → FIXED
We tend to leave these open til they've been merged+reconfiged.
Resolution: FIXED → ---

Comment 12

5 years ago
Looks like we're scheduling Android tests on Holly.  I believe we can (and should, to save resources) turn them off.  Can you confirm you don't need Android tests, Bill?
(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Looks like we're scheduling Android tests on Holly.  I believe we can (and
> should, to save resources) turn them off.  Can you confirm you don't need
> Android tests, Bill?

That's correct. How do you know we're scheduling them?

Comment 14

5 years ago
Because all the Android tests are showing up at
I've been at a workweek this week, so I haven't done any pushes since this landed. I'll be back tomorrow and do a push. We can see how well it works then :-).
It looks like Android got disabled, but we're still doing b2g builds.

What do you think is happening, Aki?
Flags: needinfo?(aki)
(In reply to Bill McCloskey (:billm) from comment #16)
> It looks like Android got disabled, but we're still doing b2g builds.
> What do you think is happening, Aki?

Looks like the 'holly' line in b2g_project_branches needs commenting out.

Could you add a "# disabled for bug 985718" comment above it, as well?
Flags: needinfo?(aki)
Posted patch disable-b2gSplinter Review
Sounds good, thanks.
Attachment #8398211 - Flags: review?(aki)


5 years ago
Attachment #8398211 - Flags: review?(aki) → review+
in production.
I'd also like to try to disable non-unified builds. It seems like a big waste of resources. (As an aside, is there any reason to do these on any project branches?)
Attachment #8398669 - Flags: review?(aki)
Comment on attachment 8398669 [details] [diff] [review]

Ideally we'd have a way to turn this off per-branch without this loop (if checked the branch's enable_nonunified_build setting) but without that check I think this is all you can do.  I don't know the reasoning behind this build.
Attachment #8398669 - Flags: review?(aki) → review+
in production.

Comment 25

5 years ago
This looks successful.  Can we close?
Yes, it looks like this is working. Thanks everyone.
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.