712244 - increase or work around hitting MAX_BROKER_REFS on test masters (too many builders per slave problem)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Seen on talos-r3-xp-037 trying to talk to buildbot-master16:

2011-12-19 11:19:51-0800 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2011-12-19 11:19:51-0800 [Broker,client] While trying to connect:
        Traceback from remote host -- Traceback (most recent call last):
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause
            self._runCallbacks()
...
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 664, in registerReference
            raise Error("Maximum PB reference count exceeded.  "
        twisted.spread.pb.Error: Maximum PB reference count exceeded.  Goodbye.

Connected fine after a reboot, so tried to hit the master during a reconfig ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Also talos-r3-xp-039 at 
2011-12-19 11:18:52-0800 [Broker,client] While trying to connect:

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

And talos-r3-xp-057 slightly older, full stack:

2011-12-16 18:22:01-0800 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2011-12-16 18:22:01-0800 [Broker,client] While trying to connect:
        Traceback from remote host -- Traceback (most recent call last):
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause
            self._runCallbacks()
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 397, in _continue
            self.unpause()
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause
            self._runCallbacks()
        --- <exception caught here> ---
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 763, in serialize
            return jelly(object, self.security, None, self)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/jelly.py", line 1122, in jelly
            return _Jellier(taster, persistentStore, invoker).jelly(object)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/jelly.py", line 475, in jelly
            return obj.jellyFor(self)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/flavors.py", line 127, in jellyFor
            return "remote", jellier.invoker.registerReference(self)
          File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 664, in registerReference
            raise Error("Maximum PB reference count exceeded.  "
        twisted.spread.pb.Error: Maximum PB reference count exceeded.  Goodbye.

Comment 3

[Armen [:armenzg]]

jhford has a bug to add a couple more masters but it is blocked on IT to open firewalls for DB access (bug 708804).

Comment 4

[Armen [:armenzg]]

jhford mention those are for tegras. I am asking dustin for more masters.
Also CPU wio is happening on that master, which is not surprising.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage:

(In reply to Armen Zambrano G. [:armenzg] - (vactions from Dec. 24th & back on Jan. 9th) from comment #3)
> jhford has a bug to add a couple more masters but it is blocked on IT to
> open firewalls for DB access (bug 708804).

bug#708804 now done.

...and so too is bug#712398.

Comment 6

[Chris Cooper [:coop] (he/him)]

Once buildbot-master21 is created, we should use our scripts to set it up as a Windows test master, enable it in slavealloc, and let it start grabbing Windows slaves from buildbot-master16.

Comment 7

[Armen [:armenzg]]

I'll set it up in here.

Comment 8

[Armen [:armenzg]]

lsblakk, catlee mentioned that you're setting up some masters.
Do you have instructions on how to? I would like to try one myself.
Thanks!

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #8)
> lsblakk, catlee mentioned that you're setting up some masters.
> Do you have instructions on how to? I would like to try one myself.
> Thanks!

Armen, I just set up buildbot-master{22..27} following https://wiki.mozilla.org/ReleaseEngineering/Master_Setup and have added a few notes where I found it necessary.  Am currently finished everything on my end and just trying to figure out the best practice for filing the IT bugs required. I will make notes in that wiki page with whatever I discover.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

If buildbot-master16 is already windows only (win32+win64) and there really are too many builders, is there a plan to split win32 vs win64 ? Simply adding another master doesn't seem like it'll help any.

Comment 11

[Chris AtLee [:catlee]]

If we're hitting the PB limit, it's a problem with number-of-builders-per-slave, not number-of-slaves-per-master. So we need to reduce number of builders and/or bump the PB limit.

Comment 12

[Armen [:armenzg]]

Should we split the win64 slaves into a separate master?
or increase the PB limit?

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

I think that the limit to the reference count in Twisted is actually reasonable, so it may be time to look for ways to have fewer builders defined.  But you can monkey-patch it.  Details in the Twisted bug (#2045)

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

sorry, *buildbot* bug - http://trac.buildbot.net/ticket/2045

Comment 15

[Armen [:armenzg]]

Does this happen when we do a reconfig that adds more builders? and then have to backout?

I just don't know when this problem happens and how often.

Comment 16

[Armen [:armenzg]]

I will leave this bug open for someone else to determine what is the way forward.

Comment 17

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #11)
> If we're hitting the PB limit, it's a problem with
> number-of-builders-per-slave, not number-of-slaves-per-master. So we need to
> reduce number of builders and/or bump the PB limit.

* Do we actually know how many builders we have per slave? 
* Does the new PB ref limit need to be a power-of-2?
* Do we actually have a way to reduce the # of builders per slave? I assume this is a direct result of having (many project branches) x (many tests split into smaller parts)

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

It doesn't need to be a power of two.

Comment 19

[Chris AtLee [:catlee]]

(In reply to Chris Cooper [:coop] from comment #17)
> (In reply to Chris AtLee [:catlee] from comment #11)
> > If we're hitting the PB limit, it's a problem with
> > number-of-builders-per-slave, not number-of-slaves-per-master. So we need to
> > reduce number of builders and/or bump the PB limit.
> 
> * Do we actually know how many builders we have per slave? 

You can count it locally by editing master.cfg for a test master and running checkconfig.

> * Does the new PB ref limit need to be a power-of-2?
> * Do we actually have a way to reduce the # of builders per slave? I assume
> this is a direct result of having (many project branches) x (many tests
> split into smaller parts)

We'd have to consolidate builders e.g. doing all the different mochitest suites in the same builder, relying on runtime information to run the right suite. This also means tbpl would need updating since it looks at the builder name to determine what type of job something is.

Comment 20

[[e-mail] (:bhearsum)]

Updating summary.

Comment 21

[Chris Cooper [:coop] (he/him)]

Not sure why this is marked as [buildduty]. This seems like more work than buildduty could hope to tackle in a given week in addition to everything else.

Comment 22

[John Hopkins (:jhopkins)]

Unblocking bug 698843: with Thunderbird builders, the highest usage is talos-r3-fed-076 has 981 builders; limit is 1012; 96 percent of max.  Still, we have very little wiggle room for new builders.

Comment 24

[Aki Sasaki (not active)]

Attachment: update slavealloc's buildbot.tac template

Dustin:

I'm guessing here, but I think this will make slavealloc give all slavealloc-enabled slaves a MAX_BROKER_REFS of 2048.

Does it look right to you?

Comment 25

[Aki Sasaki (not active)]

Attachment: hack buildbot/master/buildbot/scripts/runner.py

This patch hacks the production-0.8 copy of hg.m.o/build/buildbot to have a [larger] hardcoded MAX_BROKER_REFS.

Aiui, this doesn't affect any existing masters, but will affect any new masters.

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 623878 [details] [diff] [review]
update slavealloc's buildbot.tac template

looks right to me, untested

Comment 27

[Aki Sasaki (not active)]

Did a quick'n'dirty test:

1. Started up buildmaster with

  import twisted.spread.pb
  twisted.spread.pb.MAX_BROKER_REFS = 2048

in the master's buildbot.tac.  Connected a linux64 slave with no MAX_BROKER_REFS update.  Forced a build; the slave picked it up and started building.  Killed the job.


2. Stopped the master + slave.  Commented out the MAX_BROKER_REFS lines on the master; added those lines to the slave.  Restarted both, forced a build.  The slave picked it up and started building.

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 623884 [details] [diff] [review]
hack buildbot/master/buildbot/scripts/runner.py

http://hg.mozilla.org/build/buildbot/rev/082cd6ddcb18

Landed on the production-0.8 branch.
Any new masters created should have this fix.
We still need to land+deploy to slavealloc, and manually update the existing masters.

Catlee: is there anything else I need to do re: the buildbot repo?

Comment 29

[Aki Sasaki (not active)]

I set up a new test master on dev-master01 via 

make -f Makefile.setup \
MASTER_NAME=preproduction-tests_master \
BASEDIR=/builds/buildbot/aki/test-master \
PYTHON=/usr/bin/python26 \
VIRTUALENV="/usr/bin/python26 /tools/misc-python/virtualenv.py" \
BUILDBOTCUSTOM_BRANCH=default \
BUILDBOTCONFIGS_BRANCH=default \
virtualenv deps install-buildbot master master-makefile

The buildbot.tac had the MAX_BROKER_REFS fix.

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 623878 [details] [diff] [review]
update slavealloc's buildbot.tac template

http://hg.mozilla.org/build/tools/rev/2afbe1c9176d

This needs to be deployed, per https://wiki.mozilla.org/ReleaseEngineering/Applications/Slavealloc#Deployment .

Comment 31

[Aki Sasaki (not active)]

Attachment: fix slavealloc script

When pointing at http://staging-puppet.build.mozilla.org/staging/python-packages/ , it dies when trying to find sqlalchemy.  Pointing it at repos/python/packages works.

Comment 32

[Aki Sasaki (not active)]

Updated slavealloc with the above script; http://slavealloc.build.mozilla.org/gettac/linux-ix-slave03 gives me a tac with the MAX_BROKER_REFS lines.

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 624766 [details] [diff] [review]
fix slavealloc script

This will only work in scl1 and mtv1 right now, but since slavealloc is in scl1, this looks good.

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 624766 [details] [diff] [review]
fix slavealloc script

http://hg.mozilla.org/build/tools/rev/acbaea8e273b

Comment 35

[Aki Sasaki (not active)]

Added the MAX_BROKER_REFS lines to existing enabled masters in production-masters.json.

Comment 36

[Aki Sasaki (not active)]

Attachment: increase builder limit to 2048 in master.cfg

This shouldn't land until we restart all our test masters, or fix via manhole.

Comment 37

[Aki Sasaki (not active)]

Attachment: up builder limit for build masters

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 624777 [details] [diff] [review]
increase builder limit to 2048 in master.cfg

http://hg.mozilla.org/build/buildbot-configs/rev/6da2ccee77a0

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 627374 [details] [diff] [review]
up builder limit for build masters

http://hg.mozilla.org/build/buildbot-configs/rev/45bff744e472

Comment 40

[Chris Cooper [:coop] (he/him)]

All the enabled masters (including bm34) have had this change deployed via the manhole.

Comment 41

[Aki Sasaki (not active)]

Thanks Coop!
-> RESO FIXED