985556 - Bump MAX_BROKER_REFS to 4096

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

This is going to suck quite a bit.

Options:
* Bump the limit
* Remove one/two project branches + disable b2g18 branches
* Split linux64 test masters by product
* Run b2g reftests on linux32 VMs

Any preference?

Comment 1

[Armen [:armenzg]]

Attachment: raise_limit.puppet.diff

Comment 2

[Armen [:armenzg]]

Attachment: raise_limit.buildbot.diff

Comment 3

[Armen [:armenzg]]

Attachment: raise_limit.bc.diff

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8393613 [details] [diff] [review]
raise_limit.buildbot.diff

I think you'll need to touch slavealloc's buildbot.tac template as well.

Comment 5

[Armen [:armenzg]]

I assume that I will have to land first on puppet, then on slavealloc and then on buildbot-configs.
At that point I should start rebooting the masters?
Or better to use the manhole?

Comment 6

[Armen [:armenzg]]

Attachment: raise_limit.tools.diff

Comment 7

[Armen [:armenzg]]

https://hg.mozilla.org/build/puppet/rev/de097c0090ea
http://hg.mozilla.org/build/buildbot/rev/7ce79514a42d
http://hg.mozilla.org/build/buildbot-configs/rev/410007f6c5e6
http://hg.mozilla.org/build/tools/rev/976ac92d2f78

I need to request slavealloc to update.
Tomorrow I will go around doing graceful restarts.

Comment 8

[Armen [:armenzg]]

Fix typo: https://hg.mozilla.org/build/buildbot-configs/rev/f187ef094c33

Comment 9

[Armen [:armenzg]]

Should I back out the buildbot-configs patch before someone increases the builder limit before tomorrow?

Comment 10

[Armen [:armenzg]]

FYI, I can see slavealloc giving the right value on buildbot.tac:
> twisted.spread.pb.MAX_BROKER_REFS = 4096

Comment 11

[Armen [:armenzg]]

I will be batching the masters like this:
https://etherpad.mozilla.org/X20KMNQsXP

Steps:
1) disable the masters on slavealloc
2) use manage_masters.py to gracefull_stop update_buildbot start (3 actions)
3) enable the masters on slavealloc

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #11)
> I will be batching the masters like this:
> https://etherpad.mozilla.org/X20KMNQsXP
> 
> Steps:
> 1) disable the masters on slavealloc
> 2) use manage_masters.py to gracefull_stop update_buildbot start (3 actions)
> 3) enable the masters on slavealloc

Two comments:
1) You should leave out bm01-06 -- they're not in production yet. I'll make sure they come up with the right stuff.
2) If you disable bm51 and 52 at the same time, bm67 will end up as the only master for that pool (use1 linux tests). I recommend against doing this - the master will probably grind to a halt or die.

Comment 13

[Armen [:armenzg]]

I updated the etherpad taking collocation into consideration.
If we had a way to disable/enable a master through slavealloc we could totally script this.

Comment 14

[Chris AtLee [:catlee]]

Do we know if this works via manhole? If so, then we can use fabric to make this change via manhole to all the masters.

Comment 15

[Armen [:armenzg]]

Comment on attachment 8393613 [details] [diff] [review]
raise_limit.buildbot.diff

Landed but not deployed.

Comment 16

[Armen [:armenzg]]

Comment on attachment 8393614 [details] [diff] [review]
raise_limit.bc.diff

Backed out until the masters are ready.

Comment 17

[Armen [:armenzg]]

catlee has updated the test masters without any issues so far.
He has used fabric to accomplish it.
I don't see anything out of the ordinary:
http://builddata.pub.build.mozilla.org/reports/pending/pending.html

Comment 18

[Armen [:armenzg]]

catlee deployed the rest.

Comment 19

[Armen [:armenzg]]

This is the code that catlee used:
https://github.com/catlee/tools/commit/0cb4977d7b081c2ef882a6676d947bef489a4b53

Comment 20

[Armen [:armenzg]]

Live in production.

Comment 21

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #7)
> http://hg.mozilla.org/build/buildbot/rev/7ce79514a42d

The buildbot commit only landed on default & hasn't been merged to the production branch. Does the production branch do anything in this repo? (I'm just trying to figure out what I need to do in bug 961075).

Comment 22

[Armen [:armenzg]]

Interesting situation.
It should have landed on production and the masters use production, however, we modified the masters live by using the manhole. That is why everything still worked as we intended to.
I will land it on the right place.

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #22)
> Interesting situation.
> It should have landed on production and the masters use production, however,
> we modified the masters live by using the manhole. That is why everything
> still worked as we intended to.
> I will land it on the right place.

Please make sure you run "update_buildbot" on the masters if you're landing to the production branch. Otherwise the code on disk won't match what's running in memory.

Comment 24

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #22)
> Interesting situation.
> It should have landed on production and the masters use production, however,
> we modified the masters live by using the manhole. That is why everything
> still worked as we intended to.

Ah! :-)

Comment 25

[Armen [:armenzg]]

I went into a meeting and I did not had a chance to do it.
I've updated all masters with update_buildbot.