849002 - Create enough buildbot masters in ec2 to replace kvm-based masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

+++ This bug was initially created as a clone of Bug #847932 +++

We are killing off the buildbot masters running on kvm - see bug#846332 for details. 

We need to stand up replacement buildbot masters in AWS, split evenly between us-east-1 and us-west-2. An initial batch of 8 masters are being stood up in bug#847932 for real-life load-testing and side-by-side comparison with KVM perf. This will give us better idea of how many AWS VMs we need to replace the current set of KVM masters.

Comment 2

[Rail Aliiev [:rail]]

Attachment: add 22 masters

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 735341 [details] [diff] [review]
add 22 masters

Review of attachment 735341 [details] [diff] [review]:
-----------------------------------------------------------------

<stamp />

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 735341 [details] [diff] [review]
add 22 masters

http://hg.mozilla.org/build/tools/rev/4872fbfed2a8

Comment 5

[Rail Aliiev [:rail]]

Attachment: puppet nodes

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 735748 [details] [diff] [review]
puppet nodes

http://hg.mozilla.org/build/puppet/rev/6355249ba9a5

Comment 7

[Rail Aliiev [:rail]]

Attachment: slavealloc entries

Comment 8

[Rail Aliiev [:rail]]

Attachment: tools - enable them

I attached 2-4 slaves to each of masters and overnight results look OK to me:

http://buildbot-master59.srv.releng.use1.mozilla.com:8101/one_line_per_build
http://buildbot-master60.srv.releng.usw2.mozilla.com:8101/one_line_per_build
http://buildbot-master61.srv.releng.use1.mozilla.com:8001/one_line_per_build
http://buildbot-master62.srv.releng.use1.mozilla.com:8001/one_line_per_build
http://buildbot-master63.srv.releng.use1.mozilla.com:8001/one_line_per_build
http://buildbot-master64.srv.releng.usw2.mozilla.com:8001/one_line_per_build
http://buildbot-master65.srv.releng.usw2.mozilla.com:8001/one_line_per_build
http://buildbot-master64.srv.releng.usw2.mozilla.com:8001/one_line_per_build
http://buildbot-master67.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master68.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master69.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master70.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master71.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master72.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master73.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master74.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master75.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master76.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master77.srv.releng.use1.mozilla.com:8201/one_line_per_build
http://buildbot-master78.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master79.srv.releng.usw2.mozilla.com:8201/one_line_per_build
http://buildbot-master80.srv.releng.usw2.mozilla.com:8201/one_line_per_build

TODO: enable them in slavealloc and remove hard coded slave assignments.

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 736219 [details] [diff] [review]
tools - enable them

http://hg.mozilla.org/build/tools/rev/a222cfaeb83c

Comment 10

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #8)
> TODO: enable them in slavealloc and remove hard coded slave assignments.

Done as well.

Comment 11

[Rail Aliiev [:rail]]

Attachment: configs

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 736275 [details] [diff] [review]
configs

http://hg.mozilla.org/build/buildbot-configs/rev/427a0ba91030

Comment 13

[Rail Aliiev [:rail]]

I filed bug 867583 for tegra/panda masters - they are the only remaining masters on KVM.