847932 - Create 8 buildbot masters in ec2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Let's create 8 new buildbot masters, these ones will be in EC2.

2 build
4 test
2 try

split evenly between us-east-1 and us-west-2.

Comment 1

[Rail Aliiev [:rail]]

Attachment: [tools] add bm{51..54}

Add 4 linux tests slaves, disabled. Copy of bm17.

Comment 2

[Rail Aliiev [:rail]]

Attachment: [cloud-tools] add master instance type

* use domains from configs
* add master instance configs

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 723961 [details] [diff] [review]
[cloud-tools] add master instance type

http://hg.mozilla.org/build/cloud-tools/rev/c47c58f87eb7

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 723931 [details] [diff] [review]
[tools] add bm{51..54}

http://hg.mozilla.org/build/tools/rev/8ab60a319423

Comment 5

[Rail Aliiev [:rail]]

Attachment: [puppet] add bm5{1..4}

* added masters
* added node patters for the new domains

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 724044 [details] [diff] [review]
[puppet] add bm5{1..4}

Review of attachment 724044 [details] [diff] [review]:
-----------------------------------------------------------------

indentation looks a bit weird for the master nodes. can you fix that up when you land?

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 724044 [details] [diff] [review]
[puppet] add bm5{1..4}

http://hg.mozilla.org/build/puppet/rev/4f3a37df2fc7 with indentation fixed

Comment 8

[Rail Aliiev [:rail]]

To be done:
* add to slavealloc
* create instance_data.json 
* create a tarball with needed ssh keys
* create and assimilate instances (and probably fix the issues)
* https://wiki.mozilla.org/ReleaseEngineering/Master_Setup#IT-related_things
* https://wiki.mozilla.org/ReleaseEngineering/Master_Setup#Final_Verification
* attach a couple of slaves to test
* enable in production-master.json and slavealloc

Comment 9

[Rail Aliiev [:rail]]

Attachment: fix puppet domain patterns

forgot about "releng"

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 726162 [details] [diff] [review]
fix puppet domain patterns

http://hg.mozilla.org/build/puppet/rev/a5267cdc4944

Comment 11

[Rail Aliiev [:rail]]

Attachment: slavealloc entries for bm51-54

Comment 12

[Rail Aliiev [:rail]]

I added the masters to inventory.

Comment 13

[Rail Aliiev [:rail]]

No more blocked by bug 803823

Comment 14

[Rail Aliiev [:rail]]

Attachment: enable masters

... to make sure that they get reconfigs. They are still disabled in slavealloc though.

Comment 15

[Rail Aliiev [:rail]]

Attachment: tools - bm{55..58}

Comment 16

[Rail Aliiev [:rail]]

Attachment: puppet - add bm5{5..8}

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 734748 [details] [diff] [review]
enable masters

http://hg.mozilla.org/build/tools/rev/3922ebf25780

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 734776 [details] [diff] [review]
puppet - add bm5{5..8}

http://hg.mozilla.org/build/puppet/rev/a0715f1142f3

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 734770 [details] [diff] [review]
tools - bm{55..58}

http://hg.mozilla.org/build/tools/rev/c78e38407f3c

Comment 20

[Rail Aliiev [:rail]]

I disabled the last 4 masters in production-masters.json: http://hg.mozilla.org/build/tools/rev/59ebd84c147b

Comment 21

[Rail Aliiev [:rail]]

Attachment: slavealloc entries for bm55-58

Comment 22

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #20)
> I disabled the last 4 masters in production-masters.json:
> http://hg.mozilla.org/build/tools/rev/59ebd84c147b

Backout in  http://hg.mozilla.org/build/tools/rev/d3179ae8a2a4 - DNS is up and running!

Comment 23

[Rail Aliiev [:rail]]

The tests masters are in production now. I moved the slaves:

select * from pools where name = 'tests-aws-us-east-1-linux'; -- returns 47
select * from pools where name = 'tests-aws-us-west-2-linux'; -- returns 49

select count(*) from slaves where name like 'tst-linux%-ec2-0%' or name like 'tst-linux%-ec2-1%'; -- 298 (use1
select count(*) from slaves where name like 'tst-linux%-ec2-3%' or name like 'tst-linux%-ec2-4%'; -- 300 (usw2)

update slaves set poolid=47 where name like 'tst-linux%-ec2-0%' or name like 'tst-linux%-ec2-1%';
update slaves set locked_masterid=NULL where name like 'tst-linux%-ec2-0%' or name like 'tst-linux%-ec2-1%';
update slaves set notes=NULL where name like 'tst-linux%-ec2-0%' or name like 'tst-linux%-ec2-1%';;

update slaves set poolid=49 where name like 'tst-linux%-ec2-3%' or name like 'tst-linux%-ec2-4%';
update slaves set locked_masterid=NULL where name like 'tst-linux%-ec2-3%' or name like 'tst-linux%-ec2-4%';
update slaves set notes=NULL where name like 'tst-linux%-ec2-3%' or name like 'tst-linux%-ec2-4%';

Comment 24

[Rail Aliiev [:rail]]

2 build and 2 try masters are in production as well. Victory!