1204756 - increase size of tst-emulator64-spot instance pool

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

I've noticed recently that the wait times for the tst-emulator64-spot pool are quite long.  In watch_pending.cfg the global pool is set to 1300 with us-east-1 and us-west-2 set at 1000 each. Perhaps we could try increasing the size of the pool by 50%

Need to write patches to update range mozilla-tests/production_config.py and add new instance machines to slavealloc

This pool will need more capacity when we enable the remaining Android 4.3 debug tests in bug

Comment 1

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: tst-emulator64-spot.patch

Created the patch to increase the tst-emulator64-spot

Comment 2

[Kim Moir [:kmoir] ET]

Rail, why didn't we have a contiguous range of machines the last time we expanded the size of this pool. I recall there was a reason, but don't remember now.  Also, is there anything we need to change wrt the ip address pool assigned to this type of instance when we expand the pool size?

Comment 3

[Rail Aliiev [:rail]]

No need to extend the address pool. Just 3 steps similar to bug 1090568.

Comment 4

[Kim Moir [:kmoir] ET]

Thanks Rail.  Vlad, you also need a patch to add them to slavealloc as you can see for example in bug 1090568.  Please don't add them to slavealloc until we are ready to go with the other patches. Also, we usually add them to slavealloc as disabled at first.

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: emulator.txt

vlad had problems with his computer today and was unable to write the patch to add the slaves to slavealloc so I went ahead and did it

Comment 6

[Kim Moir [:kmoir] ET]

I looked at configs/watch_pending.cfg but and I don't think we need to change the limits
It is 1300 globally, 1000 for each us-west-2 and us-east1 which is still under the total pool size of 600 instances

Comment 7

[Amy Rich [:arr] [:arich]]

FYI, we saw a bunch of alerts for usw2 and use1 from the buildbot master twistd logs (many more than this):

[sns alert] Wed 09:09:02 PDT buildbot-master120.bb.releng.use1.mozilla.com watch_twistd_log.py: Count: 69 | First instance: 2015-09-16 08:55:53-0700 | Most recent instance: 2015-09-16 09:00:00-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.134.44.20
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 155 | First instance: 2015-09-16 08:50:43-0700 | Most recent instance: 2015-09-16 08:59:58-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.184
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 153 | First instance: 2015-09-16 08:50:57-0700 | Most recent instance: 2015-09-16 08:59:57-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.195

Comment 8

[Kim Moir [:kmoir] ET]

I can see new jobs running on the expanded pool now

Comment 9

[Kim Moir [:kmoir] ET]

kmoir think we may need to increase the tst-emulator64 pool again
kmoir	http://builddata.pub.build.mozilla.org/reports/pending/pending.html
kmoir	we have the same pool size as we used to with the pandas
kmoir	now that most tests run on android 4.3 on emulators
kmoir	but they take longer and have more chunks
kmoir	plus these instances are used for other tests other than android 4.3

Comment 10

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: production_config.py_v2.patch

Created a new patch in order to increase the number of tst-emulator64-spot

Comment 11

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: emulator.csv

Comment 12

[Vlad Ciobancai [:vladC] [:vciobancai]]

Created the patch emulator.csv to add the slaves to slavealloc

Comment 13

[Kim Moir [:kmoir] ET]

Comment on attachment 8662265 [details] [diff] [review]
production_config.py_v2.patch

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)

Comment 14

[Kim Moir [:kmoir] ET]

Comment on attachment 8662365 [details]
emulator.csv

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)

Comment 15

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: emulator.csv

Comment 16

[Kim Moir [:kmoir] ET]

Vlad, you could add these machines to mysql using this tool
https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Adding_a_slave

The machines will be disabled by default until we do the other work to enable them

I have to add more masters in bug 1205409 first

Comment 17

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Kim Moir [:kmoir] from comment #16)
> Vlad, you could add these machines to mysql using this tool
> https://wiki.mozilla.org/ReleaseEngineering/Buildduty/
> Slave_Management#Adding_a_slave
> 
> The machines will be disabled by default until we do the other work to
> enable them
> 
> I have to add more masters in bug 1205409 first

I have imported the slaves with success

Comment 18

[Kim Moir [:kmoir] ET]

I enabled the new instances in slavealloc after the reconfig had completed