Note: There are a few cases of duplicates in user autocompletion which are being worked on.

increase size of tst-emulator64-spot instance pool

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: kmoir, Assigned: vladC)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 1 obsolete attachment)

I've noticed recently that the wait times for the tst-emulator64-spot pool are quite long.  In watch_pending.cfg the global pool is set to 1300 with us-east-1 and us-west-2 set at 1000 each. Perhaps we could try increasing the size of the pool by 50%

Need to write patches to update range mozilla-tests/production_config.py and add new instance machines to slavealloc

This pool will need more capacity when we enable the remaining Android 4.3 debug tests in bug
Created attachment 8661246 [details] [diff] [review]
tst-emulator64-spot.patch

Created the patch to increase the tst-emulator64-spot
Attachment #8661246 - Flags: review?(kmoir)
Rail, why didn't we have a contiguous range of machines the last time we expanded the size of this pool. I recall there was a reason, but don't remember now.  Also, is there anything we need to change wrt the ip address pool assigned to this type of instance when we expand the pool size?
Flags: needinfo?(rail)
(Reporter)

Updated

2 years ago
Assignee: nobody → vlad.ciobancai
No need to extend the address pool. Just 3 steps similar to bug 1090568.
Flags: needinfo?(rail)
Thanks Rail.  Vlad, you also need a patch to add them to slavealloc as you can see for example in bug 1090568.  Please don't add them to slavealloc until we are ready to go with the other patches. Also, we usually add them to slavealloc as disabled at first.
(Reporter)

Updated

2 years ago
Attachment #8661246 - Flags: review?(kmoir) → review+
Created attachment 8661810 [details]
emulator.txt

vlad had problems with his computer today and was unable to write the patch to add the slaves to slavealloc so I went ahead and did it
I looked at configs/watch_pending.cfg but and I don't think we need to change the limits
It is 1300 globally, 1000 for each us-west-2 and us-east1 which is still under the total pool size of 600 instances
(Reporter)

Updated

2 years ago
Attachment #8661246 - Flags: checked-in+
FYI, we saw a bunch of alerts for usw2 and use1 from the buildbot master twistd logs (many more than this):

[sns alert] Wed 09:09:02 PDT buildbot-master120.bb.releng.use1.mozilla.com watch_twistd_log.py: Count: 69 | First instance: 2015-09-16 08:55:53-0700 | Most recent instance: 2015-09-16 09:00:00-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.134.44.20
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 155 | First instance: 2015-09-16 08:50:43-0700 | Most recent instance: 2015-09-16 08:59:58-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.184
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 153 | First instance: 2015-09-16 08:50:57-0700 | Most recent instance: 2015-09-16 08:59:57-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.195
I can see new jobs running on the expanded pool now
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
kmoir think we may need to increase the tst-emulator64 pool again
kmoir	http://builddata.pub.build.mozilla.org/reports/pending/pending.html
kmoir	we have the same pool size as we used to with the pandas
kmoir	now that most tests run on android 4.3 on emulators
kmoir	but they take longer and have more chunks
kmoir	plus these instances are used for other tests other than android 4.3
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Updated

2 years ago
Depends on: 1205409
Created attachment 8662265 [details] [diff] [review]
production_config.py_v2.patch

Created a new patch in order to increase the number of tst-emulator64-spot
Attachment #8662265 - Flags: review?(kmoir)
Created attachment 8662365 [details]
emulator.csv
Attachment #8662365 - Flags: review?(kmoir)
Created the patch emulator.csv to add the slaves to slavealloc
(Reporter)

Updated

2 years ago
Attachment #8662265 - Flags: review?(kmoir) → review+
Comment on attachment 8662265 [details] [diff] [review]
production_config.py_v2.patch

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)
Attachment #8662265 - Flags: review+ → review-
(Reporter)

Updated

2 years ago
Attachment #8662265 - Flags: review- → review+
Comment on attachment 8662365 [details]
emulator.csv

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)
Attachment #8662365 - Flags: review?(kmoir) → review-
Created attachment 8662916 [details]
emulator.csv
Attachment #8662365 - Attachment is obsolete: true
Attachment #8662916 - Flags: review?(kmoir)
(Reporter)

Updated

2 years ago
Attachment #8662916 - Flags: review?(kmoir) → review+
Vlad, you could add these machines to mysql using this tool
https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Adding_a_slave

The machines will be disabled by default until we do the other work to enable them

I have to add more masters in bug 1205409 first
(In reply to Kim Moir [:kmoir] from comment #16)
> Vlad, you could add these machines to mysql using this tool
> https://wiki.mozilla.org/ReleaseEngineering/Buildduty/
> Slave_Management#Adding_a_slave
> 
> The machines will be disabled by default until we do the other work to
> enable them
> 
> I have to add more masters in bug 1205409 first

I have imported the slaves with success
(Reporter)

Updated

2 years ago
Attachment #8662265 - Flags: checked-in+
I enabled the new instances in slavealloc after the reconfig had completed
(Reporter)

Updated

2 years ago
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.