1393786 - [tracking] AWS tcmigration cleanup

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P4)

Description

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

We need to see if we can do some cleanup for AWS to better manage the infrastructure that we have here.

We can  maybe disable some of them from the bellow pools:
- 590 t-w732-spot machines 
-200 g-w732-spot
-200 tst-linux64-spot
-300 tst-linux32-spot

And I don't know if we should also reduce the number of masters.

select masters.nickname, count(*), masters.enabled from slaves, masters where masters.nickname like '%windows%' and slaves.current_masterid = masters.masterid group by masters.nickname;
+----------------------+----------+---------+
| nickname             | count(*) | enabled |
+----------------------+----------+---------+
| bm128-tests1-windows |      134 |       1 |
| bm129-tests1-windows |      134 |       1 |
| bm137-tests1-windows |      133 |       1 |
| bm138-tests1-windows |      133 |       1 |
| bm139-tests1-windows |      133 |       1 |
| bm140-tests1-windows |      133 |       1 |
+----------------------+----------+---------+

Comment 1

[Kim Moir [:kmoir] ET]

Well, I think the question is what percentage of the AWS test pools are you going to reduce looking at the number of buildbot jobs that run on a daily basis, and reduce the number of masters by the same proportion.

https://secure.pub.build.mozilla.org/buildapi/reports/pushes

There aren't many pushes on windows on buildbot on AWS anymore on than on m-r, mozilla-esr52

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Note explaining the priority level: P4 doesn't mean we've lowered the priority, but the contrary. However, we're aligning these levels to the buildduty quarterly deliverables, where P1-P3 are taken by our daily waterline KTLO operational tasks.

Comment 3

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1393786_build.patch

Since we did some cleanup on AWS machines ([1]) and we removed from databases most of those machines that were disabled I think it's time to also lower the limit of how many spot instances are been created on these pools.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1393774#c10

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

This is the pull request[1] https://github.com/mozilla-releng/build-cloud-tools/pull/324
Seems like it failed on Travis but it also fails when I tested the previous commit from 2 November [2]

Could it be that we have the configs changed for Travis since the last commit (check [2])?

[1] https://github.com/mozilla-releng/build-cloud-tools/pull/324
[2] https://travis-ci.org/andrei987/build-cloud-tools/jobs/312436713

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Most likely all further pushes on build-cloud-tools will not pass Travis because of this.
I don't have merge rights on this repo so Rail if you think that my change will not harm the repo please merge it to production.

Comment 6

[Rail Aliiev [:rail]]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #4)
> This is the pull request[1]
> https://github.com/mozilla-releng/build-cloud-tools/pull/324
> Seems like it failed on Travis but it also fails when I tested the previous
> commit from 2 November [2]
> 
> Could it be that we have the configs changed for Travis since the last
> commit (check [2])?
> 
> [1] https://github.com/mozilla-releng/build-cloud-tools/pull/324
> [2] https://travis-ci.org/andrei987/build-cloud-tools/jobs/312436713

I fixed the issues, please rebase/merge.

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

And in production:
https://github.com/mozilla-releng/build-cloud-tools/commit/fc79bb0c6a1e3985f50a61b8ea389d4a123f907b

thank you Rail.