1438236 - Decommission proxxy instances

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Chris AtLee [:catlee]]

Proxxy isn't used any more.

We should:
1) Stop the instances for 48h to ensure that automation (especially ESR52) can continue to function without it

2) Delete the AWS instances

3) Delete the puppet code to support them.

Comment 1

[Chris AtLee [:catlee]]

Instances were stopped earlier today.

Comment 2

[Chris AtLee [:catlee]]

This caused some failures on ESR52:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr52&revision=457ddb94a90cb2e190f36c90fc05775baf8b0c5a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable

I've started the instances again, and will retrigger some failed jobs.

Comment 3

[Intermittent Failures Robot]

27 failures in 685 pushes (0.039 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr52: 27

Platform breakdown:
* linux32: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1438236&startday=2018-02-12&endday=2018-02-18&tree=all

Comment 4

[Chris AtLee [:catlee]]

A fix has landed on ESR52. I've stopped the proxxy instances in AWS again.

Comment 5

[Chris AtLee [:catlee]]

Attachment: Bug 1438236: Remove proxxy manifests.

Review commit: https://reviewboard.mozilla.org/r/229080/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/229080/

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8960332 [details]
Bug 1438236: Remove proxxy manifests.

https://reviewboard.mozilla.org/r/229080/#review234846

DIAF!

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8960332 [details]
Bug 1438236: Remove proxxy manifests.

https://hg.mozilla.org/build/puppet/rev/f9f2925f081b49458374a12efcb4d94694aac02e

Comment 8

[Chris AtLee [:catlee]]

instanced deleted.

RIP.

Comment 9

[Radu Iman[:riman]]

Hello,

Thu 11:03:06 UTC [7663] [] proxxy1.srv.releng.usw2.mozilla.com (10.132.100.234) is DOWN :PING CRITICAL - Packet loss = 100%
Thu 11:03:10 UTC [7664] [] proxxy1.srv.releng.use1.mozilla.com (10.134.100.191) is DOWN :PING CRITICAL - Packet loss = 100%


The above alerts came on IRC channel for a couple of times today. I've ACK both of them for now. 
As :mtabara suggested, I'll reopen this bug to check why Nagios alerts us given that the hosts have been decommissioned.

Thank you

Comment 10

[Radu Iman[:riman]]

Updates:

Sun 04:34:28 UTC [7814] [] proxxy1.srv.releng.use1.mozilla.com (10.134.100.191) is DOWN :PING CRITICAL - Packet loss = 100%
Sun 04:34:28 UTC [7815] [] proxxy1.srv.releng.usw2.mozilla.com (10.132.100.234) is DOWN :PING CRITICAL - Packet loss = 100%

^^ These alerts came again on IRC channel during my shift. I've acknowledge both of them again.

Comment 11

[Chris AtLee [:catlee]]

The nagios checks should be deleted.

Comment 12

[Radu Iman[:riman]]

I've created these three patches to delete the nagios checks. 

:catlee could you review them, please?

Comment 13

[Radu Iman[:riman]]

Attachment: remove-from-hostgroup.txt

Comment 14

[Radu Iman[:riman]]

Attachment: remove-from-scl3.txt

Comment 15

[Radu Iman[:riman]]

Attachment: remove-from-services-scl3.txt

Comment 16

[Attila Craciun [:arny]]

Pushed the patches to the repo.

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for taking care of these :riman!