1489061 - [Tracking] Shutdown all IX machines.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Danut Labici [:dlabici]]

Attachment: ix.txt

With BuilbBot going EOL we will have a bunch of IX machines that need to be shutdown and decommissioned. 
Attila worked on a list of IX machines (list compiled from nagios) that we know off. 

@ciduty - Don't start the shutdown process yet, Fubar and/or catlee will give us the greenlight when to do it.

@fubar/@catlee Does the list of machines (in attachment) looks good? 
Also, we will be using this command to quickly shutdown everything:

for hosts in "cat FILENAME-OF-IX"; do ipmitool -U USER -P PASSWORD -H $hosts-mgmt.build.mozilla.org chassis power down; done

But this is not working for talos-linux machines. Any idea what we are doing wrong? The error that we get when we try to run the ipmi command is:
ipmitool -U user -P [redacted] -H talos-linux64-ix-001.build.mozilla.org chassis power down
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session

Comment 1

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

> for hosts in "cat FILENAME-OF-IX"; do ipmitool -U USER -P PASSWORD -H
> $hosts-mgmt.build.mozilla.org chassis power down; done

I don't think we need "mgmt.build.mozilla.org" since the hosts will contain the full FQDN of that machine.

My go-to version for this is, splitting the attachment by type and running the script separately for each type.

Made a git repo with the modifications (has not been tested).. 
https://github.com/bccrisan/shutdown_buildbot 

What I have not tested at the moment, is if we have ssh access to the linux64 machines via ssh, afaik, have been a few discussions about this lately.

Comment 2

[Kendall Libby [:fubar] (he/him)]

Attachment: scl3-it2.txt

I'm attaching the list of RelEng systems from an audit of the SCL3 data center by IT. This list is from a couple weeks back, but was done during a walk through of SCL3, so all in this list are physically present but may not be in use. Please compare the two lists and ensure that discrepancies are handled - ie make sure that hosts that are on IT's list and not on yours are powered off.

If there are hosts on your list and not on IT's, then something weird happened with a decomm, presumably. Please verify if the machine is up and reachable. If it is, then IT missed it somehow; if not, remove it from any configs on our side.

Comment 3

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

(In reply to Kendall Libby [:fubar] from comment #2)
> Created attachment 9007781 [details]
> scl3-it2.txt

Thank you very much for the list, it is well welcomed!!

I compared the list between what CIDuty has and the list provided, put it in one big excel document.
https://docs.google.com/spreadsheets/d/1EUqj-2pj4GOJcBrglPogVi99_a78L9Zjf7AknkSlWgs/edit?usp=sharing
Yellow - Missing from one of the lists, Purple - Missing all of the machines from one list, Green - In both lists.

As a conclusion the list is as complete as possible as we can get it to be.

Meanwhile I also tested the ssh connection to the talos-linux64 machines and works in my case.
I'll also update the git list, so everyone can have it ASAP.


Another thing to mention here, we have those yosemites remained in SCL3 (t-yosemite-r7-{0001-0022}), what is going to happen with them? are they going to be moved to another DC or decommissioned?

Comment 4

[Kendall Libby [:fubar] (he/him)]

(In reply to Bogdan Crisan [:bcrisan] (UTC +3, EEST) from comment #3)
> 
> Another thing to mention here, we have those yosemites remained in SCL3
> (t-yosemite-r7-{0001-0022}), what is going to happen with them? are they
> going to be moved to another DC or decommissioned?

They will be held by IT and moved to MDC1/2 after SCL3 is fully decommissions (so probably November).

Comment 5

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

(In reply to Kendall Libby [:fubar] from comment #4)
> (In reply to Bogdan Crisan [:bcrisan] (UTC +3, EEST) from comment #3)
> > 
> > Another thing to mention here, we have those yosemites remained in SCL3
> > (t-yosemite-r7-{0001-0022}), what is going to happen with them? are they
> > going to be moved to another DC or decommissioned?
> 
> They will be held by IT and moved to MDC1/2 after SCL3 is fully
> decommissions (so probably November).

Is there any action recommended to be taken regarding them? I'm manly referring if we should also shut them down?

Comment 6

[Kendall Libby [:fubar] (he/him)]

(In reply to Bogdan Crisan [:bcrisan] (UTC +3, EEST) from comment #5)
> 
> Is there any action recommended to be taken regarding them? I'm manly
> referring if we should also shut them down?

Yes, we should power them down with the rest of the SCL3 systems.

Comment 7

[Radu Iman[:riman]]

I've managed to power off the following machines:

1) talos-linux64-ix-{001..089}
2) t-w1064-ix-{001..036 & 046..315}
3) t-w732-ix-{001..147}
4) t-w864-ix-{002 & 011..062} 
5) t-xp32-ix-{003..025}

Comment 8

[Radu Iman[:riman]]

6) t-yosemite-r7-{0001..0022}

Comment 9

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Marking this as fixed since all of the machines have been shutdown!