765021 - Incorrect and/or unavailable OOBs for node[1-8].generic.metrics.scl3

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Eric Ziegenhorn :ericz]

node[1-8].generics.metrics.scl3 (node1 has already been renamed in Inventory to node1.research.metrics.scl3) have wrong or malfunctioning OOBs listed in inventory:

node1 (https://inventory.mozilla.org/en-US/systems/show/1485/) shows http://10.22.3.64 as the OOB but that is actually the OOB for admin1b.private.scl3.
node2 (https://inventory.mozilla.org/en-US/systems/show/1486/) shows http://10.22.3.65 as the OOB which is not reachable
node3 (https://inventory.mozilla.org/en-US/systems/show/1487/) shows http://10.22.3.66 as the OOB which is not reachable
node4 (https://inventory.mozilla.org/en-US/systems/show/1488/) shows http://10.22.3.67 as the OOB which is not reachable
node5 (https://inventory.mozilla.org/en-US/systems/show/1489/) shows http://10.22.3.68 as the OOB which is not reachable
node6 (https://inventory.mozilla.org/en-US/systems/show/1490/) shows http://10.22.3.69 as the OOB which is not reachable
node7 (https://inventory.mozilla.org/en-US/systems/show/1491/) shows http://10.22.3.70 as the OOB which is not reachable
node8 (https://inventory.mozilla.org/en-US/systems/show/1492/) shows http://10.22.3.71 as the OOB which is not reachable

Comment 1

[Derek Moore [:dmoore]]

Eric,

Nodes 5-8 seem to be accessible. Could you double-check those for me?

We'll investigate 1-4

Comment 2

[Eric Ziegenhorn :ericz]

Didn't work for me yesterday, but now I can get to nodes 2-8.  1 still has the wrong IP in Inventory.

I'll get started with 2-8 then, thanks!

Comment 3

[Derek Moore [:dmoore]]

These hosts were in the middle of being moved yesterday, so they may have still been working on cabling them while you were trying to access iLO. We'll get node1 sorted today.

Comment 4

[Van Le [:van]]

looks like all the iLO ips are reachable. please let me know if you're still having any issues.

[vle@admin1a]~% fping 10.22.3.{64..71}   
10.22.3.64 is alive
10.22.3.65 is alive
10.22.3.66 is alive
10.22.3.67 is alive
10.22.3.68 is alive
10.22.3.69 is alive
10.22.3.70 is alive
10.22.3.71 is alive

thanks,
van

Comment 5

[Derek Moore [:dmoore]]

Van,

The OOB address 10.22.3.64 was previously allocated to a different host. We'll need to update the OOB IP for node1 (https://inventory.mozilla.org/en-US/systems/show/1485/) to:

10.22.3.157

Comment 6

[Van Le [:van]]

reconfigured ip and updated inventory. confirmed ip is pingable and sshable.

[[e-mail]3 ~]$ fping 10.22.3.157
10.22.3.157 is alive
[[e-mail]3 ~]$ ssh !$
ssh 10.22.3.157
The authenticity of host '10.22.3.157 (10.22.3.157)' can't be established.
RSA key fingerprint is 3c:31:79:a3:ba:cb:1f:e8:03:c0:22:65:7f:88:c5:9e.
Are you sure you want to continue connecting (yes/no)? yes

thanks,
van