switch ports incorrect for machines in https://inventory.mozilla.org/en-US/systems/racks/?rack=228

RESOLVED FIXED

Status

Infrastructure & Operations
DCOps
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: arr, Assigned: van)

Tracking

Details

(Reporter)

Description

3 years ago
The machines listed in this rack view seem to indicate that they're connected to switches that don't exist in that rack (or anywhere?). Can we please make sure the data is correct?
(Assignee)

Comment 1

3 years ago
the switch information is wrong because we renamed the switch when we renamed the racks. this would kinda suck having to manually edit 70+ entries per rack.

sheeri, is this something that the data/inventory team can whip up a script for? i'm sure we have other racks with out of date switch name information.

in the rack listed above, sample host - https://inventory.mozilla.org/en-US/systems/show/8511/ has the switch info listed as switch1.r101-18.console.scl3.mozilla.net:$interface. can we replace switch1.r101-18.console.scl3.mozilla.net with switch1.r401-5.console.scl3.mozilla.net but keep the $interface?
Flags: needinfo?(scabral)
QA Contact: jbarnell
(Assignee)

Updated

3 years ago
colo-trip: --- → scl3
:van, are you aware of the Inventory cli? You can do a lot of those changes fairly easily:
https://mana.mozilla.org/wiki/display/SYSADMIN/Inventory+CLI+Cookbook
(Assignee)

Comment 3

3 years ago
:pir, search - yes; and doing one or two is fairly simple. but was wondering if there was a way to do a mass search and replace for a specific field in specific racks since this will encompass several racks. i wanted to replace the switch name but not the switch port.
You should be able to do that with a script that takes search results and builds appropriate calls for the renames... I've not done it personally but I believe it is possible.
(Assignee)

Comment 5

3 years ago
:pir, cool thanks! is this something you can assist with as im not 100% confident in my scripting skills, especially when the cli is the production environment?
> switch1.r101-18.console.scl3.mozilla.net:$interface. can we replace switch1.r101-18.console.scl3.mozilla.net with 
> switch1.r401-5.console.scl3.mozilla.net but keep the $interface?

Absolutely. In MySQL this is just a REPLACE function, like REPLACE(name, 'switch1.r101-18', 'switch1.r401-5') - we can SELECT for it to see how many items in there first. Needinfo me if you want the Data Team to do that, it's probably quicker than writing a script (unless you want to gain the experience with the scripting).
Flags: needinfo?(scabral)
(Assignee)

Comment 7

3 years ago
yes please please perform the REPLACE for both the switch port names and the OOB name fields.

switch port: switch1.r101-18.console.scl3.mozilla.net -> switch1.r401-5.ops.releng.scl3.mozilla.net

OOB switch and port (please keep the inband* part at the beginning these hostnames as there are 2 inband switches): r101-18.console.scl3.mozilla.net -> r401-5.inband.releng.scl3.mozilla.net

example: inband2.r101-18.console.scl3.mozilla.net:ge-0/0/31 -> inband2.r401-5inband.releng.scl3.mozilla.net:ge-0/0/31
Flags: needinfo?(scabral)

Here's a comma-separated list of id's I'm changing:
mysql> select group_concat(id) from systems where switch_ports like 'switch1.r101-18.console.scl3.mozilla.net%'\G
*************************** 1. row ***************************
group_concat(id): 8448,8449,8450,8451,8452,8453,8454,8455,8456,8457,8458,8459,8460,8461,8462,8463,8464,8465,8466,8467,8468,8469,8470,8471,8472,8473,8474,8475,8476,8477,8478,8479,8480,8481,8482,8483,8484,8485,8486,8487,8488,8489,8490,8491,8492,8493,8494,8495,8496,8497,8498,8499,8500,8501,8502,8503,8504,8505,8506,8507,8508,8509,8510,8511,8512,8513,8542,8543,8544,8545
1 row in set (0.01 sec)


-----------------------------------------------------------------------------
a little test before we change things
-----------------------------------------------------------------------------
mysql> select switch_ports from systems where id=8511;
+----------------------------------------------------+
| switch_ports                                       |
+----------------------------------------------------+
| switch1.r101-18.console.scl3.mozilla.net:ge-1/0/31 |
+----------------------------------------------------+
1 row in set (0.00 sec)


-----------------------------------------------------------------------------
We change the switch_ports
-----------------------------------------------------------------------------
mysql> update systems set switch_ports=REPLACE(switch_ports,'switch1.r101-18', 'switch1.r401-5');
Query OK, 72 rows affected (0.02 sec)
Rows matched: 8892  Changed: 72  Warnings: 0

-----------------------------------------------------------------------------
post-test to make sure things look good
-----------------------------------------------------------------------------

mysql> select switch_ports from systems where id=8511;
+---------------------------------------------------+
| switch_ports                                      |
+---------------------------------------------------+
| switch1.r401-5.console.scl3.mozilla.net:ge-1/0/31 |
+---------------------------------------------------+
1 row in set (0.00 sec)
Flags: needinfo?(scabral)
Now for the OOB switch change:

The list of id's I am changing OOB for
mysql> select group_concat(id) from systems where oob_switch_port like 'inband%.r101-18.console.scl3.mozilla.net%'\G
*************************** 1. row ***************************
group_concat(id): 8448,8449,8450,8451,8452,8453,8454,8455,8456,8457,8458,8459,8460,8461,8462,8463,8464,8465,8466,8467,8468,8469,8470,8471,8472,8473,8474,8475,8476,8477,8478,8479,8480,8481,8482,8483,8484,8485,8486,8487,8488,8489,8490,8491,8492,8493,8494,8495,8496,8497,8498,8499,8500,8501,8502,8503,8504,8505,8506,8507,8508,8509,8510,8511,8512,8513,8542,8543,8544,8545
1 row in set (0.00 sec)



OOB switch and port (please keep the inband* part at the beginning these hostnames as there are 2 inband switches): r101-18.console.scl3.mozilla.net -> 

example: inband2.r101-18.console.scl3.mozilla.net:ge-0/0/31 -> inband2.r401-5inband.releng.scl3.mozilla.net:ge-0/0/31

101-18.console.scl3.mozilla.net -> r401-5.inband.releng.scl3.mozilla.ne

-----------------------------------------------------------------------------
a little test before we change things
-----------------------------------------------------------------------------
mysql> select oob_switch_port from systems where id=8511;
+----------------------------------------------------+
| oob_switch_port                                    |
+----------------------------------------------------+
| inband2.r101-18.console.scl3.mozilla.net:ge-0/0/31 |
+----------------------------------------------------+
1 row in set (0.00 sec)

-----------------------------------------------------------------------------
We change the oob_switch_port value
-----------------------------------------------------------------------------
mysql> update systems set switch_ports=REPLACE(oob_switch_port,'.r101-18.console.scl3.mozilla.net', '.r401-5.inband.releng.scl3.mozilla.net');
Query OK, 3810 rows affected (0.13 sec)
Rows matched: 8892  Changed: 3810  Warnings: 0


-----------------------------------------------------------------------------
post-test to make sure things look good
-----------------------------------------------------------------------------

mysql> select oob_switch_port from systems where id=8511;                                                                       +----------------------------------------------------+
| oob_switch_port                                    |
+----------------------------------------------------+
| inband2.r101-18.console.scl3.mozilla.net:ge-0/0/31 |
+----------------------------------------------------+
1 row in set (0.00 sec)
(Assignee)

Updated

3 years ago
Flags: needinfo?(scabral)
(Reporter)

Comment 11

3 years ago
Van reported that racks of machines now have incorrect info for oob. I'm not sure if it was this change that did that, but I've asked him to open a bug in the inventory queue with as much detail as he has.
> update systems set switch_ports=REPLACE(oob_switch_port,'.r101-18.console.scl3.mozilla.net', '.r401-5.inband.releng.scl3.mozilla.net');

should have been '..set oob_switch_port=..'

which is why the post-test failed, and why systems now have incorrect switch ports and oob switch ports :(
We had some trouble restoring backups. Currently on inventory db there is a table called systems_restore_from_me, which I will use first thing tomorrow to get the switch_ports value fixed:

mysql> select count(*) from systems left join systems_recover_from_me using (hostname) where systems.switch_ports!=systems_recover_from_me.switch_ports;
+----------+
| count(*) |
+----------+
|     2977 |
+----------+
1 row in set (0.03 sec)
Flags: needinfo?(scabral)
setting back to before the switch:

mysql> update systems left join systems_recover_from_me using (hostname) set systems.switch_ports=systems_recover_from_me.switch_ports where systems.switch_ports!=systems_recover_from_me.switch_ports;
Query OK, 2977 rows affected, 1 warning (0.11 sec)
Rows matched: 2977  Changed: 2977  Warnings: 0

-----------------------------------------------------------------------------
verification
-----------------------------------------------------------------------------
mysql> select switch_ports from systems where id=8511;
+----------------------------------------------------+
| switch_ports                                       |
+----------------------------------------------------+
| switch1.r101-18.console.scl3.mozilla.net:ge-1/0/31 |
+----------------------------------------------------+
1 row in set (0.00 sec)


-----------------------------------------------------------------------------
We change the switch_ports
-----------------------------------------------------------------------------
mysql> update systems set switch_ports=REPLACE(switch_ports,'switch1.r101-18', 'switch1.r401-5') where  switch_ports like 'switch1.r101-18.console.scl3.mozilla.net%';
Query OK, 70 rows affected (0.01 sec)
Rows matched: 70  Changed: 70  Warnings: 0

VERIFY - only 70 rows matched. good.

post-test:

mysql> select switch_ports from systems where id=8511;                                                  +---------------------------------------------------+
| switch_ports                                      |
+---------------------------------------------------+
| switch1.r401-5.console.scl3.mozilla.net:ge-1/0/31 |
+---------------------------------------------------+
1 row in set (0.00 sec)
-----------------------------------------------------------------------------
a little test before we change things
-----------------------------------------------------------------------------
mysql> select oob_switch_port from systems where id=448;
+---------------------------------------------------+
| oob_switch_port                                   |
+---------------------------------------------------+
| inband1.r101-18.console.scl3.mozilla.net:ge-0/0/1 |
+---------------------------------------------------+
1 row in set (0.00 sec)

-----------------------------------------------------------------------------
We change the oob_switch_port value
-----------------------------------------------------------------------------
mysql> update systems set oob_switch_port = REPLACE(oob_switch_port, '.r101-18.console.scl3.mozilla.net', '.r401-5.inband.releng.scl3.mozilla.net') where oob_switch_port like 'inband%.r101-18.console.scl3.mozilla.net%';
Query OK, 69 rows affected (0.01 sec)
Rows matched: 69  Changed: 69  Warnings: 0

VERIFY - only 69 rows matched

mysql> select oob_switch_port from systems where id=8448;
+--------------------------------------------------------+
| oob_switch_port                                        |
+--------------------------------------------------------+
| inband1.r401-5.inband.releng.scl3.mozilla.net:ge-0/0/1 |
+--------------------------------------------------------+
1 row in set (0.00 sec)
Duplicate of this bug: 1186237
(Assignee)

Comment 18

3 years ago
:sheeri, thanks again!
Assignee: server-ops-dcops → vle
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.