730016 - cm-webdev chain for moving

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Sheeri Cabral [:sheeri]]

In preparation for the data center move, cshields has asked me to put together a list of what connects to cm-webdev01-master01, so we can verify the dependency chain.

Comment 1

[Sheeri Cabral [:sheeri]]

After a few hours of general logging, with almost 400 connections total:

[root@cm-webdev01-master01 mysql]# grep Connect mysql-general.log  | cut -f2-100 -dC | sort | uniq -c 
      1 onnect     Access denied for user 'UNKNOWN_MYSQL_US'@'localhost' (using password: NO)
      1 onnect     clouserw@localhost on remora
     41 onnect     davedash@10.2.72.51 on d_remora
      4 onnect     fligtar@10.2.74.111 on 
     69 onnect     nagiosdaemon@10.2.74.123 on 
    254 onnect     root@10.2.74.111 on arecibo
      7 onnect     root@10.2.74.111 on c_remora
      1 onnect     root@localhost on 

So we've got:
localhost
10.2.72.51 - cm-webdev01.mozilla.org
10.2.74.111 - khan
10.2.75.123 - netops2.private.sjc1.mozilla.com (nagios)

I can post more tomorrow, but we can at least get a chain going.....The idea is to have these machines move out with the second train, so we need to know the dependencies.

Comment 2

[Sheeri Cabral [:sheeri]]

Here it is from a day of logging:

[root@cm-webdev01-master01 mysql]#  grep Connect mysql-general.log  | cut -f2-100 -dC | sort | uniq -c 
      1 onnect     Access denied for user 'UNKNOWN_MYSQL_US'@'localhost' (using password: NO)
      1 onnect     clouserw@localhost on c_remora
      1 onnect     clouserw@localhost on remora
    277 onnect     davedash@10.2.72.51 on d_remora
      4 onnect     fligtar@10.2.72.155 on fligtar_panorama
      2 onnect     fligtar@10.2.72.155 on metrics_adu
      3 onnect     fligtar@10.2.72.155 on remora
     38 onnect     fligtar@10.2.74.111 on 
    472 onnect     nagiosdaemon@10.2.74.123 on 
      1 [e-mail]', 15)
    426 onnect     root@10.2.74.111 on arecibo
      7 onnect     root@10.2.74.111 on c_remora
      2 onnect     root@localhost on 
      1 onnect     root@localhost on remora
[root@cm-webdev01-master01 mysql]# 

So we've got:
localhost
10.2.72.51 - cm-webdev01.mozilla.org
10.2.74.111 - khan
10.2.75.123 and 74.123 - nagios hosts
10.2.72.155 - hm-admin03.scl2.mozilla.com

Comment 3

[Sheeri Cabral [:sheeri]]

Possible netflows needed:

07:38 < sheeri>  10.2.74.111
07:38 <@cshields> which was khan
07:39 < sheeri> other than that we've got:
07:39 < sheeri> | fligtar_stats@10.2.75.101 | 
07:39 < sheeri> | nagiosdaemon@10.2.74.123  | 
07:39 < sheeri> | repl@10.2.72.39           | 
07:39 < sheeri> | summitr@10.2.76.43        | 
07:40 < sheeri> and a bunch of localhost, and a bunch of host=%

Comment 4

[Sheeri Cabral [:sheeri]]

cm-webdev01-master01 and -slave01 have been moved to scl3 in yesterday's train.

They needed firmware updates, dns should be done. cm-webdev01-master01 is up at 10.22.70.21, the slave will be up at 10.22.70.22 when the firmware updates complete. Replication may need to be fixed.

Comment 5

[Sheeri Cabral [:sheeri]]

cm-webdev01-slave01 seems to have come up OK.

Replication started - had to change the ip and also the grant on the host. I will post when replication is caught up.

Comment 6

[Sheeri Cabral [:sheeri]]

FWIW slave is at 620030 seconds behind right now.

Comment 7

[Sheeri Cabral [:sheeri]]

Updated the IPs on cm-webdev01 for khan to be the new ones, so the grants are still good:

mysql> update mysql.user set host='10.22.74.21' where host='10.2.74.111';
Query OK, 27 rows affected (0.00 sec)
Rows matched: 27  Changed: 27  Warnings: 0

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

Comment 8

[Sheeri Cabral [:sheeri]]

Also updated the IPs on addons-webdev1.db.phx1.mozilla.com (and replication is catching up there).

Comment 9

[Sheeri Cabral [:sheeri]]

Jason's opened a netflow between khan and the new IPs.

Comment 10

[Sheeri Cabral [:sheeri]]

Still left to do:

Copy db-related crons on addons1 to be crons on addons2
Disable db-related crons on addons1
Move backup/webdev slaves from addons1 to addons2

Comment 11

[Sheeri Cabral [:sheeri]]

ugh, wrong ticket to update in.

AT this point I believe the cm-webdev chain is complete, Dan is virtualizing them in bug 735370.