632320 - Move Bugzilla database vIPs from Netscaler to Zeus

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Miller [:justdave]]

The bugs01 database cluster currently has its load-balancing vIPs set up on the Netscaler.  In order to make failover between sjc and phx work properly we need to have it in Zeus instead (not to mention this will make some of the pool management stuff easier to automate)

Comment 1

[Dave Miller [:justdave]]

This is going to primarily affect metrics, as they're the only reader outside of the actual Bugzilla app and a couple nagios scripts.  The IP address you use to connect to the Bugzilla DB server will change.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

Please coordinate with Michael Kurze and also make sure that when the change happens, we won't be locked out of the new database due to permission changes (for instance, if our current authorization is based on some IP that will change).

Comment 3

[Michael Kurze [:michaelk]]

Thanks for the heads up! I would appreciate to know about the schedule and the new IP as that becomes available.

I take it the hostname we use stays the same (tm-bugs01-slave02), so we’re mostly fine.


If it can be arranged, it would be great if permissions could be added allowing us to also query from %.sjc1.mozilla.com (currently we tunnel over cm-metricsapp01.mozilla.org). I can file that separately though.

Comment 4

[Dave Miller [:justdave]]

Zeus vIPs set up and configured, IP-based access-control in place.

mkurze: Hostname is changing, too, actually.  The new one is tm-bugs01-ro-zeus, and should be functional already (and you already have the appropriate privs set up), so feel free to switch over at your leisure.  Let us know when you have it all switched over and we won't cut off the old one until then. :)

Comment 5

[Dave Miller [:justdave]]

Oh, forgot we need firewall holes for this, too...

The following IPs/ranges:

10.2.74.123	
10.2.74.130	
10.2.82.0/24

Need access to

10.2.70.50:3306
10.2.70.59:3306

The following IPs:

10.2.72.22	
10.2.72.28	
10.2.72.34	

Need access ONLY to:

10.2.70.59:3306

Please reassign the bug back to Server Ops and set me as assignee on completion.

Comment 6

[Michael Kurze [:michaelk]]

(In reply to comment #4)
> mkurze: Hostname is changing, too, actually.  The new one is tm-bugs01-ro-zeus,
> and should be functional already (and you already have the appropriate privs
> set up), so feel free to switch over at your leisure.  Let us know when you
> have it all switched over and we won't cut off the old one until then. :)

Dave: thanks for the info. I’ll try it when the firewall has been opened and will let you know if everything works.

Comment 7

[Derek Moore [:dmoore]]

Rules added as required, back to justdave

Comment 8

[Michael Kurze [:michaelk]]

Okay, everything works in terms of mysql-permissions.

The Bugzilla ETL is switched to querying the new instance (tunneling over cm-metricsapp01.mozilla.org as before), so the old instance can be disabled as far as metrics is concerned.

Comment 9

[Dave Miller [:justdave]]

mysql> drop user metrics@10.2.72.22;
Query OK, 0 rows affected (0.08 sec)

mysql> drop user metrics@10.2.72.28;
Query OK, 0 rows affected (0.01 sec)

mysql> drop user metrics@10.2.72.34;
Query OK, 0 rows affected (0.01 sec)

OK, your only remaining access is via the new connection given in comment 4.

Comment 10

[Dave Miller [:justdave]]

This is done, did it during (or shortly after) the zimbra outage.

Comment 11

[Dave Miller [:justdave]]

I've now removed the relevant configuration from the Netscaler and the relevant DNS entries.