731374 - use the new VIP: verify that the appropriate netflows are set up to allow the metrics machines to connect to the right VIP

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Sheeri Cabral [:sheeri]]

The appropriate VIP to use is tp-amo01-ro-zeus.phx.mozilla.com - I will see about getting that fixed on cm-metricsapp01, cm-metricsetl01, and cm-metricsetl02.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Can we have this bug be to verify that the appropriate netflows are set up to allow the metrics machines to connect to the right VIP?

Once that is confirmed, my team can take care of the reconfiguration of our processes to be querying the new VIP.

Comment 2

[Sheeri Cabral [:sheeri]]

tm-sumo-r[o|w]-zeus needs to be changed to tp-amo01-ro-zeus.phx.mozilla.com on metrics servers, make sure the netflows will allow this.

Comment 3

[Sheeri Cabral [:sheeri]]

daniel - I have changed the subject.

Comment 4

[Paula Clemente [:paula]]

We already moved to SCL3.

Could you check this now, please?

New metrics machines:
{app1,app2,etl3,etl4}.metrics.scl3.mozilla.com

Thanks

Comment 5

[Sheeri Cabral [:sheeri]]

db-sumo01-ro-zeus needs to be changed to tp-sumo01-ro-zeus.phx.mozilla.com on metrics servers, make sure the netflows will allow this.

tm-sumo-rw-zeus needs to be changed to  tp-sumo01-rw-zeus.phx.mozilla.com on metrics servers, make sure the netflows will allow this.

Comment 6

[Sheeri Cabral [:sheeri]]

See the dependent bug 748162:

[root@app2 ~]# nc -zv tp-sumo01-rw-zeus.phx.mozilla.com 3306
Connection to tp-sumo01-rw-zeus.phx.mozilla.com 3306 port [tcp/mysql] succeeded!
etc :)

This has been done! Resolving, if this is still a problem, please re-open and if you'd like you can ping me in IRC.

Comment 7

[Sheeri Cabral [:sheeri]]

passing back to :dre

db-sumo01-ro-zeus needs to be changed to tp-sumo01-ro-zeus.phx.mozilla.com on metrics servers, netflows will allow this.

tm-sumo-rw-zeus needs to be changed to  tp-sumo01-rw-zeus.phx.mozilla.com on metrics servers, netflows will allow this.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

Okay, paula, can you try to update the sumo processing with these new hostnames?

Comment 9

[Paula Clemente [:paula]]

Ok,

I will switch tm-sumo01-ro-zeus.mozilla.org to tp-sumo01-ro-zeus.phx.mozilla.com
Is this right?

We also have one connection to the host: dm-sumotools01 , user: sumo, that isn't working.
What should I use (which host) to fix that? 

Thanks.

Comment 10

[Paula Clemente [:paula]]

> We also have one connection to the host: dm-sumotools01 , user: sumo, that
> isn't working.
> What should I use (which host) to fix that? 


Not working, as:

[root@app1 mail]# nc -zv dm-sumotools01.mozilla.org 3306
nc: connect to dm-sumotools01.mozilla.org port 3306 (tcp) failed: Connection timed out

Comment 11

[Paula Clemente [:paula]]

Hum...
I'm getting Access denied with the new host:

[root@etl4 webetl]$ mysql -h tp-sumo01-rw-zeus.phx.mozilla.com -u metrics -p<password> support_mozilla_com
ERROR 1045 (28000): Access denied for user 'metrics'@'10.8.70.202' (using password: YES)


With the old one (tm-sumo01-ro-zeus.mozilla.org), it works.

Comment 12

[Sheeri Cabral [:sheeri]]

OK, I just added the proper ACL to MySQL, can you try again?

Comment 13

[Paula Clemente [:paula]]

I still get the same errors mentioned in Comment 10 and 11.

Comment 14

[Sheeri Cabral [:sheeri]]

Oh! If the connection timed out it's a netflow issue. I will have to make a bug to open that up.

Comment 15

[Paula Clemente [:paula]]

Note that we have two different issues here:

In Comment 10 I'm talking about "dm-sumotools01"
In Comment 11 I'm talking about "tp-sumo01-rw-zeus.phx.mozilla.com".

Comment 16

[Sheeri Cabral [:sheeri]]

OK, so dm-sumotools01 wasn't in the original scope but I've opened up bug 750879 to deal with it.

Can you copy/paste the error you're getting for comment 10?

mysql> select user,host from mysql.user where user='metrics';
+---------+-------------+
| user    | host        |
+---------+-------------+
| metrics | 10.2.70.200 | 
| metrics | 10.2.70.20_ | 
| metrics | 10.8.70.20_ | 
+---------+-------------+
3 rows in set (0.00 sec)

The ACL is there, so I'm not sure why your login would be denied.

Comment 17

[Sheeri Cabral [:sheeri]]

Bug 750879 has been resolved:

[root@app2 ~]# nc -zv dm-sumotools01.mozilla.org 3306
Connection to dm-sumotools01.mozilla.org 3306 port [tcp/mysql] succeeded!

As far as I can tell, both the issues should have been fixed. If you're having a problem, can you copy/paste the problem again? (obviously w/out any passwords).

Comment 18

[Paula Clemente [:paula]]

The connections are working now, thanks Sherri.