891128 - Multiple services are offline in SCL3

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Other, task)

Description

[Shyam Mani [:fox2mike]]

Still ongoing, working to resolve

Comment 1

[Shyam Mani [:fox2mike]]

List of affected services :

1) Bugzilla (should be back online now)
2) hg.mozilla.org
3) git.mozilla.org

Comment 2

[Sheeri Cabral [:sheeri]]

Done on bugzilla4 and the backups server and bugzilla1.db.phx1:

slave stop; 
 change master to master_host='bugzilla2.db.scl3.mozilla.com', master_log_pos=921472505, master_log_file='bugzilla2-bin.000322';
slave start;

needs to be done on bugzilla3 when it comes up, too. (bugzilla1 is already slaving bugzilla2)

bugzilla1.db.phx1 can't replicate because of netflows:

[[e-mail]1 ~]# nc -vz bugzilla2.db.scl3.mozilla.com 3306
nc: connect to bugzilla2.db.scl3.mozilla.com port 3306 (tcp) failed: Connection timed out

Comment 3

[Sheeri Cabral [:sheeri]]

Switched back to bugzilla1.db.scl3 as the master; if this write succeeds then everything is OK again.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

trees closed: 14:47PT
trees opened: 18:16PT

The Firefox23.0b4 release was impacted, those builds have been revived, and are now in progress... details of cleanup in bug#891165.

Like before, lets keep this bug open for root cause/postmortem.

Comment 5

[Ravi Pina [:ravi]]

We have AJTAC and BNG case 2013-0708-1012 open with Juniper and a call this morning to sync up on the issue.  We believe this may be a similar issue to bug 826609.  We believe we were able to collect full debug output and logs to help AJTAC identify a root cause.

Comment 6

[casey ransom [:casey]]

Juniper's case remains open, however, as they have not been able to track down a root cause of the issue for the past year, we are electing to remove the technology. the depending bug is the tracker for that work.  Previously in the netops roadmap, we were going to install this device in phx1 as well, however, that is no longer on the table due to instability we've experienced in scl3, which would is a similar layout.