Closed Bug 547471 Opened 14 years ago Closed 14 years ago

pv-mirror01.mozilla.org not responding causing mirror network to be out-of-sync

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: reed, Assigned: phong)

Details

rsync: failed to connect to stage-rsync.mozilla.org: Connection timed out (110)

stage-rsync.mozilla.org is pv-mirror01.mozilla.org.

That's extremely bad. Not having stage-rsync working means none of the mirrors can get new bits from add-ons or builds themselves.

Nagios has been paging non-stop over this both today and for the last few days (on and off)... Why has this been ignored?

[13:34:59] <nagios> [43] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[13:49:58] <nagios> [51] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:04:57] <nagios> [59] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:14:58] <nagios> [64] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:25:02] <nagios> [69] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:33:32] <nagios> [74] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 65 minutes
[14:40:05] <nagios> [77] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:43:35] <nagios> [80] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 75 minutes
[14:55:01] <nagios> [85] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:58:38] <nagios> [88] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 90 minutes
[15:03:33] <nagios> [91] geodns01.sj:Releases-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - mirror2.mirrors.tds.net (Madison WI) : 95 minutes
[15:05:03] <nagios> [94] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[15:18:34] <nagios> [34] geodns01.sj:Releases-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - mirror2.mirrors.tds.net (Madison WI) : 110 minutes
Assignee: server-ops → phong
[15:35:17] <nagios> [54] geodns01.sj:Releases_Mirror_Lag is CRITICAL: Mirrors CRITICAL - Too few mirrors enabled

releases.mozilla.org is going to disappear soon if this isn't fixed ASAP. :(
moved to pv-mirror02
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.