pv-mirror01.mozilla.org not responding causing mirror network to be out-of-sync

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
9 years ago
4 years ago

People

(Reporter: reed, Assigned: phong)

Tracking

Details

(Reporter)

Description

9 years ago
rsync: failed to connect to stage-rsync.mozilla.org: Connection timed out (110)

stage-rsync.mozilla.org is pv-mirror01.mozilla.org.

That's extremely bad. Not having stage-rsync working means none of the mirrors can get new bits from add-ons or builds themselves.

Nagios has been paging non-stop over this both today and for the last few days (on and off)... Why has this been ignored?

[13:34:59] <nagios> [43] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[13:49:58] <nagios> [51] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:04:57] <nagios> [59] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:14:58] <nagios> [64] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:25:02] <nagios> [69] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:33:32] <nagios> [74] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 65 minutes
[14:40:05] <nagios> [77] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:43:35] <nagios> [80] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 75 minutes
[14:55:01] <nagios> [85] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[14:58:38] <nagios> [88] geodns01.sj:Releases-Rsync_Mirror_Lag is WARNING: Mirrors WARNING - mirror2.mirrors.tds.net (Madison WI) : 90 minutes
[15:03:33] <nagios> [91] geodns01.sj:Releases-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - mirror2.mirrors.tds.net (Madison WI) : 95 minutes
[15:05:03] <nagios> [94] geodns01.sj:Stage-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - pv-mirror01.mozilla.org : No response from server!
[15:18:34] <nagios> [34] geodns01.sj:Releases-Rsync_Mirror_Lag is CRITICAL: Mirrors CRITICAL - mirror2.mirrors.tds.net (Madison WI) : 110 minutes
(Assignee)

Updated

9 years ago
Assignee: server-ops → phong
(Reporter)

Comment 1

9 years ago
[15:35:17] <nagios> [54] geodns01.sj:Releases_Mirror_Lag is CRITICAL: Mirrors CRITICAL - Too few mirrors enabled

releases.mozilla.org is going to disappear soon if this isn't fixed ASAP. :(
(Assignee)

Comment 2

9 years ago
moved to pv-mirror02
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.