Closed Bug 760659 Opened 13 years ago Closed 13 years ago

Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gkw, Assigned: mburns)

References

Details

(Whiteboard: [buildduty][outage][treeclosure])

tbpl seems down, hg.mozilla.org (esp web interface down), MDN down, some other sites down, see: http://status.mozilla.com/
Whiteboard: [buildduty][outage]
trees are closed due to HG outage
Ops is aware of the issue and working to resolve, details will be forthcoming
Assignee: server-ops → mburns
We believe we've identified the problem Zeus server and have failed it out. Sites should begin working soon. Will provide an update when we've confirmed our alerts clear
Status: NEW → ASSIGNED
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
following properties back up again: graphs.m.o hg.m.o tbpl.m.o tinderbox.m.o tree remains closed while we cross-check systems.
Summary: Various Mozilla properties down → Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01
Whiteboard: [buildduty][outage][treeclosure] → [buildduty][outage]
(In reply to John O'Duinn [:joduinn] from comment #5) > following properties back up again: > graphs.m.o > hg.m.o > tbpl.m.o > tinderbox.m.o > > tree remains closed while we cross-check systems. trees reopened. impact to various release builds-in-progress still being figured out.
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
Alerts have cleared in Nagios, external checks should clear soon. Please re-open if further issues are sen.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.