Closed Bug 760659 Opened 12 years ago Closed 12 years ago

Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gkw, Assigned: mburns)

References

Details

(Whiteboard: [buildduty][outage][treeclosure]) 

tbpl seems down, hg.mozilla.org (esp web interface down), MDN down, some other sites down, see:

http://status.mozilla.com/
Whiteboard: [buildduty][outage]
trees are closed due to HG outage
Ops is aware of the issue and working to resolve, details will be forthcoming
Assignee: server-ops → mburns
We believe we've identified the problem Zeus server and have failed it out. Sites should begin working soon.

Will provide an update when we've confirmed our alerts clear
Status: NEW → ASSIGNED
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
following properties back up again:
graphs.m.o
hg.m.o
tbpl.m.o
tinderbox.m.o

tree remains closed while we cross-check systems.
Summary: Various Mozilla properties down → Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01
Whiteboard: [buildduty][outage][treeclosure] → [buildduty][outage]
(In reply to John O'Duinn [:joduinn] from comment #5)
> following properties back up again:
> graphs.m.o
> hg.m.o
> tbpl.m.o
> tinderbox.m.o
> 
> tree remains closed while we cross-check systems.

trees reopened. 

impact to various release builds-in-progress still being figured out.
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
Alerts have cleared in Nagios, external checks should clear soon.

Please re-open if further issues are sen.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.