Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: gkw, Assigned: mburns)

Tracking

Details

(Whiteboard: [buildduty][outage][treeclosure])

(Reporter)

Description

7 years ago
tbpl seems down, hg.mozilla.org (esp web interface down), MDN down, some other sites down, see:

http://status.mozilla.com/

Updated

7 years ago
Whiteboard: [buildduty][outage]

Comment 1

7 years ago
trees are closed due to HG outage
Duplicate of this bug: 760660
Ops is aware of the issue and working to resolve, details will be forthcoming
Assignee: server-ops → mburns
We believe we've identified the problem Zeus server and have failed it out. Sites should begin working soon.

Will provide an update when we've confirmed our alerts clear
Status: NEW → ASSIGNED
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
following properties back up again:
graphs.m.o
hg.m.o
tbpl.m.o
tinderbox.m.o

tree remains closed while we cross-check systems.
Summary: Various Mozilla properties down → Outage Tracking -ZLB1 Issue in SCL3 - 2012-06-01
Whiteboard: [buildduty][outage][treeclosure] → [buildduty][outage]
(In reply to John O'Duinn [:joduinn] from comment #5)
> following properties back up again:
> graphs.m.o
> hg.m.o
> tbpl.m.o
> tinderbox.m.o
> 
> tree remains closed while we cross-check systems.

trees reopened. 

impact to various release builds-in-progress still being figured out.
Whiteboard: [buildduty][outage] → [buildduty][outage][treeclosure]
Alerts have cleared in Nagios, external checks should clear soon.

Please re-open if further issues are sen.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.