763293 - Multiple Mozilla services are down

Status: RESOLVED DUPLICATE of bug 763296

(mozilla.org Graveyard :: Server Operations, task)

Description

[Pascal Chevrel:pascalc (PTO until April 26)]

Multiple Mozilla services are currently unavailable, so far we have noticed:
- etherpad.mozilla.org is down
- can't commit to subversion (Unable to connect to a repository at URL 'svn+ssh://svn.mozilla.org/projects/granary/webdashboard-data')
- Bugzilla is not sending bugmail
- reps.mozilla.org gives a 'Service unavailable' message
- wiki.mozilla.org also gives a 'Service unavailable'

There may be more services down.

This is blocking our work at the MozFR camp in the Paris office this week end, we have 30 contributors hacking on Mozilla over the week end here)

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

people.m.o can't seem to talk to irc.m.o since around 3 am today.  Not sure if that's related.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Also nightly builds are failing with:

retry: Calling <function run_with_timeout at 0x2aaaaed1ac80> with args: (['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US']
ssh: connect to host aus3-staging.mozilla.org port 22: Connection timed out
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1ac80> with args: (['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #2
Executing: ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US']
ssh: connect to host aus3-staging.mozilla.org port 22: Connection timed out
retry: Failed, sleeping 2 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1ac80> with args: (['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #3
Executing: ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US']
ssh: connect to host aus3-staging.mozilla.org port 22: Connection timed out
retry: Failed, sleeping 4 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1ac80> with args: (['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #4
Executing: ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US']
ssh: connect to host aus3-staging.mozilla.org port 22: Connection timed out
retry: Failed, sleeping 8 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1ac80> with args: (['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #5
Executing: ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US']
ssh: connect to host aus3-staging.mozilla.org port 22: Connection timed out
retry: Giving up on <function run_with_timeout at 0x2aaaaed1ac80>
Unable to successfully run ['bash', '-c', 'ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-aurora/Linux_x86_64-gcc3/20120609042006/en-US'] after 5 attempts
program finished with exit code 1

Comment 3

[Tom Ellins [:TMZ]]

Also blog.mozilla.org

Comment 5

[Marcia Knous [:marcia]]

crash-stats.mozilla.com is down as well

Comment 6

[Dan Parsons [:lerxst]]

Many services are back online now and the rest are coming up soon, if they're not up already. Long story short, this was caused by a failure of the SPOF that is DNS & DHCP in phx1. Bug 763328 has been opened to fix this.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like that Pulse is also affected by this outage.