520773 - wikimo and MDC down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Shepherd [:sheppy]]

Both wikimo and MDC are down. Database outage?

Comment 1

[Dave Miller [:justdave]]

nope, netapp filer outage :(

Comment 3

[Dave Miller [:justdave]]

status update: we have an NFS filer (netapp) which is hanging for 10 to 20 seconds at a time every 3 or 4 minutes.  We have several leads on possible causes, and are working on moving important shares to another filer as well as trying to eliminate the cause of the hangs.

Comment 4

[Dave Miller [:justdave]]

And this is affecting several other sites in addition to mdc and wikimo (sumo, bugzilla, crash-stats, and most stuff hosted on the generic app cluster, too)

Comment 5

[Dave Miller [:justdave]]

Er, forgot to submit when I typed this earlier...

This seems to be cleared up now (as of a few hours ago now).  If there's any further problems please reopen.

Comment 6

[Eric Shepherd [:sheppy]]

MDC is still not working reliably; it may require a restart in order to function correctly now that the problem is resolved with the netapp server.

Comment 7

[Dave Miller [:justdave]]

ok, dekiwiki and httpd have both been restarted on all of the webheads.  Any better?

Comment 8

[Eric Shepherd [:sheppy]]

No, still getting occasional 500 errors. Will copy and paste details next time it happens.

Comment 9

[Dave Miller [:justdave]]

I suspect it's probably the same error we were having the other day that we ended up with the open ticket with mindtouch on... I'm having trouble finding said bug now though.

Comment 10

[Dave Miller [:justdave]]

This was fixed a while back and there's been two or three bugs for the same problem again since, that I'm sure cover recurrences of this.