808790 - ftp.m.o giving 503 Too Busy or timing out when test slaves try to download builds

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phil Ringnalda (:philor)]

e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=16763104&tree=Mozilla-Inbound for a 503, https://tbpl.mozilla.org/php/getParsedLog.php?id=16763274&tree=Mozilla-Inbound for a time out.

All trees are closed.

Comment 1

[Eric Ziegenhorn :ericz]

Server seems to have gone off the deep end.  Applied vmware keyboard fix, rebooted it, it's checking the disk now and about half way done with that.

Comment 2

[Eric Ziegenhorn :ericz]

webtools2.dmz.scl3.mozilla.com is the server that went haywire.  Will see if I can turn up anything in those links.  It is back up now and those links seem to work for me as well as tinderbox.mozilla.org

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Just to connect the dots, the load on webtools2 put pressure on the scl3 netapp, which is shared by ftp.m.o ?

Comment 4

[Phil Ringnalda (:philor)]

Reopened the trees at 14:40, whether it was coincidence or related.

Comment 5

[Eric Ziegenhorn :ericz]

No, ignore comment #1 and #2 -- they aren't related, just coincidental.  The only thing I can relate to this is that ftp4.dmz.scl3 and ftp6.dmz.scl3 briefly had their load spike and alerted shortly before this bug.  I don't know what caused the spikes as by the time I logged in they were back to normal.