712462 - ftp.m.o try-builds is unresponsive

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aki Sasaki (not active)]

This isn't mounted off cm-ixstore01, but I suspect that the nfs mount off cm-ixstore01 is hanging the server.

This is also true for stage.m.o:

[ffxbld@surf ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c1d0p2      65G   29G   33G  47% /
/dev/cciss/c1d0p1      97M   47M   46M  51% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/cciss/c0d0p1     1.9T  1.7T  180G  91% /data
10.253.0.10:/vol/ftp_stage
                      6.1T  5.5T  539G  92% /mnt/netapp/stage
10.253.0.11:/vol/stage
                      4.0T  3.7T  371G  91% /mnt/netapp/stage/archive.mozilla.org/pub/firefox
10.253.0.11:/vol/socorro/symbols
                      2.0T  1.5T  522G  74% /mnt/breakpad_symbols

(hang)

Filing for tracking; assigning to rbryce who is oncall, so the blocker bug doesn't page him.

Comment 1

[Aki Sasaki (not active)]

[15:02]	<aki>	rbryce: we may want to umount cm-ixstore01 entirely until it's back online, if that's possible. unless we have another fix

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per bear in irc, this is not causing try server to break, so he's leaving try tree open for now.

Comment 3

[Mike Taylor [:bear]]

try-builds/ indexing is not impacting try pushes as the directories are being created for each try build as needed.

this is impacting only directory scans of try-builds/ itself

Comment 4

[Rick Bryce [:rbryce]]

I had to force the removal of the cm-ixstore01 nfs mounts on surf. I also had to change the weighting on the FTP servers in the load balancer.  These 2 issues were causing the ftp load.

Comment 5

[Aki Sasaki (not active)]

Rick: are you able to get to https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/ ?  That's still not working for me. ftp://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/ as well.

Comment 6

[Aki Sasaki (not active)]

Didn't mean to reset bug status; seems like a bugzilla bug.
But we may have to reopen anyway if the above links stay broken.

Comment 7

[Alex Keybl [:akeybl]]

This is no longer blocking us because we can get to individual builds: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-c7bf93c5aa2c/

The index not coming up is still an issue though.

Comment 8

[Aki Sasaki (not active)]

Lowering severity to normal.

Comment 9

[Rick Bryce [:rbryce]]

aki: I can not get to those links.  try-builds is being served from cm-ixstore01 which is down.  I will remove the mounts from the ftp servers

Comment 10

[Aki Sasaki (not active)]

I'm pretty certain cm-ixstore01's try-builds is bind mounted to try-builds/old only, and is hanging ftp.m.o.

Comment 11

[Rick Bryce [:rbryce]]

dumitru removed the cm-ixstore01 mounts from puppet.  try-builds and tinderbox-builds are no longer hanging on ftp.