706607 - NFS mounts from cm-ixstore01 are missing (Firefox trees closed for missing firefox/tinderbox-builds)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Alice0775 White]

Just now, I can not access http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ due to 403 Forbidden.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Missing tinderbox builds has all sorts of bad effects on uploading Firefox compile jobs, tests jobs doing downloads, logs getting uploaded (impacting TBPL and developers debugging compile/test failures). --> blocker

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

More specifically
[03] dm-ftp01:NFS mount - /mnt/cm-ixstore01/try-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/try-builds dont exists in filesystem :
[04] ftp1.dmz.sjc1:NFS mount - /mnt/cm-ixstore01/try-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/try-builds dont exists in filesystem :
[05] ftp2.dmz.sjc1:NFS mount - /mnt/cm-ixstore01/tinderbox-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/tinderbox-builds dont exists in filesystem :
[09] surf:NFS mount - /mnt/cm-ixstore01/tinderbox-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/tinderbox-builds dont exists in filesystem :
[11] dm-ftp01:NFS mount - /mnt/cm-ixstore01/tinderbox-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/tinderbox-builds dont exists in filesystem :
[12] ftp1.dmz.sjc1:NFS mount - /mnt/cm-ixstore01/tinderbox-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/tinderbox-builds dont exists in filesystem :
[13] ftp2.dmz.sjc1:NFS mount - /mnt/cm-ixstore01/try-builds is CRITICAL: CRITICAL: /mnt/cm-ixstore01/try-builds dont exists in filesystem :

So looks like NFS isn't working on cm-ixstore01.

Comment 3

[Mike Taylor [:bear]]

Trees closed by buildduty and developers notified

Comment 5

[Dave Miller [:justdave]]

summary of fix instructions from IRC for future reference:

- shut down httpd and xinetd
- drop in an /etc/nologin file
- lsof | grep {path to mountpoint}
  - see what has open files on it
  - shut down any app that has files open on it
- unmount the entire tree (in the correct order to get them all unmounted)
- mount -a
- remove /etc/nologin
- start up httpd and xinetd (and anything else you had to kill to get it to unmount)

Comment 6

[Dave Miller [:justdave]]

and of course, the above only helps if the server is working, which it's not.

The /data partition on cm-ixstore01 is offline with i/o errors.

still debugging

Comment 7

[Dave Miller [:justdave]]

unknown what caused the external array to drop offline, but the machine wouldn't unmount it and wouldn't shut down in whatever condition it was in.  I power-cycled the box, and it came back up and mounted everything, so stuff should be coming back online now.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

:justdave, was there any data loss as a result of this outage?

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

trees reopened and developers notified.

Comment 10

[Dave Miller [:justdave]]

(In reply to John O'Duinn [:joduinn] from comment #8)
> :justdave, was there any data loss as a result of this outage?

You guys can answer that better than I can, I have no idea what's supposed to be there.

There was an abundance of items present in that directory after it mounted, that's all I know.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

I would expect there would have been quite a lot of builds which could not be uploaded to firefox/tinderbox-builds while that directory was not available, which shows up as red on TBPL. By the looks of the Mozilla-Inbound tree most of that has been rebuilt.