709488 - Builds and logs from trees other than try are not available on ftp.m.o where tbpl thinks they will be

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

Logs from builds on tryserver (which as I understand it is a different mount) are where they ought to be, like https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-1cb485c4f804/

Logs from what little is being done (most trees are closed already from a different problem) on non-try ought to be in places like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/1323525604/ but are not there to be found.

Comment 1

[Ben Kero [:bkero]]

It looks like this could be related to a cm-ixstore outage:

[root@dm-ftp01 ftp.mozilla.org-include]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol01
                      73514160  11913224  57806404  18% /
/dev/md0                124323     32192     85712  28% /boot
tmpfs                  8316632         0   8316632   0% /dev/shm
10.253.0.10:/vol/ftp_stage
                     6442450976 5671325568 771125408  89% /mnt/netapp/stage
10.253.0.11:/vol/stage
                     4294967296 3959550848 335416448  93% /mnt/netapp/stage/archive.mozilla.org/pub/firefox
df: `/mnt/cm-ixstore01/try-builds': Stale NFS file handle
df: `/mnt/cm-ixstore01/tinderbox-builds': Stale NFS file handle
df: `/mnt/netapp/stage/archive.mozilla.org/pub/firefox/try-builds/old': Stale NFS file handle
df: `/mnt/netapp/stage/archive.mozilla.org/pub/firefox/tinderbox-builds/old': Stale NFS file handle


[root@cm-ixstore01 ~]# dmesg|tail -5
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.

[root@cm-ixstore01 ~]# cd /data
[root@cm-ixstore01 data]# ls
ls: reading directory .: Input/output error
[root@cm-ixstore01 data]#

Comment 2

[Phil Ringnalda (:philor)]

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/1323525604/ does not exist, but http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94/ does.

Comment 3

[Ben Kero [:bkero]]

This doesn't seem actionable from an IT perspective.  Both your working and non-working link examples are both located on the same (accessible) mount, and both writable by surf.

1323525604 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94

I have no idea why this is symlinking to /home/ftp/pub, however that directory does not exist.

[root@dm-ftp01 mozilla-central-linux-pgo]# ls /home/ftp/pub
ls: /home/ftp/pub: No such file or directory

Could it be possible to be a problem with the tools that generate these files (and symlinks?)

Comment 4

[Phil Ringnalda (:philor)]

Yes, that's why along with that comment I moved it to releng and marked it as blocking bug 671450, which even though it doesn't look like it thought it was going into production at 1am on a Saturday seems like it must have, since according to my understanding of it, it switches us from creating 1323525604/ to creating 20111210060004-63bff373cb94/ with a symlink for 1323525604/. Or without a symlink. Or with a symlink that doesn't work.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The problem is that bug 671450 is making absolute symlinks to a mount that probably doesn't exist on ftp.m.o

nthomas@surf:/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo$ ls -l
total 26608
...
drwxr-xr-x 2 ffxbld firefox     4096 Dec  9 19:30 1323482404
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 00:48 1323504004 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210000004-63bff373cb94
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 04:09 1323515510 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210031150-63bff373cb94
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 06:48 1323525604 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 01:30 20111210000004-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 04:47 20111210031150-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 07:38 20111210060004-63bff373cb94

If it made a relative symlink they it would be fine.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've backed out bug 671450 and updated stage. Will fix the existing symlinks now.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Done.