Builds and logs from trees other than try are not available on ftp.m.o where tbpl thinks they will be

RESOLVED FIXED

Status

Release Engineering
General
--
blocker
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: philor, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
Logs from builds on tryserver (which as I understand it is a different mount) are where they ought to be, like https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bmo@edmorley.co.uk-1cb485c4f804/

Logs from what little is being done (most trees are closed already from a different problem) on non-try ought to be in places like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/1323525604/ but are not there to be found.

Comment 1

7 years ago
It looks like this could be related to a cm-ixstore outage:

[root@dm-ftp01 ftp.mozilla.org-include]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol01
                      73514160  11913224  57806404  18% /
/dev/md0                124323     32192     85712  28% /boot
tmpfs                  8316632         0   8316632   0% /dev/shm
10.253.0.10:/vol/ftp_stage
                     6442450976 5671325568 771125408  89% /mnt/netapp/stage
10.253.0.11:/vol/stage
                     4294967296 3959550848 335416448  93% /mnt/netapp/stage/archive.mozilla.org/pub/firefox
df: `/mnt/cm-ixstore01/try-builds': Stale NFS file handle
df: `/mnt/cm-ixstore01/tinderbox-builds': Stale NFS file handle
df: `/mnt/netapp/stage/archive.mozilla.org/pub/firefox/try-builds/old': Stale NFS file handle
df: `/mnt/netapp/stage/archive.mozilla.org/pub/firefox/tinderbox-builds/old': Stale NFS file handle


[root@cm-ixstore01 ~]# dmesg|tail -5
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.
Filesystem sdb1: xfs_log_force: error 5 returned.

[root@cm-ixstore01 ~]# cd /data
[root@cm-ixstore01 data]# ls
ls: reading directory .: Input/output error
[root@cm-ixstore01 data]#

Updated

7 years ago
Assignee: server-ops → bkero
Depends on: 707270
(Reporter)

Comment 2

7 years ago
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/1323525604/ does not exist, but http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94/ does.
Assignee: bkero → nobody
Blocks: 671450
Component: Server Operations → Release Engineering
No longer depends on: 707270
QA Contact: cshields → release
Summary: Builds and logs from trees other than try are missing from ftp.m.o → Builds and logs from trees other than try are not available on ftp.m.o where tbpl thinks they will be

Comment 3

7 years ago
This doesn't seem actionable from an IT perspective.  Both your working and non-working link examples are both located on the same (accessible) mount, and both writable by surf.

1323525604 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94

I have no idea why this is symlinking to /home/ftp/pub, however that directory does not exist.

[root@dm-ftp01 mozilla-central-linux-pgo]# ls /home/ftp/pub
ls: /home/ftp/pub: No such file or directory

Could it be possible to be a problem with the tools that generate these files (and symlinks?)
Component: Release Engineering → Server Operations
(Reporter)

Comment 4

7 years ago
Yes, that's why along with that comment I moved it to releng and marked it as blocking bug 671450, which even though it doesn't look like it thought it was going into production at 1am on a Saturday seems like it must have, since according to my understanding of it, it switches us from creating 1323525604/ to creating 20111210060004-63bff373cb94/ with a symlink for 1323525604/. Or without a symlink. Or with a symlink that doesn't work.
Component: Server Operations → Release Engineering
(Assignee)

Comment 5

7 years ago
The problem is that bug 671450 is making absolute symlinks to a mount that probably doesn't exist on ftp.m.o

nthomas@surf:/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo$ ls -l
total 26608
...
drwxr-xr-x 2 ffxbld firefox     4096 Dec  9 19:30 1323482404
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 00:48 1323504004 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210000004-63bff373cb94
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 04:09 1323515510 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210031150-63bff373cb94
lrwxrwxrwx 1 ffxbld firefox       92 Dec 10 06:48 1323525604 -> /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux-pgo/20111210060004-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 01:30 20111210000004-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 04:47 20111210031150-63bff373cb94
drwxr-xr-x 2 ffxbld firefox     4096 Dec 10 07:38 20111210060004-63bff373cb94

If it made a relative symlink they it would be fine.
Assignee: nobody → nrthomas
(Assignee)

Comment 6

7 years ago
I've backed out bug 671450 and updated stage. Will fix the existing symlinks now.
(Assignee)

Comment 7

7 years ago
Done.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.