615963 - Symlink build directories on try when moved to old/

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mounir Lamouri (:mounir)]

It would be nice if, when builds are moved from ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds to ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/old
a symlink is created so a link to ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/foo/ automatically point to ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/old/foo

Because, if a give a link like:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-f791551a81ea I got a 404 error but it lives in old/. So, in practice, links have a very small time life :(

This happened to me in bug 600154 unfortunately.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

This is not in fact a simple thing to do, and highlights the fact that the reason things are being moved to /old is to prevent stage from running out of space. 

Right now try-builds are only using 
18% of /mnt/cm-ixstore01/try-builds 
18% of /mnt/netapp/stage/archive.mozilla.org/pub/firefox/try-builds/old


Seems like we might be able to stop moving things to /old if it's possible to bump the disk space on cm-ixstore01, alternately perhaps move the storage location for try-builds to a larger storage location?

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Ignore that last comment, I put:

cd /home/ftp/pub/firefox/try-builds && nice -n 19 find . -mindepth 1 -maxdepth 1 -mtime +3 -type d | cut -c 3- | while read dir; do mv $dir old/; ln -sf old/$dir $dir; done


into place and that should work for now.

Comment 3

[Jake Maul [:jakem]]

Out of our (ServerOps) hands now, but wanted to comment anyway.

/mnt/netapp/stage/archive.mozilla.org/pub/firefox/try-builds
Filesystem            Size  Used Avail Use% Mounted on
10.253.0.11:/vol/stage
                      3.5T  3.1T  472G  87% /mnt/netapp/stage/archive.mozilla.org/pub/firefox

/mnt/netapp/stage/archive.mozilla.org/pub/firefox/try-builds/old
Filesystem            Size  Used Avail Use% Mounted on
10.253.0.139:/data/try-builds
                       17T  2.9T   14T  18% /mnt/cm-ixstore01/try-builds

There's lots of space available on cm-ixstore01/try-builds (aka: try-builds), but comparatively little on stage.

A bigger NetApp volume is unfortunately not really feasible right now. That filer is mostly out of space... that volume could grow a bit, but not more than a few hundred GB... not enough to eliminate the '/old' directory. This may change in the near future, as we're moving to a new datacenter, which would have new storage devices to choose from.

It also might be possible to move the entire thing over to cm-ixstore01, instead of just /old, and thus eliminate /old altogether (just go with some suitable purge time on old dirs). There might be performance implications with this though.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Thanks for the information.  I'm sure this will come up again in the future and we'll have to look at moving builds but for now the symlinking is working and meeting the request of this bug so I'm closing.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I've added
 symlinks -d /home/ftp/pub/firefox/try-builds/ > /dev/null
to the cron job that cleans up old/, so that we leave dangling symlinks behind.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Or even .. so we don't leave dangling symlinks behind.