725838 - please create a firefox/candidates mount

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Aki Sasaki (not active)]

Per #infra,

We're planning to create a 500gb mount off netapp-d to mount on surf and ftp*.m.o on /mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates

The candidates directory/mount doesn't exist yet.  We'll use this for release candidates to unblock our release work.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Filling in details from the IRC conversation:

The problem we're trying to solve is disk IO and network bandwidth problems on netapps A and B, both of which are heavily pegged on both metrics.  Rather than try to fix this on A and B, which requires data we don't have, we chose to move the particular pain point to a less-heavily-loaded netapp -- D, in this case.  To be clear, we are not trying to solve the needs-more-space problem, as suggested by concurrent requests for thumpers.

The available space is small, but getting decent performance is important to turning around releases in a reasonable amount of time.  500GB isn't much, but releng estimates it will be sufficient for somewhere around six weeks until we can start using space in scl3.  We're aware that this is all the space that's available -- killing VMs will not help, as their space is not allocated per-VM.

The mount will need to take place on the ftp heads, on surf, and on dm-download02.

Releng will take care of copying in-progress releases into this directory, and changing the release mechanics to put subsequent builds there.  Releng will (reluctantly!) use symlinks for older/non-current releases.

Nick believes there are no implications of this change for mirrors.

Dan, can you set up the volume, with access parameters identical to 10.253.0.11:/vol/stage ?

Then bump back to me and I'll get the mountpoints set up and coordinate with releng to get things moved over.

Priority is to have this in place before Tuesday for pending releases, but preferably ASAP.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Raising to critical, as part of downgrading bug#725811.

Details in https://bugzilla.mozilla.org/show_bug.cgi?id=725811#c2.

Comment 3

[Dan Parsons [:lerxst]]

The volume has been created. You should be able to mount it as 10.253.0.3:/vol/fx_candidates from any of the hosts that can mount 10.253.0.11:/vol/stage.

Passing it back to dustin to coordinate getting the volume mounted on the necessary servers.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

[root@surf ~]# df -h /mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates
Filesystem            Size  Used Avail Use% Mounted on
10.253.0.3:/vol/fx_candidates
                      500G   64K  500G   1% /mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates

Similarly:
dm-download02:/mnt/ftp/archive.mozilla.org/pub/firefox/candidates
dm-ftp01:/mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates
ftp{1,2}.dmz.sjc1:/mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates (via puppet)

Please re-open if the mounts are incorrect.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I fixed up the permissions on surf:/pub/mozilla.org/firefox/candidates but otherwise this was a really smooth setup. Thanks all!

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

All the existing candidates dirs have been moved into this new mount.