708865 - Move firefox/nightly/20{04,05,06,07,08,09,10} to 18TB disk on cm-ixstore01.san.m.o, and create softlinks/bind mount on stage

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Aki Sasaki (not active)]

We have a backup, so we're not blocked on Ops.
We may want to have a cm-ixstore01 nightly/old mount so we're not in danger of cleaning up old nightlies when we clean up tinderbox-builds/old or try-builds/old.

Coop: this is a RelEng bug; please don't move to serverops.
If there's a dependency, let's file it in serverops and block this bug on it.

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to Aki Sasaki [:aki] from comment #0)
> Coop: this is a RelEng bug; please don't move to serverops.
> If there's a dependency, let's file it in serverops and block this bug on it.

We've decided to go forward with this. Given that no one on releng has the required access to do this move, I *am* reassigning to relops.

Amy is going to make sure the tape backup is tested first, possibly by restoring directly to cm-ixstore01 to avoid the copy step, but we're unsure whether that will work due to the netapp backup format.

Comment 2

[Amy Rich [:arr] [:arich]]

Are we talking about /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/ or some other directory?  If that's the correct directory, it's not the one where we keep running out of space (though it also has high usage):

df -m:

4299162   3961193    337970  93% /mnt/netapp/stage/archive.mozilla.org/pub/firefox
6765007   6608403    156605  98% /mnt/netapp/stage

The latter is the partition where the biggest space crunch is.

Comment 3

[Aki Sasaki (not active)]

We run out of space in both all the time.
The reason we're less than 95% right now is because we mounted netapp-d for firefox/candidates.

We can potentially do the same thing with older mobile,thunderbird,seamonkey nightlies, which would free up space on the other mount.  We don't have tape backups of those yet.

Comment 4

[Aki Sasaki (not active)]

also xulrunner.

Comment 5

[Justin Wood (:Callek)]

(In reply to Aki Sasaki [:aki] from comment #3)
> We run out of space in both all the time.
> The reason we're less than 95% right now is because we mounted netapp-d for
> firefox/candidates.
> 
> We can potentially do the same thing with older mobile,thunderbird,seamonkey
> nightlies, which would free up space on the other mount.  We don't have tape
> backups of those yet.

I for one am perfectly happy with SeaMonkey older nightlies getting moved like this, but I also want a sane backup solution for them like the Firefox ones.

Comment 6

[Aki Sasaki (not active)]

(In reply to Justin Wood (:Callek) from comment #5)
> (In reply to Aki Sasaki [:aki] from comment #3)
> > We run out of space in both all the time.
> > The reason we're less than 95% right now is because we mounted netapp-d for
> > firefox/candidates.
> > 
> > We can potentially do the same thing with older mobile,thunderbird,seamonkey
> > nightlies, which would free up space on the other mount.  We don't have tape
> > backups of those yet.
> 
> I for one am perfectly happy with SeaMonkey older nightlies getting moved
> like this, but I also want a sane backup solution for them like the Firefox
> ones.

Right, hence the key word "yet".
I would strongly object to moving any of these without a tested tape backup.

Comment 7

[Amy Rich [:arr] [:arich]]

And just for info on how much space we're talking about:

/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly

du -sk 200[4-9] 2010
13615380	2004
25917548	2005
104438708	2006
80179260	2007
112655140	2008
168015676	2009
181753152	2010

Comment 8

[Amy Rich [:arr] [:arich]]

Based on our conversation yesterday, there's not action for relops right now, so I'm going to hand this back over to release for more information or resolution.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I've marked bug 715840 resolved by way of moving firefox/nightly/{2004..2008} to the other partition, which got a lot bigger along the way, so there's no need to move ahead with this bug.