646709 - mount cm-ixstore01 drive on staging-stage

Status: RESOLVED WORKSFORME

(Infrastructure & Operations :: RelOps: General, task)

Description

[Aki Sasaki (not active)]

We keep on running out of disk on staging-stage.build.mozilla.org, which turns all of our staging testing red and makes it difficult for us to test our changes before we roll them out.

In the meantime, we have 17T of storage on cm-ixstore01 with a bit less than 16T free.

Could we:

a) mount that drive on staging-stage.b.m.o
b) such that we can never overwrite or corrupt production data (/data/try-builds, /data/tinderbox-builds)
c) a quota would be awesome, but otherwise the massive amount of free space and some diligent cleanup on our end should be good

This will save us many headaches on staging-stage.

Comment 1

[Aki Sasaki (not active)]

This may be in addition to, or instead of, bug 640990.

Comment 2

[Amy Rich [:arr] [:arich]]

Since cm-ixstore01 will be going away when we move to scl3, just checking back in to see what the current plan is here (also ccing cshields since staging-stage is managed by his group, I believe, and lerxst since he's the storage guy).

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #0)
> We keep on running out of disk on staging-stage.build.mozilla.org, which
> turns all of our staging testing red and makes it difficult for us to test
> our changes before we roll them out.

You mean dev-stage01, right?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> (In reply to Aki Sasaki [:aki] from comment #0)
> > We keep on running out of disk on staging-stage.build.mozilla.org, which
> > turns all of our staging testing red and makes it difficult for us to test
> > our changes before we roll them out.
> 
> You mean dev-stage01, right?

Arr just pointed out that this bug is 11 months old.....so Aki was correct at the time! bug 664144 is where dev-stage01 was set-up/swapped in.

Comment 5

[Aki Sasaki (not active)]

I think we were consistently running out of space on staging-stage so this was a workaround to try to fix that.
The dev-stage01 replacement appears to be currently sufficient.
Closing.