517015 - Verify/Increase NETAPP_STORAGE space

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Wil Clouser [:clouserw]]

I don't know how much space the NETAPP_STORAGE mount has on the AMO boxes, but plans are coming together to pull the images out of the database(!) in bug 482837 and that's going to be about 2-3G extra right there, and then any new images will be put on disk as well.  Can you verify that we'll have enough space for this now and in the future?  Also, what is the nagios disk space monitoring threshold on this mount since it'll be growing faster after we close that bug?

Comment 1

[Jeremy Orem [:oremj]]

Right now amo is sharing a mount with the rest of the webapps. That mount has 14G free. I just created a new mount with 90G usable space that I want to migrate amo to soon, but that'll probably need some downtime.

Comment 2

[Wil Clouser [:clouserw]]

Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I can add it to the production push steps

Comment 3

[matthew zeier [:mrz]]

What's the largest image size?  Want to make sure ZXTM is set to cache something that big (Amsterdam is already set to 20MB).

Comment 4

[Wil Clouser [:clouserw]]

god, I hope it's under that

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

 1.6M is the largest image I found.

Comment 6

[matthew zeier [:mrz]]

(In reply to comment #2)
> Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I
> can add it to the production push steps

Did we do that?

Comment 7

[Jeremy Orem [:oremj]]

Nope, but the mount is ready whenever. Just need some time to rsync over. Is it possible to disable new addon uploads?

Comment 8

[Wil Clouser [:clouserw]]

We had a "disable submissions" flag but the latest dev tools apparently no longer respects it.  So, no.

Comment 9

[matthew zeier [:mrz]]

Wil, I believe this can wait till the next content push - do you know when that is?

Comment 10

[Wil Clouser [:clouserw]]

The next AMO cycle is freeze on the 13th, push on the 20th.  We hope to get this in this cycle but with the search bugs taking up time I'm not sure we'll be able to.

We'll need to preview/verify this first so preview.amo will need to have ample space as well.

Comment 11

[Wil Clouser [:clouserw]]

Status?

Comment 12

[Jeremy Orem [:oremj]]

Preview is already using the new mount. We can switch over addons whenever. Only catch is we'll need a way to disable submissions while we sync the files, which could take an hour or so.

Comment 13

[Wil Clouser [:clouserw]]

Can't we sync them ahead of time and then in our regularly scheduled downtime do another sync to pick up any last minute changes?

Comment 14

[Jeremy Orem [:oremj]]

Yeah, I plan on doing that, but the netapp is crazy slow so I'd still expect it to take a little while.

Comment 15

[Wil Clouser [:clouserw]]

It sounds like we're going to have some downtime on Thursday for database upgrades.  Can we do this at the same time?

Comment 16

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

We should be able to keep the diffs fairly low, if we start sooner than later with syncing.

Comment 17

[Wil Clouser [:clouserw]]

(In reply to comment #15)
> It sounds like we're going to have some downtime on Thursday for database
> upgrades.  Can we do this at the same time?

Tomorrow is Thursday.  Can we plan for this?

Comment 18

[Wil Clouser [:clouserw]]

What is the new ETA for this?

Comment 19

[matthew zeier [:mrz]]

IT fail on docs to carry this out last Thurs.  This coming Tues @ 9pm work for everyone?

Comment 20

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Any idea how long this will take?

It seems to me we can start earlier with no downtime in trying to keep the two storage units in sync, and then that should minimize downtime when we cutover.

Comment 21

[matthew zeier [:mrz]]

I don't but what you suggest is what we'd do anyways.

Comment 22

[Jeremy Orem [:oremj]]

Any reason not to knock this out tonight? I've done the initial rsync and am timing a resync right now. I'll post the estimate in a bit.

Comment 23

[Jeremy Orem [:oremj]]

We are delaying for the release.

Comment 24

[Jeremy Orem [:oremj]]

The rersync took 36 min.

Comment 25

[Jeremy Orem [:oremj]]

All done.