Last Comment Bug 517015 - Verify/Increase NETAPP_STORAGE space
: Verify/Increase NETAPP_STORAGE space
10/29/2009 @ 9pm
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
: -- normal (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Depends on:
Blocks: 482837
  Show dependency treegraph
Reported: 2009-09-16 11:06 PDT by Wil Clouser [:clouserw]
Modified: 2013-10-09 10:29 PDT (History)
2 users (show)
mzeier: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Description Wil Clouser [:clouserw] 2009-09-16 11:06:58 PDT
I don't know how much space the NETAPP_STORAGE mount has on the AMO boxes, but plans are coming together to pull the images out of the database(!) in bug 482837 and that's going to be about 2-3G extra right there, and then any new images will be put on disk as well.  Can you verify that we'll have enough space for this now and in the future?  Also, what is the nagios disk space monitoring threshold on this mount since it'll be growing faster after we close that bug?
Comment 1 Jeremy Orem [:oremj] 2009-09-17 16:32:38 PDT
Right now amo is sharing a mount with the rest of the webapps. That mount has 14G free. I just created a new mount with 90G usable space that I want to migrate amo to soon, but that'll probably need some downtime.
Comment 2 Wil Clouser [:clouserw] 2009-09-17 16:35:16 PDT
Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I can add it to the production push steps
Comment 3 matthew zeier [:mrz] 2009-09-17 16:37:37 PDT
What's the largest image size?  Want to make sure ZXTM is set to cache something that big (Amsterdam is already set to 20MB).
Comment 4 Wil Clouser [:clouserw] 2009-09-17 16:39:27 PDT
god, I hope it's under that
Comment 5 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2009-09-17 16:41:26 PDT
 1.6M is the largest image I found.
Comment 6 matthew zeier [:mrz] 2009-10-02 11:40:05 PDT
(In reply to comment #2)
> Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I
> can add it to the production push steps

Did we do that?
Comment 7 Jeremy Orem [:oremj] 2009-10-02 14:53:25 PDT
Nope, but the mount is ready whenever. Just need some time to rsync over. Is it possible to disable new addon uploads?
Comment 8 Wil Clouser [:clouserw] 2009-10-02 16:37:24 PDT
We had a "disable submissions" flag but the latest dev tools apparently no longer respects it.  So, no.
Comment 9 matthew zeier [:mrz] 2009-10-07 16:21:45 PDT
Wil, I believe this can wait till the next content push - do you know when that is?
Comment 10 Wil Clouser [:clouserw] 2009-10-07 16:26:12 PDT
The next AMO cycle is freeze on the 13th, push on the 20th.  We hope to get this in this cycle but with the search bugs taking up time I'm not sure we'll be able to.

We'll need to preview/verify this first so preview.amo will need to have ample space as well.
Comment 11 Wil Clouser [:clouserw] 2009-10-12 12:56:31 PDT
Comment 12 Jeremy Orem [:oremj] 2009-10-13 00:51:35 PDT
Preview is already using the new mount. We can switch over addons whenever. Only catch is we'll need a way to disable submissions while we sync the files, which could take an hour or so.
Comment 13 Wil Clouser [:clouserw] 2009-10-13 08:36:06 PDT
Can't we sync them ahead of time and then in our regularly scheduled downtime do another sync to pick up any last minute changes?
Comment 14 Jeremy Orem [:oremj] 2009-10-13 09:57:08 PDT
Yeah, I plan on doing that, but the netapp is crazy slow so I'd still expect it to take a little while.
Comment 15 Wil Clouser [:clouserw] 2009-10-21 09:04:15 PDT
It sounds like we're going to have some downtime on Thursday for database upgrades.  Can we do this at the same time?
Comment 16 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2009-10-21 09:17:20 PDT
We should be able to keep the diffs fairly low, if we start sooner than later with syncing.
Comment 17 Wil Clouser [:clouserw] 2009-10-21 22:35:27 PDT
(In reply to comment #15)
> It sounds like we're going to have some downtime on Thursday for database
> upgrades.  Can we do this at the same time?

Tomorrow is Thursday.  Can we plan for this?
Comment 18 Wil Clouser [:clouserw] 2009-10-23 10:11:35 PDT
What is the new ETA for this?
Comment 19 matthew zeier [:mrz] 2009-10-24 10:49:51 PDT
IT fail on docs to carry this out last Thurs.  This coming Tues @ 9pm work for everyone?
Comment 20 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2009-10-24 11:01:48 PDT
Any idea how long this will take?

It seems to me we can start earlier with no downtime in trying to keep the two storage units in sync, and then that should minimize downtime when we cutover.
Comment 21 matthew zeier [:mrz] 2009-10-24 11:08:11 PDT
I don't but what you suggest is what we'd do anyways.
Comment 22 Jeremy Orem [:oremj] 2009-10-27 11:19:38 PDT
Any reason not to knock this out tonight? I've done the initial rsync and am timing a resync right now. I'll post the estimate in a bit.
Comment 23 Jeremy Orem [:oremj] 2009-10-27 11:25:32 PDT
We are delaying for the release.
Comment 24 Jeremy Orem [:oremj] 2009-10-29 10:20:23 PDT
The rersync took 36 min.
Comment 25 Jeremy Orem [:oremj] 2009-10-29 21:29:06 PDT
All done.

Note You need to log in before you can comment on or make changes to this bug.