Verify/Increase NETAPP_STORAGE space

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: clouserw, Assigned: oremj)

Tracking

other
All
Other
Bug Flags:
needs-downtime +

Details

(Whiteboard: 10/29/2009 @ 9pm)

(Reporter)

Description

8 years ago
I don't know how much space the NETAPP_STORAGE mount has on the AMO boxes, but plans are coming together to pull the images out of the database(!) in bug 482837 and that's going to be about 2-3G extra right there, and then any new images will be put on disk as well.  Can you verify that we'll have enough space for this now and in the future?  Also, what is the nagios disk space monitoring threshold on this mount since it'll be growing faster after we close that bug?
(Assignee)

Updated

8 years ago
Assignee: server-ops → jeremy.orem+bugs
(Assignee)

Comment 1

8 years ago
Right now amo is sharing a mount with the rest of the webapps. That mount has 14G free. I just created a new mount with 90G usable space that I want to migrate amo to soon, but that'll probably need some downtime.
(Reporter)

Comment 2

8 years ago
Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I can add it to the production push steps

Comment 3

8 years ago
What's the largest image size?  Want to make sure ZXTM is set to cache something that big (Amsterdam is already set to 20MB).
(Reporter)

Comment 4

8 years ago
god, I hope it's under that
 1.6M is the largest image I found.

Comment 6

8 years ago
(In reply to comment #2)
> Sweet.  Do you want to do that at the same time that we upgrade AMO (9/29)?  I
> can add it to the production push steps

Did we do that?
(Assignee)

Comment 7

8 years ago
Nope, but the mount is ready whenever. Just need some time to rsync over. Is it possible to disable new addon uploads?
(Reporter)

Comment 8

8 years ago
We had a "disable submissions" flag but the latest dev tools apparently no longer respects it.  So, no.

Comment 9

8 years ago
Wil, I believe this can wait till the next content push - do you know when that is?
(Reporter)

Comment 10

8 years ago
The next AMO cycle is freeze on the 13th, push on the 20th.  We hope to get this in this cycle but with the search bugs taking up time I'm not sure we'll be able to.

We'll need to preview/verify this first so preview.amo will need to have ample space as well.
(Reporter)

Comment 11

8 years ago
Status?
(Assignee)

Comment 12

8 years ago
Preview is already using the new mount. We can switch over addons whenever. Only catch is we'll need a way to disable submissions while we sync the files, which could take an hour or so.
(Reporter)

Comment 13

8 years ago
Can't we sync them ahead of time and then in our regularly scheduled downtime do another sync to pick up any last minute changes?
(Assignee)

Comment 14

8 years ago
Yeah, I plan on doing that, but the netapp is crazy slow so I'd still expect it to take a little while.
(Reporter)

Comment 15

8 years ago
It sounds like we're going to have some downtime on Thursday for database upgrades.  Can we do this at the same time?
We should be able to keep the diffs fairly low, if we start sooner than later with syncing.
(Reporter)

Updated

8 years ago
Blocks: 482837
(Reporter)

Comment 17

8 years ago
(In reply to comment #15)
> It sounds like we're going to have some downtime on Thursday for database
> upgrades.  Can we do this at the same time?

Tomorrow is Thursday.  Can we plan for this?
(Reporter)

Comment 18

8 years ago
What is the new ETA for this?
IT fail on docs to carry this out last Thurs.  This coming Tues @ 9pm work for everyone?
Flags: needs-downtime+
Whiteboard: 10/27/2009 @ 9pm
Any idea how long this will take?

It seems to me we can start earlier with no downtime in trying to keep the two storage units in sync, and then that should minimize downtime when we cutover.
I don't but what you suggest is what we'd do anyways.

Updated

8 years ago
Whiteboard: 10/27/2009 @ 9pm → 10/29/2009 @ 9pm
(Assignee)

Comment 22

8 years ago
Any reason not to knock this out tonight? I've done the initial rsync and am timing a resync right now. I'll post the estimate in a bit.
(Assignee)

Comment 23

8 years ago
We are delaying for the release.
(Assignee)

Comment 24

8 years ago
The rersync took 36 min.
(Assignee)

Comment 25

8 years ago
All done.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.