Closed Bug 725826 Opened 12 years ago Closed 12 years ago

genericadm.private.phx1 is running out of space

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mburns, Assigned: dparsons)

References

Details

[14:01:56] <nagios-phx1> [168] genericadm.private.phx1:root partition is CRITICAL: DISK CRITICAL - free space: / 785 MB (2% inode=29%):
[14:04:26] <nagios-phx1> [169] genericadm.private.phx1:disk - / is CRITICAL: DISK CRITICAL - free space: / 801 MB (3% inode=29%):
This is a VM... can we grow the volume, and/or make a new (bigger) one and move all of /data to the new one?
FYI, I already did what I could to shrink the .git directories. Running out of space here is basically just a function of how every site exists on this about 6 times... dev, stage, prod, in src and www trees.

I imagine deduplication is already enabled on the underlying netapp volume, but I'm not sure how that would appear inside a VM. Still, might be worth checking.
I will power off the VM today at 8 PM PST to add more disk space (I can give it an extra 10GB).
Assignee: server-ops → dgherman
@dumitru if you can please increase this to 15GB or more that would be good.
Actually, I would push for a bit more than comment 4 or 5...

mradm02's /data/generic/www is 7GB right now (plus some stuff outside of that, I think). Some of that is probably not moving, but a lot of it is... figure around 5GB will get migrated eventually. Some of that will want to grow dev or stage environments, and all of it will grow a separate src/ tree that's not included in that estimate.

So that's 10GB just for src and www, just for prod. Figure maybe 1/2 of it will get a dev or stage env (some will probably get both)... many of these sites might already have dev or stage instances on mrapp-stage02, mrapp-stage04, or other random server(s) that will get merged in to genericrhel6 at some point. That's over 15GB already, and that only accounts for straight migrations from generic in SJC1 to genericrhel6 in PHX1... doesn't even touch other potential migrations or growth.

Can we get this increased by, say, 30GB (up to 60GB total)? That should be enough to finish migrations and allow for some growth. I fear if we do less then we'll be back in the same boat pretty quickly.
The VM runs on the 'old' ESX in PHX. That volume has currently only 57GB free disk space left.
That's... not good. I don't want to eat fully half of the remaining space with this, but at the same time if we don't do this we're going to have problems very soon... like, SCL3-migration-trains-soon.

Is it feasible to have this moved to the newer ESX cluster and grown there? Not HCI, obviously... the other one. I don't know what storage is currently backing that. Same NetApps I presume, but maybe different volumes... might have more space open.

CC'ing Dan for input.
I can easily migrate it to the new ESX (non-hci) cluster in phx1. I can handle adding more disk space to it then, too. 

(1) Is 8PM a good time, like dumitru suggested?

(2) How much more disk space should I give it? 30GB more, to 60GB total? We have room.
<3's to both... 8PM is fine, and +30GB is perfect. Thanks!
Assignee: dgherman → dparsons
This is done. The VM has been migrated to the new ESX cluster and disk space expanded. Inventory updated too.

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       57G   25G   30G  46% /
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.