1034237 - re-kick hgweb8

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Kendall Libby [:fubar] (he/him)]

hgweb8 has new 600gb disks and needs to be re-kicked (with extra inodes, a la https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem). (we're gonna need to re-kick all of them in short order to fix the 95% disk space issues)

I tried to give it a shot, but the ilo console always hangs and I haven't a clue why or what I'm doing wrong. Also, I swear we had more documentation on using the hp ilo from when I tried kickstarting the new gitweb servers, but it seems to have disappeared. :-(

Comment 1

[Kendall Libby [:fubar] (he/him)]

Hi, disks on the other hgweb heads are up to 96%, so this is critical. I either need someone to kick hgweb8, or help me figure out wtf I'm doing wrong.

Comment 2

[Rick Bryce [:rbryce]]

Kendall Im on it. Hopefully have this within the hour

Comment 3

[Rick Bryce [:rbryce]]

rekicked and puppetized.  Let me know when this is ready for hg specific monitoring.

Comment 4

[Rick Bryce [:rbryce]]

nevermind about the monitoring. Its already in place with downtime

Comment 5

[Kendall Libby [:fubar] (he/him)]

We need a re-install, not enough inodes:

hgweb8.dmz.scl3# df -i
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda3         29M   67K   29M    1% /
tmpfs            1.5M     1  1.5M    1% /dev/shm
/dev/sda1         32K    45   32K    1% /boot

hgweb2.dmz.scl3# df -i
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda3         48M   22M   26M   46% /
tmpfs            1.5M     1  1.5M    1% /dev/shm
/dev/sda1         26K    45   26K    1% /boot


We'll also want to reduce the reserved block percentage, so we're not wasting ~18gb, but that can be done after the kickstart.

Comment 6

[Rick Bryce [:rbryce]]

rekicked. Still need to tune the reserved blocks

Filesystem       Inodes IUsed    IFree IUse% Mounted on
/dev/sda3      80026800 56257 79970543    1% /
tmpfs           1522603     1  1522602    1% /dev/shm
/dev/sda1         32768    45    32723    1% /boot

Comment 7

[Rick Bryce [:rbryce]]

:fubar closing this for now.  Im not sure what setting you want for the reserved blocks. Let me know if you need anything else.

Comment 8

[Kendall Libby [:fubar] (he/him)]

c.f. bug 1026620 - wrong disks were installed, so we'll need to re-re-install once correct disks are in.

Comment 9

[Rick Bryce [:rbryce]]

done.