Status

RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: fubar, Assigned: rbryce)

Tracking

Details

(Reporter)

Description

4 years ago
hgweb8 has new 600gb disks and needs to be re-kicked (with extra inodes, a la https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem). (we're gonna need to re-kick all of them in short order to fix the 95% disk space issues)

I tried to give it a shot, but the ilo console always hangs and I haven't a clue why or what I'm doing wrong. Also, I swear we had more documentation on using the hp ilo from when I tried kickstarting the new gitweb servers, but it seems to have disappeared. :-(
(Reporter)

Comment 1

4 years ago
Hi, disks on the other hgweb heads are up to 96%, so this is critical. I either need someone to kick hgweb8, or help me figure out wtf I'm doing wrong.
(Assignee)

Updated

4 years ago
Assignee: server-ops → rbryce
(Assignee)

Comment 2

4 years ago
Kendall Im on it. Hopefully have this within the hour
(Assignee)

Comment 3

4 years ago
rekicked and puppetized.  Let me know when this is ready for hg specific monitoring.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

4 years ago
nevermind about the monitoring. Its already in place with downtime
(Reporter)

Comment 5

4 years ago
We need a re-install, not enough inodes:

hgweb8.dmz.scl3# df -i
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda3         29M   67K   29M    1% /
tmpfs            1.5M     1  1.5M    1% /dev/shm
/dev/sda1         32K    45   32K    1% /boot

hgweb2.dmz.scl3# df -i
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda3         48M   22M   26M   46% /
tmpfs            1.5M     1  1.5M    1% /dev/shm
/dev/sda1         26K    45   26K    1% /boot


We'll also want to reduce the reserved block percentage, so we're not wasting ~18gb, but that can be done after the kickstart.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 6

4 years ago
rekicked. Still need to tune the reserved blocks

Filesystem       Inodes IUsed    IFree IUse% Mounted on
/dev/sda3      80026800 56257 79970543    1% /
tmpfs           1522603     1  1522602    1% /dev/shm
/dev/sda1         32768    45    32723    1% /boot
(Assignee)

Comment 7

4 years ago
:fubar closing this for now.  Im not sure what setting you want for the reserved blocks. Let me know if you need anything else.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

4 years ago
c.f. bug 1026620 - wrong disks were installed, so we'll need to re-re-install once correct disks are in.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 9

4 years ago
done.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Reporter)

Updated

4 years ago
Blocks: 1038867
(Reporter)

Updated

4 years ago
No longer blocks: 1038867
(Reporter)

Updated

4 years ago
Blocks: 1061937
(Reporter)

Updated

4 years ago
Blocks: 1061938
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.