Closed Bug 559215 Opened 15 years ago Closed 15 years ago

Need to take down tinderbox for about 15 seconds to reload the partition table

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: justdave, Assigned: justdave)

References

Details

tinderbox ran out of disk space this morning and lots of stuff died. We cleaned up a little bit, but there really isn't much to clean up, there's been LOTs of builds added to the build farm in recent months, and the data's just coming in too fast, even with the limited retention time. We've already allocated an additional 50 GB to it on the fileserver, and the partition table has already been resized, however in order to pick up the size change, we need to unmount/remount the partition, which requires shutting down both apache and sendmail on the tinderbox server for the 10 to 15 seconds it will take to do this. Filing the bug here to get approval to do this and set a time to do it at that'll cause the least interruption in services.
Blocks: 558858
This is done. It ran out of space again, so it was dead anyway. The fact that it only took 2 days to run out again when I nuked 10 days worth of data to buy time worries me a little though.
Assignee: nobody → justdave
Status: NEW → RESOLVED
Closed: 15 years ago
Component: Release Engineering → Server Operations
QA Contact: release → mrz
Resolution: --- → FIXED
06:46:54 <@nagios> justdave: dm-webtools04:disk - /var/www/iscsi is OK: DISK OK - free space: /var/www/iscsi 49686 MB (24% inode=96%):
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.