548566 - Clean up breakpad nfs store.

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aravind Gottipati [:aravind]]

We are running out of inode space on the nfs filestore (on the freebsd box).  This box is currently storing the incoming raw breakpad dumps.  In the past, I haven't cleaned up this space because it was negatively impacting performance.  I have since switched the filestore to create individual filesystems daily.  This allows us to drop them easily with relatively less impact.  However, its finally time to clean up the original large zfs fs to free up inode space.

I plan on working on this tonight (after the socorro 1.5 deployment tonight).  While this zfs store is being deleted, it will negatively impact our ability to accept crashes.  To help with this, I plan on setting the reject percentage in the collector to something high.  Even that may not be sufficient (but it should help).  I am not sure on how long this deletion will take.

This will also require me to redo some symlinks etc, that depend on the fs volume, but that should be relatively harmless.

Comment 1

[Aravind Gottipati [:aravind]]

Kicked off the delete, will probably take a while to finish.

Comment 2

[Aravind Gottipati [:aravind]]

So the cleanup didn't go so great and hung nfsd on the box.  I had to abort the delete and reboot the box.  Its currently recovering the filesystems on it and will probably be a while before its available.  In the meantime, I setup another box to accept incoming crashes etc.

This also means that the nightly cron reports didn't run tonight.  The data should be intact on the system, and I will run the reports when the FS is available again.

Comment 3

[Aravind Gottipati [:aravind]]

More specifically, I am talking about Dbarons library report here, the url reports etc, should run fine.

Comment 4

[matthew zeier [:mrz]]

serving data off different nfs store, no longer necessary.