Last Comment Bug 548566 - Clean up breakpad nfs store.
: Clean up breakpad nfs store.
Product: Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All Other
-- major (vote)
: ---
Assigned To: Aravind Gottipati [:aravind]
: matthew zeier [:mrz]
Depends on:
Blocks: 548833
  Show dependency treegraph
Reported: 2010-02-25 07:28 PST by Aravind Gottipati [:aravind]
Modified: 2015-03-12 08:17 PDT (History)
2 users (show)
mzeier: needs‑downtime-
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Description User image Aravind Gottipati [:aravind] 2010-02-25 07:28:22 PST
We are running out of inode space on the nfs filestore (on the freebsd box).  This box is currently storing the incoming raw breakpad dumps.  In the past, I haven't cleaned up this space because it was negatively impacting performance.  I have since switched the filestore to create individual filesystems daily.  This allows us to drop them easily with relatively less impact.  However, its finally time to clean up the original large zfs fs to free up inode space.

I plan on working on this tonight (after the socorro 1.5 deployment tonight).  While this zfs store is being deleted, it will negatively impact our ability to accept crashes.  To help with this, I plan on setting the reject percentage in the collector to something high.  Even that may not be sufficient (but it should help).  I am not sure on how long this deletion will take.

This will also require me to redo some symlinks etc, that depend on the fs volume, but that should be relatively harmless.
Comment 1 User image Aravind Gottipati [:aravind] 2010-02-25 21:00:34 PST
Kicked off the delete, will probably take a while to finish.
Comment 2 User image Aravind Gottipati [:aravind] 2010-02-26 02:24:40 PST
So the cleanup didn't go so great and hung nfsd on the box.  I had to abort the delete and reboot the box.  Its currently recovering the filesystems on it and will probably be a while before its available.  In the meantime, I setup another box to accept incoming crashes etc.

This also means that the nightly cron reports didn't run tonight.  The data should be intact on the system, and I will run the reports when the FS is available again.
Comment 3 User image Aravind Gottipati [:aravind] 2010-02-26 02:27:16 PST
More specifically, I am talking about Dbarons library report here, the url reports etc, should run fine.
Comment 4 User image matthew zeier [:mrz] 2010-03-10 10:08:39 PST
serving data off different nfs store, no longer necessary.

Note You need to log in before you can comment on or make changes to this bug.