1097178 - Figure out size of daily symbols cleanup

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[(not currently active) Ted Mielczarek]

As part of migrating symbols to S3 we'll need to move all our existing symbols from the NetApp to S3 storage we'll want to have a period where we turn off the cleanup scripts so that we're not racing to upload things that are going to be deleted. gcox thinks that we have enough headroom on the NetApp volume that we could think on the order of months here, but I think we'll only need weeks.

Comment 1

[Daniel Maher [:phrawzty]]

06:38:43 < phrawzty> ted: comment #0 is.. confusing to read.
06:39:42 <@ted> so we have this netapp volume with all our symbols on it, right
06:39:49 <@ted> and we have this python script that runs daily and does cleanup
06:40:31 < phrawzty> with you so far.
06:42:32 <@ted> okay
06:42:49 <@ted> as part of the migration from netapp->s3 we're going to want to disable the cleanup script, because migration will take a while (~10TB of symbols)
06:43:00 <@ted> and we don't want to deal with trying to sync stuff that's being removed from under us
06:43:34 <@ted> gcox has a graph of our current storage growth over time (our cleanup is not super effective)
06:43:49 <@ted> so we'd like to know how much we're cleaning up every day with the cleanup script so that we can predict what our growth rate would be without cleanup running
06:44:01 <@ted> which tells us how long we could string that migration step along before we have a problem
06:44:26 < phrawzty> ah, ok.
06:48:58 < phrawzty> so i guess you just df before and after the cleanup, which gives you the delta
06:52:40 <@ted> phrawzty: yeah, that might work
06:54:22 <@ted> did i just hook you into fixing this bug for me? because that would be awesome
06:55:09 < phrawzty> yeah, looks like you did.

Comment 2

[Daniel Maher [:phrawzty]]

Added the following lines to the bash wrapper[1] that runs the cleanup script[2][3]:

BEFORE=`df --sync -P /mnt/netapp/breakpad | grep '/' | cut -d ' ' -f 4`
echo "Before: ${BEFORE}" >&2
# [the cleaner is here]
AFTER=`df --sync -P /mnt/netapp/breakpad | grep '/' | cut -d ' ' -f 4`
echo "After: ${AFTER}" >&2
DELTA=$(($AFTER-$BEFORE))
echo "Delta: ${DELTA}" >&2

Output is to stdout so that an email will be generated and sent to ted[4] on completion.


[1] /mnt/netapp/breakpad/cleanup-breakpad-symbols.sh
[2] /mnt/netapp/breakpad/cleanup-breakpad-symbols.py
[3] http://hg.mozilla.org/build/tools/file/tip/buildfarm/breakpad/cleanup-breakpad-symbols.py
[4] Puppet SVN commit 97582

Comment 3

[(not currently active) Ted Mielczarek]

Thanks!

Comment 4

[(not currently active) Ted Mielczarek]

Over the last 8 days, the average cleanup (in bytes) is 25543504, so only about 25MB(!).

If I look at the actual free space available day-to-day, it's pretty erratic (for 6/8 days the free size decreased, for 2 days it increased), averaging out to only ~6MB of growth/day.

Comment 5

[(not currently active) Ted Mielczarek]

I think I've got enough data here. phrawzty: can you undo that change? You can close this bug when that's done.

Comment 6

[Daniel Maher [:phrawzty]]

$ svn ci -m 'revert r97582; bug 1097178'
Sending        manifests/nodes/socorro.pp
Transmitting file data .
Committed revision 97987.

Also removed the additions from /mnt/netapp/breakpad/cleanup-breakpad-symbols.sh on symbols1.dmz.phx1 .