540713 - Running out of space for symbols

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

From nagios:
dm-symbolpush01:disk - /mnt/netapp/breakpad is WARNING: DISK WARNING - free space: /mnt/netapp/breakpad 82194 MB (8% inode=94%)

We run 
 http://hg.mozilla.org/build/tools/file/default/buildfarm/breakpad/cleanup-breakpad-symbols.py
on stage by using crontabs on various accounts. At some point in October last year the mountpoint changed and the script stopped working.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Updated crontab for ffxbld, seabld, calbld (for sunbird), tbirdbld, xrbld. caminobld doesn't have a cron job but a dry-run doesn't report anything to remove so I'm going to leave it that way.

For Firefox there are a bunch of old electrolysis, places, and tracemonkey manifests, where old = last june. Not sure why they're not being removed, perhaps a shared file with something that doesn't deserve to die yet ?

(There are only symbols stored on this mount now, so no more invisible minidumps and the df on stage reads true.)

Comment 2

[(not currently active) Ted Mielczarek]

The cleanup script tries to preserve 30 days of symbols per branch, where a branch is "version number base + branch name", so like "1.9.3-tracemonkey". If a version bump happens, like 1.9.2 -> 1.9.3, we'll probably strand a set of symbols per branch, like all the 1.9.2 tracemonkey and electrolysis symbols. We should probably either make the script smarter somehow or just give it a mode where we can say "remove all 1.9.2*pre-tracemonkey symbols". We can't just directly remove all older branch stuff, because we want to keep 30 days of 1.9.{0,1,2} branch symbols. We could just make it remove old *project* branch stuff, I suppose.

Comment 3

[(not currently active) Ted Mielczarek]

(Either way, we should file a followup on that.)

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We could just let the (age < 180 day) condition apply.

Comment 5

[(not currently active) Ted Mielczarek]

As long as we only handle nightly symbols (versions containing pre), then that should be fine.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

All the old release manifests look like they're there. And we have nearly 400G free now the cleanup is working correctly for current branches, so we an afford to wait for the 180 day limit to start chopping through 3.6a1pre electrolysis/places files (late June to late Aug '09).

Comment 7

[(not currently active) Ted Mielczarek]

Oh, I forgot I had written that! I thought you were talking about adding some new feature. :)