342972 - Implement stage cleaning policy

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p1     1.9T  1.8T     0 100% /data

So full!

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Build log snippet (from URL):

./upload-packages.sh
ssh -2 -l cltbld stage.mozilla.org mkdir -p /home/ftp/pub/thunderbird/nightly/2006-06-28-03-trunk /home/ftp/pub/thunderbird/nightly/latest-trunk
mkdir: cannot create directory `/home/ftp/pub/thunderbird/nightly/2006-06-28-03-trunk': No space left on device
command failed!

Comment 2

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Looks like about 200G in ~cltbld, might be some easy pruning to be had there.

Comment 3

[Justin Fitzhugh]

Not much we can do here - build, can you clean some things up?

Comment 4

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I think this is keeping extensions from being propagated, too -- I didn't realize that one could affect the other.

Can I get added to the nagios notification list for disk-space warnings/errors on stage?  This is pretty bad times for me, not sure what's happened to extensions that were approved during the full state. :(

Comment 5

[Dave Carlson]

perhaps you could remove some of the OLD OLD.
there are still nightlies from august 2005 hanging about for firefox and thunderbird.

surely we dont need that many still saved, i doubt anyone downloads them.

Comment 6

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Thanks, the people who actually have access to the machine know what needs to be kept and what doesn't, and what's taking up appreciable amounts of space.  Please help us help you by keeping the noise level in this bug low.

justdave has freed up some space by moving some backup data to dracula, but I think we still need to look at clearing out more, and figuring out how to keep cltbld, f.e., from filling up with crap again. :)  Reducing severity, but please do look at this ASAP.

Comment 7

[J. Paul Reed [:preed]]

I've moved ancient staging areas for FFx 1.0.x, Tbird 1.0.x, and Mozilla 1.x to the Netapp for now.

There's now ~120 gigs on stage, which should hold us for... awhile.

But we need to tackle this with a build expiration policy and some good ol' stage.m.o scrubbing.

Comment 8

[J. Paul Reed [:preed]]

Over to TR.

Comment 9

[T.R. Fullhart]

tfullhart: Let's see if I understand the requirements for that script you wanted: 1) based on directory name, delete all nightlies older than a month except for 1 representative nightly per week. 2) configurable
preed: I can shoot you an email, but basically:
preed: given a directory, delete all builds except a representative set of builds for periods of time
preed: as an example, keep all builds < 6 months, 6-9 months, keep a weekly build, 9+ months, keep 2 builds
preed: as configurable as possible is good, since those requirements may become something like
preed: keep all builds < 6 months, keep every other day 6-9 months, keep a weekly build 9-18 months, keep 2 builds/month > 18 months

Comment 10

[T.R. Fullhart]

Attachment: cleaner script

This version isn't completed but I promised I would post it today. I need to go home to run an errand. I will finish it tomorrow.

Comment 11

[T.R. Fullhart]

Attachment: Configuration file for cleaner script

This configuration file includes rules to make the cleaner script do the jobs of ftp-trim-archive.sh, ftp-trim-contrib.sh, trim-downloads, and trim-downloads.conf. It also includes some example rules that express requirements that preed requested.

Comment 12

[T.R. Fullhart]

Attachment: cleaner script

Comment 13

[T.R. Fullhart]

Attachment: Configuration file for cleaner script

This has a bunch of example rules. The actual rules need to be defined.

Comment 14

[T.R. Fullhart]

Attachment: preed-compatible formatted cleaner script

Comment 15

[T.R. Fullhart]

Attachment: perltidy profile

For reformatting perl scripts using the "perltidy" script.

Comment 16

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Please get reasonably wide review on any cleaning policy before implementing it -- having old nightlies is extremely important for finding regressions quickly.

Comment 17

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'd also note that I think the rules given here as examples are probably deleting too much -- especially too much since the last branch.  As we approach the 1.9 release we'll hear about regressions early in the development cycle (i.e., starting August 12, 2005), and we'll want to isolate what caused them.  And there are probably still regressions shipped in 1.8 that we've gotten reports of but haven't yet isolated.

Comment 18

[T.R. Fullhart]

Attachment: Configuration file for cleaner script

Comment 19

[T.R. Fullhart]

Attachment: reformatted cleaner script

Comment 20

[T.R. Fullhart]

Attachment: perltidy profile

Comment 21

[J. Paul Reed [:preed]]

Taking TR's bug; I'll retriage these shortly.

Comment 22

[J. Paul Reed [:preed]]

Reassigning bugs I'm not actively working on back into the triage pool.

Comment 23

[(not currently active) Ted Mielczarek]

I may consider taking this, I need to write symbol cleanup for Breakpad anyway.

Comment 24

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Nick, can you look at this as part of your stage migration work?

Comment 25

[J. Paul Reed [:preed]]

Comment on attachment 246988 [details]
Configuration file for cleaner script

Canceling old reviews.

Comment 26

[J. Paul Reed [:preed]]

Comment on attachment 246989 [details]
reformatted cleaner script

Canceling old reviews.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

We have plenty of room on the netapp at the moment (>400GB), so this is not an urgent problem. Back to the pool for now.

Comment 28

[Justin Fitzhugh]

Does this actually block the stage migration?  It's listed as such...if not, can it be removed from blocking list?

Comment 29

[Robert Helmer [:rhelmer]]

(In reply to comment #28)
> Does this actually block the stage migration?  It's listed as such...if not,
> can it be removed from blocking list?

Can we continue to grow the FTP archive indefinitely? Not sure if it needs to block, but it seems important if the answer is "no" :)

Comment 30

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #29)
> Can we continue to grow the FTP archive indefinitely?

But it would be really nice if the answer is yes.  (Isn't disk cheap these days?)

Comment 31

[Robert Helmer [:rhelmer]]

(In reply to comment #30)
> (In reply to comment #29)
> > Can we continue to grow the FTP archive indefinitely?
> 
> But it would be really nice if the answer is yes.  (Isn't disk cheap these
> days?)

I'd love for the answer to be yes, for what its worth :)

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Consumption graph

Here's some info on how we're doing currently. Of the 2.9TB of space allocated, we have about 150GB still available, and nightly builds are consuming it at a rate of 1.75GB/day. When a release comes along we take an additional bite by keeping multiple copies around, which is what causes the big jumps on the gentle slope.

KaiRo recently pointed out that there isn't much point keeping the nightly update (mar) files around, because they are superseded by the next build. That could save us quite a bit of space, eg 72GB for the Firefox nightlies from 2007. Provided we can't think of a reason to keep them, that is.

Comment 33

[Justin Fitzhugh]

Nope - disk space does cost, both in maintenance and initial cost and is not an infinite resource.  I think the right thing to do is implement a policy that keeps the most used files on disk such as all releases, etc but archives nighties off to tape.  We could always restore if needed relatively quickly and we won't waste the electricity to spin disks constantly when they are almost never accessed.  As for what should be kept on/off line, I'll leave that to build - thoughts?

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Not working on this right now.

Comment 35

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 36

[Chris Cooper [:coop] (he/him)]

(In reply to comment #32)
> KaiRo recently pointed out that there isn't much point keeping the nightly
> update (mar) files around, because they are superseded by the next build. That
> could save us quite a bit of space, eg 72GB for the Firefox nightlies from
> 2007. Provided we can't think of a reason to keep them, that is.

I was going to close this bug out, but this ^^ still seems like an easy win.

Comment 37

[Chris Cooper [:coop] (he/him)]

(In reply to comment #36)
> (In reply to comment #32)
> > KaiRo recently pointed out that there isn't much point keeping the nightly
> > update (mar) files around, because they are superseded by the next build. That
> > could save us quite a bit of space, eg 72GB for the Firefox nightlies from
> > 2007. Provided we can't think of a reason to keep them, that is.
> 
> I was going to close this bug out, but this ^^ still seems like an easy win.

Can someone from IT please comment on whether the above fix is already in place on stage? I can only see the crontabs for cltbld and ffxbld and we're only cleaning up breakpad symbols and experimental builds in those.

Comment 38

[Dave Miller [:justdave]]

IT doesn't manage this, RelEng does.  nthomas is the one with access to it last time I looked.

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

We don't have an automated solution for this. When we've hit a disk usage crunch in the past I've gone through with the help of some shell commands to clean up. It's getting towards that time again - we're down to 200G free on the non-firefox partition and 300G for firefox.

Comment 40

[Robert Kaiser]

IMHO, we really should have some automated way of clearing out MARs for older nightlies for all products.
Esp. as we are producing complete _and_ partial MARs for most of our products and branches now, this is a lot of space that is used up by those even though they are not hit by any current update (which only points to the latest of those).

Comment 41

[Chris Cooper [:coop] (he/him)]

Alright then, let's figure out a plan and put it in place. That this has been open for almost 4 years is ridiculous.

Comment 42

[Chris Cooper [:coop] (he/him)]

nthomas: do you think we can resurrect TR's attached cleanup scripts (assuming we revisit his/preed's cleanup criteria first to make sure they're current)?

I'm compiling some current disk usage numbers to aid in our assessment.

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

TR's scripts may be overkill if we just want to remove older nightly mar files. Bonus points if we don't scan over all of firefox/nightly/YYYY every run.

FYI, we've left the likes of YYYY-MM-DD-firefox3.0.19{,-l10n} alone since they are release files. Bug 562261 for the cleanup I did recently.

Comment 44

[Chris Cooper [:coop] (he/him)]

(In reply to comment #43)
> TR's scripts may be overkill if we just want to remove older nightly mar files.
> Bonus points if we don't scan over all of firefox/nightly/YYYY every run.

Not sure how we'd avoid a complete scan without relying on some sort of script-based solution. Eliminating nightly mars would be simple enough to do via cron, but I think we want to do more. Maybe more elaborate cronjobs that only run weekly?
 
> FYI, we've left the likes of YYYY-MM-DD-firefox3.0.19{,-l10n} alone since they
> are release files. Bug 562261 for the cleanup I did recently.

Can you be more specific here? Which subdir oddballs do we have floating around that we shouldn't be purging?

Comment 45

[Nick Thomas [:nthomas] (UTC+12)]

Could you elaborate on what else you'd like to clean up ? I took your comment #41 as a response to #40.

(In reply to comment #44)
> Can you be more specific here? Which subdir oddballs do we have floating around
> that we shouldn't be purging?

eg
/pub/mozilla.org/firefox/nightly/2010/03/2010-03-12-18-firefox3.0.19/
/pub/mozilla.org/firefox/nightly/2010/03/2010-03-12-18-firefox3.0.19/
/pub/mozilla.org/firefox/nightly/2010/03/2010-03-12-19-firefox3.0.19-l10n/

Tinderbox pushes the files to those locations, then we copy to the candidates directory. They're only really relevant if that copy fails, so perhaps we don't actually care if the mars are deleted a week later.

Comment 46

[Chris Cooper [:coop] (he/him)]

(In reply to comment #45)
> Could you elaborate on what else you'd like to clean up ? I took your comment
> #41 as a response to #40.

I'd like to automatically prune nightlies and tinderbox-builds as TR's script was trying to do, but perhaps in a more straightforward manner. 

We'll clean up the MARs too, of course, but having a policy for how we expire/archive everything is a good idea.

Comment 47

[Chris Cooper [:coop] (he/him)]

I've made a complete blog post on the subject here (http://coop.deadsquid.com/2010/05/reclaiming-space-on-stage-mozilla-org/), but here are my specific proposals:

1) Move Firefox releases that are no longer supported (< 3.0, including firebird) to separate storage.

2) Remove Firefox nightlies prior to 2007, freeing 260G. These can be deleted if they're not going to be used, or archived if we think they might.

3) Remove nightlies for products other than Firefox prior to 2007, freeing 174G. Again, "remove" can mean either deletion or archiving.

4) Automate the deletion of nightly MAR files older than one month. Only the most recent MAR files are required. This would be done across all products.

5) Delete builds from older candidates directories after official release. This will reclaim up to 13G per build attempt per release. This will be a manual process.

6) For every new year going forward, remove the oldest remaining year of nightlies, e.g. for a 3-year history of nightly builds, remove nightly builds from 2007 in January 2011. This will be a manual process.

Comment 48

[Chris Cooper [:coop] (he/him)]

(In reply to comment #47)
> I've made a complete blog post on the subject here

...and I've also posted to dev-builds, dev-planning, and dev-tree-management now.

Comment 49

[Timothy Nikkel (:tnikkel)]

I only started bisecting about a year ago and I have one build from 2004, 10 from 2005, 17 from 2006, 45 from 2007 in my bisecting archive. I would appreciate the ability to continue bisecting things that may have changed long ago.

Comment 50

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

In the past year, I've gone back as far as January 2006 (Firefox 1.6a1) trying to track down a regression of some sort (I can tell by the builds in my Firefox folder).  

Philippe Wittenbergh just last month went back to March 2007 trunk builds tracking down a Core regression that only manifests in Camino (for which we really needed both Camino and Firefox nightlies in order to verify it never appeared in Firefox), but we had to start checking from when 1.8 branched from the trunk on 2005-08-12.

Speaking from a Camino bugs perspective, I know there are still several Gecko/Gecko-tickled regressions that require older builds (including 1.7-was-trunk/1.8-was-trunk-era) to track down (because I've started, but never finished, tracking down the regression ranges.)

I'm certainly OK with moving older nightly builds to a separate http://archive.mozilla.org/ again (like they used to be in the old days, before ftp.m.o and archive.m.o were merged) to relieve pressure on stage/ftp.m.o, but I'm opposed to taking them offline entirely and very strongly opposed to deleting them.

(I remember this being mentioned before in another bug, but disk *is* cheap.  I just bought a nice small, portable bus-powered, quad-interface 7200 RPM 300 GB HD for ~$150.  I know it's not "server-grade" (but it has those other features that make it more costly than your average disk, which is even cheaper), but that would cover 75% of the non-Firefox nightlies since 2001, or a recent year of Firefox nightlies, or several older years of Firefox nightlies.)

Comment 51

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #50)
> (I remember this being mentioned before in another bug, but disk *is* cheap.  I

IIRC, disk was specifically "cheap" in comparison to employing a layout hacker to recreate those ancient builds needed for doing his bisection (or to come up with an alternate way of tracking down the regression).  Sorry, I was not trying to be completely cavalier with that part of the previous comment.

Comment 52

[Jo Hermans]

In my company, we start deleting nightly builds after 6 months, but we leave 1 build per week (there's 1 weekly build which has a semi-alpha status) for regression lookups.

But we can always recreate from CVS sources, as we only have automated builds and regression tests. To make you guys jealous : even the checkins in release builds (as opposed to developer builds) are more-or-less automated, except for merging conflicts of course.

Comment 53

[Samuel Sidler (:ss)]

(In reply to comment #51)
> IIRC, disk was specifically "cheap" in comparison to employing a layout hacker
> to recreate those ancient builds needed for doing his bisection (or to come up
> with an alternate way of tracking down the regression).

I'd also note that modern toolchains might not be able to even build from source that's four, five, or six years old. e.g., with most developers on 10.5 and 10.6 and on Intel, it might be impossible to build the 1.8 branch or the 1.7 branch. I doubt releng wants to stand by for creating old builds on outdated hardware whenever required by developers, which would be one alternative.

Comment 54

[Chris Cooper [:coop] (he/him)]

Based on the feedback I've received, I have revised my policy proposals:

1) Keep all releases for all products online and available. There's no need to remove them.

2) Keep all en-US nightly builds for all products online and available. There's no need to remove them.

3) Delete nightly artifacts for all products that are not useful in regression hunting. Specifically, this means deleting installer files (linux and windows) and xpis from 2009 and earlier. This could represent a one-time space recovery of almost 900GB. Individual products can opt out of this cleanup with sufficient cause.

4) Automate the deletion of nightly MAR files older than one month. Only
the most recent MAR files are required. This would be done across all
products. (unchanged)

5) Delete builds from older candidates directories after official
release. This will reclaim up to 13G per build attempt per release. This
will be a manual process. (unchanged)

6) Automate the removal of nightly artifacts older than 6 months for all products that are not useful in regression hunting.

Comment 55

[Robert Kaiser]

(In reply to comment #54)
> 5) Delete builds from older candidates directories after official
> release. This will reclaim up to 13G per build attempt per release. This
> will be a manual process. (unchanged)

Note that I cleaned up a good number of SeaMonkey candidates this week, which also should have reclaimed some space.

Comment 56

[Chris Cooper [:coop] (he/him)]

(In reply to comment #55)
> Note that I cleaned up a good number of SeaMonkey candidates this week, which
> also should have reclaimed some space.

Thanks, KaiRo. Very helpful. Simon has also cleaned up a bunch of old Calendar stuff which is also appreciated.

Project leads have been contacted. I will likely start pruning the Firefox and old Mozilla installers/xpis later this week, but will wait to hear back the from the other project leads before I do cleanup for their projects or put any cron jobs in place.

Comment 57

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #56)
> Project leads have been contacted.

coop, who did you contact for Camino?  None of pink, smorgan, and mento report receiving an email.

Comment 58

[Chris Cooper [:coop] (he/him)]

Attachment: Proposed crontab for ffxbld user on stage

3 entries for the ffxbld crontab to be run on a weekly basis:

* delete installer files older that 6 months
* delete xpi files older than 6 months
* delete empty directories older than 6 months

nthomas: can you sanity check this for me before I put it in place? Do we want to run the rm in verbose mode?

Comment 59

[Rail Aliiev [:rail]]

I would modify the following:
- quote the file patterns with *, shell may expand it if there is a file with the same pattern in the current directory (*.installer.* should be '*.installer.*')
- quote {} to properly handle spaces ({} should be '{}')
- don't use -r for for files, add "-type f" to be more specific.

Comment 60

[Chris Cooper [:coop] (he/him)]

Attachment: Proposed crontab for ffxbld user on stage, v2

(In reply to comment #59)
> I would modify the following:
> - quote the file patterns with *, shell may expand it if there is a file with
> the same pattern in the current directory (*.installer.* should be
> '*.installer.*')
> - quote {} to properly handle spaces ({} should be '{}')
> - don't use -r for for files, add "-type f" to be more specific.

New version contains these fixes.

Comment 61

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 448594 [details] [diff] [review]
Proposed crontab for ffxbld user on stage, v2

Might as well save a crawl through the disk tree and combine the two file searches, I think this does the operator precedence correctly:

@weekly nice -n 19 find /home/ftp/pub/firefox/nightly -mtime +180 -type f \( -name '*.installer.*' -o -name '*xpi' \) -exec rm -f '{}' \;

or use a regex.

What's the -mtime +180 on the empty directory search for ? Without it I found some *-candidates/contrib dirs, but we could exclude the candidates or just search 20?? originally. With it I think we'll empty some dirs and then wait 180 days to remove the dir.

Comment 62

[Chris Cooper [:coop] (he/him)]

Attachment: Proposed crontab for ffxbld user on stage, v3

Merges two file deletions into a single command and removes the mtime check for empty directory removal.

Comment 63

[Chris Cooper [:coop] (he/him)]

Attachment: Proposed crontab for ffxbld user on stage, v4

Adds the MAR file expiry, but only for non-candidates dirs.

Comment 64

[Chris Cooper [:coop] (he/him)]

Attachment: Proposed crontab for ffxbld user on stage, v5

Same as previous patch, but removes the 2010/ directory I was using for as a limiter for MAR file find testing.

Comment 65

[Chris Cooper [:coop] (he/him)]

Comment on attachment 449104 [details] [diff] [review]
Proposed crontab for ffxbld user on stage, v5

># Delete all installer and xpi files/dirs older than 6 months
>@weekly nice -n 19 find /home/ftp/pub/firefox/nightly -mtime +180 \( -name '*.installer.*' -o -name '*xpi' \) -exec rm -rf '{}' \;

I took out the -type f limiter, changed the xpi search to *xpi, and returned to doing a rm -rf here. This allows us to delete entire directories like "windows-xpi" which contain both .xpi files and uninstaller .zips, and wouldn't get cleaned up otherwise.

Comment 66

[Chris Cooper [:coop] (he/him)]

I've created https://wiki.mozilla.org/ReleaseEngineering:StageCleanupPolicy as a lasting artifact of this bug. That wiki page includes crontabs and such, but note that none of these have been run or enabled yet. Still waiting on the ownership changes in bug 569461.

Comment 67

[Jesse Ruderman]

Please don't automatically delete the last remaining build of a given type. My fuzzing scripts broke due to finding https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/ empty after a day-long tree closure.

Comment 68

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449104 [details] [diff] [review]
Proposed crontab for ffxbld user on stage, v5

Sorry for the delay coop, I'll get this early next week.

Comment 69

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449104 [details] [diff] [review]
Proposed crontab for ffxbld user on stage, v5

I did some spot checks on a few YYYY/MM dirs and the matches seemed OK.

Please set a MAILTO so we can catch errors. Worth doing verbose rm for the first week or two, or doing it manually the first time to a log ?

># Delete all installer and xpi files/dirs older than 6 months
>@weekly nice -n 19 find /home/ftp/pub/firefox/nightly -mtime +180 \( -name '*.installer.*' -o -name '*xpi' \) -exec rm -rf '{}' \;

You'll need to add a '-depth' to prevent errors on matches like this:
 .../2009-01-01-04-mozilla1.8/windows-xpi
 .../2009-01-01-04-mozilla1.8/windows-xpi/xforms.xpi

r+ with those changes.

># Delete all MAR files older than 1 month that aren't in a candidates dir.
>@weekly nice -n 19 find /home/ftp/pub/firefox/nightly -wholename '*-candidates' -prune -o -mtime +30 -name '*.mar' -exec rm -f '{}' \;

We could probably remove the tests files we started publishing on 2009/09/24 on the same age criterion. The extension changed from tar.bz2 to zip on some branches so matching on '*.tests.*' probably makes sense.

Comment 70

[Chris Cooper [:coop] (he/him)]

(In reply to comment #69)
> Please set a MAILTO so we can catch errors. Worth doing verbose rm for the
> first week or two, or doing it manually the first time to a log ?

I'm running a first-pass now by hand and logging the output. I'll make the log available if there's anything out of the ordinary in it. 

I'll add a MAILTO to the crontabs when I put them in place.

> You'll need to add a '-depth' to prevent errors on matches like this:
>  .../2009-01-01-04-mozilla1.8/windows-xpi
>  .../2009-01-01-04-mozilla1.8/windows-xpi/xforms.xpi
> 
> r+ with those changes.

Added.

> We could probably remove the tests files we started publishing on 2009/09/24 on
> the same age criterion. The extension changed from tar.bz2 to zip on some
> branches so matching on '*.tests.*' probably makes sense.

Added.

Comment 71

[Chris Cooper [:coop] (he/him)]

The crontabs for all products are in place now. I've run all the individual commands by hand once and collected logs, but haven't seen anything I didn't expect.

We didn't reclaim as much space as originally estimated because we preserved the contrib dirs largely intact. We did still manage to recover almost 350G of space across all products, bringing our %used number on stage down to 79%. It was at 95% yesterday.

I've documented the new policy in the wiki:
https://wiki.mozilla.org/ReleaseEngineering:StageCleanupPolicy

We'll still have to grow the disk eventually, or shuffle stuff around, but at least now we have *something* in place.

Comment 72

[Robert Kaiser]

Ouch, it looks like those scripts killed the *.mar files in the seamonkey/nightly/2.1a1-candidates/ directory, which made my 2.1a2 update builds fail. I've copied them back from the 2.1a1 release directory, but it would be good if we cared that this doesn't happen again.