Last Comment Bug 596852 - purge_builds should delete directories older than some threshold
: purge_builds should delete directories older than some threshold
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: P3 normal (vote)
: ---
Assigned To: Chris AtLee [:catlee]
:
:
Mentors:
Depends on:
Blocks: releng-downtime end_to_end
  Show dependency treegraph
 
Reported: 2010-09-15 20:16 PDT by Chris AtLee [:catlee]
Modified: 2013-08-12 21:54 PDT (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
delete directories older than 7 days (5.20 KB, patch)
2010-09-16 05:48 PDT, Chris AtLee [:catlee]
bhearsum: review+
nthomas: checked‑in+
Details | Diff | Splinter Review
reduce max age to 14 days (880 bytes, patch)
2010-10-26 18:57 PDT, Chris AtLee [:catlee]
nthomas: review+
catlee: checked‑in+
Details | Diff | Splinter Review

Description Chris AtLee [:catlee] 2010-09-15 20:16:51 PDT
If purge_builds were to delete directories older than a week or so, we would
a) clean out old cruft from the /builds/slave directory
b) make it so that we're not always running as close to full as possible
Comment 1 Chris AtLee [:catlee] 2010-09-16 05:48:51 PDT
Created attachment 475827 [details] [diff] [review]
delete directories older than 7 days
Comment 2 Ben Hearsum (:bhearsum) 2010-09-20 10:03:58 PDT
Comment on attachment 475827 [details] [diff] [review]
delete directories older than 7 days

Looks good
Comment 3 Ben Hearsum (:bhearsum) 2010-10-01 10:11:33 PDT
This will land on Monday.
Comment 4 Justin Wood (:Callek) 2010-10-03 15:12:39 PDT
So the summary and comments do not match the code I see here...

It seems like we _always_ preserve builds newer than 7 days, rather than always delete older than 7 days. [am I wrong]

I expected this bug to be a "We want to delete all builds older than 7 days, and/or newer builds if we still do not have the free-space we want"

Am I either missing something or wrong?
Comment 5 Justin Wood (:Callek) 2010-10-03 15:17:23 PDT
(In reply to comment #4)
> So the summary and comments do not match the code I see here...

Nope just confused.

> Am I either missing something or wrong?

Missed the addition of |dirs.sort()| before the while... sorry for noise.
Comment 6 Ben Hearsum (:bhearsum) 2010-10-04 07:42:41 PDT
This is landed and working.
Comment 7 Ben Hearsum (:bhearsum) 2010-10-04 07:47:35 PDT
Comment on attachment 475827 [details] [diff] [review]
delete directories older than 7 days

Landed in e179cf0fd98b
Comment 8 Ben Hearsum (:bhearsum) 2010-10-04 13:35:45 PDT
This got backed out because it caused windows 7 test machines to fail (because they timed out when cleaning up).
Comment 9 Armen Zambrano [:armenzg] (EDT/UTC-4) 2010-10-15 08:25:00 PDT
could we include this change with a bigger threshold at first and then reduce it every week?
I believe the machines that burned the day we enabled had many many GBs of old stuff and that is why it timed out as it had too many things to remove. Think of the IX machines that barely need to clobber as their disk always have more than 12GB free and everything is just getting older.

Maybe we could ask the script to check if they have spend X amount of mins deleting stuff and then stop if they have passed it. Luckily the machine would stop between X and the buildbot timeout.
Comment 10 Aki Sasaki [:aki] 2010-10-15 15:41:16 PDT
Why is this needs-treeclosure? if the patch that broke win7 hasn't been changed?
Comment 11 Chris AtLee [:catlee] 2010-10-16 07:05:10 PDT
(In reply to comment #10)
> Why is this needs-treeclosure? if the patch that broke win7 hasn't been
> changed?

We expect that after the win7 slaves cycle through and clean up, that this will work ok going forward.  So it needs tree closure because we expect burning the first time stuff is getting run, but it should be ok after.
Comment 12 Chris AtLee [:catlee] 2010-10-16 07:07:51 PDT
I'm also fine with Armen's suggestion of increasing the threshold.  The max_age default at line 130 needs to be changed.  I'd recommend 28 days to start with.
Comment 13 Nick Thomas [:nthomas] 2010-10-17 14:18:22 PDT
Comment on attachment 475827 [details] [diff] [review]
delete directories older than 7 days

Checked in with 28 day default:
http://hg.mozilla.org/build/tools/rev/7920b159c3e6
Comment 14 Nick Thomas [:nthomas] 2010-10-18 02:37:46 PDT
Leaving open to crank down the threshold.
Comment 15 Chris AtLee [:catlee] 2010-10-26 18:57:22 PDT
Created attachment 486252 [details] [diff] [review]
reduce max age to 14 days
Comment 16 Chris AtLee [:catlee] 2010-10-27 06:09:53 PDT
Doesn't need a reconfig per se, but needs somebody to watch it for a bit.
Comment 17 Chris AtLee [:catlee] 2010-11-01 11:02:14 PDT
Comment on attachment 486252 [details] [diff] [review]
reduce max age to 14 days

changeset:   883:5a99b6e600c6
Comment 18 Chris AtLee [:catlee] 2010-11-01 12:59:18 PDT
14 days is a good number to finish with.

Note You need to log in before you can comment on or make changes to this bug.