861733 - Clean up old release dirs on slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Adding 00000 suffixes to build directories (bug 827306) also changed several release builders. The old dirs aren't cleared up by purge_builds.py because we specifically exempt them. We've now gone a full release cycle so any old directories are taking up space and causing more churn than necessary in non-release builds.

eg bld-centos6-hp-017 happened to run out of space today, and 66GB was freed by running
  cd /builds/slave
  ls -d rel-m-rel* | grep -v 0$ | xargs rm -r
  ls -d rel-m-beta* | grep -v 0$  | xargs rm -r

That's nearly 30% of the disk. A sample size of 1 is totally legit, right ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

What if we had some code that looked in the twistd.log for lines like

2013-04-28 14:32:57-0700 [Broker,client] I have a leftover directory 'rel-m-beta-xr-psh-mrrrs' that is not being used by the buildmaster: you can delete it now
2013-04-28 14:32:57-0700 [Broker,client] I have a leftover directory 'rel-m-esr17-lnx-update-verify-4' that is not being used by the buildmaster: you can delete it now

from the last buildbot startup ? I'm not sure if this is best in runslave or purge_builds, or even if patching buildbot slave to (optionally) delete those directories.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

catlee suggested we teach purge_builds.py to expire rel-* after N days, say 40 or 60, so that we still have cleanup while retaining release builds for post-release debugging. With that directory renaming is eventually cleaned up without human intervention.

Comment 3

[John Hopkins (:jhopkins)]

Attachment: [tools] support time-based expiry for skipped directories

This will require a separate buildbotcustom patch that makes use of the expiry syntax.  Tested script locally on its own.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 770875 [details] [diff] [review]
[tools] support time-based expiry for skipped directories

I'm finding the second lambda fairly opaque...

Comment 5

[John Hopkins (:jhopkins)]

Comment on attachment 770875 [details] [diff] [review]
[tools] support time-based expiry for skipped directories

https://hg.mozilla.org/build/tools/rev/d1e287abf65c

Comment 6

[John Hopkins (:jhopkins)]

Attachment: [buildbotcustom] call purge_builds with expiry time for release dirs

I'm assuming a 45 day expiry for release dirs - please let me know if you have another preference.
Test build in staging:
 http://dev-master01.build.scl1.mozilla.com:8900/builders/Linux%20mozilla-central%20build/builds/0/steps/clean_old_builds/logs/stdio

Comment 7

[John Hopkins (:jhopkins)]

Comment on attachment 770954 [details] [diff] [review]
[buildbotcustom] call purge_builds with expiry time for release dirs

https://hg.mozilla.org/build/buildbotcustom/rev/edf2327007cf

Comment 8

[John Hopkins (:jhopkins)]

Comment on attachment 770954 [details] [diff] [review]
[buildbotcustom] call purge_builds with expiry time for release dirs

Cherry-picked onto production-0.8 branch: http://hg.mozilla.org/build/buildbotcustom/rev/359b0c3d670f

Comment 9

[John Hopkins (:jhopkins)]

Confirmed that release directories are being cleaned up as expected.