699219 - Add automated clean up of hg-shared directory

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We currently accumulate repositories when shared repositories is enabled, and need some way to bound the disk usage. We could remove repositories in which the tip is older than some age, or limit the space the whole of hg-shared is allowed to use and expire the oldest repo's first. 

purge_builds.py may be a good place to do this, but would have to be taught how to look up hg revisions, and the way directories are mapped out in hg-share (eg mozilla-central, releases/mozilla-beta, rather than flat in a master dir).

Comment 1

[Chris AtLee [:catlee]]

Attachment: use purge_builds.py to delete hg shared dirs

This isn't quite as smart as your suggestion, but easier to write.

It looks for directories under $HG_SHARE_BASE_DIR with a .hg subdirectory, and then calls the regular purge() logic on that set.

This means that max_age will be enforced for these directories as well.

HG_SHARE_BASE_DIR must be set in the environment for this to work (requires a buildbotcustom change)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 588855 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs

>+    # Clean up empty directories
>+    for d in hg_dirs:
>+        if not os.path.exists(os.path.join(d, '.hg')):
>+            print "Deleting", d

Maybe say |'Cleaning up', d| since there will already be a log line with "Deleting d" from purge().

So how might this go in practice ? We delete a lot of shared clones before we zap *any* builds, and so might do a lot more cloning ? Having a longer max-age may be useful for hg-shared.

Comment 3

[Chris AtLee [:catlee]]

Attachment: use purge_builds.py to delete hg shared dirs

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 589255 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs

Lets give it a go.

Comment 5

[Chris AtLee [:catlee]]

Attachment: use self.env for purge_builds step

Comment 6

[[e-mail] (:bhearsum)]

Merged to production today.