Add automated clean up of hg-shared directory

RESOLVED FIXED

Status

Release Engineering
General
P5
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: nthomas, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildslaves])

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

7 years ago
We currently accumulate repositories when shared repositories is enabled, and need some way to bound the disk usage. We could remove repositories in which the tip is older than some age, or limit the space the whole of hg-shared is allowed to use and expire the oldest repo's first. 

purge_builds.py may be a good place to do this, but would have to be taught how to look up hg revisions, and the way directories are mapped out in hg-share (eg mozilla-central, releases/mozilla-beta, rather than flat in a master dir).
(Assignee)

Comment 1

7 years ago
Created attachment 588855 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs

This isn't quite as smart as your suggestion, but easier to write.

It looks for directories under $HG_SHARE_BASE_DIR with a .hg subdirectory, and then calls the regular purge() logic on that set.

This means that max_age will be enforced for these directories as well.

HG_SHARE_BASE_DIR must be set in the environment for this to work (requires a buildbotcustom change)
Attachment #588855 - Flags: feedback?(nrthomas)
(Assignee)

Updated

7 years ago
Assignee: nobody → catlee
(Reporter)

Comment 2

7 years ago
Comment on attachment 588855 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs

>+    # Clean up empty directories
>+    for d in hg_dirs:
>+        if not os.path.exists(os.path.join(d, '.hg')):
>+            print "Deleting", d

Maybe say |'Cleaning up', d| since there will already be a log line with "Deleting d" from purge().

So how might this go in practice ? We delete a lot of shared clones before we zap *any* builds, and so might do a lot more cloning ? Having a longer max-age may be useful for hg-shared.
Attachment #588855 - Flags: feedback?(nrthomas) → feedback+
(Assignee)

Comment 3

7 years ago
Created attachment 589255 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs
Attachment #588855 - Attachment is obsolete: true
Attachment #589255 - Flags: review?(nrthomas)
(Reporter)

Comment 4

7 years ago
Comment on attachment 589255 [details] [diff] [review]
use purge_builds.py to delete hg shared dirs

Lets give it a go.
Attachment #589255 - Flags: review?(nrthomas) → review+
(Assignee)

Updated

7 years ago
Attachment #589255 - Flags: checked-in+
(Assignee)

Comment 5

7 years ago
Created attachment 589954 [details] [diff] [review]
use self.env for purge_builds step
Attachment #589954 - Flags: review?(nrthomas)
(Reporter)

Updated

7 years ago
Attachment #589954 - Flags: review?(nrthomas) → review+
(Assignee)

Updated

7 years ago
Attachment #589954 - Flags: checked-in+
Merged to production today.
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.