771017 - Set up cron jobs to clean up b2g/ dirs

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Off the top of my head, we should be removing *tests.zip older than 30 days (do we need them at all ?). Possibly more, I'd need to look at the files on disk and what we do for ffxbld.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Add b2g cleanup

This will clean up test archives in nightly dirs, trim the number of symlinks in b2g/nightly/ down to 30 days worth, and keep 2 days of tinderbox-builds. 

I'm not sure if the last is correct or if we should be taking the last 30 days. There's a big collection of logs from the discontinued *-b2g-fedora16 which will get cleaned up, but how long do we need *gb_armv7a_gecko* for ? Are these going to get used for tests ?

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 639202 [details] [diff] [review]
[svn] Add b2g cleanup

>Index: files/cron/b2gbld
>===================================================================
>--- files/cron/b2gbld	(revision 0)
>+++ files/cron/b2gbld	(revision 0)
>@@ -0,0 +1,6 @@
>+# Delete all test archives older than 1 month that aren't in a candidates dir.
>+0 1 * * 0 nice -n 19 find /home/ftp/pub/b2g/nightly -wholename '*-candidates' -prune -o -mtime +30 -name '*.tests.*' -exec rm -f '{}' \;

Is this one here in anticipation of having release builds? Because of the *-candidates match it won'tdo anything yet.



Unfortunately, /pub/mozilla.org/b2g/* is owned by ffxbld/firefox until bug 767515 is fixed - are these running as root? If not, it may have to wait to land.

> I'm not sure if the last is correct or if we should be taking the last 30
> days. There's a big collection of logs from the discontinued *-b2g-fedora16
> which will get cleaned up, but how long do we need *gb_armv7a_gecko* for ?

At this time I don't think we have any plans to run any tests on gb_armv7a_gecko builds. We don't even upload the binaries.

I think 2 days worth of tinderbox-build is fine - we have nightlies for everything, too.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> Is this one here in anticipation of having release builds? Because of the
> *-candidates match it won'tdo anything yet.

It's a copy from the ffxbld one, but given that it's fairly likely we'll start doing b2g releases and are very unlikely to remember to tweak the cron jobs then it's probably better to leave it.
 
> Unfortunately, /pub/mozilla.org/b2g/* is owned by ffxbld/firefox until bug
> 767515 is fixed - are these running as root? If not, it may have to wait to
> land.

It'll run as b2gbld, so I'll block on that bug. Good catch.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 639202 [details] [diff] [review]
[svn] Add b2g cleanup

The ftp-trim-links.sh part runs s root, so I landed that to keep the symlinks in b2g/nightly/ under control:

Sending        bin/ftp-trim-links.sh
Transmitting file data .
Committed revision 43588.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Morphing to consider b2g/tinderbox-builds too. Do we want to keep the panda builds for 30 days like desktop dep builds, or can we go with something shorter ?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 639202 [details] [diff] [review]
[svn] Add b2g cleanup

>Index: files/cron/ffxbld
>===================================================================
>-# Delete all MAR files older than 1 month that aren't in a candidates dir.
>+# Delete all MAR files & test archives older than 1 month that aren't in a candidates dir.
> 0 1 * * 0 nice -n 19 find /home/ftp/pub/{firefox,mobile}/nightly -wholename '*-candidates' -prune -o -mtime +30 \( -name '*.mar' -o -name '*.tests.*' \) -exec rm -f '{}' \;

Landed already.

>Index: files/cron/b2gbld
>===================================================================
>+# Delete all test archives older than 1 month that aren't in a candidates dir.
>+0 1 * * 0 nice -n 19 find /home/ftp/pub/b2g/nightly -wholename '*-candidates' -prune -o -mtime +30 -name '*.tests.*' -exec rm -f '{}' \;

Dropping this given the latest info on b2g.

>+# Clean up tinderbox-builds
>+@hourly   nice -n 19 find /home/ftp/pub/b2g/tinderbox-builds -mindepth 2 -maxdepth 2 -type d -mtime +1 -name 1?????????

Added this to ffxbld's cron, saving 1.3T of space. 

Also modified trybld's cron to clean up /pub/mozilla.org/b2g/try-builds after 14 days (to match desktop), saving 1.1TB for files going back to June 2012.

Both of those in revision 64957.

>Index: files/bin/ftp-trim-links.sh
>===================================================================

Landed already.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Remove mar files after 7 days on pvtbuilds

Could possibly be more aggressive here as the files get used on update.boot2gecko.org. Open to suggestions.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 784899 [details] [diff] [review]
[svn] Remove mar files after 7 days on pvtbuilds

Committed revision 72429.