608230 - Should have an automated way to remove old nightly versions from the latest-* directories on FTP

Status: RESOLVED FIXED

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Description

[Mark Banner (:standard8)]

We currently have to go into the latest-* directories the day after we've bumped a nightly version to remove the old versions of the builds.

We should find a way to automate this.

Comment 1

[John Hopkins (:jhopkins)]

A cron job on stage.m.o would be the best place for this to happen.

Comment 2

[John Hopkins (:jhopkins)]

Attachment: proposed script to automate nightly pruning

What is a good pruning threshold?  I chose 365 days.

Comment 3

[Mark Banner (:standard8)]

Comment on attachment 504616 [details]
proposed script to automate nightly pruning

This is not what was intended.

MoCo already has cron scripts to prune nightly builds and basically they get rid of certain files at certain times.

What I want to see is something *just* for the latest-* directories. Something like this:

1) Check current version of comm-central (from default, via hg.mozilla.org).
2) Look at the latest-comm-central (and -l10n) directories on ftp.

2a) if there are files that match the current version *and* there are files of previous versions, remove the files belonging to the previous version.

=> The and case in the above accounts for cases where the version has been bumped but the new nightlies haven't run yet.

2b) else do nothing.

3) Repeat for active branches

Comment 4

[John Hopkins (:jhopkins)]

Attachment: proposed script

Mark: I believe the attached script will do what you want.  Here is how we would use it in a cron job (plus a trailing pipe to 'sh' which I've omitted for safety):

cd /pub/mozilla.org/thunderbird/nightly/latest-comm-central && python ~/trim-nightlies.py http://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt
cd /pub/mozilla.org/thunderbird/nightly/latest-comm-central-l10n && python ~/trim-nightlies.py http://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt
cd /pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.2 && python ~/trim-nightlies.py http://hg.mozilla.org/releases/comm-1.9.2/raw-file/default/mail/config/version.txt
cd /pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.2-l10n && python ~/trim-nightlies.py http://hg.mozilla.org/releases/comm-1.9.2/raw-file/default/mail/config/version.txt

Comment 5

[Philippe M. Chiasson (:gozer)]

(In reply to comment #4)
> Created attachment 505682 [details]
> proposed script
> 
> Mark: I believe the attached script will do what you want.  Here is how we
> would use it in a cron job (plus a trailing pipe to 'sh' which I've omitted for
> safety):

That will bite you when the filenames have spaces in them. Why not just have a -n flag for dry-run mode, and otherwise, just do it and output what was deleted ?

(Even better, not delete builds but move them somewhere else, and clean that up every week)

> cd /pub/mozilla.org/thunderbird/nightly/latest-comm-central && python
> ~/trim-nightlies.py
> http://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt

Wouldn't invoking something like this make more sense like:

$> python ~/trim-nightlies.py /pub/mozilla.org/thunderbird/nightly/latest-comm-central http://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt

I would prefer relying on arguments instead of the current working directory.

Comment 6

[John Hopkins (:jhopkins)]

Philippe, thanks for the input.  You're right, I should add quotes around the filenames in case there are spaces in the filenames.

The shell output is meant as a dry-run feature, and it requires less code than handling internal dry-run/delete functionality.  It's also grep-friendly if the user wants to further modify what will get deleted.

I could move old builds to an old/ subdirectory but then nightly pruning isn't fully automatic.  Do we want the burden of removing old nightlies "for real" on a regular basis?

About passing in a directory path - could do, but my feeling is it's more work for the user for little, if any, gain.

Comment 7

[John Hopkins (:jhopkins)]

gozer: we discussed escaping the filenames, but looking at the code, the names don't come from user input (rather, from glob()) so I think we're safe there.  Please reopen if you disagree.

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 505682 [details]
proposed script

The only thing I'd be concerned about here is if l10n nightlies were only partially generated and we wiped out all the older ones (because one file with the new version was in the directory)

However I guess that's probably going to be a rare case so I doubt it'll matter that much.