633288 - Keep tinderbox-builds/${branch}-l10n clean

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Armen [:armenzg]]

Attachment: remove binaries with older version for dependent L10n packages

I was thinking of adding this cronjob.

nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-l10n \! -name `wget --quiet -O- http://hg.mozilla.org/mozilla-central/raw-file/default/browser/config/version.txt`
nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-1.9.2-l10n \! -name `wget --quiet -O- http://hg.mozilla.org/releases/mozilla-1.9.2/raw-file/default/browser/config/version.txt`

What do you say?

I will add 1.9.1 once I figure out why it is not there.

Comment 1

[Armen [:armenzg]]

What do you think the frequency should be?
Every day? Every hour?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 511474 [details] [diff] [review]
remove binaries with older version for dependent L10n packages

Don't think you've tested this properly. It's returning 4.0b12pre builds when you want any non-4.0b12pre ones. May I suggest something like 

VERSION=`wget --quiet -O- http://hg.mozilla.org/mozilla-central/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-l10n -type f ! -name "*$VERSION*"

and you'll need something to actually remove files.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

RE timing of the cronjob - it's tricky. If it runs hourly then the directory will get emptied pretty quickly after we do a build1 for a new release.

tinderbox-builds only gets the on-change builds right ? I don't remember why we don't upload nightly builds there but this problem would go away if we did and used a daily cronjob to clean up. Easier for the localiser too, in that they only have to go to one place to get the latest build for their locale.

What say you Axel ?

Comment 4

[Armen [:armenzg]]

Attachment: remove binaries with older version for dependent L10n packages

My bad. I missed copy/paste for "-depth -exec rm -rf '{}' \;"

the "\! -name" catches the opposite of current version.

I also added your comments into this new patch.

I tested it as well.

Comment 5

[Axel Hecht [:Pike]]

The incremental updates will only work on the nightlies and not on deps, right? I'd rather not throw those into one bucket.

I would think that cleaning up once a day would suffice, though I'm not sure when. Can't wrap my head around how the dep builds interact with version bumps. Probably the builds fail anyway 'til the nightlies are spun with the bumped version anyway.

Maybe this should be two different purge algorithms? One for old builds, something like "anything older than a week", and one for "old version". Whether it makes sense to tie the latter to the actual version bump process or just run it as part of the other, don't know.

(The benefit of purging old builds with the current version is to just keep the list of files in that dir somewhat low to make searching easier)

Comment 6

[Armen [:armenzg]]

The dep builds seem to download
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-4.0b12pre.en-US.linux-i686.tar.bz2
so it might keep the version of the latest nightly (previous version).
We should add +mtime <number_of_days> (at least one day) into the mix to avoid removing repacks after a version bump.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 511499 [details] [diff] [review]
remove binaries with older version for dependent L10n packages

r+ if you remove the -depth, and feel free to add '-mtime +0' and run it hourly. You don't need to escape the ! or quote the {}.

Comment 8

[Armen [:armenzg]]

I have deployed this in ffxbld@stage's crontab:
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/mozilla-central/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-l10n -type f ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/mozilla-aurora/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-aurora-l10n -type f ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/release/mozilla-1.9.2/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-1.9.2-l10n -type f ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/release/mozilla-2.0/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-2.0-l10n -type f ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;

I will report back if I see things going well.

Comment 9

[Armen [:armenzg]]

I have substituted "release" for "releases".
I have also added -name "firefox*" to only delete binaries and leave the logs alone.

How long should we keep L10n logs around? Is 30 days good?
coop do we have a policy somewhere?

Currently they follow this format:
> mozilla-2.0-linux-l10n-dep-ast-build48.txt.gz

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to comment #9)
> How long should we keep L10n logs around? Is 30 days good?
> coop do we have a policy somewhere?

No official policy, no. 

Right now we're only keeping 3 days of logs in latest-*. How long we keep logs under tinderbox-builds/ is really a question for Axel (if he finds them useful at all), but 30 days should definitely be the max.

Comment 11

[Axel Hecht [:Pike]]

What's the retention policy for en-US these days? I overheard a discussion that went along the lines of "keep the logs as long as the builds", which would make sense to me, too.

Comment 12

[Armen [:armenzg]]

en-US tinderbox-builds are 30 days (builds & logs) from looking in ftp. Even though I can't find where is the cronjob that does the cleaning.

Does anyone know where the cronjobs for tinderbox-builds are? I can't see anything relevant on fxbld@stage.

Comment 13

[Armen [:armenzg]]

I have added cronjob to ffxbld@stage for the L10n logs in the same style as we do for en-US (as we do in root@stage).

@daily nice -n 19 find /pub/mozilla.org/firefox/tinderbox-builds/*-l10n -type f -mtime +29 -name *txt.gz -exec rm -f {} \;

Comment 14

[Armen [:armenzg]]

I asked nthomas on IRC and said it looked fine.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

We keep l10n nightlies for a few days, so I don't follow why we keep logs for on-change l10n builds for 30 days. Is there a reason for that ?

Comment 16

[Armen [:armenzg]]

No, no special reason.
I just chose a value that would keep things tidy and the logs are not big to be cleaning more aggressively.
Feel free to adjust it.