565357 - figure out TTL for nightly snippets and snippet backups

Status: RESOLVED DUPLICATE of bug 596391

(Release Engineering :: General, defect, P5)

Description

[[e-mail] (:bhearsum)]

We had some nightlies fail today with errors like:
mkdir: cannot create directory `/opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20100512030754': No space left on device

Strangely, the partition has 7MB free still, so we could be out of inodes, not space. Nevertheless, we'll need more space since we're doing nightly backups now.

Comment 1

[[e-mail] (:bhearsum)]

I freed some space by moving all snippet backups from 2007 to ~/old-backups and rekicked all the failed nightlies.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I removed /opt/aus2/build-with-nightly-l10n to get us the 7MB free, hoping to avoid blowing out any nightlies still in progress. That was an old directory from when we enabled nightly updates for l10n, and were using prometheus-vm for the partials. 

/opt/aus2/build is probably every nightly we ever did updates for, and we only need the Tb2 and Fx3 parts now.

Comment 3

[[e-mail] (:bhearsum)]

We're running low on space here again (down to 300MB). We should probably figure out a long term solution here in addition to adding more space. Here's a proposal:
* Only keep 1 nightly snippet backup per week, for backups older than a month
* Delete all nightly snippets older than a month

Anyone have objections are agreements?

Comment 4

[[e-mail] (:bhearsum)]

After chatting with nthomas on IRC I moved /opt/aus2/build.old to ~cltbld and removed the root cronjob that synced build -> build.old. This has freed up 1GB of space on /opt.

Comment 5

[[e-mail] (:bhearsum)]

Partition size has been increased, but we still need to figure out an aging-off plan for nightly snippets and the backups.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to comment #5)
> Partition size has been increased, but we still need to figure out an aging-off
> plan for nightly snippets and the backups.

We're removing all nightly MARs from stage after thirty days if we want to apply the same logic here.

Comment 7

[[e-mail] (:bhearsum)]

This ended up happening in bug 596391 AFAICT