Closed Bug 565357 Opened 10 years ago Closed 9 years ago

figure out TTL for nightly snippets and snippet backups

Categories

(Release Engineering :: General, defect, P5)

All
macOS
defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 596391

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

(Whiteboard: [updates])

We had some nightlies fail today with errors like:
mkdir: cannot create directory `/opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20100512030754': No space left on device

Strangely, the partition has 7MB free still, so we could be out of inodes, not space. Nevertheless, we'll need more space since we're doing nightly backups now.
I freed some space by moving all snippet backups from 2007 to ~/old-backups and rekicked all the failed nightlies.
I removed /opt/aus2/build-with-nightly-l10n to get us the 7MB free, hoping to avoid blowing out any nightlies still in progress. That was an old directory from when we enabled nightly updates for l10n, and were using prometheus-vm for the partials. 

/opt/aus2/build is probably every nightly we ever did updates for, and we only need the Tb2 and Fx3 parts now.
We're running low on space here again (down to 300MB). We should probably figure out a long term solution here in addition to adding more space. Here's a proposal:
* Only keep 1 nightly snippet backup per week, for backups older than a month
* Delete all nightly snippets older than a month

Anyone have objections are agreements?
After chatting with nthomas on IRC I moved /opt/aus2/build.old to ~cltbld and removed the root cronjob that synced build -> build.old. This has freed up 1GB of space on /opt.
Blocks: 570512
No longer blocks: 570512
Partition size has been increased, but we still need to figure out an aging-off plan for nightly snippets and the backups.
Summary: AUS snippet partition ran out of space → figure out TTL for nightly snippets and snippet backups
(In reply to comment #5)
> Partition size has been increased, but we still need to figure out an aging-off
> plan for nightly snippets and the backups.

We're removing all nightly MARs from stage after thirty days if we want to apply the same logic here.
Whiteboard: [updates]
Priority: -- → P5
This ended up happening in bug 596391 AFAICT
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 596391
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.