Periodically cleanup files on update.boot2gecko.org

RESOLVED FIXED

Status

Release Engineering
Buildduty
P1
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: jhopkins, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
We need more disk space for unagi builds as we keep hitting out of disk space issues when promoting a build to dogfooders.

Current space allocated to /data is 32GB with 24KB free.

This is blocking the latest promotion to dogfooders.
(Assignee)

Comment 1

5 years ago
This is ec2-184-73-70-191.compute-1.amazonaws.com which is not in releng AWS.

I rsynced and deleted some files:

[cltbld@buildbot-master36 ~]$ rsync -avP -e "ssh -i .ssh/b2gbld_dsa" ec2-user@update.boot2gecko.org:/data/update-channels/stable/ ~/stable-bug877224/

[ec2-user@ip-10-39-70-110 stable]$ rm /data/update-channels/stable/*2012*
(Assignee)

Comment 2

5 years ago
jgriffin, can you add another disk to this instance(i-8609e8e4), let's say 150G?
Flags: needinfo?(jgriffin)
(Assignee)

Updated

5 years ago
Summary: Add disk space to ip-10-39-70-110 → Add disk space to update.boot2gecko.org
(Assignee)

Comment 3

5 years ago
/data/update-channels/nightly/*201301* and /data/update-channels/nightly/*201302* are synced to bm36:~cltbld/nightly-bug877224 and deleted.

/dev/xvdf1             32G   23G  7.6G  75% /data
I can't; Andreas has the only admin access to this VM's config.  Andreas, this VM is used for our dogfood update server, can you add another 150G disk?

This is ec2-184-73-70-191.compute-1.amazonaws.com
Flags: needinfo?(jgriffin) → needinfo?(gal)
(Assignee)

Comment 5

5 years ago
We should cleanup old files instead. At some point we will switch to AUS.
Flags: needinfo?(gal)
Summary: Add disk space to update.boot2gecko.org → Periodically cleanup files on update.boot2gecko.org
Opening the bug, there's nothing sensitive here. We hit full disk again, looking for cleanups.
Group: mozilla-corporation-confidential
Blocks: 864069
I've removed all the mar and data files in /data/update_channels/nightly/, since we map requests for the old /nightly/update.html to /unagi/1.1.0/nightly/ now. Which leaves 6.5G free, which is probably only good for about 10 days.

So, perhaps we want a small python script that keeps the last 10 (say) files with names like b2g_update_20*.mar, application_20*.ini, b2g_update_source_20*.xml, while leaving everything else alone. We could also ask if we really need to the doing things like two inari v1.0.1 nightlies / day.
(Assignee)

Comment 8

5 years ago
Nick, how about this one:

find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -name '*.mar' -or -name '*.xml' -or -name '*.ini' -type f -mmin +20160

It deletes nightly mar/xml/ini files older than 2 weeks, not touching other files like beta.
Flags: needinfo?(nthomas)
That would certainly keep disk usage under control. Using -mtime +13 would be a little clearer (at least to me), and we could possibly keep things for less time. Looks like we're keeping about 5 weeks at the moment.

The downside is that branches that stop getting builds will get cleaned up, rather than leaving the last update. That's a feature for deprecated jobs, unless you need that last build to change the way updates are queried for (eg old -> midpoint build -> latest). It's also regression for any trees that you only build occasionally, which is admittedly not what we do for b2g.
Flags: needinfo?(nthomas)
(Assignee)

Comment 10

5 years ago
Since the updates are for in-house use only I tend to not overkill with clean up logic.

I added the following to crontab:

MAILTO=release+ec2.$HOSTNAME@mozilla.com
@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.mar' -delete
@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.ini' -delete
@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.xml' -delete
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Woot! I've added docs at
 https://intranet.mozilla.org/Auto-tools/B2G_Automation/Adding_New_Update_URLs#Update_cron_jobs
and generally cleaned that page up for clarity.

FTR, the current space usage is 13G of 32G (42%).
Created attachment 762788 [details]
bash script to email on low disk space

Attaching script for posterity -- driven from crontab an documented on the wiki page in comment 11.

Also updated https://wiki.mozilla.org/ReleaseEngineering/How_To/Process_release_email for these emails.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.