Periodically cleanup files on



Release Engineering
5 years ago
5 years ago


(Reporter: jhopkins, Assigned: rail)


Firefox Tracking Flags

(Not tracked)



(1 attachment)



5 years ago
We need more disk space for unagi builds as we keep hitting out of disk space issues when promoting a build to dogfooders.

Current space allocated to /data is 32GB with 24KB free.

This is blocking the latest promotion to dogfooders.

Comment 1

5 years ago
This is which is not in releng AWS.

I rsynced and deleted some files:

[cltbld@buildbot-master36 ~]$ rsync -avP -e "ssh -i .ssh/b2gbld_dsa" ~/stable-bug877224/

[ec2-user@ip-10-39-70-110 stable]$ rm /data/update-channels/stable/*2012*

Comment 2

5 years ago
jgriffin, can you add another disk to this instance(i-8609e8e4), let's say 150G?
Flags: needinfo?(jgriffin)


5 years ago
Summary: Add disk space to ip-10-39-70-110 → Add disk space to

Comment 3

5 years ago
/data/update-channels/nightly/*201301* and /data/update-channels/nightly/*201302* are synced to bm36:~cltbld/nightly-bug877224 and deleted.

/dev/xvdf1             32G   23G  7.6G  75% /data
I can't; Andreas has the only admin access to this VM's config.  Andreas, this VM is used for our dogfood update server, can you add another 150G disk?

This is
Flags: needinfo?(jgriffin) → needinfo?(gal)

Comment 5

5 years ago
We should cleanup old files instead. At some point we will switch to AUS.
Flags: needinfo?(gal)
Summary: Add disk space to → Periodically cleanup files on
Opening the bug, there's nothing sensitive here. We hit full disk again, looking for cleanups.
Group: mozilla-corporation-confidential
Blocks: 864069
I've removed all the mar and data files in /data/update_channels/nightly/, since we map requests for the old /nightly/update.html to /unagi/1.1.0/nightly/ now. Which leaves 6.5G free, which is probably only good for about 10 days.

So, perhaps we want a small python script that keeps the last 10 (say) files with names like b2g_update_20*.mar, application_20*.ini, b2g_update_source_20*.xml, while leaving everything else alone. We could also ask if we really need to the doing things like two inari v1.0.1 nightlies / day.

Comment 8

5 years ago
Nick, how about this one:

find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -name '*.mar' -or -name '*.xml' -or -name '*.ini' -type f -mmin +20160

It deletes nightly mar/xml/ini files older than 2 weeks, not touching other files like beta.
Flags: needinfo?(nthomas)
That would certainly keep disk usage under control. Using -mtime +13 would be a little clearer (at least to me), and we could possibly keep things for less time. Looks like we're keeping about 5 weeks at the moment.

The downside is that branches that stop getting builds will get cleaned up, rather than leaving the last update. That's a feature for deprecated jobs, unless you need that last build to change the way updates are queried for (eg old -> midpoint build -> latest). It's also regression for any trees that you only build occasionally, which is admittedly not what we do for b2g.
Flags: needinfo?(nthomas)

Comment 10

5 years ago
Since the updates are for in-house use only I tend to not overkill with clean up logic.

I added the following to crontab:

@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.mar' -delete
@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.ini' -delete
@daily find /data/update-channels/{hamachi,inari,leo,unagi}/*/nightly -type f -mtime +13 -name '*.xml' -delete
Last Resolved: 5 years ago
Resolution: --- → FIXED
Woot! I've added docs at
and generally cleaned that page up for clarity.

FTR, the current space usage is 13G of 32G (42%).
Created attachment 762788 [details]
bash script to email on low disk space

Attaching script for posterity -- driven from crontab an documented on the wiki page in comment 11.

Also updated for these emails.
Product: → Release Engineering
You need to log in before you can comment on or make changes to this bug.