649886 - free-space clobber requirement set too low

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

moz2-linux-slave16 just ran out of space and killed itself.

Running a 'du', these are the interestingly large dirs:

966M    191-lnx-l10n-ntly
3.2G    192-lnx-dbg
2.8G    aurora-mob-andrd-r7-ntly/
1.3G    cen-lnx-l10n-dep/
1.5G    cen-lnx-l10n-ntly/
2.8G    cen-mob-andrd-r7-bld/
1.8G    valgrind-linux
767M    fuzzer-linux/
3.1G    plc-mob-lnx-bld/
694M    tm-lnx-dbg-spidermonkey-nomethodjit/
1.8G    valgrind-linux/
----
~22G

/dev/sdb1              30G   29G     0 100% /builds

Comment 1

[Chris AtLee [:catlee]]

(In reply to comment #0)
> moz2-linux-slave16 just ran out of space and killed itself.
> 
> Running a 'du', these are the interestingly large dirs:
> 
> 966M    191-lnx-l10n-ntly
> 3.2G    192-lnx-dbg
> 2.8G    aurora-mob-andrd-r7-ntly/
> 1.3G    cen-lnx-l10n-dep/
> 1.5G    cen-lnx-l10n-ntly/
> 2.8G    cen-mob-andrd-r7-bld/
> 1.8G    valgrind-linux
> 767M    fuzzer-linux/
> 3.1G    plc-mob-lnx-bld/
> 694M    tm-lnx-dbg-spidermonkey-nomethodjit/
> 1.8G    valgrind-linux/
> ----
> ~22G
> 
> /dev/sdb1              30G   29G     0 100% /builds

[cltbld@moz2-linux-slave16 builds]$ du -hs ccache hg-shared
1.8G	ccache
2.8G	hg-shared/

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I deleted some build dirs on the slave and rebooted, so this slave itself should be OK for a while - we probably still need to adjust the clobberer though, and I don't know how to do that.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The key is to figure out which build it ran out of space on, look at how much space it was set to require in the clobber call, then dig around in config.py until you find that setting.

Comment 5

[Chris Cooper [:coop] (he/him)]

So we know that tracemonkey currently has this problem (bug 646318), but I'm not sure by how much we need to bump up the space requirements.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like I either forgot to reboot, or the reboot failed to start buildbot for some reason; at any rate:
2011-04-13 18:07:16-0700 [Broker,client]   in dir /builds/moz2_slave/192-lnx-dbg/build/obj-firefox (timeout 1200 secs)

So to answer nick's questions in comment 3, that means 1.9.2 linux debug build, which hit 3.2GB in this build.  *Now* I'll get the slave back up.

Comment 7

[Chris Cooper [:coop] (he/him)]

I'll make the adjustments for tracemonkey on Linux since that's the only recent example of this I can find in the logs.

Comment 8

[Chris Cooper [:coop] (he/him)]

Attachment: Bump build space for tracemonkey on Linux

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 528595 [details] [diff] [review]
Bump build space for tracemonkey on Linux

http://hg.mozilla.org/build/buildbot-configs/rev/ca5f4080e95e

Comment 10

[Chris Cooper [:coop] (he/him)]

Merged into production today.