Closed
Bug 801607
Opened 12 years ago
Closed 10 years ago
Make EC2 instances less susceptible to "abort: No space left on device"
Categories
(Release Engineering :: General, defect, P2)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Unassigned)
References
Details
(Keywords: sheriffing-untriaged)
Latest instance: slave: bld-linux64-ec2-020 https://tbpl.mozilla.org/php/getParsedLog.php?id=16115372&tree=Mozilla-Inbound { ========= Started clone build tools failed (results: 2, elapsed: 19 secs) (at 2012-10-15 05:45:08.363782) ========= hg clone http://hg.mozilla.org/build/tools tools in dir /builds/slave/m-in-lnx/. (timeout 1320 secs) watching logfiles {} argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-ec2-020.build.aws-us-west-1.mozilla.com LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/m-in-lnx SHELL=/bin/bash SHLVL=1 TERM=linux USER=cltbld _=/tools/buildbot/bin/python using PTY: False requesting all changes adding changesets adding manifests adding file changes added 3062 changesets with 6319 changes to 1063 files updating to branch default abort: No space left on device program finished with exit code 255 elapsedTime=19.700516 ========= Finished clone build tools failed (results: 2, elapsed: 19 secs) (at 2012-10-15 05:45:28.080959) ========= }
Reporter | ||
Comment 1•12 years ago
|
||
and: s: bld-linux64-ec2-033 https://tbpl.mozilla.org/php/getParsedLog.php?id=16113771&tree=Mozilla-Inbound { ========= Started clone build tools failed (results: 2, elapsed: 20 secs) (at 2012-10-15 04:16:27.915769) ========= hg clone http://hg.mozilla.org/build/tools tools in dir /builds/slave/m-in-lnx-dbg/. (timeout 1320 secs) watching logfiles {} argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-ec2-033.build.aws-us-west-1.mozilla.com LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/m-in-lnx-dbg SHELL=/bin/bash SHLVL=1 TERM=linux USER=cltbld _=/tools/buildbot/bin/python using PTY: False requesting all changes adding changesets adding manifests adding file changes added 3061 changesets with 6318 changes to 1063 files updating to branch default abort: No space left on device: /builds/slave/m-in-lnx-dbg/tools/lib/python program finished with exit code 255 elapsedTime=20.961564 ========= Finished clone build tools failed (results: 2, elapsed: 20 secs) (at 2012-10-15 04:16:48.912714) ========= }
Comment 2•12 years ago
|
||
Sounds like we need to bump buildSpace requirements due to mock overhead -- downloaded RPMs and mock chroot.
Comment 3•12 years ago
|
||
Having some initial cleanup on boot would be a good thing to do too.
Depends on: 712206
Updated•12 years ago
|
Severity: critical → major
Priority: -- → P2
Reporter | ||
Updated•12 years ago
|
Comment 4•12 years ago
|
||
The EC2 instances have ~100G build space, and the ix hardware they're replacing has close to twice that. This could also be summarised as "update build space requirements to reality, since we've been getting away with them being wrong by having a lot of free space".
Comment 5•12 years ago
|
||
linux64 needs another gig - http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#190 should be a 7 (inbound overrides the 6, and I don't think it's been starting these).
Comment 6•12 years ago
|
||
I bumped the build space for linux64 to 7 - http://hg.mozilla.org/build/buildbot-configs/rev/eebcc5ed460e
Comment 7•12 years ago
|
||
in production
Reporter | ||
Comment 8•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=17171227&tree=Firefox bld-linux64-ec2-306 android
Reporter | ||
Updated•12 years ago
|
Keywords: sheriffing-untriaged
Whiteboard: [sheriff-want]
Assignee | ||
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Comment 9•10 years ago
|
||
It doesn't look like there's an overarching problem to fix here...just the normal individual buildSpace bumps.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Assignee | ||
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•