801607 - Make EC2 instances less susceptible to "abort: No space left on device"

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Ed Morley [:emorley]]

Latest instance:

slave: bld-linux64-ec2-020
https://tbpl.mozilla.org/php/getParsedLog.php?id=16115372&tree=Mozilla-Inbound

{
========= Started clone build tools failed (results: 2, elapsed: 19 secs) (at 2012-10-15 05:45:08.363782) =========
hg clone http://hg.mozilla.org/build/tools tools
 in dir /builds/slave/m-in-lnx/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-ec2-020.build.aws-us-west-1.mozilla.com
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/slave/m-in-lnx
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
requesting all changes
adding changesets
adding manifests
adding file changes
added 3062 changesets with 6319 changes to 1063 files
updating to branch default
abort: No space left on device
program finished with exit code 255
elapsedTime=19.700516
========= Finished clone build tools failed (results: 2, elapsed: 19 secs) (at 2012-10-15 05:45:28.080959) =========
}

Comment 1

[Ed Morley [:emorley]]

and:

s: bld-linux64-ec2-033
https://tbpl.mozilla.org/php/getParsedLog.php?id=16113771&tree=Mozilla-Inbound

{
========= Started clone build tools failed (results: 2, elapsed: 20 secs) (at 2012-10-15 04:16:27.915769) =========
hg clone http://hg.mozilla.org/build/tools tools
 in dir /builds/slave/m-in-lnx-dbg/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-ec2-033.build.aws-us-west-1.mozilla.com
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/slave/m-in-lnx-dbg
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
requesting all changes
adding changesets
adding manifests
adding file changes
added 3061 changesets with 6318 changes to 1063 files
updating to branch default
abort: No space left on device: /builds/slave/m-in-lnx-dbg/tools/lib/python
program finished with exit code 255
elapsedTime=20.961564
========= Finished clone build tools failed (results: 2, elapsed: 20 secs) (at 2012-10-15 04:16:48.912714) =========
}

Comment 2

[Rail Aliiev [:rail]]

Sounds like we need to bump buildSpace requirements due to mock overhead -- downloaded RPMs and mock chroot.

Comment 3

[Chris AtLee [:catlee]]

Having some initial cleanup on boot would be a good thing to do too.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The EC2 instances have ~100G build space, and the ix hardware they're replacing has close to twice that. This could also be summarised as "update build space requirements to reality, since we've been getting away with them being wrong by having a lot of free space".

Comment 5

[Phil Ringnalda (:philor)]

linux64 needs another gig - http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#190 should be a 7 (inbound overrides the 6, and I don't think it's been starting these).

Comment 6

[Chris AtLee [:catlee]]

I bumped the build space for linux64 to 7 - http://hg.mozilla.org/build/buildbot-configs/rev/eebcc5ed460e

Comment 7

[Rail Aliiev [:rail]]

in production

Comment 8

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17171227&tree=Firefox
bld-linux64-ec2-306
android

Comment 9

[[e-mail] (:bhearsum)]

It doesn't look like there's an overarching problem to fix here...just the normal individual buildSpace bumps.