988657 - Android 2.3 abort: No space left on device

Status: RESOLVED DUPLICATE of bug 988262

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Geoff Brown [:gbrown]]

Android 2.3 tests on Ash have not been running for the last couple of days - bug  987759. Now that they are running, they are failing with "No space on device" fairly frequently:

 https://tbpl.mozilla.org/php/getParsedLog.php?id=36766705&tree=Ash&full=1

hg clone https://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness scripts
 in dir /builds/slave/test/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'https://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness', 'scripts']
 environment:
  COLORTERM=gnome-terminal
  DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-mppc4hEcJa,guid=cc1e7403ee195aab6fd74e1b0000003c
  DEFAULTS_PATH=/usr/share/gconf/ubuntu-2d.default.path
  DISPLAY=:0
  GNOME_DESKTOP_SESSION_ID=this-is-deprecated
  GNOME_KEYRING_CONTROL=/tmp/keyring-yVT7GB
  GPG_AGENT_INFO=/tmp/keyring-yVT7GB/gpg:0:1
  HOME=/home/cltbld
  LANG=en_US.UTF-8
  LANGUAGE=en_US:en
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANDATORY_PATH=/usr/share/gconf/ubuntu-2d.mandatory.path
  NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
  OPENGL_IS_BROKEN_HERE=1
  PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PWD=/builds/slave/test
  SESSION_MANAGER=local/tst-linux64-spot-751.test.releng.usw2.mozilla.com:@/tmp/.ICE-unix/1690,unix/tst-linux64-spot-751.test.releng.usw2.mozilla.com:/tmp/.ICE-unix/1690
  SHELL=/bin/bash
  SHLVL=1
  SSH_AGENT_PID=1718
  SSH_AUTH_SOCK=/tmp/keyring-yVT7GB/ssh
  TERM=xterm
  TMOUT=86400
  UBUNTU_MENUPROXY=libappmenu.so
  USER=cltbld
  WINDOWID=35651590
  XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu-2d:/etc/xdg
  XDG_CURRENT_DESKTOP=Unity
  XDG_DATA_DIRS=/usr/share/ubuntu-2d:/usr/share/gnome:/usr/local/share/:/usr/share/
  XDG_SESSION_COOKIE=9ca12473fbb1d023794ffd180000023c-1395875381.515413-335749518
  _=/etc/X11/Xsession
  __GL_YIELD=NOTHING
 using PTY: False
requesting all changes
adding changesets
adding manifests
adding file changes
added 2705 changesets with 4763 changes to 381 files (+8 heads)
updating to branch default
abort: No space left on device
program finished with exit code 255
elapsedTime=4.480796
========= Finished 'hg clone ...' failed (results: 5, elapsed: 4 secs) (at 2014-03-26 16:10:04.152956) =========

========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) =========
========= Started set_script_properties exception (results: 4, elapsed: 0 secs) (at 2014-03-26 16:10:04.153572) =========

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/twisted/spread/pb.py", line 840, in _recvMessage
    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/twisted/spread/flavors.py", line 114, in remoteMessageReceived
    state = method(*args, **kw)
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/buildslave/bot.py", line 141, in remote_startCommand
    d = self.command.doStart()
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/buildslave/commands/base.py", line 140, in doStart
    d = defer.maybeDeferred(self.start)
--- <exception caught here> ---
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
    result = f(*args, **kw)
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/buildslave/commands/shell.py", line 77, in start
    logEnviron=args.get('logEnviron', True),
  File "/tools/buildbot-0.8.4-pre-moz2/local/lib/python2.7/site-packages/buildslave/runprocess.py", line 281, in __init__
    os.makedirs(workdir)
  File "/tools/buildbot-0.8.4-pre-moz2/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
exceptions.OSError: [Errno 28] No space left on device: '/builds/slave/test/properties'
]

Comment 1

[Geoff Brown [:gbrown]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=36766987&tree=Ash&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=36767164&tree=Ash&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=36763240&tree=Ash&full=1

Comment 2

[Geoff Brown [:gbrown]]

This might be a dup of 988262.

Comment 3

[Armen [:armenzg]]

That bug is the root cause over here.

Comment 4

[John Hopkins (:jhopkins)]

Armen: do we need to keep this bug open given that the root cause is tracked elsewhere?

Comment 5

[Geoff Brown [:gbrown]]

fwiw, I have not been seeing this happen much on ash; it is not hampering the Android 2.3 efforts.

Comment 6

[Armen [:armenzg]]

Let's close it.