Intermittent failure to running rmtree on Linux, "WARNING: Command failed. See logs for output.", "# umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/tmp/ccache", etc.

RESOLVED WORKSFORME

Status

Firefox Build System
General
RESOLVED WORKSFORME
2 years ago
4 months ago

People

(Reporter: aryx, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

+++ This bug was initially created as a clone of Bug #1244446 +++

https://treeherder.mozilla.org/logviewer.html#?job_id=37602672&repo=mozilla-inbound

14:58:05     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --cwd /builds/slave/m-in-lx-0000000000000000000000/build/src/obj-firefox --unpriv --shell "/usr/bin/env MOZ_AUTOMATION=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" TMOUT=86400 LOGNAME=cltbld USER=cltbld PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 HG_SHARE_BASE_DIR=/builds/hg-shared MOZ_SOURCE_REPO=https://hg.mozilla.org/integration/mozilla-inbound CCACHE_DIR=/builds/ccache MOZ_BUILD_DATE=20161013141619 TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/m-in-lx-0000000000000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-lx-0000000000000000000000/build/signing_cache -t /builds/slave/m-in-lx-0000000000000000000000/token -n /builds/slave/m-in-lx-0000000000000000000000/nonce -c /builds/slave/m-in-lx-0000000000000000000000/build/tools/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110\" LC_ALL=C TOOLTOOL_HOME=/builds _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib MAIL=/var/spool/mail/cltbld HOSTNAME=bld-linux64-spot-248.build.releng.use1.mozilla.com MOZBUILD_STATE_PATH=/builds/slave/m-in-lx-0000000000000000000000/.mozbuild HISTCONTROL=ignoredups PWD=/builds/slave/m-in-lx-0000000000000000000000 PROPERTIES_FILE=/builds/slave/m-in-lx-0000000000000000000000/buildprops.json MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 TOOLTOOL_CACHE=/builds/tooltool_cache python2.7 /builds/slave/m-in-lx-0000000000000000000000/build/src/mach python /builds/slave/m-in-lx-0000000000000000000000/build/src/config/printconfigsetting.py /builds/slave/m-in-lx-0000000000000000000000/build/src/obj-firefox/dist/bin/application.ini App SourceStamp"
14:58:08    ERROR - Errors received:
14:58:08     INFO - Reading from file tmpfile_stderr
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/tmp/ccache
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/targetdata/
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/tmp/.X11-unix
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/tooltool_cache
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/mercurial-certs
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/git-shared
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/hg-shared
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/ccache
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/builds/slave
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/var/cache/yum
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/sys
14:58:08    ERROR -  WARNING: Command failed. See logs for output.
14:58:08    ERROR -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64/root/proc
14:58:08    ERROR -  Traceback (most recent call last):
14:58:08    ERROR -    File "/usr/sbin/mock_mozilla", line 862, in <module>
14:58:08    ERROR -      main(retParams)
14:58:08    ERROR -    File "/usr/sbin/mock_mozilla", line 740, in main
14:58:08    ERROR -      sys.exit(chroot.shell(options, cmd))
14:58:08    ERROR -    File "/usr/lib/python2.6/site-packages/mock_mozilla/backend.py", line 659, in shell
14:58:08    ERROR -      self._setupDev()
14:58:08    ERROR -    File "<peak.util.decorators.rewrap wrapping mock_mozilla.backend._setupDev at 0x02918230>", line 3, in _setupDev
14:58:08    ERROR -    File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
14:58:08    ERROR -      result = func(*args, **kw)
14:58:08    ERROR -    File "/usr/lib/python2.6/site-packages/mock_mozilla/backend.py", line 386, in _setupDev
14:58:08    ERROR -      mock_mozilla.util.rmtree(self.makeChrootPath("dev"), selinux=self.selinux)
14:58:08    ERROR -    File "<peak.util.decorators.rewrap wrapping mock_mozilla.util.rmtree at 0x02905938>", line 3, in rmtree
14:58:08    ERROR -    File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
14:58:08    ERROR -      result = func(*args, **kw)
14:58:08    ERROR -    File "/usr/lib/python2.6/site-packages/mock_mozilla/util.py", line 87, in rmtree
14:58:08    ERROR -      shutil.rmtree(path, *args, **kargs)
14:58:08    ERROR -    File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree
14:58:08    ERROR -      rmtree(fullname, ignore_errors, onerror)
14:58:08    ERROR -    File "/usr/lib64/python2.6/shutil.py", line 221, in rmtree
14:58:08    ERROR -      onerror(os.rmdir, path, sys.exc_info())
14:58:08    ERROR -    File "/usr/lib64/python2.6/shutil.py", line 219, in rmtree
14:58:08    ERROR -      os.rmdir(path)
14:58:08    ERROR -  OSError: [Errno 16] Device or resource busy: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/dev/shm'
14:58:08    ERROR - Return code: 1
14:58:08  WARNING - setting return code to 2

Comment 1

2 years ago
11 failures in 623 pushes (0.018 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 7
* autoland: 4

Platform breakdown:
* linux64: 7
* android-4-0-armv7-api15: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1310038&startday=2016-11-21&endday=2016-11-27&tree=all
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → WORKSFORME

Updated

4 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.