Closed Bug 1042790 Opened 10 years ago Closed 10 years ago

clobberer step is failing to clobber

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlund, Unassigned)

References

Details

07:33:04     INFO -  Removing lost+found.deleteme/
07:33:04     INFO -  Couldn't clobber properly, bailing out.
07:33:04    ERROR - Return code: 1
07:33:04     INFO - retry: Failed, sleeping 60 seconds before retrying
07:34:04     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #2
07:34:04     INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/'] in /builds/slave
07:34:04     INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-020 http://buildbot-master70.srv.releng.use1.mozilla.com:8001/
07:34:04     INFO -  Checking clobber URL: http://clobberer.pvt.build.mozilla.org/index.php?master=http%3A%2F%2Fbuildbot-master70.srv.releng.use1.mozilla.com%3A8001%2F&slave=bld-linux64-spot-020&builddir=%2Fbuilds%2Fslave&branch=b2g-inbound&buildername=b2g_b2g-inbound_emulator_dep
07:34:04     INFO -  b2g_b2g-in_emu-jb-d_dep-000000:Our last clobber date:  None
07:34:04     INFO -  b2g_b2g-in_emu-jb-d_dep-000000:Server clobber date:    2014-07-23 07:24:05
07:34:04     INFO -  tb-c-cen-l64-l10n-dep-00000000:Our last clobber date:  None
07:34:04     INFO -  tb-c-cen-l64-l10n-dep-00000000:Server clobber date:    2014-07-23 06:07:03
07:34:04     INFO -  /builds/slave:Our last clobber date:  2014-07-23 04:34:13
07:34:04     INFO -  /builds/slave:Server clobber date:    2014-07-23 07:24:05
07:34:04     INFO -  /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com
07:34:04     INFO -  /builds/slave:Clobbering...
07:34:04     INFO -  Removing lost+found.deleteme/
07:34:04     INFO -  Couldn't clobber properly, bailing out.
07:34:04    ERROR - Return code: 1
07:34:04     INFO - retry: Failed, sleeping 120 seconds before retrying
07:36:04     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #3
07:36:04     INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/'] in /builds/slave
07:36:04     INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-020 http://buildbot-master70.srv.releng.use1.mozilla.com:8001/
07:36:05     INFO -  Checking clobber URL: http://clobberer.pvt.build.mozilla.org/index.php?master=http%3A%2F%2Fbuildbot-master70.srv.releng.use1.mozilla.com%3A8001%2F&slave=bld-linux64-spot-020&builddir=%2Fbuilds%2Fslave&branch=b2g-inbound&buildername=b2g_b2g-inbound_emulator_dep
07:36:05     INFO -  b2g_b2g-in_emu-jb-d_dep-000000:Our last clobber date:  None
07:36:05     INFO -  b2g_b2g-in_emu-jb-d_dep-000000:Server clobber date:    2014-07-23 07:24:05
07:36:05     INFO -  tb-c-cen-l64-l10n-dep-00000000:Our last clobber date:  None
07:36:05     INFO -  tb-c-cen-l64-l10n-dep-00000000:Server clobber date:    2014-07-23 06:07:03
07:36:05     INFO -  /builds/slave/b2g_m-aurora_flame_eng_dep-000 doesn't exist, skipping
07:36:05     INFO -  b2g_b2g-in_emu_dep-00000000000:Our last clobber date:  None
07:36:05     INFO -  b2g_b2g-in_emu_dep-00000000000:Server clobber date:    2014-07-23 07:24:05
07:36:05     INFO -  /builds/slave:Our last clobber date:  2014-07-23 04:34:13
07:36:05     INFO -  /builds/slave:Server clobber date:    2014-07-23 07:24:05
07:36:05     INFO -  /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com
07:36:05     INFO -  /builds/slave:Clobbering...
07:36:05     INFO -  Removing lost+found.deleteme/
07:36:05     INFO -  Couldn't clobber properly, bailing out.
07:36:05    ERROR - Return code: 1
07:36:05    ERROR - <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> failed after 3 tries!
07:36:05    FATAL - failed to clobber build
07:36:05    FATAL - Running post_fatal callback...
07:36:05    FATAL - Exiting 2
program finished with exit code 2
thus far this has only been seen on b2g platforms. and all "07:34:04     INFO -  /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com"

So it looks like it is limited to b2g builders. clobberer hosts and AWS instances are suspects.
jobs are still occasionally failing. callek pointed out this could be fall out of my fx desktop build patch: http://hg.mozilla.org/build/mozharness/rev/da2e0cf4a096
it looks like they keep clobbering builddirs fine until it gets to:
07:36:05     INFO -  Removing lost+found.deleteme/
07:36:05     INFO -  Couldn't clobber properly, bailing out.


looking here you can see it is owned by root:
[cltbld@bld-linux64-spot-024.build.releng.use1.mozilla.com slave]$ ls -la
total 5388
drwx------    2 root   root          16384 Jul 23  2014 lost+found.deleteme
it appears to not just be limited to lost+found.deleteme

07:38:24     INFO -  Removing mock_mozilla/
07:38:24     INFO -  Couldn't clobber properly, bailing out.
07:38:24    ERROR - Return code: 1
07:38:24     INFO - retry: Failed, sleeping 60 seconds before retrying
07:39:24     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #2
07:39:24     INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/'] in /builds/slave
07:39:24     INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-027 http://buildbot-master71.srv.releng.use1.mozilla.com:8001/
07:39:24     INFO -  python: can't open file '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py': [Errno 2] No such file or directory
07:39:24    ERROR - Return code: 2
07:39:24     INFO - retry: Failed, sleeping 120 seconds before retrying
07:41:24     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #3
07:41:24     INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/'] in /builds/slave
07:41:24     INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-027 http://buildbot-master71.srv.releng.use1.mozilla.com:8001/
07:41:24     INFO -  python: can't open file '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py': [Errno 2] No such file or directory
07:41:24    ERROR - Return code: 2
07:41:24    ERROR - <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> failed after 3 tries!
07:41:24    FATAL - failed to clobber build
07:41:24    FATAL - Running post_fatal callback...
07:41:24    FATAL - Exiting 2

I think there is something going on due to my previous patch:
diff --git a/external_tools/clobberer.py b/external_tools/clobberer.py
--- a/external_tools/clobberer.py
+++ b/external_tools/clobberer.py
@@ -1,10 +1,10 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # vim:sts=2 sw=2
 import sys
 import shutil
 import urllib2
 import urllib
 import os
 import traceback
 import time
 if os.name == 'nt':
diff --git a/mozharness/mozilla/purge.py b/mozharness/mozilla/purge.py
--- a/mozharness/mozilla/purge.py
+++ b/mozharness/mozilla/purge.py
@@ -27,38 +27,42 @@ class PurgeMixin(object):
     clobber_tool = os.path.join(external_tools_path, 'clobberer.py')
 
     default_skips = ['info', 'rel-*', 'tb-rel-*']
     default_maxage = 14
     default_periodic_clobber = 7 * 24
 
     def purge_builds(self, basedirs=None, min_size=None, skip=None, max_age=None):
         # Try clobbering first
         c = self.config
+        dirs = self.query_abs_dirs()
         if 'clobberer_url' in c:
             self.clobberer()
 
         min_size = min_size or c['purge_minsize']
         max_age = max_age or c.get('purge_maxage') or self.default_maxage
         skip = skip or c.get('purge_skip') or self.default_skips
 
         if not basedirs:
             # some platforms using this method (like linux) supply more than
             # one basedir
             basedirs = []
-            assert self.buildbot_config
-            basedirs.append(os.path.dirname(self.buildbot_config['properties']['basedir']))
+            basedirs.append(os.path.dirname(dirs['base_work_dir']))
             if self.config.get('purge_basedirs'):
                 basedirs.extend(self.config.get('purge_basedirs'))
 
+        try:
+            python = self.query_python_path()
+        except AttributeError:
+            # we are not inheriting from VirtualenvMixin
+            python = self.query_exe('python')
+
         # Add --dry-run if you don't want to do this for realz
-        cmd = [self.purge_tool,
-               '-s', str(min_size),
-               ]
+        cmd = [python, self.purge_tool, '-s', str(min_size)]
 
         if max_age:
             cmd.extend(['--max-age', str(max_age)])
 
         for s in skip:
             cmd.extend(['--not', s])
 
         cmd.extend(basedirs)
 
@@ -68,50 +72,57 @@ class PurgeMixin(object):
         share_base = c.get('vcs_share_base', os.environ.get("HG_SHARE_BASE_DIR", None))
         if share_base:
             env['HG_SHARE_BASE_DIR'] = share_base
         retval = self.run_command(cmd, env=env)
         if retval != 0:
             self.fatal("failed to purge builds", exit_code=2)
 
     def clobberer(self):
         c = self.config
+        dirs = self.query_abs_dirs()
         if not self.buildbot_config:
             self.fatal("clobberer requires self.buildbot_config (usually from $PROPERTIES_FILE)")
 
         periodic_clobber = c.get('periodic_clobber') or self.default_periodic_clobber
         clobberer_url = c['clobberer_url']
 
-        builddir = os.path.basename(self.buildbot_config['properties']['basedir'])
+        builddir = os.path.dirname(dirs['base_work_dir'])
         branch = self.buildbot_config['properties']['branch']
         buildername = self.buildbot_config['properties']['buildername']
         slave = self.buildbot_config['properties']['slavename']
         master = self.buildbot_config['properties']['master']
 
+
+        try:
+            python = self.query_python_path()
+        except AttributeError:
+            # we are not inheriting from VirtualenvMixin
+            python = self.query_exe('python')
+
         # Add --dry-run if you don't want to do this for realz
-        cmd = [self.clobber_tool]
+        cmd = [python, self.clobber_tool]
         # TODO configurable list
         cmd.extend(['-s', 'scripts'])
         cmd.extend(['-s', 'logs'])
         cmd.extend(['-s', 'buildprops.json'])
         cmd.extend(['-s', 'token'])
 
         if periodic_clobber:
             cmd.extend(['-t', str(periodic_clobber)])
 
         cmd.extend([clobberer_url, branch, buildername, builddir, slave, master])
         error_list = [{
             'substr': 'Error contacting server', 'level': ERROR,
             'explanation': 'Error contacting server for clobberer information.'
         }]
 
         retval = self.retry(self.run_command, attempts=3, good_statuses=(0,), args=[cmd],
-                 kwargs={'cwd':os.path.dirname(self.buildbot_config['properties']['basedir']),
-                         'error_list':error_list})
+                            kwargs={'cwd': builddir, 'error_list': error_list})
         if retval != 0:
             self.fatal("failed to clobber build", exit_code=2)
 
     def clobber(self, always_clobber_dirs=None):
         """ Mozilla clobberer-type clobber.
             """
         c = self.config
         if c.get('is_automation'):
             # Nightly builds always clobber


you can see that it may affect the interpreter. I am going to back this out
resolved per c#5
Status: NEW → RESOLVED
Closed: 10 years ago
Depends on: 858797
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.