Closed Bug 1042790 Opened 10 years ago Closed 10 years ago

clobberer step is failing to clobber

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlund, Unassigned)

References

Details

07:33:04 INFO - Removing lost+found.deleteme/ 07:33:04 INFO - Couldn't clobber properly, bailing out. 07:33:04 ERROR - Return code: 1 07:33:04 INFO - retry: Failed, sleeping 60 seconds before retrying 07:34:04 INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #2 07:34:04 INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/'] in /builds/slave 07:34:04 INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-020 http://buildbot-master70.srv.releng.use1.mozilla.com:8001/ 07:34:04 INFO - Checking clobber URL: http://clobberer.pvt.build.mozilla.org/index.php?master=http%3A%2F%2Fbuildbot-master70.srv.releng.use1.mozilla.com%3A8001%2F&slave=bld-linux64-spot-020&builddir=%2Fbuilds%2Fslave&branch=b2g-inbound&buildername=b2g_b2g-inbound_emulator_dep 07:34:04 INFO - b2g_b2g-in_emu-jb-d_dep-000000:Our last clobber date: None 07:34:04 INFO - b2g_b2g-in_emu-jb-d_dep-000000:Server clobber date: 2014-07-23 07:24:05 07:34:04 INFO - tb-c-cen-l64-l10n-dep-00000000:Our last clobber date: None 07:34:04 INFO - tb-c-cen-l64-l10n-dep-00000000:Server clobber date: 2014-07-23 06:07:03 07:34:04 INFO - /builds/slave:Our last clobber date: 2014-07-23 04:34:13 07:34:04 INFO - /builds/slave:Server clobber date: 2014-07-23 07:24:05 07:34:04 INFO - /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com 07:34:04 INFO - /builds/slave:Clobbering... 07:34:04 INFO - Removing lost+found.deleteme/ 07:34:04 INFO - Couldn't clobber properly, bailing out. 07:34:04 ERROR - Return code: 1 07:34:04 INFO - retry: Failed, sleeping 120 seconds before retrying 07:36:04 INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #3 07:36:04 INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-020', u'http://buildbot-master70.srv.releng.use1.mozilla.com:8001/'] in /builds/slave 07:36:04 INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-020 http://buildbot-master70.srv.releng.use1.mozilla.com:8001/ 07:36:05 INFO - Checking clobber URL: http://clobberer.pvt.build.mozilla.org/index.php?master=http%3A%2F%2Fbuildbot-master70.srv.releng.use1.mozilla.com%3A8001%2F&slave=bld-linux64-spot-020&builddir=%2Fbuilds%2Fslave&branch=b2g-inbound&buildername=b2g_b2g-inbound_emulator_dep 07:36:05 INFO - b2g_b2g-in_emu-jb-d_dep-000000:Our last clobber date: None 07:36:05 INFO - b2g_b2g-in_emu-jb-d_dep-000000:Server clobber date: 2014-07-23 07:24:05 07:36:05 INFO - tb-c-cen-l64-l10n-dep-00000000:Our last clobber date: None 07:36:05 INFO - tb-c-cen-l64-l10n-dep-00000000:Server clobber date: 2014-07-23 06:07:03 07:36:05 INFO - /builds/slave/b2g_m-aurora_flame_eng_dep-000 doesn't exist, skipping 07:36:05 INFO - b2g_b2g-in_emu_dep-00000000000:Our last clobber date: None 07:36:05 INFO - b2g_b2g-in_emu_dep-00000000000:Server clobber date: 2014-07-23 07:24:05 07:36:05 INFO - /builds/slave:Our last clobber date: 2014-07-23 04:34:13 07:36:05 INFO - /builds/slave:Server clobber date: 2014-07-23 07:24:05 07:36:05 INFO - /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com 07:36:05 INFO - /builds/slave:Clobbering... 07:36:05 INFO - Removing lost+found.deleteme/ 07:36:05 INFO - Couldn't clobber properly, bailing out. 07:36:05 ERROR - Return code: 1 07:36:05 ERROR - <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x1a6b5d0>> failed after 3 tries! 07:36:05 FATAL - failed to clobber build 07:36:05 FATAL - Running post_fatal callback... 07:36:05 FATAL - Exiting 2 program finished with exit code 2
thus far this has only been seen on b2g platforms. and all "07:34:04 INFO - /builds/slave:Server is forcing a clobber, initiated by cbook@mozilla.com" So it looks like it is limited to b2g builders. clobberer hosts and AWS instances are suspects.
jobs are still occasionally failing. callek pointed out this could be fall out of my fx desktop build patch: http://hg.mozilla.org/build/mozharness/rev/da2e0cf4a096
it looks like they keep clobbering builddirs fine until it gets to: 07:36:05 INFO - Removing lost+found.deleteme/ 07:36:05 INFO - Couldn't clobber properly, bailing out. looking here you can see it is owned by root: [cltbld@bld-linux64-spot-024.build.releng.use1.mozilla.com slave]$ ls -la total 5388 drwx------ 2 root root 16384 Jul 23 2014 lost+found.deleteme
it appears to not just be limited to lost+found.deleteme 07:38:24 INFO - Removing mock_mozilla/ 07:38:24 INFO - Couldn't clobber properly, bailing out. 07:38:24 ERROR - Return code: 1 07:38:24 INFO - retry: Failed, sleeping 60 seconds before retrying 07:39:24 INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #2 07:39:24 INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/'] in /builds/slave 07:39:24 INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-027 http://buildbot-master71.srv.releng.use1.mozilla.com:8001/ 07:39:24 INFO - python: can't open file '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py': [Errno 2] No such file or directory 07:39:24 ERROR - Return code: 2 07:39:24 INFO - retry: Failed, sleeping 120 seconds before retrying 07:41:24 INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> with args: [['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #3 07:41:24 INFO - Running command: ['python', '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'http://clobberer.pvt.build.mozilla.org/index.php', u'b2g-inbound', u'b2g_b2g-inbound_emulator_dep', '/builds/slave', u'bld-linux64-spot-027', u'http://buildbot-master71.srv.releng.use1.mozilla.com:8001/'] in /builds/slave 07:41:24 INFO - Copy/paste: python /builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php b2g-inbound b2g_b2g-inbound_emulator_dep /builds/slave bld-linux64-spot-027 http://buildbot-master71.srv.releng.use1.mozilla.com:8001/ 07:41:24 INFO - python: can't open file '/builds/slave/b2g_b2g-in_emu_dep-00000000000/scripts/external_tools/clobberer.py': [Errno 2] No such file or directory 07:41:24 ERROR - Return code: 2 07:41:24 ERROR - <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0xfaf5d0>> failed after 3 tries! 07:41:24 FATAL - failed to clobber build 07:41:24 FATAL - Running post_fatal callback... 07:41:24 FATAL - Exiting 2 I think there is something going on due to my previous patch: diff --git a/external_tools/clobberer.py b/external_tools/clobberer.py --- a/external_tools/clobberer.py +++ b/external_tools/clobberer.py @@ -1,10 +1,10 @@ -#!/usr/bin/python +#!/usr/bin/env python # vim:sts=2 sw=2 import sys import shutil import urllib2 import urllib import os import traceback import time if os.name == 'nt': diff --git a/mozharness/mozilla/purge.py b/mozharness/mozilla/purge.py --- a/mozharness/mozilla/purge.py +++ b/mozharness/mozilla/purge.py @@ -27,38 +27,42 @@ class PurgeMixin(object): clobber_tool = os.path.join(external_tools_path, 'clobberer.py') default_skips = ['info', 'rel-*', 'tb-rel-*'] default_maxage = 14 default_periodic_clobber = 7 * 24 def purge_builds(self, basedirs=None, min_size=None, skip=None, max_age=None): # Try clobbering first c = self.config + dirs = self.query_abs_dirs() if 'clobberer_url' in c: self.clobberer() min_size = min_size or c['purge_minsize'] max_age = max_age or c.get('purge_maxage') or self.default_maxage skip = skip or c.get('purge_skip') or self.default_skips if not basedirs: # some platforms using this method (like linux) supply more than # one basedir basedirs = [] - assert self.buildbot_config - basedirs.append(os.path.dirname(self.buildbot_config['properties']['basedir'])) + basedirs.append(os.path.dirname(dirs['base_work_dir'])) if self.config.get('purge_basedirs'): basedirs.extend(self.config.get('purge_basedirs')) + try: + python = self.query_python_path() + except AttributeError: + # we are not inheriting from VirtualenvMixin + python = self.query_exe('python') + # Add --dry-run if you don't want to do this for realz - cmd = [self.purge_tool, - '-s', str(min_size), - ] + cmd = [python, self.purge_tool, '-s', str(min_size)] if max_age: cmd.extend(['--max-age', str(max_age)]) for s in skip: cmd.extend(['--not', s]) cmd.extend(basedirs) @@ -68,50 +72,57 @@ class PurgeMixin(object): share_base = c.get('vcs_share_base', os.environ.get("HG_SHARE_BASE_DIR", None)) if share_base: env['HG_SHARE_BASE_DIR'] = share_base retval = self.run_command(cmd, env=env) if retval != 0: self.fatal("failed to purge builds", exit_code=2) def clobberer(self): c = self.config + dirs = self.query_abs_dirs() if not self.buildbot_config: self.fatal("clobberer requires self.buildbot_config (usually from $PROPERTIES_FILE)") periodic_clobber = c.get('periodic_clobber') or self.default_periodic_clobber clobberer_url = c['clobberer_url'] - builddir = os.path.basename(self.buildbot_config['properties']['basedir']) + builddir = os.path.dirname(dirs['base_work_dir']) branch = self.buildbot_config['properties']['branch'] buildername = self.buildbot_config['properties']['buildername'] slave = self.buildbot_config['properties']['slavename'] master = self.buildbot_config['properties']['master'] + + try: + python = self.query_python_path() + except AttributeError: + # we are not inheriting from VirtualenvMixin + python = self.query_exe('python') + # Add --dry-run if you don't want to do this for realz - cmd = [self.clobber_tool] + cmd = [python, self.clobber_tool] # TODO configurable list cmd.extend(['-s', 'scripts']) cmd.extend(['-s', 'logs']) cmd.extend(['-s', 'buildprops.json']) cmd.extend(['-s', 'token']) if periodic_clobber: cmd.extend(['-t', str(periodic_clobber)]) cmd.extend([clobberer_url, branch, buildername, builddir, slave, master]) error_list = [{ 'substr': 'Error contacting server', 'level': ERROR, 'explanation': 'Error contacting server for clobberer information.' }] retval = self.retry(self.run_command, attempts=3, good_statuses=(0,), args=[cmd], - kwargs={'cwd':os.path.dirname(self.buildbot_config['properties']['basedir']), - 'error_list':error_list}) + kwargs={'cwd': builddir, 'error_list': error_list}) if retval != 0: self.fatal("failed to clobber build", exit_code=2) def clobber(self, always_clobber_dirs=None): """ Mozilla clobberer-type clobber. """ c = self.config if c.get('is_automation'): # Nightly builds always clobber you can see that it may affect the interpreter. I am going to back this out
resolved per c#5
Status: NEW → RESOLVED
Closed: 10 years ago
Depends on: 858797
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.