06:01 < nagios-releng> Wed 06:01:55 PDT  cruncher.srv.releng.scl3.mozilla.com:Pending builds is CRITICAL: CRITICAL Pending Builds: 8005 (http://m.mozilla.org/Pending+builds) ..integrations trees closed for backlog
Looks like a puppet change caused permission problems with runner that manifested in the golden amis so buildbot couldn't start when new instances were spun up. No twistd long found, can't find last_build_name 2015-09-09 06:48:24,481 - DEBUG - 0-checkout_tools: starting (max time 600s) Traceback (most recent call last): File "/opt/runner/bin/runner", line 9, in <module> load_entry_point('runner==2.0', 'console_scripts', 'runner')() File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 256, in main runner(config, args.taskdir, args.times) File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 224, in runner if not process_taskdir(config, taskdir): File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 138, in process_taskdir r = run_task(task_cmd, env, max_time=task_config['max_time']) File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 23, in run_task proc = subprocess.Popen(t, stdin=open(os.devnull, 'r'), env=env) File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child raise child_exception OSError: [Errno 13] Permission denied Amy has de-registered last night's AMIs, Marko is going to land a fix
Created attachment 8658755 [details] [diff] [review] Bug1203104.patch r+ by Callek in irc as a quick fix. https://hg.mozilla.org/build/puppet/rev/031561b5703d https://hg.mozilla.org/build/puppet/rev/dc8119d6ce40 The issues was the variables, particular $mode, were undefined. This a quick fix to get things moving in the correct direction.
I just closed try branches too to stop the bleeding.
Amy has started the process to regenerate the golden AMIs now that markco's patch has landed As an aside, bug 1146341 (upon implementation) should prevent this type of failure in the future.
assigning to markco since he provided the patch and also so we stop getting alerts in #buildduty
Assignee: nobody → mcornmesser
Trees reopened by around 2pm.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.