Closed
Bug 1203104
Opened 9 years ago
Closed 9 years ago
Integration Trees closed for backlog
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Infrastructure & Operations Graveyard
CIDuty
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: cbook, Assigned: markco)
Details
Attachments
(1 file)
1.24 KB,
patch
|
markco
:
review+
markco
:
checked-in+
|
Details | Diff | Splinter Review |
06:01 < nagios-releng> Wed 06:01:55 PDT [4228] cruncher.srv.releng.scl3.mozilla.com:Pending builds is CRITICAL: CRITICAL Pending Builds: 8005 (http://m.mozilla.org/Pending+builds) ..integrations trees closed for backlog
Comment 1•9 years ago
|
||
Looks like a puppet change caused permission problems with runner that manifested in the golden amis so buildbot couldn't start when new instances were spun up. No twistd long found, can't find last_build_name 2015-09-09 06:48:24,481 - DEBUG - 0-checkout_tools: starting (max time 600s) Traceback (most recent call last): File "/opt/runner/bin/runner", line 9, in <module> load_entry_point('runner==2.0', 'console_scripts', 'runner')() File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 256, in main runner(config, args.taskdir, args.times) File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 224, in runner if not process_taskdir(config, taskdir): File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 138, in process_taskdir r = run_task(task_cmd, env, max_time=task_config['max_time']) File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 23, in run_task proc = subprocess.Popen(t, stdin=open(os.devnull, 'r'), env=env) File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child raise child_exception OSError: [Errno 13] Permission denied Amy has de-registered last night's AMIs, Marko is going to land a fix
Assignee | ||
Comment 2•9 years ago
|
||
r+ by Callek in irc as a quick fix. https://hg.mozilla.org/build/puppet/rev/031561b5703d https://hg.mozilla.org/build/puppet/rev/dc8119d6ce40 The issues was the variables, particular $mode, were undefined. This a quick fix to get things moving in the correct direction.
Attachment #8658755 -
Flags: review+
Attachment #8658755 -
Flags: checked-in+
Comment 3•9 years ago
|
||
I just closed try branches too to stop the bleeding.
Comment 4•9 years ago
|
||
Amy has started the process to regenerate the golden AMIs now that markco's patch has landed As an aside, bug 1146341 (upon implementation) should prevent this type of failure in the future.
Comment 5•9 years ago
|
||
assigning to markco since he provided the patch and also so we stop getting alerts in #buildduty
Assignee: nobody → mcornmesser
Comment 6•9 years ago
|
||
Trees reopened by around 2pm.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•