1203104 - Integration Trees closed for backlog

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

06:01 < nagios-releng> Wed 06:01:55 PDT [4228] cruncher.srv.releng.scl3.mozilla.com:Pending builds is CRITICAL: CRITICAL Pending Builds: 8005 (http://m.mozilla.org/Pending+builds)


..integrations trees closed for backlog

Comment 1

[Kim Moir [:kmoir] ET]

Looks like a puppet change caused permission problems with runner that manifested in the golden amis so buildbot couldn't start when new instances were spun up.

No twistd long found, can't find last_build_name
2015-09-09 06:48:24,481 - DEBUG - 0-checkout_tools: starting (max time 600s)
Traceback (most recent call last):
  File "/opt/runner/bin/runner", line 9, in <module>
    load_entry_point('runner==2.0', 'console_scripts', 'runner')()
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 256, in main
    runner(config, args.taskdir, args.times)
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 224, in runner
    if not process_taskdir(config, taskdir):
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 138, in process_taskdir
    r = run_task(task_cmd, env, max_time=task_config['max_time'])
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 23, in run_task
    proc = subprocess.Popen(t, stdin=open(os.devnull, 'r'), env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

Amy has de-registered last night's AMIs, Marko is going to land a fix

Comment 2

[Mark Cornmesser [:markco]]

Attachment: Bug1203104.patch

r+ by Callek in irc as a quick fix.

https://hg.mozilla.org/build/puppet/rev/031561b5703d
https://hg.mozilla.org/build/puppet/rev/dc8119d6ce40

The issues was the variables, particular $mode, were undefined. This a quick fix to get things moving in the correct direction.

Comment 3

[Chris Cooper [:coop] (he/him)]

I just closed try branches too to stop the bleeding.

Comment 4

[Kim Moir [:kmoir] ET]

Amy has started the process to regenerate the golden AMIs now that markco's patch has landed

As an aside, bug 1146341 (upon implementation) should prevent this type of failure in the future.

Comment 5

[Kim Moir [:kmoir] ET]

assigning to markco since he provided the patch and also so we stop getting alerts in #buildduty

Comment 6

[Phil Ringnalda (:philor)]

Trees reopened by around 2pm.