Integration Trees closed for backlog

RESOLVED FIXED

Status

Infrastructure & Operations
Buildduty
--
blocker
RESOLVED FIXED
3 years ago
13 days ago

People

(Reporter: Tomcat, Assigned: markco)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
06:01 < nagios-releng> Wed 06:01:55 PDT [4228] cruncher.srv.releng.scl3.mozilla.com:Pending builds is CRITICAL: CRITICAL Pending Builds: 8005 (http://m.mozilla.org/Pending+builds)


..integrations trees closed for backlog

Comment 1

3 years ago
Looks like a puppet change caused permission problems with runner that manifested in the golden amis so buildbot couldn't start when new instances were spun up.

No twistd long found, can't find last_build_name
2015-09-09 06:48:24,481 - DEBUG - 0-checkout_tools: starting (max time 600s)
Traceback (most recent call last):
  File "/opt/runner/bin/runner", line 9, in <module>
    load_entry_point('runner==2.0', 'console_scripts', 'runner')()
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 256, in main
    runner(config, args.taskdir, args.times)
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 224, in runner
    if not process_taskdir(config, taskdir):
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 138, in process_taskdir
    r = run_task(task_cmd, env, max_time=task_config['max_time'])
  File "/opt/runner/local/lib/python2.7/site-packages/runner/__init__.py", line 23, in run_task
    proc = subprocess.Popen(t, stdin=open(os.devnull, 'r'), env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

Amy has de-registered last night's AMIs, Marko is going to land a fix
(Assignee)

Comment 2

3 years ago
Created attachment 8658755 [details] [diff] [review]
Bug1203104.patch

r+ by Callek in irc as a quick fix.

https://hg.mozilla.org/build/puppet/rev/031561b5703d
https://hg.mozilla.org/build/puppet/rev/dc8119d6ce40

The issues was the variables, particular $mode, were undefined. This a quick fix to get things moving in the correct direction.
Attachment #8658755 - Flags: review+
Attachment #8658755 - Flags: checked-in+

Comment 3

3 years ago
I just closed try branches too to stop the bleeding.

Comment 4

3 years ago
Amy has started the process to regenerate the golden AMIs now that markco's patch has landed

As an aside, bug 1146341 (upon implementation) should prevent this type of failure in the future.

Comment 5

3 years ago
assigning to markco since he provided the patch and also so we stop getting alerts in #buildduty
Assignee: nobody → mcornmesser
Trees reopened by around 2pm.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

13 days ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.