527229 - win32 build slaves not running buildbot

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Getting tons of this in IRC:
11:10 < nagios> [13] moz2-win32-slave15.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)
11:10 < nagios> [14] moz2-win32-slave22.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)
11:14 < nagios> [21] moz2-win32-slave26.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)
11:15 < nagios> [22] moz2-win32-slave35.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)
11:15 < nagios> [23] moz2-win32-slave36.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)
11:17 < nagios> [24] moz2-win32-slave08.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)                                  11:17 < nagios> [25] moz2-win32-slave02.build:buildbot is CRITICAL: CRITICAL:
                python.exe: stopped (critical)

Comment 1

[[e-mail] (:bhearsum)]

XP talos slaves appear to be affected, too.

load has spiked a lot on production-opsi. Looks like the same thing we had last time we had a spike like this - slaves are stuck at the screensaver. Once the mouse is moved, they login and get back to normal. I'll start kicking the slaves.

Comment 2

[[e-mail] (:bhearsum)]

Oh, I restarted the opsiconfd service first, too, to stop OPSI from eating the memory.

Comment 3

[[e-mail] (:bhearsum)]

Alright, all of the win32 and Talos slaves that were down are now back up. The root cause of this is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=522078