720138 - What's wrong with WinXP?

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Phil Ringnalda (:philor)]

Rhetorical summary, but something's gone wrong with the testpool. https://build.mozilla.org/buildapi/running with winnt 5.1 in the search box shows 35 running jobs, but sorted by running-since, 10 have been "running" for  21+ hours, and 7 have been running for 7+ hours, so only 18 can really be counted as working, which is apparently not enough - there's 325 pending jobs on try, going back over 15 hours.

Comment 1

[Phil Ringnalda (:philor)]

http://build.mozilla.org/builds/last-job-per-slave.html#test says that around 23 hours ago, so maybe the Friday reconfig, we lost 27 WinXP slaves, then around 8 hours ago, so probably the bug 720118 outage, we lost another 8.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

As you already discussed on IRC, the running builds list has about 20 builds which actually finished but the db wasn't updated. I think we can let that sit for another day.

For the many slaves which haven't done a job in ages it comes down to OPSI being teh suck. Fix up strategy
 * connect to slave with VNC
 * expect to find the windows logon screen saver
 * click in the VNC window, some machines will log on at this point and need no further help
 * if the pcptch dialog is visible then OPSI is stuck, so
   * connect using ssh
   * run 'taskkill /IM pcptch.exe /F'
   * when you see the VNC session start to logon and start the cmd window, do 'shutdown -r -t 0 -f' to ensure a clean state

I've gotten up to talos-r3-xp-028 so far, working upwards.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I got up to talos-r3-xp-060, just touching the slaves that hadn't done work recently. The non-try backlog has already cleared and hopefully there are now enough to keep up for 24 hours while clearing the try backlog. 

I'll check again tomorrow but for now real life (what's that ??!?) asks me to be elsewhere.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I finished up the XP slaves. Still need to do the DB fixes.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Buildbot mostly cleaned up the dead jobs in the db, but I took the opportunity to clean up 4 try jobs from 4 days ago. Bug 721290 for the three pending jobs.