Created attachment 561723 [details] problematic pending jobs After the downtime yesterday we triggered a bunch of jobs via the Rebuild feature of self-serve. Most of them completed successfully, but it appears that some of them have somehow gotten stuck as "pending". Attached is a dump of the problematic ones from schedulerdb. I'm not 100% sure if the Tegras are part of the same issue.
I bet the "claimed_at" column being 0 doesn't help here. Here's a more recently pending job, which has claimed_at being NULL: mysql> select id, buildsetid, buildername, priority, claimed_at, claimed_by_name, claimed_by_incarnation, complete, results, from_unixtime(submitted_at), complete_at from buildrequests where complete=0 and submitted_at order by submitted_at desc limit 1000; | 5635183 | 2698252 | Linux try leak test build | 0 | 0 | NULL | NULL | 0 | NULL | 2011-09-22 08:18:11 | NULL | Since these aren't causing any harm, I'm not going to do anything with them right now, because I don't fully understand what's going on. Catlee might be able to shed some light when he's back from PTO.
Looks like most (all?) of these are due to the talos suite changes in bug 661918. I've deleted the talos pending builds, leaving us with some stray misc. pending builds. I think there's a bug somewhere else about automatically cleaning these up, but I can't find it right now.