858659 - many jobs not starting (or taking a long time to start) on linux test masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

We have over 300 fedora jobs pending right now, and tons of idle slaves. The jobs aren't starting though.

13:41 <+catlee> but I think bm24 is so overloaded it's taking hours to process RPC 
                calls

Comment 1

[[e-mail] (:bhearsum)]

bm18 seems to be in the worst shape - i don't see it handing out any jobs to r3 machines. It's got a graceful shutdown started, but I'm not sure who did it. I also don't see anything in twistd.log about the graceful shutdown being started...this master is very broken right now.

Comment 2

[[e-mail] (:bhearsum)]

bm18's slaves haven't run a single job since march 28th, i'm restarting it the hard way.

I'm guessing that someone initiated a graceful shutdown on the 28th and something happened, and it didn't shut down. $5 says something to do with ec2 slaves mucked it up.

Comment 3

[[e-mail] (:bhearsum)]

bm18 is back up and fedora pending is down to 73.

14:00 <+catlee> that's the same hung slave issue
14:00 <+catlee> we're only protected against it in a disconnect step
14:00 <+catlee> if the slave dies in other steps, we can still hang