1127365 - Log parser queues not listed after updates/restart for glibc

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Carsten Book [:Tomcat]]

07:54 <&edmorley> processor[1-3] are idle
07:54 <&edmorley> mdoglio: fubar: shall I restart the processors again then?
07:55 < camd> edmorley: probably, yeah.  Dang, those things get disconnected so easily!
07:55 -!- bc|afk is now known as bc
07:55 < mdoglio> the queues are not there
07:55 < mdoglio> apparently
07:56 < Tomcat|sheriffduty> mdoglio: edmorley i guess this means yes there is a problem :) 
07:56 < mdoglio> Tomcat|sheriffduty: indeed
07:56 <&edmorley> Tomcat|sheriffduty: yeah
07:56 < Tomcat|sheriffduty> RyanVM: ^ :)
07:56 < Tomcat|sheriffduty> cool 
07:56 < mdoglio> edmorley: http://pastebin.mozilla.org/8422417
07:57 < Tomcat|sheriffduty> i should rename myself to Tomcat|nagiosbot or so maybe :) 
07:57 < mdoglio> it's not listing those queues at all
07:57 < RyanVM> Tomcat|sheriffduty: darn it, don't make me start paying attention to things
07:57 < Tomcat|sheriffduty> or newrelibot :)
07:58 <&edmorley> Tomcat|sheriffduty: there were system updates applied for glibc (same as all the other mozilla services), and then subsequent restarts, something isn't right after
07:58 <&edmorley> if that gives more context 
07:58 < Tomcat|sheriffduty> edmorley: yeah releng systems got this too
07:58 < Tomcat|sheriffduty> about 30 minutes ago
07:58 < fubar> edmorley: mdoglio: was just looking. noticed the normal log queues were missing
07:59 < fubar> restarting procs across the board didn't help. celery job on log nodes starts up fine and connects to rabbitmq, but then does nothing
07:59 <&edmorley> "how to reduce load in 3 easy steps... 1) delete those darn log parser queues!, ..."

Comment 1

[Carsten Book [:Tomcat]]

closing trees for this problem

Comment 2

[Ryan VanderMeulen [:RyanVM]]

We're catching back up now. Most trees have been opened except for the ones that were backlogged the worst. I'll reopen them as they're cleaned up.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Inbound is having issue ingesting jobs still. Everything else is reopened.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Just reopened inbound at Ed's request.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

comm-* trees were also reopened around 10:38am Pacific.