07:54 <&edmorley> processor[1-3] are idle 07:54 <&edmorley> mdoglio: fubar: shall I restart the processors again then? 07:55 < camd> edmorley: probably, yeah. Dang, those things get disconnected so easily! 07:55 -!- bc|afk is now known as bc 07:55 < mdoglio> the queues are not there 07:55 < mdoglio> apparently 07:56 < Tomcat|sheriffduty> mdoglio: edmorley i guess this means yes there is a problem :) 07:56 < mdoglio> Tomcat|sheriffduty: indeed 07:56 <&edmorley> Tomcat|sheriffduty: yeah 07:56 < Tomcat|sheriffduty> RyanVM: ^ :) 07:56 < Tomcat|sheriffduty> cool 07:56 < mdoglio> edmorley: http://pastebin.mozilla.org/8422417 07:57 < Tomcat|sheriffduty> i should rename myself to Tomcat|nagiosbot or so maybe :) 07:57 < mdoglio> it's not listing those queues at all 07:57 < RyanVM> Tomcat|sheriffduty: darn it, don't make me start paying attention to things 07:57 < Tomcat|sheriffduty> or newrelibot :) 07:58 <&edmorley> Tomcat|sheriffduty: there were system updates applied for glibc (same as all the other mozilla services), and then subsequent restarts, something isn't right after 07:58 <&edmorley> if that gives more context 07:58 < Tomcat|sheriffduty> edmorley: yeah releng systems got this too 07:58 < Tomcat|sheriffduty> about 30 minutes ago 07:58 < fubar> edmorley: mdoglio: was just looking. noticed the normal log queues were missing 07:59 < fubar> restarting procs across the board didn't help. celery job on log nodes starts up fine and connects to rabbitmq, but then does nothing 07:59 <&edmorley> "how to reduce load in 3 easy steps... 1) delete those darn log parser queues!, ..."
closing trees for this problem
Component: Treeherder → Treeherder: Infrastructure
Priority: -- → P1
QA Contact: laura
Hardware: x86 → All
Summary: Treeherder Log Parsing broken → Log parser queues not listed after updates/restart for glibc
We're catching back up now. Most trees have been opened except for the ones that were backlogged the worst. I'll reopen them as they're cleaned up.
Inbound is having issue ingesting jobs still. Everything else is reopened.
Just reopened inbound at Ed's request.
Assignee: nobody → klibby
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
comm-* trees were also reopened around 10:38am Pacific.
You need to log in before you can comment on or make changes to this bug.