Log parser queues not listed after updates/restart for glibc

RESOLVED FIXED

Status

P1
blocker
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: cbook, Assigned: fubar)

Tracking

Details

(Reporter)

Description

4 years ago
07:54 <&edmorley> processor[1-3] are idle
07:54 <&edmorley> mdoglio: fubar: shall I restart the processors again then?
07:55 < camd> edmorley: probably, yeah.  Dang, those things get disconnected so easily!
07:55 -!- bc|afk is now known as bc
07:55 < mdoglio> the queues are not there
07:55 < mdoglio> apparently
07:56 < Tomcat|sheriffduty> mdoglio: edmorley i guess this means yes there is a problem :) 
07:56 < mdoglio> Tomcat|sheriffduty: indeed
07:56 <&edmorley> Tomcat|sheriffduty: yeah
07:56 < Tomcat|sheriffduty> RyanVM: ^ :)
07:56 < Tomcat|sheriffduty> cool 
07:56 < mdoglio> edmorley: http://pastebin.mozilla.org/8422417
07:57 < Tomcat|sheriffduty> i should rename myself to Tomcat|nagiosbot or so maybe :) 
07:57 < mdoglio> it's not listing those queues at all
07:57 < RyanVM> Tomcat|sheriffduty: darn it, don't make me start paying attention to things
07:57 < Tomcat|sheriffduty> or newrelibot :)
07:58 <&edmorley> Tomcat|sheriffduty: there were system updates applied for glibc (same as all the other mozilla services), and then subsequent restarts, something isn't right after
07:58 <&edmorley> if that gives more context 
07:58 < Tomcat|sheriffduty> edmorley: yeah releng systems got this too
07:58 < Tomcat|sheriffduty> about 30 minutes ago
07:58 < fubar> edmorley: mdoglio: was just looking. noticed the normal log queues were missing
07:59 < fubar> restarting procs across the board didn't help. celery job on log nodes starts up fine and connects to rabbitmq, but then does nothing
07:59 <&edmorley> "how to reduce load in 3 easy steps... 1) delete those darn log parser queues!, ..."
(Reporter)

Comment 1

4 years ago
closing trees for this problem
Component: Treeherder → Treeherder: Infrastructure
Priority: -- → P1
QA Contact: laura
Hardware: x86 → All
Summary: Treeherder Log Parsing broken → Log parser queues not listed after updates/restart for glibc
We're catching back up now. Most trees have been opened except for the ones that were backlogged the worst. I'll reopen them as they're cleaned up.
Inbound is having issue ingesting jobs still. Everything else is reopened.
Just reopened inbound at Ed's request.
Assignee: nobody → klibby
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
comm-* trees were also reopened around 10:38am Pacific.
You need to log in before you can comment on or make changes to this bug.