1270463 - Stage rabbitmq has 3.8 million queued messages

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

https://rpm.newrelic.com/accounts/677903/dashboard/6293252/page/4?tw%5Bend%5D=1462450695&tw%5Bstart%5D=1461845895

Looks like one of the event queues is stuck.

However I can't clear it using the rabbitmq admin panel, since the password I have doesn't work (guess it was rotated recently).

Kendall, could you PM it to me? :-)


[[e-mail]3 ~]$ sudo rabbitmqctl -p treeherder list_queues
Listing queues ...
autoclassify    0
bug_suggestions 0
buildapi        0
buildapi_4hr    0
buildapi_pending        0
buildapi_running        0
calculate_durations     0
calculate_eta   0
[e-mail]    0
[e-mail]    0
[e-mail]        0
[e-mail]     0
celery@log_parser.treeherder-processor1.stage.private.scl3.mozilla.com.celery.pidbox    0
celery@log_parser.treeherder-processor2.stage.private.scl3.mozilla.com.celery.pidbox    0
celery@log_parser.treeherder-processor3.stage.private.scl3.mozilla.com.celery.pidbox    0
[e-mail]     0
[e-mail]     0
celeryev.36e1bc03-30a7-41c2-80ab-8c5dc78f29b8   0
celeryev.4f11c515-377d-4588-9167-43bbb6f4053c   0
celeryev.78ee1285-9505-4fe1-8cd2-8177e44583b7   0
celeryev.7ced2084-e947-4cca-906c-2d058998458f   3850064
celeryev.7fdcbba8-665e-434a-96dc-91817104320a   0
celeryev.85a6ed9b-fdab-4746-b695-49c7cbaa7f0e   0
celeryev.c666cdde-c724-44f9-9c8a-e8135c3a4eaf   0
celeryev.f40fe3e2-18c6-459c-87da-a3911a0bcc5c   0
celeryev.f95de67c-a428-4e21-8fe4-e9e0fe69c286   0
celeryev.fd9755e0-8ccf-4547-bcf7-d33f7e593359   0
classification_mirroring        0
cycle_data      0
default 0
detect_intermittents    0
error_summary   0
fetch_allthethings      0
fetch_bugs      0
fetch_missing_push_logs 0
generate_perf_alerts    0
high_priority   0
log_autoclassify        40
log_autoclassify_fail   0
log_autoclassify_hp     0
log_crossreference_error_lines  3
log_crossreference_error_lines_fail     60
log_crossreference_error_lines_hp       0
log_parser      181
log_parser_fail 4
log_parser_hp   0
log_parser_json 0
log_store_failure_lines 137
log_store_failure_lines_fail    0
log_store_failure_lines_hp      0
parse_job_logs  0
parse_job_logs_fail     0
parse_job_logs_hp       0
populate_performance_series     0
process_objects 0
publish_to_pulse        0
pushlog 0
store_error_summary     0
...done.

Comment 1

[Ed Morley [:emorley]]

I've killed some stuck processes, eg:

[[e-mail]3 ~]$ multi treeherder-stage "ps ax -o ppid,pid,stime,command | egrep '^\s* 1\s+.*[p]ython'"
...
[treeherder-processor1.stage.private.scl3.mozilla.com] out: 1  2755 Apr25 python2.7 /data/www/treeherder.allizom.org/venv/bin/celery -A treeherder worker -Q parse_job_logs,parse_job_logs_fail,parse_job_logs_hp,log_parser,log_parser_fail,log_parser_hp,log_store_failure_lines,log_store_failure_lines_fail,log_store_failure_lines_hp,log_crossreference_error_lines,log_crossreference_error_lines_fail,log_crossreference_error_lines_hp,log_autoclassify,log_autoclassify_fail,log_autoclassify_hp,error_summary --concurrency=5 --logfile=/var/log/celery/celery_worker_log_parser.log -l INFO --maxtasksperchild=500 -n log_parser.%h

And the stuck queue has been removed.

It would still be useful to have the new rabbitmq admin password though :-)

Comment 2

[Kendall Libby [:fubar] (he/him)]

sent!