1296320 - treeherder-rabbitmq log parser queue backlog due to slow transit

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Justin Lazaro [:jlaz] (use needinfo)]

Nagios alert:

07:46:36 <@nagios-scl3> (IRC) Thu 07:46:36 PDT [5137] treeherder-rabbitmq2.private.scl3.mozilla.com:Rabbit Unread Messages is WARNING: RABBITMQ_OVERVIEW WARNING - messages WARNING(2839) messages_ready WARNING (2721), messages_unacknowledged OK (118) (http://m.mozilla.org/Rabbit+Unread+Messages)

It looks like the log_parser queue is backlogged, but it was mentioned that this is caused by archive.m.o's CDN being routed via Asia.  Filing a bug for tracking this issue.

Comment 2

[Dave Williams [:daveio]]

Happened again, logging for tracking

10:56 <@nagios-scl3> (IRC) Wed 02:56:19 PDT [5070] treeherder-rabbitmq2.private.scl3.mozilla.com:Rabbit Unread Messages is CRITICAL: RABBITMQ_OVERVIEW CRITICAL - messages CRITICAL (20370) messages_ready CRITICAL (20308), messages_unacknowledged OK (62) (http://m.mozilla.org/Rabbit+Unread+Messages)

Comment 4

[Ed Morley [:emorley]]

See:
https://rpm.newrelic.com/accounts/677903/applications/4180461/externals

In this latest case is affecting requests to queue.taskcluster.net too - which is served by CloudFront.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

and there are still problems today- jobs finished and 50 minutes later, no perf data is being ingested.

Comment 6

[Ed Morley [:emorley]]

Fixed by moving to Heroku.