Closed Bug 1287894 Opened 9 years ago Closed 9 years ago

relengapi under heavy load (Max Queue Times is CRITICAL)

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1285454

People

(Reporter: garbas, Unassigned)

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/3219])

here is last 30min from the #sysadmins channels. could we tell what is causing this heavy load? ``` 20:23:48 │ (IRC) Tue 11:23:48 PDT [5422] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is OK: OK: All queue times less than 1000 ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:33:47 │ (IRC) Tue 11:33:47 PDT [5425] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is CRITICAL: CRITICAL: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=2788.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:34:47 │ (IRC) Tue 11:34:47 PDT [5426] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is OK: OK: All queue times less than 1000 ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:48:48 │ (IRC) Tue 11:48:48 PDT [5431] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is CRITICAL: CRITICAL: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=4146.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:49:48 │ (IRC) Tue 11:49:48 PDT [5432] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is WARNING: WARNING: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=1350.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:51:08 │ (IRC) Tue 11:51:08 PDT [5433] wde1.corpdmz.scl3.mozilla.com:Puppet freshness is WARNING: Last run had 2 errors (http://m.mozilla.org/Puppet+freshness) 20:53:48 │ (IRC) Tue 11:53:48 PDT [5434] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is WARNING: WARNING: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=1036.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:54:49 │ (IRC) Tue 11:54:48 PDT [5435] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is OK: OK: All queue times less than 1000 ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 20:58:29 │ Tue 11:58:29 PDT [5438] fuzzer-linux1.sec.scl3.mozilla.com:HP Health is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds. (http://m.mozilla.org/HP+Health) 21:01:48 │ (IRC) Tue 12:01:48 PDT [5440] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is WARNING: WARNING: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=1399.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) 21:02:58 │ (IRC) Tue 12:02:58 PDT [5441] treeherder-rabbitmq2.private.scl3.mozilla.com:Rabbit Unread Messages is CRITICAL: RABBITMQ_OVERVIEW CRITICAL - messages CRITICAL (6791) messages_ready CRITICAL (6693), messages_unacknowledged OK (98) (http://m.mozilla.or/Rabbit+Unread+Messages) 21:05:48 │ (IRC) Tue 12:05:48 PDT [5445] nagios2.private.scl3.mozilla.com:Zeus Max Queue Times is WARNING: WARNING: zeus.zlb1_ops_scl3_mozilla_com.pools.releng-http.max_queue_time=1352.0ms (http://m.mozilla.org/Zeus+Max+Queue+Times) ```
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/3219]
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.