1343831 - [Alert] Cloudamqp - Queue total messages alarm: treeherder-prod log_parser 2017-03-02

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

Description

[Ed Morley [:emorley]]

CloudAMQP just alerted about a log parser backlog.

Checking New Relic shows a spike in transaction time:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1488455025&tw%5Bstart%5D=1488451425#id=5b224f746865725472616e73616374696f6e2f43656c6572792f6c6f672d706172736572222c22225d

...with some logs taking upto 200s to parse. eg:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1488455025&tw%5Bstart%5D=1488451425#id=5b224f746865725472616e73616374696f6e2f43656c6572792f6c6f672d706172736572222c22225d&app_trace_id=3a76d3f5-ff3b-11e6-a5be-f8bc124256a0_29280_30847

For that example, the log URL is:
https://queue.taskcluster.net/v1/task/MUxLsC8LQDWI6A6HkTBkzw/runs/0/artifacts/public/logs/live_backing.log

...and the the log is 316MB.

Comment 1

[Ed Morley [:emorley]]

Things that would helps:
1) Bug 1295997 - Add limit to size of log we will parse
2) Bug 1294544 - Treeherder should limit the task time for log parsing so we don't get a backlog on long logs