1110754 - plugins1/2/3.webapps are under high load.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ludovic Hirlimann [:Usul]]

I've restarted httpd/ and rebooted the hosts. but it seems new relic processe are high on cpu with these.

Started at Fri 05:04:32 PST [1493] plugins1.webapp.phx1.mozilla.com:Load is CRITICAL: CRITICAL - load average: 55.30, 32.44, 21.96 (http://m.mozilla.org/Load)

I've pinged opsec because I was thinking we where being DOSed :
kang: top requester is newrelic at 50.31.164.230
kang: acually "10.8.81.115\|10.8.81.114\|10.8.81.113" for all 3 hosts

Comment 1

[Ludovic Hirlimann [:Usul]]

paged gozer at :26:

Comment 2

[Philippe M. Chiasson (:gozer)]

(In reply to Ludovic Hirlimann [:Usul] from comment #0)
> [...]
> I've pinged opsec because I was thinking we where being DOSed :
> kang: top requester is newrelic at 50.31.164.230
> kang: acually "10.8.81.115\|10.8.81.114\|10.8.81.113" for all 3 hosts

Doesn't make that much sense, since 10.8.81.113-115 are the plugins[1-3].webapp.phx1.mozilla.com..

Comment 3

[Philippe M. Chiasson (:gozer)]

From the ZLBs point of view, traffic is at what appears to be somewhat usual levels.

However, looking at the traffic sent to backend nodes, it is nearly triple what it should normally be.

I suspect something isn't caching in Zeus as it should be.

Comment 4

[Philippe M. Chiasson (:gozer)]

Scratch my previous comment, inbound traffic is massively more than usual

Comment 5

[Philippe M. Chiasson (:gozer)]

Attachment: 7-day traffic history for that virtual server

Comment 6

[Philippe M. Chiasson (:gozer)]

Attachment: 7-day traffic history for the backend pool

Comment 7

[Guillaume Destuynder [:kang] [this account is no longer active]]

paste from irc of comment 0 is out of context, the 3 ips where for the grep command of the netflow logs (ie all 3 plugin hosts) i.e.:

ssh syslog1.private.phx1.mozilla.com sudo tail -f /var/log/syslog/network/2014-12-12.log | grep "10.8.81.115\|10.8.81.114\|10.8.81.113"

Not the offenders :)

Comment 8

[Ludovic Hirlimann [:Usul]]

(In reply to Guillaume Destuynder [:kang] from comment #7)
> paste from irc of comment 0 is out of context, the 3 ips where for the grep
> command of the netflow logs (ie all 3 plugin hosts) i.e.:
> 
> ssh syslog1.private.phx1.mozilla.com sudo tail -f
> /var/log/syslog/network/2014-12-12.log | grep
> "10.8.81.115\|10.8.81.114\|10.8.81.113"
> 
> Not the offenders :)

In the middle of 3 alerts. My bad.

Comment 9

[Philippe M. Chiasson (:gozer)]

Storm has passed.