1235400 - genericX.webapp.phx1.mozilla.com apps down

Status: RESOLVED DUPLICATE of bug 1234772

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Marlena]

8:33 AM <nagios-phx1> (IRC) Mon 08:33:51 PST [1017] generic4.webapp.phx1.mozilla.com:Out of memory - killed process is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds. (http://m.mozilla.org/Out+of+memory+-+killed+process)

8:33 AM <nagios-phx1> Mon 08:33:51 PST [1018] moztrap.mozilla.org:HTTP - / is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 630 bytes in 0.006 second response time (http://m.mozilla.org/HTTP+-+/)

8:34 AM <nagios-phx1> (IRC) Mon 08:34:11 PST [1019] generic4.weba

Comment 1

[Marlena]

8:34 AM <nagios-phx1> (IRC) Mon 08:34:11 PST [1019] generic4.webapp.phx1.mozilla.com:Uptime is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. (http://m.mozilla.org/Uptime)
8:34 AM <nagios-phx1> (IRC) Mon 08:34:21 PST [1020] generic5.webapp.phx1.mozilla.com:Out of memory - killed process is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds. (http://m.mozilla.org/Out+of+memory+-+killed+process)
8:34 AM <nagios-phx1> Mon 08:34:41 PST [1021] generic2.webapp.phx1.mozilla.com:PHX Zeus - Port 81 is CRITICAL: generic2.webapp.phx1.mozilla.com down(2) (http://m.mozilla.org/PHX+Zeus+-+Port+81)
8:35 AM <nagios-phx1> (IRC) Mon 08:35:11 PST [1022] generic2.webapp.phx1.mozilla.com:Uptime is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. (http://m.mozilla.org/Uptime)

Comment 2

[C. Liang [:cyliang]]

We found a site that was aggressively *not* caching a sizable video asset that appears on its home page.  There is currently a workaround in place to force caching until the site owner addresses the issue.  We're going to go into a holding pattern for several days to see if the issue crops up again.

Comment 3

[C. Liang [:cyliang]]

Issue reoccurred.  Folding this bug into 1234772