1124662 - MDN downtime (2015-01-22)

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[John Karahalis [:openjck]]

MDN is currently down. We're discussing in #mdndev.

Comment 1

[Ryan Watson [:w0ts0n]]

Just before this bug was filed I was getting alerts:

nagios-scl3	Thu 06:21:21 PST [5287] developer3.webapp.scl3.mozilla.com:httpd max clients is WARNING: Using 512 out of 512 Clients

Once confirmed down, I graceful'd httpd which bought it back up and it hasn't spiked back up since.. 

Leaving this with webops for investigation.

Comment 2

[C. Liang [:cyliang]]

Attachment: mdn_historical.png

Definite spike in traffic seen on the Zeus end.

Comment 3

[C. Liang [:cyliang]]

Typically, the top visitor is usually the firewall (usually double the traffic of anything else).

This morning, the top visitors were 145.253.116.106 and 222.148.56.200.

145.253.116.106 looks like it was mostly grabbing one URL:
   4 "GET /en-US/jsi18n/build:db3598d HTTP/1.1"
  14 " " -
18139 "GET /en-US/docs/Web/API/Simple_Push_API HTTP/1.1"

Traffic from 222.148.56.200 looks like:
   2 "GET /favicon.ico HTTP/1.1"
 966 "GET /en-US/docs/tag/ECMAScript6 HTTP/1.1"
17326 " " -

Comment 4

[C. Liang [:cyliang]]

As a point of comparison, requests for "GET /en-US/docs/Web/API/Simple_Push_API HTTP/1.1", sorted by IP:
    2 143.164.102.13
    2 222.117.185.70
18139 145.253.116.106

Those asking for " ":
 3028 91.103.41.1
 8862 115.114.191.92
17743 63.245.214.162
18168 145.253.116.106
18296 222.148.56.200

Comment 5

[C. Liang [:cyliang]]

Closing this as the incident has passed.