1203528 - Investigate MDN down-time incident 2015-09-09

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect)

Description

[Luke Crouch [:groovecoder]]

https://rpm.newrelic.com/accounts/263620/applications/3172075/downtime/17280568

http://logs.glob.uno/?c=mozilla%23mdndev#c104071

Comment 1

[Luke Crouch [:groovecoder]]

:cyliang - you mentioned getting the apache core dumps. Can you pastebin them and link here, or maybe attach them here?

Comment 2

[C. Liang [:cyliang]]

The core dumps are on developeradm.private.scl3, in /tmp/cores.  It turns out that they are too big to attach to the bug, even after being compressed.

Comment 3

[Luke Crouch [:groovecoder]]

Dang, I missed them. :(

[[e-mail]3 ~]$ ls -l /tmp/cores/
total 0

In any case, this incident and a couple others were immediately preceded by a spike in $compare transactions. We previously made some changes to mitigate the risks of the $compare transaction overwhelming us [1][2], but Monday I made a more major change to prevent the $compare transaction from executing its CPU-intensive operations at all during an HTTP request. [3] It may just shift the load from the web processes to the celery processes, but at least it will prevent these run-away $compare operations from overwhelming our web node CPUs.

So, I'm going to call this incident investigated and resolved, knowing that we still haven't quite cleaned up everything.

[1] https://github.com/mozilla/kuma/pull/3386
[2] https://github.com/mozilla/kuma/pull/3447
[3] https://github.com/mozilla/kuma/pull/3497