715384 - support-dev.allizom.org is slow, has connection resets, etc.

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

http://support-dev.allizom.org seems fraught with timeouts, slow loads, and connection resets, as our automation (http://qa-selenium.mv.mozilla.com:8080/job/sumo.fft.saucelabs/501/console) and the following curl output attest:

host-2-173:~ moco$ curl -i http://support-dev.allizom.org
curl: (52) Empty reply from server

It should be, of course, returning this:

host-2-173:~ moco$ curl -i http://support-dev.allizom.org
HTTP/1.1 302 FOUND
Server: Apache
Vary: Accept-Language,X-Mobile,User-Agent
X-Backend-Server: node252
Content-Type: text/html; charset=utf-8
Date: Thu, 05 Jan 2012 00:31:59 GMT
Location: http://support-dev.allizom.org/en-US/
x-frame-options: DENY
Content-Length: 0

Comment 1

[Jason Thomas [:jason]]

How often does automation run on -dev?

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jason Thomas [:jason] from comment #1)
> How often does automation run on -dev?

It's set up to run on changes to http://support-dev.allizom.org/media/revision.txt, so, I think, we're typically seeing problems when there are a lot of checkins -- perhaps this has to do with process restarts (or do we not restart Django/Apache?)

Comment 3

[Jason Thomas [:jason]]

Yes we do restart wsgi daemon on deploy by touching the wsgi entry point. After some additional testing I feel the prime_app would work better if there was a sleep between curl requests.

Created the following pull request with modification https://github.com/mozilla/kitsune/pull/425

Comment 4

[Jason Thomas [:jason]]

Going to move -dev and -stage off of seamicro hardware onto VM's. Bug 717597 for VM creation.

Comment 5

[Jason Thomas [:jason]]

-dev and -stage are now hosted on VM's instead of seamicro. This should help to reduce the performance issues we have been seeing in the past.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Going to verify this, thanks, Jason; it seems a ton better, and we can always file new, specific bugs as we come across them.