709171 - https://affiliates.allizom.org/en-US/ sometimes takes 4s or more

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Anthony Ricaud (:rik)]

I don't know the frequency at which it happens but maybe every 3 or 4 requests.

$ time curl -vI https://affiliates.allizom.org/fr/
real	0m7.730s
and a normal one:
real	0m0.629s

Looking at the output, it seems the SSL handshake is happening quickly.

Maybe it is the Python process initialisation that we are hitting too often.

Comment 1

[James Socol [:jsocol, :james]]

On SUMO's staging servers we were seeing a lot of timeouts in automated tests, so we added a deploy task to prime the WSGI processes by hitting the local Apache with a bunch of parallel curl requests. We determined that WSGI starts up in a round-robin, on-demand fashion, and is not quick, especially on the SeaMicro boxes.

Comment 2

[Chris More [:cmore]]

(In reply to James Socol [:jsocol, :james] from comment #1)
> On SUMO's staging servers we were seeing a lot of timeouts in automated
> tests, so we added a deploy task to prime the WSGI processes by hitting the
> local Apache with a bunch of parallel curl requests. We determined that WSGI
> starts up in a round-robin, on-demand fashion, and is not quick, especially
> on the SeaMicro boxes.

Yeah, the WSGI daemon spin up on the SeaMicro boxes seems to take forever. We were hoping that after we had enough traffic to the site that the daemons would keep spun up, but we think sometimes they go to sleep and we hit a node that is not awake. Dev and stage being slow with daemon spinning up is fairly normal given that they are not constantly used.

Comment 3

[James Socol [:jsocol, :james]]

What are the WSGIDaemonProcess settings? They shouldn't go to sleep or die unless inactivity-timeout is set, and you almost certainly don't want that. maximum-requests could also be too low.

Comment 4

[Jason Thomas [:jason]]

I made a modification to the update script on the web heads on engagement-stage:

* removed apache graceful restart - we touch the wsgi entry point on code push for stage
* curl requests to affiliates.allizom.org as mentioned in comment 1

This should help with the slowness on -stage.