681513 - Anonymous visitor to https://mozillians-dev.allizom.org/en-US/ gets homepage HTML after 6 seconds

Status: VERIFIED FIXED

(Participation Infrastructure :: Phonebook, defect, P1)

Description

[Austin King [:ozten]]

Steps to repro:
1) time curl https://mozillians-dev.allizom.org/en-US/

Expected: 
This should be a couple hundred milliseconds.

My dev server (Apache WSGI in a virtualbox VM) is
real	0m0.011s

Actual:
real	0m5.985s

Comment 1

[Austin King [:ozten]]

Attachment: Screenshot of hotshot profile

I've profiled this case and don't see anything unusual.

Note: no mysql or ldap calls.

Comment 2

[Justin Dow [:jabba]]

Perhaps zeus isn't caching this VIP. Corey, can you take a look? I know these seamicros are pretty weak and slow when we don't enable caching...

Comment 3

[Corey Shields [:cshields]]

Try now.

Comment 4

[Austin King [:ozten]]

Zeus - Cool!
I made 4 requests. 3 looked good (around 300 milliseconds) and 1 took 5+ seconds.

But...

Is 5-6 seconds expected for Playdoh on the VM? Much of our app cannot be cached at the Zues layer due to privacy.

What are the WSGI settings? (processes, threads, max-requests, etc).

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Even a weak machine should be able to serve a page in 200ms uncached - just not a lot of concurrency, which is fine for dev.

ozten, what do you get when you curl for unzeus-cached static assets.

Comment 6

[Corey Shields [:cshields]]

WSGISocketPrefix /var/run/wsgi
<IfModule prefork.c>
    StartServers      200
    MinSpareServers   10
    MaxSpareServers   50
    ServerLimit     260
    MaxClients      260
    MaxRequestsPerChild  25
</IfModule>

don't have any WSGI-specific settings..  Open to suggestions if we should add some.

Comment 7

[Austin King [:ozten]]

(In reply to Corey Shields [:cshields] from comment #6)
I'm not familiar with this configuration. I've only seen:
WSGIDaemonProcess playdoh processes=4 threads=1 maximum-requests=1

If we have a really high number of python processes, I'm not sure how long it would take them to warm up with a low number of users on the system.

The timing for /en-US/ is in the range of the first request to `python manage.py runserver 0.0.0.0:8001`. Subsequent calls would then drop an order of magnitude.

Another possibility is python is getting restarted after every request, which would have the same effect - cold server.

Troubleshooting the static assets first is a good strategy, adding these notes in case we end up looking at WSGI.

Comment 8

[Corey Shields [:cshields]]

We have no WSGIDaemonProcess settings for mozillians..

Comment 9

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

What's the status on this?  I'm still noticing slowness.  I won't have time myself to look into good WSGI Settings.

Comment 10

[Corey Shields [:cshields]]

We have been fighting a lot of fires this week, have had no time to look further, sorry.

Comment 11

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

I get a lot of sub 200ms requests, I'm considering this fixed.

Thanks cshields and ozten.

Comment 12

[Tobias (:Tobbi) Markus]

Verified fixed.