921480 - [sumo][prod] Site slow / Lots of time spent "request queueing" lately

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task, P3)

Description

[Ricky Rosario [:rrosario, :r1cky]]

We are getting complaints of su.m.o being slow on and off quite frequently lately. I took a look at the chart in new relic and apparently we are spending a lot of time Request Queueing at certain times:
http://cl.ly/image/0O0Q1c1B3W45

https://rpm.newrelic.com/accounts/263620/applications/2779374

Has something changed in our apache/wsgi config? Is it time to make changes?

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

Can somebody take a look now that summit and no-change period is over?

It is pretty bad right now: http://cl.ly/image/2n000K1N2342

Thanks!

Comment 2

[Sheeri Cabral [:sheeri]]

Attachment: ksar graphing of sar for 9 days on support3, memory and other associated graphs

early on 10/6 there seems to be a drop in memory misc buffers.... This is from sar, so I'm not sure exactly what that represents. Buffers available? buffers used?

Comment 3

[Ricky Rosario [:rrosario, :r1cky]]

As discussed in IRC, can we try adding this to settings_local.py for prod only:

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
)

This is just to see if it makes a difference, the real fix would be somewhere else.

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

(In reply to Ricky Rosario [:rrosario, :r1cky] from comment #3)
> AUTHENTICATION_BACKENDS = (
>     'django.contrib.auth.backends.ModelBackend',
> )

BTW, this just removes the persona auth backend. Persona isn't enabled on prod so there should be no affect except hopefully no more calls to persona.verify() service.

Comment 5

[Daniel Maher [:phrawzty]]

(In reply to Ricky Rosario [:rrosario, :r1cky] from comment #3)
> AUTHENTICATION_BACKENDS = (
>     'django.contrib.auth.backends.ModelBackend',
> )

The change has been made as requested, and pushed to the webheads, which will pick up the new configuration shortly (httpd graceful).

Comment 6

[Ricky Rosario [:rrosario, :r1cky]]

So far this looks super promising! We'll keep monitoring. If it stays this way we can close this out.

I've filed bug 924975 to get us on a newer and less buggy django-browserid. Once that happens we can undo the change in settings_local.py.

Thanks :phrawzty!!

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

Site is whistling now :). No request queue since Comment 5 - http://cl.ly/image/0M3e01183B28

Thanks for helping us find the issue was on our end! Once we update that libary, I'll file a bug to reverse the settings_local.py change.

\o/