693694 - Affiliates: 500 errors on stage with banner link

Status: VERIFIED INVALID

(Firefox Affiliates Graveyard :: affiliates.mozilla.org, defect)

Description

[Chris More [:cmore]]

We are randomly getting 500 errors on banner links. 

Here are the times:

Thread Name: Thread Group 1-7
Sample Start: 2011-10-11 10:59:57 PDT
Load time: 11425
Latency: 11425
Size in bytes: 743
Headers size in bytes: 202
Body size in bytes: 541
Sample Count: 1
Error Count: 1
Response code: 500
Response message: Internal Server Error

Thread Name: Thread Group 1-4
Sample Start: 2011-10-11 10:59:57 PDT
Load time: 18684
Latency: 18684
Size in bytes: 743
Headers size in bytes: 202
Body size in bytes: 541
Sample Count: 1
Error Count: 1
Response code: 500
Response message: Internal Server Error

Thread Name: Thread Group 1-9
Sample Start: 2011-10-11 10:59:57 PDT
Load time: 18697
Latency: 18697
Size in bytes: 743
Headers size in bytes: 202
Body size in bytes: 541
Sample Count: 1
Error Count: 1
Response code: 500
Response message: Internal Server Error

Thread Name: Thread Group 1-13
Sample Start: 2011-10-11 10:59:57 PDT
Load time: 18636
Latency: 18636
Size in bytes: 743
Headers size in bytes: 202
Body size in bytes: 541
Sample Count: 1
Error Count: 1
Response code: 500
Response message: Internal Server Error

Since these are on stage, we will need to have IT look at the apache log files.

Comment 1

[Chris More [:cmore]]

These errors were random and when load testing on stage with 100 concurrent requests.

Comment 2

[Chris More [:cmore]]

I'm having IT pull the log files for these times. I am also able to replicate it with additional load tests, but it is still random.

Comment 3

[Chris More [:cmore]]

Mkelly: I've looked over all of the apache error logs and there is only one error that stands out, but it looks to be just a warning. I don't think this would cause a 500 error:

[Tue Oct 11 11:49:31 2011] [error] /usr/lib/python2.6/site-packages/jinja2/__init__.py:31: UserWarning: Module memcache was already imported from /data/www/affiliates.allizom.org/affiliates-app/vendor/lib/python/memcache.py, but /usr/lib/python2.6/site-packages is being added to sys.path
[Tue Oct 11 11:49:31 2011] [error]   __version__ = __import__('pkg_resources') \\

Comment 4

[Chris More [:cmore]]

Here are some additional errors that I got from the error logs:

[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi
[Wed Oct 12 10:50:07 2011] [error] [client 10.8.33.248] Premature end of script headers: playdoh.wsgi

Comment 5

[Chris More [:cmore]]

Here is some information on the premature end of script error:

http://stackoverflow.com/questions/4501817/django-apache-mod-wsgi-error-premature-end-of-script-headers

Comment 6

[Chris More [:cmore]]

This was due to just overloading dev/stage due to the limited amount of concurrent requests they can handle and sessions being dropped. Invalid