1014157 - Snippet Service python performance plummets with large snippet images

Status: RESOLVED FIXED

(Snippets :: Service, defect)

Description

[Osmose [:osmose, :mkelly]]

Whenever we create a snippet with an image with a large filesize, the service's performance plummets. On one hand, we should avoid large snippets if possible, but it would also be nice to have an idea of why the service gets so angry when dealing with snippets over 150kb and how we can fix that.

Here's a link to New Relic during the time of our last incident: https://rpm.newrelic.com/accounts/263620/applications/2904874?tw[end]=1397499960&tw[start]=1396895160

Here's the transaction view for the main snippets view at that time: https://rpm.newrelic.com/accounts/263620/applications/2904874/transactions?tw[end]=1397499960&tw[start]=1396895160&type=app#id=309252923

Almost all the time is spent in WSGI/Response, which according to http://stackoverflow.com/questions/20214508/analyzing-high-wsgi-response-time-data-of-django-with-mod-wsgi-on-newrelic is the time it takes for WSGI to pull strings out of our response and send them to the client.

Could this be roughly as fast as we could send that data out? Could we be any faster? Is there a reason why, even with the load balancer, the Python app is still getting hammered like this?

Comment 1

[Chris More [:cmore]]

We'll be working with jakem on the infrastructure side of this project.

Comment 2

[Osmose [:osmose, :mkelly]]

We just finished testing the effects of bug 1091939 and were able to send out 4 megabytes of snippets data to release with no issues, so I think we can call this resolved.