850784 - Geolocated Snippets: Load Test

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

We'd like to load test the new geo service and infrastructure. This service will serve a very high volume: Up to 4 billion page requests per month. Therefore we should anticipate anywhere from 60 million to 140 million requests per day.

Suggested tests: 
* A stress test seeking the breaking point and identifying the bottleneck that caused it (iterating on this if the breaking point is less than our anticipated max load)
* A load test sending appx. 5,500,000 (4,000,000,000 / 30 / 24) requests over the course of an hour or more, to tease out any issues that might appear under long loads

Comment 1

[Jake Maul [:jakem]]

Hang on a second... I believe our data is skewed a bit here. We talked about this in a meeting, but I suspect we've still got it wrong.

We will only hit the geo.mozilla.org service once per month per client (per client that uses about:home, that is). This is in contrast to the main snippets service itself, which should get 1 hit per *day* per client, and in contrast to the snippets-stats service, which should get 1 hit per *impression*.

We should have approximately 1/30th as much traffic as snippets itself gets.

If snippets.mozilla.com gets up to 4B hits per month, then we should expect roughly 1/30th of that many hits to geo.mozilla.org... or 133M hits/month. Slightly more, to account for situations where the cache doesn't last a full month (cache purges, profile wipes, reinstalls... random chaos).

That means our estimate is off by a factor of 30x too high. We actually only have to serve 140M hits/month, not per day. This is much closer to what I would expect to see, based on hits to our load balancers for snippets.mozilla.com.

Comment 2

[Jake Maul [:jakem]]

I have tested this a bit... we are capable of bursting up to around 5000 requests/second, at least. More might be possible... I'm not sure if I'm out of client testing resources, or server/geodude resources at that point. :)

That works out to be around 300K/min, which is far more than snippets.mozilla.com *and* snippets-stats.mozilla.org serve, combined.


I am currently trying a longer-running test, but at the moment it appears that 2000 requests/sec sustained indefinitely is no problem. I suspect double this would be feasible.

This works out to (at least) 120K/min, or 7.2M/hour, or 172M/day. Well beyond even our original estimates, which I still believe are 30x too high.

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

This is very promising! 

Was there any additional information after the long-running test, or did the service continue to perform as expected?

Comment 4

[Jake Maul [:jakem]]

We're good here... no problems encountered. :)