1160218 - Pocket Load Testing

Status: RESOLVED FIXED

(Firefox :: Pocket, defect)

Description

[Stuart Philp :sphilp]

Rather than guessing what will happen, we should plan for the worst case and then scale back as needed post launch once we get real usage data, that way we won't have to worry about capacity/outages due to load.

It seems that 120m users is likely to be the max we can possibly see at launch as users click around and explore the Pocket integration based on overall Firefox usage and en-us locales. 

In requests per second, that works out to roughly 5500rps (at 4 requests per user per 24h period). So that's the goal right now. 

We need to clarify which FxA servers need to be at that level (is it across the board or just oauth or some combination?), and how many requests per user per day we think we'll see.

Comment 1

[Matt Koidin]

If helpful, right now we are making calls to:

https://stable.dev.lcip.org/profile/
https://oauth-stable.dev.lcip.org/

(Obviously we will make calls to their production counterparts when going live.)

These calls are for validating the FxA auth on our backend and pulling the required profile information. This does not include anything that is happening in browser - defer to others on that.

Comment 2

[Stuart Philp :sphilp]

Thanks Matt! When are those calls being made, as in what user action, just on login? I'll check on the firefox side today.

Comment 3

[Stuart Philp :sphilp]

So it's looking like we only hit the FxA servers on account creation and login, which means we can scale this back a bit as those calls will be less frequent than the pocket api calls. Chatting with :ckolos it seems around 200k requests per minute would be comfortable for us, which is about 3300rps.

Comment 4

[Matt Koidin]

Stuart - sorry for the delay responding here - yes, you are correct, we only hit the FxA servers on account creation and login.

Comment 5

[Stuart Philp :sphilp]

Kicked off some load testing yesterday, the basics look good. Ran into a bit of an issue with FxA signup returning some 500s which we think is due to the current staging config. The other endpoints look pretty good so far and we'll finish off those today.

Comment 6

[Stuart Philp :sphilp]

Hit the 4 endpoints (signup, save, get, getSuggestedTags) with heavier loads today and found some issues, some that have been resolved. We look to be good now on the signup, get, and getSuggestedTags endpoints. There seems to be an issue on the save endpoint that the Pocket guys are investigating, so we'll try pushing that endpoint again when they have a solution.

Comment 7

[Stuart Philp :sphilp]

Completed load testing today with big help from Chris and Matt from Pocket. They had a DB issue with hitting the same record over and over causing row locks, so randomizing that a bit helped. We were able to successfully test all endpoints to their desired load.