878107 - Move Clients collection to a memcache writethrough

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect, P2)

Description

[Toby Elliott [:telliott]]

Clients is a small (but important!) set of records (1-3 items in the vast majority of cases) that currently accounts for approximately 25% of our database read requests. 

Dropping them into memcache as a writethrough (a la metaglobal) would both speed up performance on that query and reduce query load on the database. Note that, unlike metaglobal, there may be multiple records in the client collection.

Comment 1

[Ryan Kelly [:rfkelly]]

Adding Bob because we should do this for sync1.5.  In theory it's a simple matter of adding "clients" to the "cached_collections" setting, as the new memcached layer was explicitly designed with this sort of tweak in mind.

Comment 2

[Toby Elliott [:telliott]]

make sure it's a writethrough in the config, not a memcache-only configuration.

Comment 3

[Ryan Kelly [:rfkelly]]

> make sure it's a writethrough in the config,

yes, very important - next to the "meta" collection

Comment 4

[Bob Micheletto [:bobm]]

We are currently sizing memcache needs at 6k per user. I wouldn't expect this to change that by much, but would like to get some testing / sizing in stage before turning it on in production.  What will the configuration item be?

Comment 5

[Ryan Kelly [:rfkelly]]

In the [storage] section, add "clients" to the "cached_collections" setting.

Comment 6

[Bob Micheletto [:bobm]]

This has been enabled in Stage.  Any spare cycles to test this?

Comment 7

[James Bonacci [:jbonacci]]

If it can wait until tomorrow, then definitely. Just drop a window of time in the bug when you are around and I will do some testing...

Comment 8

[James Bonacci [:jbonacci]]

And by Stage, I assume you mean the 1.5 nodes living in AWS right now?
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net

So then a mix of tokenserver, sync, and both through load tests...

Comment 9

[Ryan Kelly [:rfkelly]]

Loadtests won't help us here, as they don't hit the clients collection (Bug 967987).  You'll have to do an actual sync against the stage servers to test this out.

Comment 10

[James Bonacci [:jbonacci]]

Got it. Will point some desktop and Androidy devices at TS Stage and we can watch what happens...
Some tokenserver and sync node log watching too...

Comment 11

[Bob Micheletto [:bobm]]

(In reply to James Bonacci [:jbonacci] from comment #7)
> If it can wait until tomorrow, then definitely. Just drop a window of time
> in the bug when you are around and I will do some testing...

Any time between 11 and 3 PM Pacific works for me.

Comment 12

[James Bonacci [:jbonacci]]

We worked on this in Stage. What's the status on moving forward?

Comment 13

[Ryan Kelly [:rfkelly]]

We need to do a loadtest run with the changes from Bug 967987, which I just landed today.  After that I think we should be fine to roll it out to prod.

Comment 14

[James Bonacci [:jbonacci]]

:rfkelly ok, let's run some load tomorrow afternoon while we chat about stuff...

Comment 15

[James Bonacci [:jbonacci]]

Marking this as Verified. We have been load testing sync nodes for over a month.