756226 - utilities to read/copy/delete data stored in the cache

Status: VERIFIED INCOMPLETE

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[:Atoll]

We discovered yesterday that all of our hand-crafted Python utilities for talking to the sync storage cache do not use the same sharding algorithm as sync storage, on at least one platform so far.

So that's been temporarily fixed with an adhoc script from Toby.  However long-term we should create and maintain an actual set of maintenance tools for the cache.

Please create a set of tools that use the syncstorage CacheManager to:

* read all sync storage data for a STDIN userid list
* clear all sync storage data for a STDIN userid list

Comment 1

[Ryan Kelly [:rfkelly]]

I'm taking a crack at this for sync2.0 as part of Bug 817941, attaching in three parts:

  * a new script for clearning memcache data, syncstorage/scripts/mcclear.py
  * a new script for reading memcache data, syncstorage/scripts/mcread.py
  * supporting code patch, including testcases

Comment 2

[Ryan Kelly [:rfkelly]]

Attachment: script to clear memcache data for a set of users

This script is designed to be run as follows:

    python syncstorage/scripts/mcclear.py syncstorage.ini < list_of_uids.txt

It will iterate through the uids given on stdin and clear the memcache data for each one in turn.

It's basically a port of an existing script, but I've tried to make it re-use as much of the application code as possible.  So for example rather than specifying the memcache server settings, you just specify the app config file and it reads everything out of there.

This script is required as part of the node-management procedure, where migrating users off a node involves clearing all of their memcache data.  Does it have any other uses in our current sync1.1 setup?

Comment 3

[Ryan Kelly [:rfkelly]]

Attachment: script to read memcache data for a set of users

This script is designed to be run as:

    python syncstorage/scripts/mcread.py syncstorage.ini < list_of_uids.txt

It will iterate through the uids given on stdin and print out a line for each memcache key of each user.  Currently is just prints <key><space><value> and doesn't do any escaping of e.g. newlines in the values.  What output format is required by ops for this script?

It's also not clear to me under what circumstances this script would be used.  Debugging only?

Comment 4

[Ryan Kelly [:rfkelly]]

Attachment: patch to support running of cache-management scripts

Part 3 is changes to the code to support running of these scripts:

 * make it easy to load a Configurator object like the one the running app would use
 * provide ability to iterate over all storage backends loaded into an app
 * make iter_cache_keys a public API method of the memcache backend, rather than duplicating this list of keys
 * tests!

The last part is the whole reason I'm trying to pull these scripts into the repo - testcases that actually run the script and assert that it's behaving correctly.

Comment 5

[Ryan Kelly [:rfkelly]]

Also: at Ops request, I will be glad to backport these scripts to the sync1.1 codebase, it should be quite straightforward modulo the config-loading logic.

Comment 6

[Bob Micheletto [:bobm]]

The output format for this should be fine.  I believe the operational purpose is to identify and resolve any run-away user problems.

Comment 7

[Ryan Kelly [:rfkelly]]

Committed for sync1.5 in https://github.com/mozilla-services/server-syncstorage/commit/23f0ef70ed32ba169d6feee45c47c4a38fef218b

:bobm are we happy to WONTFIX this for sync1.1 and just leave the current scripts as they are?

Comment 8

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #7)
> 
> :bobm are we happy to WONTFIX this for sync1.1 and just leave the current
> scripts as they are?

Yes.