985794 - More robust handling of broken storage nodes in tokenserver cleanup script

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

This is a follow-up to Bug 984297, which added a script for purging old records from the tokenserver database.  That script directly uses the tokenserver database as its work queue, so if there's a problem with deleting the data for any row, all subsequent rows are blocked until it can be resolved.

We should try to be a little more robust here.  One suggestion (Bug 984297 Comment 8) is to adjust the replaced_at timestamp on failed records to simulate "moving it to the back of the queue".  A more general approach might be to use an explicit workflow solution based on AWS SWF.

Comment 1

[Ryan Kelly [:rfkelly]]

I'm realistically not going to be able to work on this bug, relinquishing ownership.

Comment 2

[Ryan Kelly [:rfkelly]]

This script (tokenserver/scripts/purge_old_records.py) is an important part of our end-to-end data handling story, as it's the thing that ultimately deletes a user's abandoned sync data from the storage nodes.  What sort of monitoring do we have for it in production to ensure that it's working as intended?

Comment 3

[Ryan Kelly [:rfkelly]]

This is an old bug, but it still seems important; +:pjenvey for consideration.

Comment 4

[David Durst [:ddurst]]

Transferring to internal issue tracking and closing this.