784598 - [meta] outstanding Sync1.1 operational improvements

Status: RESOLVED INCOMPLETE

(Cloud Services Graveyard :: Server: Sync, defect, P4)

Description

[Ryan Kelly [:rfkelly]]

We have been deferring or WONTFIXing quite a few operational improvements for Sync1.1, with the expectation that we can just fix them in Sync2.0.  Given that Sync1.1 is going to be with us for some time yet, this bug is for collecting those improvements that we should specifically still target for Sync1.1.

I'll add them as blockers to this bug, y'all speak up if you don't think it's worth implementing them over the remaining lifetime of Sync1.1.  (All have been or will be implemented for Sync2.0).

Comment 1

[Ryan Kelly [:rfkelly]]

Bug 735085: prevent overlapping writes to same user/collection by different clients

This would involve doing a "SELECT FOR UPDATE" when reading the id for a user's collection, so that no other writes can be performed to that collection while the request is outstanding.  No changes required to database schema, but would need some refactoring of how db connections are handled in python.  I've basically already done this for Sync2.0 and that would could be partially re-used.

Comment 2

[Ryan Kelly [:rfkelly]]

Bug 750964: Prevent a single unresponsive database from bringing down the webheads

This will just involve me refactoring the DB connection code so that we can detect and avoid downed DBs.

Comment 3

[Ryan Kelly [:rfkelly]]

Bug 599096: deadlock when delete_objects and store_object collide

I think this will be resolved by the locking suggested in Bug 735085.

Comment 4

[Ryan Kelly [:rfkelly]]

Bug 624542: probably faster queries for username, collection, MAX(modified)

I've no idea how or why these queries would be faster, but am happy to bench them when we all get a few free cycles :-)

Comment 5

[Ryan Kelly [:rfkelly]]

Bug 624791: Mysql error "Query execution was interrupted" should lead to 503+Retry-After

Since this relates to handling a specific MySQL error code, we should be able to special-case it in the same way as we did for Bug 692544.

Comment 6

[Ryan Kelly [:rfkelly]]

Bug 637523: syncserver should never connect to a db marked as down in the adm1 db

This sounds pretty similar to Bug 750964, they may have a common solution.

Comment 7

[Ryan Kelly [:rfkelly]]

Bug 693375: Defer deletes to operational discretion

I recall us working around this problem by increasing the worker timeouts, giving them long enough for the deletes to complete successfully.  Ops, are you still interested in the deferred-delete solution described in that bug?

Comment 8

[Ryan Kelly [:rfkelly]]

Bug 766026: Gracefully handle lots of records with closely-spaced ttls

I think is a specific example of a more generic request: paginate SELECT queries so that one query is able to monopolize all the database resources.  We could implement it with an internal-to-the-server variation of the pagination techniques discussed in Bug 735102.

Comment 9

[Ryan Kelly [:rfkelly]]

Note to self, :atoll has identified Bug 750964 as the top priority from this collection so far.

Comment 10

[Ryan Kelly [:rfkelly]]

Adding Bug 668664.  Bob or Toby, anything in this list that we want to revive after the latest round of adventures?

Comment 11

[Ryan Kelly [:rfkelly]]

Unlinking this from the sync1.5-prod-ready tree, I've added the relevant sub-bugs directly to it.

Comment 12

[Ryan Kelly [:rfkelly]]

sync is dead, long live sync