1364005 - Prepare for new formautofill collections

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, enhancement)

Description

[Mark Hammond [:markh] [:mhammond]]

The formautofill work is likely to create 2 new collections. Both the record size and record count for should be small (I expect "a few" to be typical, > 10 to be outliers) - however, we do expect to keep tombstones aggressively - at this stage they will never be deleted locally and will be synced down and stored to new devices. However it seems unlikely there will be a significant number of tombstones (the client isn't going to want to store a large number locally either!).

I believe :rfkelly suggested there might be some choices to be made regarding memcached configurations etc, but that's all largely over my head :) I'll let bobm and rfkelly decide if this bug should be a meta for various issues, or whether we can nut it all out in this bug.

Comment 1

[Ryan Kelly [:rfkelly]]

It sounds like it should be OK to leave these collections with the default storage/caching parameters, at least until we get operational experience telling us otherwise.  What we should do ahead-of-time is get them added to the list of "standard collections" here:

  https://github.com/mozilla-services/server-syncstorage/blob/master/syncstorage/storage/sql/__init__.py#L48

Which will make operations on them slightly more efficient for the server.

We've cleverly left a gap between collection ids of standard vs custom collections, so in theory this will be a straightforward matter of adding the collection names to that list.  :markh, what are the intended names of these two new collections?

:bobm, let's do some sanity-checks to make sure that this addition will go smoothly.  The intended collection ids for these new collections will be 12 and 13, so can we please check the sync storage dbs to ensure that:

* No users have any existing collections with ids of 12 or 13
* No users have existing custom collections with names that match the ones :markh is going to provide

We should be able to confirm both of these things by looking in the "collections" table(s) on each node:

  https://github.com/mozilla-services/server-syncstorage/blob/master/syncstorage/storage/sql/dbconnect.py#L80

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #1)
I think we'll want to get the load tests changed to reflect the new collections via Bug 1326243.

Comment 3

[Bob Micheletto [:bobm]]

Here is the complete list of unique collection names from the collections tables on Sync servers by running "SELECT DISTINCT(name) FROM weave0.collections;":
adblocker
adblockplus
adshield
awe
col1
col2
ephy-bookmarks
extension-storage
floatnotes
formautofill
greasemonkey
meh
mozcn.ntab
name
pan
passwordtags
policy
qddons
reqpolsync
stopallads
stylishsync
theadblockerproject
userstyles
xxxx

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #1)

> :bobm, let's do some sanity-checks to make sure that this addition will go
> smoothly.  The intended collection ids for these new collections will be 12
> and 13, so can we please check the sync storage dbs to ensure that:
> 
> * No users have any existing collections with ids of 12 or 13
> * No users have existing custom collections with names that match the ones
> :markh is going to provide

Based on the following query: SELECT DISTINCT(collection) FROM weave0.user_collections WHERE collection = 12 or collection = 13;
There are no collections with an ID of 12 or 13.

Comment 5

[Mark Hammond [:markh] [:mhammond]]

(In reply to Ryan Kelly [:rfkelly] from comment #1)
> :markh, what are the intended names of these two new collections?

addresses and creditcards are what we have in our current patches (but they aren't user-visible, so I'm happy to change them).

Comment 6

[Ryan Kelly [:rfkelly]]

Attachment: PR adding the new standard collections

Mark, could you please give this a quick once-over?

Comment 7

[Mark Hammond [:markh] [:mhammond]]

Comment on attachment 8878291 [details] [review]
PR adding the new standard collections

(In reply to Ryan Kelly [:rfkelly] from comment #6)
> Mark, could you please give this a quick once-over?

Yay for partial github integration (I also gave this r+ on github)

Comment 8

[Mark Hammond [:markh] [:mhammond]]

https://github.com/mozilla-services/server-syncstorage/pull/65 was merged on Jun 19.