713801 - Sync server support for add-on sync

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Gregory Szorc [:gps]]

Add-on sync is in Firefox 11. It utilizes a new server-side collection: addons. It is my understanding that the Sync server code has special considerations to well-known collections. "addons" probably needs to be added to this list.

There might be other server-side considerations as well. Please use this bug to track them.

Comment 1

[Toby Elliott [:telliott]]

Ryan, make sure you're familiar with Bug 688623 before tackling this one.

Comment 2

[Ryan Kelly [:rfkelly]]

Quoting from Bug 688623: "There also appears to be a problem in the python code where custom codes start right after the constants, rather than with 100. This gives us no room to add new constants in the future and will need to be fixed."

There are 10 existing standard collections, with ids 1 through 10.  To do this properly in the absence of bugs, we would hard-code "addons" as standard collection id 11.  But that doesn't seem to be an option since this will collide with the accidental custom collections created by Bug 688623.

I see two options for moving forward:

1) Fix the data.  As described in Bug 688623 comment 4, except also renumber any other custom collections to have id > 100.  Then "addons" can safely become standard collection id 11.

2) Hack around it.  Find the highest custom collection id in production and allocate new standard collections from there.  Say for example that no-one has more than 10 custom collections, with ids 11 through 20.  Then we would reserve the id range 11 through 20 for compatibility, set "addons" as standard collection id 21, and start allocating new custom collections ids from 100 so we don't have this problem again.

Thoughts?

Comment 3

[Toby Elliott [:telliott]]

Ugh, nicely spotted.

OK, that means that the current nodes are pretty much fubared. Here's what I suggest we do:

1) Nothing on the current nodes.

2) Come up with a new storage engine for sync that implements collections correctly, and install it on all new nodes. I'll file a separate bug with thoughts on how to do this.

3) Slowly migrate users off old nodes and onto new nodes.

Comment 4

[:Atoll]

I don't believe this should block add-on sync. In the new world of collections, there's no inefficiency in custom collections, and we don't need to hard-code it for any particular reason otherwise.

Comment 5

[Toby Elliott [:telliott]]

System will work fine but inefficiently for now, and we can look at actually fixing these issues in 714318. No need to block.

Comment 6

[Toby Elliott [:telliott]]

Let's make sure that addons is a defined collection for 2.0 (and that 2.0 leaves a good-sized gap before custom collections), and not worry about the rest of this.

Comment 7

[Ryan Kelly [:rfkelly]]

Attachment: patch to make addons a standard collection

Assuming the fixes from Bug 714318, this patch should be all that's needed.

Comment 8

[Toby Elliott [:telliott]]

Comment on attachment 601101 [details] [diff] [review]
patch to make addons a standard collection

I dunno. There's a lot of complexity here.

Comment 9

[Ryan Kelly [:rfkelly]]

https://github.com/mozilla-services/server-syncstorage/commit/de2fb1b8cd12857eb86dd0693445bb2f62407f0f