1314171 - Reintroduce 'older' query parameter

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Richard Newman [:rnewman]]

In Sync 1.5 we removed sort=oldest and older=X from the API. No clients were using them, so we took the opportunity to reduce the API surface. Ryan summarized this in Bug 1212189 Comment 2.

In that bug we reintroduced sort=oldest to support batch download.

I'd like to finish the original scope of that bug, reintroducing older=X.

The motivation for this:

In Bug 1291821, Android is about to begin batch-downloading all kinds of things.

History is huge. It's hard to batch download the last 5000 items with the Sync API as it stands: our only options are to batch newest-to-oldest (which isn't high-watermark resumable in the case of interruption), to download IDs and then download in chunks by ID, or to try to guess a date range to fetch oldest-to-newest.

Instead we download everything… but that might be 60,000 records, and we block the sync with no visible history until we're done!

To remedy that we download just your 500 most recent items, finish the rest of the sync, then come back and backfill from t=0 using the usual batching code, oldest-to-newest.

Of course, that re-downloads at least 500 items, because we're unable to say "give me records between these two timestamps, sorted by date ascending" to exclude the ones we already grabbed.

Reintroducing older=X would allow us to batch-fetch only old records.

It would also, as described in Bug 1212189, allow us to batch-fetch *backwards*, which is ideal in some circumstances.

This is a backwards-compatible change, and judging by the discussion in that bug, it should be pretty easy to do.

Comment 1

[Ryan Kelly [:rfkelly]]

Attachment: docs-sync-15-older.diff

Sounds fine to me; r? this trivial docs change to be sure we're on the same page?

Comment 2

[Richard Newman [:rnewman]]

Comment on attachment 8807416 [details] [diff] [review]
docs-sync-15-older.diff

Review of attachment 8807416 [details] [diff] [review]:
-----------------------------------------------------------------

Perfect.

Comment 3

[Ryan Kelly [:rfkelly]]

Server patch here: https://github.com/mozilla-services/server-syncstorage/pull/53

Comment 4

[Ryan Kelly [:rfkelly]]

Merged, this will ship as part of Bug 1315425.

Of course, there's also the question of how the client would determine whether 'older' is supported or not.  It's OK to specify it to older versions of the server, it will just be ignored.  But the client needs to be prepared for it being ignored.

Comment 5

[Richard Newman [:rnewman]]

Thanks for the quick fix!

I think that clients will be OK: in the situation we expect to use it, the parameter will simply cause additional newer records to be returned, which is exactly the safe redundancy we're trying to avoid. It's not damaging, just inefficient.

Comment 6

[:Grisha Kruglov]

Doesn't seem that the 1.5 sync storage API docs were updated for this change. I don't see `older` parameter mentioned in https://docs.services.mozilla.com/storage/apis-1.5.html#individual-collection-interaction

Comment 7

[Ryan Kelly [:rfkelly]]

Whoops, apparently I never committed that docs patch.  Push at https://github.com/mozilla-services/docs/commit/061598e76dd23b9d96bdbcee050d08e2c90f48ee and it should show up in the deployed site shortly.