546715 - Go to 10ths of a second for synchronization

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Toby Elliott [:telliott]]

Realistically, it seems like 10ths of a second should be plenty, and exposing 100ths of a second does have some potential, if fairly unlikely, security issues. 

If the client needs the %2d value, I can simply make it .x0

Can we do this?

Comment 1

[Mike Connor [:mconnor]]

I think if we're worried about race conditions measured in tenths of a second, we've probably won already.  Mardak, your thoughts?

Comment 2

[Ed Lee :Mardak]

There's a header for X-If-Unmodified-Since header that the server supports but the client doesn't use. However, the time granularity still affects what can be done. For example, if it was a day granularity, client A uploads on day 1, client 2 can fetch new items on day 1 and on the upload of data, ask If-Unmodified-Since <last fetch time, day 1> and fail because the original data was uploaded on day 1.

So only one write is allowed per time granularity, if that's what we want.

Doing that would complicate sync in that we do multiple posts/deletes per sync. Do we want to have X-If-Unmodified-Since cancel the action or just inform that something was modified? If it's just informative, it could hint to the client that data should be refetched from newer=<value for unmodified snice> and older=<time of the response>. Otherwise, the client would need to break out of its upload process and start downloading items again and reconcile, etc before continuing.

Comment 3

[Toby Elliott [:telliott]]

This one's an API change, needs more discussion, and should be part of 2.0

Comment 4

[Mike Connor [:mconnor]]

Punting this to Future, we need to discuss this further before it gets onto the engineering roadmap.

Comment 5

[Toby Elliott [:telliott]]

Would really be better if we could make this seconds...

Comment 6

[Chris Lyon [:clyon]]

Any plans on getting this done soon?

Comment 7

[Toby Elliott [:telliott]]

soon is a relative term, but I'd like to think it's on the 2.0 roadmap

Comment 8

[:Atoll]

+2 to "X-If-Unmodified_Since", because it's good practice regardless and I think it will also prevent a known issue where "upload new record" and "reset sync" trigger MySQL deadlocks.  I am okay with the slightly increased load due to the client aborting its upload and reconciling, as long as we fuzz the delay between those two steps.

Comment 9

[Mike Connor [:mconnor]]

Let's spin that part off to another bug.

The larger "when can we switch to a new API rev" question is post-Fx4, and then we'll have to think long and hard about how that'll work.

Comment 11

[Richard Newman [:rnewman]]

Something I jokingly suggested to Toby at a lunch a little while back, which got a positive reception. I realized I hadn't appended it to a bug, so here goes. This is mostly for posterity.

It seems, from that conversation at least, that the current concern from the server side is about using excessive space per timestamp: 5,000 WBOs * 1M users = 37GB of timestamps.

I didn't get the impression that privacy was a big issue. This does not seem to be captured in this bug, so I might be way off base.

The obvious answer to the server space concern is to truncate the value, losing precision.

The client, however, cares about granularity in order to do its work; coarse timestamps put more load on reconciliation, and introduce ambiguity. (Imagine an extreme world where timestamps had a granularity measured in hours. Which prefs bundle should we take?)

My suggestion was to truncate at the top end of the value, not reduce precision. From what I can see, WBO lastModified is a bigint. Dropping to an int saves 4 bytes per WBO. 2^32 unsigned is 4,294,967,295, which gives a timespan of 497 days down to 1/100sec precision, which we could conceivably pin to a useful epoch. 1/10sec gives a span of 13 years. At the cost of complicating access, another byte or two could be added in an additional column to blow this up into a worry-free size.

Yet another alternative is to share a single bigint between, say, timestamp (6 bytes) and collection (2 bytes), or timestamp and sortindex.

Just somethin' to think about!

Comment 12

[:Atoll]

Average size per wbo body is ~550 bytes. Minimum disk space allocated per innodb record is 768 bytes. Using 8 bytes for the timestamp is unlikely to have any particular impact, vs 4 bytes currently.

I don't see any discussion in *this* bug about the security interest in lowering precision to tenths or even whole seconds.  :mcoates, is there a separate bug for that or is this it?

Comment 13

[Toby Elliott [:telliott]]

We're going to stay with hundredths of seconds, but look at other ways to compress this data.