969555 - Reduce single device sync interval

Status: VERIFIED FIXED

(Firefox :: Sync, defect, P2)

Description

[Tracy Walker [:tracy]]

https://bugzilla.mozilla.org/show_bug.cgi?id=968341#c1:

(In reply to Richard Newman [:rnewman] from comment #1)
> I'd flip this around a little bit: newly-added clients should schedule a
> sync for a few minutes later, and they'll switch into the right mode
> automatically. (Optimally it would sync immediately after the other devices
> on the account.)
> 
> Related problem: if the *first* device you use is a desktop, then you add
> another device, the first device won't know it's now in a multi-device
> scenario until the next day. Unlike with the pairing flow, there's no way to
> notify it to switch.
> 
> It might be worth dropping the single-device interval down to something
> smaller.


with J-pake pairing gone, client A has no idea that client B has just connected to the FxA.  24 hours, possibly more, is a long time for devices to not be aware of each other, particularly when it is not clear to the user their devices aren't syncing to each other yet.

Is one hour for the singleDevice interval too short?  me thinks that is what it was before j-pake in legacy sync.

Comment 1

[Richard Newman [:rnewman]]

f+ from me. I'm planning to use 8 hours on a phone, but use of the browser will actually trigger a sync, so…

Comment 2

[Tim Taubert [:ttaubert] (inactive)]

Attachment: 0001-Bug-969555-Reduce-single-device-sync-interval.patch

Comment 3

[Richard Newman [:rnewman]]

Comment on attachment 8389417 [details] [diff] [review]
0001-Bug-969555-Reduce-single-device-sync-interval.patch

Review of attachment 8389417 [details] [diff] [review]:
-----------------------------------------------------------------

Is svc-ops prepared for a 16x increase in traffic for Old Sync (albeit staged over the release trains), as well as New Sync?

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Richard Newman [:rnewman] from comment #3)
> Comment on attachment 8389417 [details] [diff] [review]
> 0001-Bug-969555-Reduce-single-device-sync-interval.patch
> 
> Review of attachment 8389417 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Is svc-ops prepared for a 16x increase in traffic for Old Sync (albeit
> staged over the release trains), as well as New Sync?

No.

How do we stop or delay that from happening in old Sync?

We are expecting the increase in New Sync and TS as part of the beta release because of ~4.5 million user base, not based solely on the increase in client Sync intervals.  We have been sizing our plan for Sync 1.5 in production based on extrapolating user behaviors from current Sync 1.1 and the much smaller user population in Sync 1.5 production.  I believe it would be most beneficial for the influx of the beta users and the increase in the Sync interval to happen at different stages.  Will that be possible?

Comment 5

[Toby Elliott [:telliott]]

A message saying "It may take 24 hours for all your devices to be syncing. To speed that up, select Sync Now on your other devices" when adding a device might be a reasonable compromise. I don't recall this being a source of frequent complaints in 1.1.

Comment 6

[Tracy Walker [:tracy]]

(In reply to Toby Elliott [:telliott] from comment #5)
>  I don't recall this being a source
> of frequent complaints in 1.1.

Because with J-Pake pairing, clientA was able to be notified that it will no longer be a singleDevice at the time clientB is added. And thus clientA would immediately start syncing based on the instant sync rules of a multi-device account.

Comment 7

[Tim Taubert [:ttaubert] (inactive)]

Attachment: 0001-Bug-969555-Reduce-single-device-sync-interval.patch, v2

So how about only applying the 1 hour single device interval to FxA users? Does that still break assumptions that were made when estimating server load?

Comment 8

[Richard Newman [:rnewman]]

(In reply to Bob Micheletto [:bobm] from comment #4)

> No.
> 
> How do we stop or delay that from happening in old Sync?

We can tweak the pref during the FxA setup process instead of hard-preffing it.

But bear in mind that single-device users will already be syncing as a result of activity (e.g., bookmarking), return from idle, syncing after restarting the browser, etc., and that not every single-device user will be leaving their browser open 24 hours a day.

16x is therefore an upper bound. I don't know what the real traffic increase would be.

Bear in mind also that for many single-device users, each of these syncs will be an info/collections hit and nothing more, or an info/collections hit and then an upload to history and tabs.


> We are expecting the increase in New Sync and TS as part of the beta release
> because of ~4.5 million user base, not based solely on the increase in
> client Sync intervals.  We have been sizing our plan for Sync 1.5 in
> production based on extrapolating user behaviors from current Sync 1.1 and
> the much smaller user population in Sync 1.5 production.

1.5 is quite likely to attract a different user population, so there's a limit to how useful our prior models will be, alas :/

Comment 9

[Richard Newman [:rnewman]]

Comment on attachment 8389447 [details] [diff] [review]
0001-Bug-969555-Reduce-single-device-sync-interval.patch, v2

Review of attachment 8389447 [details] [diff] [review]:
-----------------------------------------------------------------

::: services/sync/modules/policies.js
@@ +45,5 @@
> +    if (service.fxAccountsEnabled) {
> +      // Reduce single device interval for FxA to 1 hour.
> +      this.singleDeviceInterval = 3600 * 1000;
> +    } else {
> +      this.singleDeviceInterval = Svc.Prefs.get("scheduler.singleDeviceInterval") * 1000;

scheduler.fxa.singleDeviceInterval, scheduler.sync11.singleDeviceInterval?

Comment 10

[Richard Newman [:rnewman]]

(In reply to Toby Elliott [:telliott] from comment #5)
> A message saying "It may take 24 hours for all your devices to be syncing.
> To speed that up, select Sync Now on your other devices" when adding a
> device might be a reasonable compromise. I don't recall this being a source
> of frequent complaints in 1.1.

UI is frozen for 29.

Comment 11

[Tim Taubert [:ttaubert] (inactive)]

(In reply to Richard Newman [:rnewman] from comment #9)
> > +    if (service.fxAccountsEnabled) {
> > +      // Reduce single device interval for FxA to 1 hour.
> > +      this.singleDeviceInterval = 3600 * 1000;
> > +    } else {
> > +      this.singleDeviceInterval = Svc.Prefs.get("scheduler.singleDeviceInterval") * 1000;
> 
> scheduler.fxa.singleDeviceInterval, scheduler.sync11.singleDeviceInterval?

Thought about doing that as well. I'm happy to make that change if we don't kill our servers with that.

Comment 12

[Richard Newman [:rnewman]]

Another, slightly more complex, approach to this is to independently poll clients more frequently than doing a full sync -- that is, have some portion of these scheduled syncs be download-only syncs of just clients (and tabs?).

Most of the time this will be just an info/collections fetch, and it'll allow the client to detect a change in the number of collected clients and adjust scheduling accordingly.

Comment 13

[Ryan Kelly [:rfkelly]]

> Bear in mind also that for many single-device users, each of these syncs will be an info/collections
> hit and nothing more, or an info/collections hit and then an upload to history and tabs.

This is the really important point.  Better, if it's possible to make it *just* an /info/collections hit and eliminate the possibility of tab/history syncing during these checks, it will be a lot more manageable on both New Sync and any hypothetical spillage to Old Sync.

(Not awesome mind you, but much much better).

Comment 14

[Bob Micheletto [:bobm]]

(In reply to Tim Taubert [:ttaubert] from comment #7)
> Created attachment 8389447 [details] [diff] [review]
> 0001-Bug-969555-Reduce-single-device-sync-interval.patch, v2
> 
> So how about only applying the 1 hour single device interval to FxA users?
> Does that still break assumptions that were made when estimating server load?

To the extent that we have useful information to extrapolate with, any changes to the Sync 1.5 client behavior at this point make it much more difficult for us to size our production environment.  The download only / memecached hit only approach seems the safest to me.

Any changes that might impact Sync 1.1 behavior are highly discouraged from an operations stand-point.  We have delayed some operational improvements in the Sync 1.1 environment, are working on some disk issues at present at the caching-tier, and this additional load would be negative for the service.

Comment 15

[Bob Micheletto [:bobm]]

(In reply to Richard Newman [:rnewman] from comment #8)
> (In reply to Bob Micheletto [:bobm] from comment #4)

> We can tweak the pref during the FxA setup process instead of hard-preffing
> it.

Sounds good.

> 16x is therefore an upper bound. I don't know what the real traffic increase
> would be.
> 
> Bear in mind also that for many single-device users, each of these syncs
> will be an info/collections hit and nothing more, or an info/collections hit
> and then an upload to history and tabs.

That is much more tenable than a full 16x traffic increase.  Because of the local caching on the Sync 1.5 storage nodes, absorbing the traffic _probably_ won't lower the user to node ratio by much.  Because of the operational issues in the Sync 1.1 caching tier, extra info/collections and tabs hits there wouldn't be absorbed as easily.

> 1.5 is quite likely to attract a different user population, so there's a
> limit to how useful our prior models will be, alas :/

Agreed, but it's the only game in town.

Comment 16

[Bob Micheletto [:bobm]]

Comment on attachment 8389447 [details] [diff] [review]
0001-Bug-969555-Reduce-single-device-sync-interval.patch, v2

Review of attachment 8389447 [details] [diff] [review]:
-----------------------------------------------------------------

FxA only looks good to me, but again prefer the download only option.

Comment 17

[Tim Taubert [:ttaubert] (inactive)]

Attachment: 0001-Bug-969555-Reduce-single-device-sync-interval-for-Fx.patch, v3

Looks like we will go with an fxa-specific pref for 29.

Comment 18

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/integration/fx-team/rev/4609ebf86a67

Comment 19

[Tim Taubert [:ttaubert] (inactive)]

Forgot to update a test and had to push a bustage fix:

https://hg.mozilla.org/integration/fx-team/rev/d2f55c8689ff

Comment 20

[Tim Taubert [:ttaubert] (inactive)]

Comment on attachment 8389937 [details] [diff] [review]
0001-Bug-969555-Reduce-single-device-sync-interval-for-Fx.patch, v3

[Approval Request Comment]
Bug caused by (feature/regressing bug #):
User impact if declined: Secondary devices can take a long time (up to 24 hrs) before syncing.
Testing completed (on m-c, etc.): 
Risk to taking this patch (and alternatives if risky): Low risk.
String or IDL/UUID changes made by this patch: None.

Comment 21

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b1eafb2118dc

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4609ebf86a67
https://hg.mozilla.org/mozilla-central/rev/d2f55c8689ff

Comment 23

[Tracy Walker [:tracy]]

Much better! :-) ClientA synced up with clientB at the new one hour interval.