Closed Bug 1154991 Opened 9 years ago Closed 9 years ago

Please set sync migration header throttle to a specific amounts

Categories

(Cloud Services Graveyard :: Server: Sync, defect)

Product:
Cloud Services Graveyard
Component:
Server: Sync
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: edwong, Assigned: bobm)

References

Details

Please send out sync migration header to 5% of users on Fx37 or greater with a single device.  Please do this at 4/16 morning PDT.
Assignee: nobody → bobm
OS: Mac OS X → All
Hardware: x86 → All
We are planning to release 37.0.2 tomorrow. In order not to conflate any issues that may arise from enabling this notification, I request that we hold off on enabling the notification until sometime next week.
Thanks for the heads up.  Holding this change until further review and rescheduling.
Blocks: 1142276
Lawrence, are we safely past the 37.0.X deployments at this stage?  If we delay our ramp-up much longer we risk bumping up against the 38 release, where we *definitely* don't want to conflate any issues from this versus other features.
Flags: needinfo?(lmandel)
Yes. There is still a decent user base on 37.0.1 (37.0.2 is latest) but I don't think that needs to hold up this work. Please send an e-mail to release-drivers with details of what you plan to do for sync migration.
Flags: needinfo?(lmandel)
:bobm what kind of numbers are we user agent version numbers are we seeing form the wild.  If a majority is 37.x or greater- let's turn on today for 5% of single device Fx37+.
Flags: needinfo?(bobm)
(In reply to Edwin Wong [:edwong] from comment #5)

Marking this employee only, because of metrics.

Top 10 UAs by percentage of traffic:
37.0.2 63.82%
37.0.1 11.84%
36.0.4 2.23%
35.0.1 1.5%
36.0.1 1.16%
31.6.0 1.02%
38.0 0.86%
34.0.5 0.74%
35.0 0.45%
36.0 0.41%

If 75% is sufficient to meet the majority, looks like we should be a go for today.  

:edwong are you sending the release-drivers mail?  Or shall I?
Group: mozilla-employee-confidential
Flags: needinfo?(bobm) → needinfo?(edwong)
email sent to release-drivers.
Flags: needinfo?(edwong)
(In reply to Bob Micheletto [:bobm] from comment #6)
> Marking this employee only, because of metrics.

Is this really data that needs to be kept secret?
> Is this really data that needs to be kept secret?

It's a long-standing cloud services habit to mark anything with metrics as confidential, and open it up if we feel like it's safe.  In this case it seems fine to me, particularly since we're talking about percentage breakdown rather than total numbers.
(In reply to :Gavin Sharp [email: gavin@gavinsharp.com] from comment #8)
> (In reply to Bob Micheletto [:bobm] from comment #6)

> Is this really data that needs to be kept secret?
I err on the side of caution when it comes to metrics.  I'm happy to lift the employee-only flag if you feel it is warranted.
OK. Fine to be cautious, but I don't see any harm here.
Group: mozilla-employee-confidential
Setting Sync 1.1 EOL headers roll-out percentage to 5%.
Status: NEW → ASSIGNED
This setting has been updated.  Closing bug.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
From email: rfkelly, bobm, and I agree we should move to 20% as there is no FHR data that can be pulled this week. Please do this at your earliest convenience.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Setting this to 20% now.
We want to take a somewhat aggressive schedule to getting users to update prior to Fx38 for single device, and start multi-device after Fx38 is stable

Here's the plan (can change due to dependent dates):
may 18 - 60%
may 25 - 100%
jun 15 - 5% multi-device
Depending on how many deployments :bobm wants to do, we could also do half-weekly increments like this:

may 18 - 40%
may 21 - 60%
may 25 - 80%
may 27 - 100%

It might smooth out the load on tokenserver node assignments, at the cost of more of Bob's time.
Flags: needinfo?(edwong)
:edwong Added needinfo flag for verification that we're clear to move forward today.
rfkelly and I are both locked out release drivers. we don't want to delay and in agreement that we should move fwd to 40% as no one has contacted us otherwise.
Flags: needinfo?(edwong)
Moving to 40% throttle.
(In reply to Bob Micheletto [:bobm] from comment #20)
> Moving to 40% throttle.

Done.
Per Sync weekly meeting, we're a go for moving to a 60% throttle tomorrow.
(In reply to Bob Micheletto [:bobm] from comment #22)
> Per Sync weekly meeting, we're a go for moving to a 60% throttle tomorrow.

Done.
Moving to 80%.
(In reply to Bob Micheletto [:bobm] from comment #24)
> Moving to 80%.

Done.
Dropping this down to zero in preparation for the 38.0.5 release.  I should note that due to capacity concerns (that turned out to be a single user with a problem) we did not move to 100% throttling.
(In reply to Bob Micheletto [:bobm] from comment #26)
> Dropping this down to zero in preparation for the 38.0.5 release.  I should
> note that due to capacity concerns (that turned out to be a single user with
> a problem) we did not move to 100% throttling.

Done.
comment #6 says we're over 75% Fx37 or greater saturation. All systems go for Jun 15 throttle to 5% for multi-device.  

rfkelly brought up that all Fx versions will receive migration headers, the code currently doesn't filter by UA version.  Knowing that we're ~80% Fx37 or greater gives confidence that multi-device users won't encounter mixed version issues.  bobm is going to get another UA distribution of sync users prior to Monday for a Go/NoGo.
Summary: Please set sync migration header throttle to a specific amount for Fx37 or greater - single device → Please set sync migration header throttle to a specific amounts
(In reply to Edwin Wong [:edwong] from comment #28)

> version issues.  bobm is going to get another UA distribution of sync users
> prior to Monday for a Go/NoGo.

Traffic from 6/8 was 75.28% Fx37+, which makes sense since those are the target audience for migrations.  We should expect to see that to continue to go down as we migrate more users.  Android specific Fx37+ clients are at 74.46%.
> all Fx versions will receive migration headers, the code currently doesn't filter by UA version

Refering back to earlier experiments we did with client behaviour in Bug 1014411 Comment 27, when older browsers (back as far as Fx29 IIRC) see a "soft-eol" header they will show a grey error bar saying:

 "Your Firefox Sync service is shutting down soon. Upgrade Nightly to keep syncing. [Learn More]".

ISTM this is appropriate messaging at this point in time, so we're safe to proceed with general release of these headers.

Users on super-old versions of Firefox will get a warning message logged in their console and sync debug logs, but AFAIK will not see any UI in response to our EOL headers.
Set to 5% throttle for all users.
Setting throttle to 20% for all users per decision made during weekly Sync stand-up meeting.
(In reply to Bob Micheletto [:bobm] from comment #32)
> Setting throttle to 20% for all users per decision made during weekly Sync
> stand-up meeting.

Done.
Setting throttle to 40%.
Set throttle to 60%.
Setting throttle to 80%.
(In reply to Bob Micheletto [:bobm] from comment #36)
> Setting throttle to 80%.

Done.
Dialing this up to 11.
(In reply to Bob Micheletto [:bobm] from comment #38)
> Dialing this up to 11.

Dialed.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.