Closed Bug 1387651 Opened 7 years ago Closed 7 years ago

Staged rollout of Safe Browsing V4 to the release population using Shield

Categories

(Toolkit :: Safe Browsing, defect, P1)

Product:
Toolkit
Component:
Safe Browsing
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox56 + fixed

People

(Reporter: francois, Assigned: francois)

References

Details

(Whiteboard: #sbv4-m9) 

Google would like us to gradually move users to the Safe Browsing V4 server (from the existing V2 server).

They suggested 10%, then 50%, then 100%.
After talking to Mike Kelly, we have decided to implement this roll-out in the form of a system add-on, the same mechanism that was used for e10s.

Since the default is already V4, the add-on will do a "reverse roll-out" (or roll back) of 90% of users. Then 50%, etc.
Summary: Staged rollout of Safe Browsing V4 to the release population → Staged rollout of Safe Browsing V4 to the release population using a system add-on
Depends on: 1390698
Michael, other than producing the add-on (which is covered by bug 1390698) do you know what the process is for shipping a system add-on? Or who I should talk to?
Flags: needinfo?(mkelly)
I believe https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process is up-to-date on the process. rdalal is currently managing the release process for system add-ons and can also answer further questions.
Flags: needinfo?(mkelly)
Whiteboard: #sbv4-m9 → #sbv4-m9 [go-faster-system-addon]
Do you want to approach it by shipping with the default on? 

Or turn it off later in beta?
You could try turning it off in beta 56 and ship a system addon to beta as well, to flip the prefs on.
status-firefox56: --- → affected
tracking-firefox56: --- → +
Flags: needinfo?(francois)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #4)
> Do you want to approach it by shipping with the default on? 
> 
> Or turn it off later in beta?
> You could try turning it off in beta 56 and ship a system addon to beta as
> well, to flip the prefs on.

I was thinking of leaving it default enabled and then shipping a system add-on to disable it. That way, we could simply remove the system add-on when we're ready to go 100%.

But I'm not very familiar with the system add-on process so if you think it would be easier to first make it default off on beta and then slowly enable it via the add-on, I can reverse the plan.
Flags: needinfo?(francois)
rehan, what do you think?
Flags: needinfo?(rdalal)
I think Felipe would also have some good insight for Francois, here.
Flags: needinfo?(felipc)
Sorry about the delayed response. I think either plan *should* work fine. 

The only concern I have is we are supposed to be shipping the tls13-middlebox addon to 5% of Beta 56.* users. Since we can't set different rates per addon we will need to do one of the following:

- Path 1:
  - Ship to 5% of users.
  - After some time, ship to 100% of users.

- Path 2:
  - Ship to 100% of users.

- Path 3:
  - Sync up with the tls13-middlebox folks and match rates of roll-out.

- Path 4:
  - Ship to 100% of users and handling sampling in the addon.

Hope that helps!

Let me know if any of that needs further explanation.
Flags: needinfo?(rdalal)
One good example to look it at is the flash clicktoplay rollout, which uses Path 4 that Rehan mentioned (ships to 100% and handle the sampling in the code): https://hg.mozilla.org/releases/mozilla-release/file/10a244c0f835/browser/extensions/clicktoplay-rollout/bootstrap.js

I can talk more about it in the meeting today
Flags: needinfo?(felipc)
Depends on: 1393954
No longer depends on: 1390698
I spoke with ekr about the tls13-middlebox addon. They cannot change the rate to match these rollout plans. However they have plans to stop shipping their addon by Friday of next week (Sept 1) at the latest, but probably on Wednesday. If this addon is not going to ship before then we should be fine to let Balrog handle the roll out, if not it will need to be baked in to the addon.
Depends on: 1393980
(In reply to Rehan Dalal [:rehan, :rdalal] from comment #10)
> However they have plans to stop shipping
> their addon by Friday of next week (Sept 1) at the latest, but probably on
> Wednesday. If this addon is not going to ship before then we should be fine
> to let Balrog handle the roll out, if not it will need to be baked in to the
> addon.

We can wait until Monday 4 Sep before shipping this add-on.
Here is the proposed roll-out plan (approved by Google):

2017-09-26: 10% on V4
2017-10-03: 50% on V4
2017-10-10: 100% on V4

I assume we could easily roll back (by removing the add-on) or adjust the numbers if something goes wrong and Google asks us to slow down the roll-out?
Priority: -- → P1
Summary: Staged rollout of Safe Browsing V4 to the release population using a system add-on → Staged rollout of Safe Browsing V4 to the release population using Shield
Whiteboard: #sbv4-m9 [go-faster-system-addon] → #sbv4-m9
Francois, do you have an update here? Did this ship for 56?
Flags: needinfo?(francois)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #13)
> Francois, do you have an update here? Did this ship for 56?

It's ready to go. The Shield add-on has been signed and uploaded to Normandy.

I assume we need to wait for the release to happen before starting the roll-out of the add-on? Or can we send it to the release56 channel already?
Flags: needinfo?(francois)
We should be shipping this Thursday morning. I'll just keep this tracked for 56 for now, to make sure we ship after the release is live. Thanks.
We can go ahead here with the rollout (We are not yet at 100% updates for 56, but that seems even better to give the servers a gradual rollout).
10% is live.

Should we shift the rollout back 4 days accordingly?
(In reply to Rob from comment #17)
> 10% is live.
> 
> Should we shift the rollout back 4 days accordingly?

Thanks Rob.

We can ignore the dates in comment 12 for now since the release was delayed.

I will liaise with Google and ping you when Google is ready for the next batch of users.
You may still want to shift the dates a little. If I enable updates at 100% tomorrow (Oct 3) for 56.0 on release, then there will be a lot of updates by Oct 4.
Flags: needinfo?(francois)
To be clear, the new rollout plan is:

2017-10-02: 10%
TBD: 50%
TBD: 100%

The two TBD dates will be determined by Google as they watch our update requests and ramp up capacity.
Flags: needinfo?(francois)
Google has approved bumping the # of users to 50% of release.

Rob, can we change the experiment target to 50%?
Flags: needinfo?(rrayborn)
Gladly. Liz? You ok with that change?
Flags: needinfo?(lhenry)
Sound good to me!
Flags: needinfo?(lhenry)
We are now live at 50% for Release
Flags: needinfo?(rrayborn)
Ready for 100?
Flags: needinfo?(francois)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #25)
> Ready for 100?

I asked Google this afternoon, but haven't heard back yet.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #25)
> Ready for 100?

Google has asked for more time to ensure they have enough capacity to handle the rest of our users.

ETA for 100%: 2017-10-30

I will confirm again with them once we are closer to the date.
Flags: needinfo?(francois)
Just checking in here. Are we ready to expand to 100%?
(In reply to Matt Grimes [:Matt_G] from comment #28)
> Just checking in here. Are we ready to expand to 100%?

I'm waiting on Google to confirm whether or not they are ready.
Flags: needinfo?(francois)
New ETA for 100%: early next week (week of 6 Nov)

Google wants to increase the capacity further to play it safe.
Flags: needinfo?(francois)
Thanks for the heads up. We're standing by for confirmation.
Google has approved 100% on V4. Let's do this!
Flags: needinfo?(mgrimes)
Update is queued. Waiting on sign off from Liz!
Flags: needinfo?(mgrimes) → needinfo?(lhenry)
Sounds good to me!
Flags: needinfo?(lhenry)
This is no live for 100% of the population. Thanks Liz!
Thanks Liz, Matt! I guess we are done here.

57 defaults to V4 and the Shield add-on will automatically uninstall itself once people upgrade their profile to 57.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Great job!  Thank you, everyone!  ^o^
Kamyar has disabled new enrollments into the Shield "study".
You need to log in before you can comment on or make changes to this bug.