Closed Bug 1484465 Opened Last year Closed Last year

Re-enable FxA commands pref by default

Categories

(Firefox :: Sync, enhancement)

enhancement
Not set

Tracking

()

RESOLVED FIXED
Firefox 64
Tracking Status
firefox64 --- fixed

People

(Reporter: eoger, Assigned: rfkelly)

References

Details

Attachments

(1 file)

Depends on https://github.com/mozilla/fxa-auth-db-mysql/issues/384.
Re-enabling the pref should be enough to force clients to re-register their device. [0]


[0] https://searchfox.org/mozilla-central/source/services/fxaccounts/FxAccounts.jsm#723
The fix for the server issue will be going live in FxA train-119, tracked in Bug 1484995.

We'll need to QA the enablement procedure again.  I've done the first step, which is connecting two devices to stage, flipping `identity.fxaccounts.commands.enabled` to true, and confirming that they can still send tabs to each other by falling back to the old sync-clients-collection mechanism.

From here we'll need to do the following:

1. Get FxA train-119 deployed to prod with `pushbox.enabled` configured to `false`.
2. Re-deploy FxA train-119 to stage with `pushbox.enabled` flipped to `true`.
3. Check that two devices connected to stage can send tabs to each other, and that they go via the new pushbox system.
4. Run a bit of a loadtest in stage to simulate device registrations and confirm performance issue has been fixed.
5. Re-deploy FxA train-119 to prod with `pushbox.enabled` flipped to `true`.
6. Repeat (3) for two devices connected to production.
7. Land a patch to flip `identity.fxaccounts.commands.enabled` to true by default in desktop.
8. Carefully monitor FxA server load over the following few hours/days to ensure it's behaving as expected, rolling back the deploy from (5) if it isn't.

:jrgm does this sound like a good plan?  Any thoughts on the specifics of step (4) from your perspective?
Depends on: 1484995
Flags: needinfo?(jrgm)
Update on data for point (4):

> $ for f in sessionTokens accounts emails devices deviceCommands; do echo -n "$f "; mysql fxa -NBe "select count(*) from $f"; done
> sessionTokens 902285
> accounts 2841799
> emails 2848980
> devices 10883449
> deviceCommands 10035
> 
> MySQL [fxa]> source ~/query-15.sql
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> | id | select_type | table | type   | possible_keys                     | key        | key_len | ref                | rows | Extra       |
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> |  1 | SIMPLE      | t     | const  | PRIMARY                           | PRIMARY    | 32      | const              |    1 | NULL        |
> |  1 | SIMPLE      | a     | const  | PRIMARY                           | PRIMARY    | 16      | const              |    1 | NULL        |
> |  1 | SIMPLE      | e     | ref    | emails_uid                        | emails_uid | 16      | const              |    1 | Using where |
> |  1 | SIMPLE      | d     | ref    | PRIMARY,UQ_devices_sessionTokenId | PRIMARY    | 16      | const              |    1 | Using where |
> |  1 | SIMPLE      | dc    | ref    | PRIMARY                           | PRIMARY    | 32      | fxa.d.uid,fxa.d.id |    1 | NULL        |
> |  1 | SIMPLE      | ci    | eq_ref | PRIMARY                           | PRIMARY    | 4       | fxa.dc.commandId   |    1 | NULL        |
> |  1 | SIMPLE      | ut    | const  | PRIMARY                           | PRIMARY    | 32      | const              |    1 | Using where |
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> 7 rows in set (0.00 sec)
> 
> MySQL [fxa]> source ~/query-14.sql
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> | id | select_type | table | type  | possible_keys                     | key         | key_len | ref   | rows  | Extra                                              |
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> |  1 | SIMPLE      | t     | const | PRIMARY                           | PRIMARY     | 32      | const |     1 | NULL                                               |
> |  1 | SIMPLE      | a     | const | PRIMARY                           | PRIMARY     | 16      | const |     1 | NULL                                               |
> |  1 | SIMPLE      | e     | ref   | emails_uid                        | emails_uid  | 16      | const |     1 | Using where                                        |
> |  1 | SIMPLE      | d     | ref   | PRIMARY,UQ_devices_sessionTokenId | PRIMARY     | 16      | const |     1 | Using where                                        |
> |  1 | SIMPLE      | ci    | index | PRIMARY                           | commandName | 766     | NULL  |     1 | Using index; Using join buffer (Block Nested Loop) |
> |  1 | SIMPLE      | dc    | ALL   | NULL                              | NULL        | NULL    | NULL  | 10125 | Using where; Using join buffer (Block Nested Loop) |
> |  1 | SIMPLE      | ut    | const | PRIMARY                           | PRIMARY     | 32      | const |     1 | Using where                                        |
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> 7 rows in set (0.01 sec)
> 
> So, same, sub-optimal join with sessionWithDevice_14, but not in sessionWithDevice_15.
> 
> 
> I did back-to-back-to-back extended runs, first with 10K rows in deviceCommands (CPU from RDS metrics:
> 
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  250.24s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  295.81s MySQL CPU use: ~95%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  241.69s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  285.60s MySQL CPU use: ~99%
> 
> Then with an empty deviceCommands table:
> 
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  224.73s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  226.32s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  221.57s MySQL CPU use: ~5%%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  224.30s MySQL CPU use: ~5%
> 
> (Note: the overall declining wall clock time correlates with steadily lower DB Read IO over the test runs)
Flags: needinfo?(jrgm)
Fantastic, thanks :jrgm!
I have the auth 119 server in stage with pushbox enabled and confirmed pushbox usage (i.e., auth logs say: "pushbox.retrieve.response" on sync with two browsers enabled with identity.fxaccounts.commands.enabled true, and synced tabs sidebar showing each other's tabs). Also, one additional row in deviceCommands now.

So, should I put this live now, or tomorrow morning?
I'm able to send tabs between two desktop profiles connected to stage, and can confirm that they're going via the new pushbox infra.  So I think this looks good to go live whenever you're ready :jrgm.

Let's wait until next week to do (7) and start enabling the rollout to Nightly profiles though, we don't need another hiccup over a weekend :-)
auth-server 119 in production is now 100% enabled for push.
OK, so for anyone wanting to test this out, flipping `identity.fxaccounts.commands.enabled` to true in Desktop should now cause your device to (attempt to) send and receive tabs via the new pushbox infra.  I'm going to go ahead and flip it in my profile and do a couple of tests, and we can look at enabling it by default next week.
:rfkelly, :jrgm, Thank you!! 🙌🏻
We're ready to try this again on the server side, I'm going to put up the patch today so that we can land it early in the week, giving us a few work days to keep an eye on things as it rolls out into Nightly (the client-side equivalent of "don't deploy on a Friday").
This switches send-tab to use the new FxA "device commands" feature
by default, rather than the old sync client commands channel.
Assignee: nobody → rfkelly
Comment on attachment 9006122 [details]
Bug 1484465 - Enable FxA Commands feature by default.

Vlad Filippov (:vladikoff) has approved the revision.
Attachment #9006122 - Flags: review+
Keywords: checkin-needed
Pushed by dluca@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3115b98972da
Enable FxA Commands feature by default. r=vladikoff
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/3115b98972da
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → Firefox 64
You need to log in before you can comment on or make changes to this bug.