1484465 - Re-enable FxA commands pref by default

Status: RESOLVED FIXED

(Firefox :: Sync, enhancement)

Description

[Edouard Oger [:eoger]]

Depends on https://github.com/mozilla/fxa-auth-db-mysql/issues/384.
Re-enabling the pref should be enough to force clients to re-register their device. [0]


[0] https://searchfox.org/mozilla-central/source/services/fxaccounts/FxAccounts.jsm#723

Comment 1

[Ryan Kelly [:rfkelly]]

The fix for the server issue will be going live in FxA train-119, tracked in Bug 1484995.

We'll need to QA the enablement procedure again.  I've done the first step, which is connecting two devices to stage, flipping `identity.fxaccounts.commands.enabled` to true, and confirming that they can still send tabs to each other by falling back to the old sync-clients-collection mechanism.

From here we'll need to do the following:

1. Get FxA train-119 deployed to prod with `pushbox.enabled` configured to `false`.
2. Re-deploy FxA train-119 to stage with `pushbox.enabled` flipped to `true`.
3. Check that two devices connected to stage can send tabs to each other, and that they go via the new pushbox system.
4. Run a bit of a loadtest in stage to simulate device registrations and confirm performance issue has been fixed.
5. Re-deploy FxA train-119 to prod with `pushbox.enabled` flipped to `true`.
6. Repeat (3) for two devices connected to production.
7. Land a patch to flip `identity.fxaccounts.commands.enabled` to true by default in desktop.
8. Carefully monitor FxA server load over the following few hours/days to ensure it's behaving as expected, rolling back the deploy from (5) if it isn't.

:jrgm does this sound like a good plan?  Any thoughts on the specifics of step (4) from your perspective?

Comment 2

[John Morrison [:jrgm]]

Update on data for point (4):

> $ for f in sessionTokens accounts emails devices deviceCommands; do echo -n "$f "; mysql fxa -NBe "select count(*) from $f"; done
> sessionTokens 902285
> accounts 2841799
> emails 2848980
> devices 10883449
> deviceCommands 10035
> 
> MySQL [fxa]> source ~/query-15.sql
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> | id | select_type | table | type   | possible_keys                     | key        | key_len | ref                | rows | Extra       |
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> |  1 | SIMPLE      | t     | const  | PRIMARY                           | PRIMARY    | 32      | const              |    1 | NULL        |
> |  1 | SIMPLE      | a     | const  | PRIMARY                           | PRIMARY    | 16      | const              |    1 | NULL        |
> |  1 | SIMPLE      | e     | ref    | emails_uid                        | emails_uid | 16      | const              |    1 | Using where |
> |  1 | SIMPLE      | d     | ref    | PRIMARY,UQ_devices_sessionTokenId | PRIMARY    | 16      | const              |    1 | Using where |
> |  1 | SIMPLE      | dc    | ref    | PRIMARY                           | PRIMARY    | 32      | fxa.d.uid,fxa.d.id |    1 | NULL        |
> |  1 | SIMPLE      | ci    | eq_ref | PRIMARY                           | PRIMARY    | 4       | fxa.dc.commandId   |    1 | NULL        |
> |  1 | SIMPLE      | ut    | const  | PRIMARY                           | PRIMARY    | 32      | const              |    1 | Using where |
> +----+-------------+-------+--------+-----------------------------------+------------+---------+--------------------+------+-------------+
> 7 rows in set (0.00 sec)
> 
> MySQL [fxa]> source ~/query-14.sql
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> | id | select_type | table | type  | possible_keys                     | key         | key_len | ref   | rows  | Extra                                              |
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> |  1 | SIMPLE      | t     | const | PRIMARY                           | PRIMARY     | 32      | const |     1 | NULL                                               |
> |  1 | SIMPLE      | a     | const | PRIMARY                           | PRIMARY     | 16      | const |     1 | NULL                                               |
> |  1 | SIMPLE      | e     | ref   | emails_uid                        | emails_uid  | 16      | const |     1 | Using where                                        |
> |  1 | SIMPLE      | d     | ref   | PRIMARY,UQ_devices_sessionTokenId | PRIMARY     | 16      | const |     1 | Using where                                        |
> |  1 | SIMPLE      | ci    | index | PRIMARY                           | commandName | 766     | NULL  |     1 | Using index; Using join buffer (Block Nested Loop) |
> |  1 | SIMPLE      | dc    | ALL   | NULL                              | NULL        | NULL    | NULL  | 10125 | Using where; Using join buffer (Block Nested Loop) |
> |  1 | SIMPLE      | ut    | const | PRIMARY                           | PRIMARY     | 32      | const |     1 | Using where                                        |
> +----+-------------+-------+-------+-----------------------------------+-------------+---------+-------+-------+----------------------------------------------------+
> 7 rows in set (0.01 sec)
> 
> So, same, sub-optimal join with sessionWithDevice_14, but not in sessionWithDevice_15.
> 
> 
> I did back-to-back-to-back extended runs, first with 10K rows in deviceCommands (CPU from RDS metrics:
> 
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  250.24s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  295.81s MySQL CPU use: ~95%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  241.69s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  285.60s MySQL CPU use: ~99%
> 
> Then with an empty deviceCommands table:
> 
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  224.73s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  226.32s MySQL CPU use: ~5%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_15 Time elapsed:  221.57s MySQL CPU use: ~5%%
> Trying 20 concurrent runs of 10000 calls to sessionWithDevice_14 Time elapsed:  224.30s MySQL CPU use: ~5%
> 
> (Note: the overall declining wall clock time correlates with steadily lower DB Read IO over the test runs)

Comment 3

[Ryan Kelly [:rfkelly]]

Fantastic, thanks :jrgm!

Comment 4

[John Morrison [:jrgm]]

I have the auth 119 server in stage with pushbox enabled and confirmed pushbox usage (i.e., auth logs say: "pushbox.retrieve.response" on sync with two browsers enabled with identity.fxaccounts.commands.enabled true, and synced tabs sidebar showing each other's tabs). Also, one additional row in deviceCommands now.

So, should I put this live now, or tomorrow morning?

Comment 5

[Ryan Kelly [:rfkelly]]

I'm able to send tabs between two desktop profiles connected to stage, and can confirm that they're going via the new pushbox infra.  So I think this looks good to go live whenever you're ready :jrgm.

Let's wait until next week to do (7) and start enabling the rollout to Nightly profiles though, we don't need another hiccup over a weekend :-)

Comment 6

[John Morrison [:jrgm]]

auth-server 119 in production is now 100% enabled for push.

Comment 7

[Ryan Kelly [:rfkelly]]

OK, so for anyone wanting to test this out, flipping `identity.fxaccounts.commands.enabled` to true in Desktop should now cause your device to (attempt to) send and receive tabs via the new pushbox infra.  I'm going to go ahead and flip it in my profile and do a couple of tests, and we can look at enabling it by default next week.

Comment 8

[Vlad Filippov (:vladikoff)]

:rfkelly, :jrgm, Thank you!! 🙌🏻

Comment 9

[Ryan Kelly [:rfkelly]]

We're ready to try this again on the server side, I'm going to put up the patch today so that we can land it early in the week, giving us a few work days to keep an eye on things as it rolls out into Nightly (the client-side equivalent of "don't deploy on a Friday").

Comment 10

[Ryan Kelly [:rfkelly]]

Attachment: Bug 1484465 - Enable FxA Commands feature by default.

This switches send-tab to use the new FxA "device commands" feature
by default, rather than the old sync client commands channel.

Comment 11

[Vlad Filippov (:vladikoff)]

Comment on attachment 9006122 [details]
Bug 1484465 - Enable FxA Commands feature by default.

Vlad Filippov (:vladikoff) has approved the revision.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3115b98972da
Enable FxA Commands feature by default. r=vladikoff

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/3115b98972da