1805700 - Some desktop devices disappear from send tab list

Status: RESOLVED FIXED

(Firefox :: Firefox Accounts, defect)

Description

[Tarik Eshaq [:teshaq]]

We are getting reports of some desktop devices disappearing from mobile send tab list after we filtered the list to only include active devices (devices active in the last 21 days)

See: https://github.com/mozilla-mobile/fenix/issues/28035

My theory is that since last access data for session tokens doesn't propagate well into SQL database, the latest data for the user's devices is stored in the redis store, but is kicked out of the redis store at a faster rate than 21 days.

However, the fact the some users need to go to their settings menu or sign out/sign in tells me that it's possible the redis store for the device data is not updated either.

The fix here is most likely a server fix, in an ideal world the last access timestamp would be written back, but that's infeasible given how many users we have.

A good solution would probably ensure that our redis store actually hold the information and are updated as often as we think they are (on every use of the API)

Comment 1

[Tarik Eshaq [:teshaq]]

Maybe Vijay or Dan, do any of your know off-hand how often the session token's last access date gets updated in the Redis store?

And a related question, how hard would it be to increase that frequency? Currently, for mobile and refresh tokens, the Redis store is updated on every request - but I'm unsure that's the right approach here for session tokens.

Maybe with /devices requests? I see us accessing the redis store there, maybe we can touch the current devices session token as well?)

Regardless I am careful to suggest anything else without knowing what the impact from increasing Redis store write throughput would be.

As a side note: I can't reproduce this myself - and I always assumed session token redis store got updated on every request 😅

Comment 2

[Tarik Eshaq [:teshaq]]

I also want to note that the reports so far are from Linux firefox desktops. Client wise, they should be the same - is possible the server discriminates here somehow? I'll keep my eye for reports from other platforms

Comment 3

[Barry Chen]

Maybe Vijay or Dan, do any of your know off-hand how often the session token's last access date gets updated in the Redis store?

I'm neither Vijay nor Dan, but I'll try to answer some of your questions. The session token's last access date gets updated whenever the client makes a request to /account/attached_clients on FxA's auth server.

And a related question, how hard would it be to increase that frequency? Currently, for mobile and refresh tokens, the Redis store is updated on every request - but I'm unsure that's the right approach here for session tokens.

I think there are two questions here: a) when to update the last access date, and b) can Redis handle the additional I/O? The first question is the hard one I believe...

Maybe with /devices requests? I see us accessing the redis store there, maybe we can touch the current devices session token as well?)

...yes, I think this is do-able. We'll need to check with the SREs about the additional Redis writes. But for device commands this is the most straightforward approach.

I also want to note that the reports so far are from Linux firefox desktops. Client wise, they should be the same - is possible the server discriminates here somehow? I'll keep my eye for reports from other platforms

It's likely because those older devices are using a session token, not an OAuth refresh token. I'd think a new sign-in from all the latest versions of Firefox would be using a refresh token, not a session token.

Comment 4

[Allan Laal]

can confirm this on firefox v109 on Linux

only my mobile phone is visible in the Send Tab to Device menu

on the same machine on Firefox ESR v91 and all other Linux laptops with there is a full menu with all of the devices

tried re-logging in to sync on v109, but nothing changed

Comment 5

[Tarik Eshaq [:teshaq]]

Hi Allan!

Thanks a ton for commenting! Have you tried re-logging into the other devices (i.e the devices that are missing from the list, not the device that has the list)

I would also like to confirm, it seems like all the devices that are missing are Linux devices, correct?

Comment 6

[Tarik Eshaq [:teshaq]]

It's likely because those older devices are using a session token, not an OAuth refresh token. I'd think a new sign-in from all the latest versions of Firefox would be using a refresh token, not a session token.

I'm not sure that's the case, from what I'm seeing in the FxA HTTP client in desktop it uses the session token still, I know for sure both Firefox Android and Firefox iOS do use refresh tokens though.

...yes, I think this is do-able. We'll need to check with the SREs about the additional Redis writes. But for device commands this is the most straightforward approach.

I'd be interested in checking in with SRE to see the impact of this - but I'd first like to accurately diagnose what I'm seeing with this bug, since it seems to only impact Linux devices and that's odd (and not all Linux devices, a person on my team has a Linux machine and couldn't reproduce)

Since the sessionToken last access date is updated only on attached_clients requests, some linux devices don't seem to be sending a attached_client request. I see a couple of uses of that API in searchfox - I'll see if I can track down if anything there could hint why some devices do not have an accurate sessionToken last access date

Comment 7

[Vijay Budhram]

Hey Tarik,

I agree it is a bit odd that this is only showing up for Linux devices. As you mentioned, the /account/devices endpoint[1] doesn't appear to update the lastAccessTime on the sessionToken. From this comment [2], I think we can safely update the last access time in this route. We can still check with SRE but I think it should be ok. I'll prep a PR for consideration.

[1] - https://github.com/mozilla/fxa/blob/613ad4c213637cf537f2d006c66ffeeb91086a43/packages/fxa-auth-server/lib/routes/devices-and-sessions.js#L581
[2] - https://github.com/mozilla/fxa/blob/442daf50a03cc1b20c1e21ed7c99f2d40c6bd7de/packages/fxa-auth-server/lib/db/index.js#L465

Comment 8

[Tarik Eshaq [:teshaq]]

Thanks a ton Vijay!

Updating the session token lastAccessTime on /devices endpoint will definitely help keep it (reasonably) up to date. There is also logic in the clients that makes sure we don't overwhelm the server of /devices endpoints. For example Desktop will keep a cache alive for 1 minute before talking to the server again, so that should also help in keeping the I/O down.

Please let me know if I can be of any help - in that PR or otherwise! In the meantime, I'll try to figure out why only Linux devices seem to be acting up

Comment 9

[Allan Laal]

(In reply to Tarik Eshaq from comment #5)

Thank You!
re-logging in from sync on every firefox Firefox FIXED the issue for me.

now all of them appear on v108 Send To menu

Comment 14

[Tarik Eshaq [:teshaq]]

For visibility, a server PR that fixes this bug landed in https://github.com/mozilla/fxa/pull/14742 and should rollout in the next firefox accounts release this week.

In the meantime, for anyone experiencing this, try logging out and back into the device that is missing (The linux device) and that should fix it. The device shouldn't disappear anymore after that unless you don't use it for over 21 days.

Comment 15

[Oliver]

(In reply to Tarik Eshaq from comment #2)

I also want to note that the reports so far are from Linux firefox desktops. Client wise, they should be the same - is possible the server discriminates here somehow? I'll keep my eye for reports from other platforms

I was seeing this with Fennec on Android which didn't seem to know about the Linux desktop Firefox instances anymore.

Fennec is:

109.1.1 (Build #1091120), 8f4899e33+
AC: 109.1.1, 1dd31ba60c
GV: 109.0-20230118234008
AS: 96.1.3

A Firefox on the same Android device shows the same symptoms:

109.1.1 (Build #2015927171), 8f4899e33+
AC: 109.1.1, 1dd31ba60c
GV: 109.0-20230112150232
AS: 96.1.3

One of the missing devices was a Ubuntu 20.04 with Firefox 109.0 (64-bit).

The trick described by user mans0n helped resolve it, however. I can now see the Linux desktop device again from the Android instances. The gist: log into https://accounts.firefox.com/ and "fiddle around" a bit and the problem may vanish.

Comment 16

[Oliver]

In the meantime, for anyone experiencing this, try logging out and back into the device that is missing (The linux device) and that should fix it. The device shouldn't disappear anymore after that unless you don't use it for over 21 days.

FYI, Tarik: this alone didn't resolve it for me (the Fennec instance in question was signed out last night and I signed back in, yet the issue persisted). But the trick from the GitHub issue I commented on did resolve it.

Comment 17

[Niels]

Several weeks ago or months I had this problem, that several devices disappeared from list of "Send tab to device". A user provided me with a link to this bugzilla page but myself I tried to logout the missing device and login again. One device is Firefox 109.0 (64 bit) on Ubuntu 22.04 64 bit (debian package, not snap because of keepassxc). Another device is with Firefox 109.0 (universal or 64 bit) on latest macOS (13.2 - macOS Ventura). Now both appear in list again.

So I confirm that a 'relogin' can solve this nasty problem. As I read, perhaps fiddling around in settings of firefox account (https://accounts.firefox.com/settings) may be needed, too, but I had not to do this.

Comment 19

[Tarik Eshaq [:teshaq]]

A server fix for this bug was rolled out yesterday that should keep the access date for linux devices reasonably up to date!

For anyone who used to experience this, you can verify if this is fixed by the following:

• Launch Firefox on your linux device at least once (no need to log out!)
• On your phone (or any other devices) verify that the linux device is now on your list. If you have another desktop device, you can even hover over the name of the device and it should show you when the device was last seen

I'll keep this open for a few more days to catch any other reports so please comment here if it's not fixed for you yet!

Thanks!

Comment 20

[holocronweaver]

(In reply to Tarik Eshaq from comment #19)

Confirmed fixed on Ubuntu 22.04, Firefox Android Beta 110.0b4!

Synced my desktop Firefox, force closed Firefox Android and re-opened, and lo and behold the desktop device showed up.

I'm guessing desktop FF Send Tab to Device does not filter devices in this way since its device list was never effected? I tend to lean towards sorting devices by tab sending frecency on Android rather than filtering. Example: when I need to send important travel tabs over to my laptop just before departing on a trip, I don't want to suddenly lose ability to send tabs to my laptop because I haven't used in awhile since I've been using my desktop while WFH. Perhaps there is a server-side optimization reason for filtering?

Comment 21

[Tarik Eshaq [:teshaq]]

There haven't been any more reports since the fix was deployed last week so I'll close this as Fixed. If anyone is still experiencing this try the above advice in comment 19:

For anyone who used to experience this, you can verify if this is fixed by the following:

• Launch Firefox on your linux device at least once (no need to log out!)
• On your phone (or any other devices) verify that the linux device is now on your list. If you have another desktop device, you can even hover over the name of the device and it should show you when the device was last seen

:holocronweaver thanks a ton for the comment and feedback! It describes a very useful use case for sending tabs (Sending tabs to other devices for long-term safekeeping). We can do better for that use case. In the meantime, we suggest using bookmarks for preserving important sites as those also sync and we try not to trim them, regardless of any activity.

Comment 22

[Gian-Carlo Pascutto [:gcp]]

Tarik, has it been root caused why only Linux devices were affected by this bug? (As per https://bugzilla.mozilla.org/show_bug.cgi?id=1805700#c8).

Comment 23

[Tarik Eshaq [:teshaq]]

Hi :gcp!

Partially, the main issue was that the timestamp we used for activity was only updated on /attached_clients requests, and that's not a request that's predictably triggered by the browser on regular bases (as opposed to the /devices endpoint that devices trigger when syncing).

As for why it was only linux affected, The browser only sends those requests in a UITour event and deeply nested in ASRouterTargeting. The UITour case isn't triggered anywhere I could find.

The ASRouterTargeting case is triggered when ASRouterTargeting is initialized and the stack goes all the way up to A startup idle task. My suspicion is somehow that code path doesn't execute the same way in Linux as other platforms, but I haven't had time to dig further there.

The fact the timestamp was only updated on /attached_clients requests was the bug fixed here, so the access dates correctly reflect user's activity regardless of platform