1752609 - Firefox on my desktop computers can send tabs, but can't receive sent tabs

Status: RESOLVED FIXED

(Firefox :: Sync, defect)

Description

[Chris Peterson [:cpeterson]]

A couple weeks ago, I started having problems sending tabs between some Firefox instances. My Firefox Sync account links a Windows laptop, a MacBook Pro, two Android devices, and one iPhone. I have both regular Firefox 96.0 (7403) and TestFlight Firefox Beta 96.0 (7278) installed. I am running Firefox Nightly 98 on my desktop computers.

• I can successfully send tabs back and forth between my iPhone and Android devices.
• I can successfully send tabs from my desktop computers (Windows laptop and MacBook Pro) to my mobile devices.
• When I try to send a tab from a mobile device to a desktop computer, the tab appears to have sent (Firefox for iOS shows the "Tab Sent" toast message) but the tab never arrives at the destination desktop computer.
• Tabs sent from one desktop computer to another never arrive either.

On my desktop computers, Firefox's "Synced Tabs" list can see current list of open tabs on my mobile devices, so I know that some Sync state from my mobile devices is successfully reaching my desktop computers, just not any sent tabs.

Comment 1

[Mark Hammond [:markh] [:mhammond]]

This is very interesting - I've not come across this kind of failure on desktop. What I have seen is push notifications failing to arrive, but never the tab failing to arrive after some delay. At least once per day, and every time "sync now" is pressed, we explicitly check for tabs that we didn't get a push notification for.

Regarding logs: On the sending device, as soon as a tab is sent we create a new sync log file, which should be visible via about:sync-log. "success" logs are captured by default on Nightly, so it would be great if you can attach such a log. Specifically, we should see something like:

1643601519396 Sync.Browser INFO Sending a tab to 7941d7f16911de1c0b861532cc362ce8 using FxA commands.
1643601519396 FirefoxAccounts INFO Sending a tab to 1 devices.
...
1643601519687 FirefoxAccounts INFO Successfully sent: {"enqueued":true,"notified":true}

This is telling us that we both submitted the sent tab to the server and it sent a push notification.

On the receiving device, even if the push fails, we should see:

1643602100935 FirefoxAccounts INFO Polling device commands.

soon after startup. It would be great if you could attach the relevant logs from each of the desktop devices that shows these entries.

(A bit of an unfun fact, is that sending a tab doesn't actually use any sync infrastructure at all (although it does use the sync keys for encryption) - instead it uses completely different infrastructure, so the feature works even if Sync is disabled. I'll need to bring in someone from the server team if the client logs don't offer any clues.)

Comment 2

[Chris Peterson [:cpeterson]]

Attachment: 1_Sender_log_success-sync-1643670405286.txt

Here are some logs from my about:sync-log.

When I send a tab from my Windows laptop, two log files are generated.

Comment 3

[Chris Peterson [:cpeterson]]

Attachment: 2_Sender_log_success-sync-1643670415862.txt

Comment 4

[Chris Peterson [:cpeterson]]

Attachment: 3_Receiver_log_error-sync-1643670428587.txt

And one error log on the receiver computer. This error in the receiver log looks suspicious:

1643670415652 FirefoxAccounts ERROR Error while handling incoming Send Tab payload.: TypeError: undefined has no properties(resource://gre/modules/FxAccountsCommands.js:357:20) JS Stack trace: [e-mail]:357:20

The error was logged from here:

https://searchfox.org/mozilla-central/rev/09819cb9d36b7052d715cbfe392216dcb5c0709e/services/fxaccounts/FxAccountsCommands.js#357

Comment 5

[Mark Hammond [:markh] [:mhammond]]

Thanks Chris,
I suspect this is a regression from bug 1738898.

Comment 6

[Chris Peterson [:cpeterson]]

(In reply to Mark Hammond [:markh] [:mhammond] from comment #5)

I suspect this is a regression from bug 1738898.

FWIW, I don't use a Firefox primary password.

Comment 7

[Tarik Eshaq [:teshaq]]

Thanks a ton for the logs and for filing this Chris!! I'll be trying to reproduce this - to help narrow it down, do happen to have changed your firefox account password recently? (within the past 2/3 months)

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Tarik Eshaq from comment #7)

Thanks a ton for the logs and for filing this Chris!! I'll be trying to reproduce this - to help narrow it down

Thanks! I'm happy to provide any logs or test any debug builds or settings. Sending tabs is one of my favorite Firefox features, so I'm very motivated to help. :)

do happen to have changed your firefox account password recently? (within the past 2/3 months)

No. I've been using the same Firefox Account password for over a year.

The timing might be purely coincidental, but I first noticed this tab sending problem when setting up my new work laptop. I had Sync configured on my old work laptop, my new work laptop, and my personal computer. My new work laptop temporarily used the same machine name as my old work laptop (which I then renamed). My work laptops' FxA Device Names were also similar (and long strings). At the time, I wondered whether that similarity might have caused some problems, so I tried changing all my desktop profile names but that had no effect.

I did not copy my Firefox profile directory from my old laptop to my new laptop. I used Sync to synchronize my old bookmarks and open tabs to my new computer's profile. So there shouldn't be any local profile state or IDs that are duplicated on the different devices.

Comment 9

[Tarik Eshaq [:teshaq]]

Thanks a lot Chris! I'm looking into it, I can't seem to reproduce with a fresh firefox build, but I'll experiment more with using a computer that doesn't already have any Firefox installed

I'll add more information here as I go, but given the logs it seems that the device doesn't have its send-tab keys persisted. I have a theory, but hard to validate without reproducing: We persist two pieces of information regarding your sent-tab keys,

• The keys themselves
• some ciphertext, which is your public key encrypted using your sync keys (this is eventually sent to fxa so other devices have it)

The latter is what we usually check for, if it exists, we assume that the keys are also there. If not, we will check if the keys are there - and possibly generate them. My theory is that you're in a state where the encrypted keys are there, but not the keys themselves.

I have a protective patch in mind that might fix this, so I'll start with that and investigate how this could have happened in the first place

Comment 10

[Tarik Eshaq [:teshaq]]

Attachment: Bug 1752609: Recover if persisted send tab keys are lost. r=markh

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1cb395e11d5
Recover if persisted send tab keys are lost. r=markh

Comment 13

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fad7bf499945
Backed out changeset f1cb395e11d5 for causing build bustages on FuzzingFunctions.cpp. CLOSED TREE

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/159237dacf20
Recover if persisted send tab keys are lost. r=markh

Comment 15

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/159237dacf20

Comment 16

[Tarik Eshaq [:teshaq]]

Thank you for your patience Chris! Once you update to a version of Nightly with the patch I landed last night, try sending a few tabs, the first might fail, but it should recover you from the state you're in. The patch also tries to ensure that you never hit this state again!

I filed Bug 1753566 for some clean-ups to make sure bugs like this don't show up again

Comment 17

[Chris Peterson [:cpeterson]]

(In reply to Tarik Eshaq from comment #16)

Thank you for your patience Chris! Once you update to a version of Nightly with the patch I landed last night, try sending a few tabs, the first might fail, but it should recover you from the state you're in. The patch also tries to ensure that you never hit this state again!

Unfortunately, I am still experiencing the bug with today's Nightly build 98.0a1 (2022-02-05). I still can't send tabs to either of my desktop computers (from my Android or iPhone or from the other desktop computer).

Should I reopen this bug or file a new one?

Comment 18

[Tarik Eshaq [:teshaq]]

I'll re-open this one - So sorry this didn't fix your problem.

This is odd, and it would be incredibly helpful if you can try sending a few more tabs and upload your receiver error log again - I'll take another look!

Comment 19

[Tarik Eshaq [:teshaq]]

I reproduced the state you were in a local build (by adding code to delete the send tab keys) - then I applied my fix. It didn't work right away, but once I signed out, and back into firefox I was able to send tabs again.

Let me know if signing out and back into your firefox account doesn't work for you and I can explore more!

Comment 20

[Mark Hammond [:markh] [:mhammond]]

(In reply to Tarik Eshaq from comment #19)

Let me know if signing out and back into your firefox account doesn't work for you and I can explore more!

Thanks Tarik - but why is this necessary? I don't think we can live with send tab being silently broken until the user signs out.

Comment 21

[Tarik Eshaq [:teshaq]]

(In reply to Mark Hammond [:markh] [:mhammond] from comment #20)

(In reply to Tarik Eshaq from comment #19)

Let me know if signing out and back into your firefox account doesn't work for you and I can explore more!

Thanks Tarik - but why is this necessary? I don't think we can live with send tab being silently broken until the user signs out.

Absolutely, I'm trying to understand it better myself - it seems that the FxA (and thus the sending device) isn't getting the updated sendtab keys.

We end up in a state where the receiver has recovered, but the keys the receiver has can't decrypt the bundle as sent by the sender (since the sender uses the outdated keys)

I'm trying to come up with a patch that forces the keys to be updated right away (more accurately get the device record updated since https://searchfox.org/mozilla-central/source/services/fxaccounts/FxAccountsDevice.jsm#339-356 wouldn't trigger an update if the keys themselves change). Incrementing the device registration number might work, but I need to make sure I understand the implications there. Either ways, I agree we should make sure sendtab can automatically recover from this state!

Comment 22

[Chris Peterson [:cpeterson]]

(In reply to Tarik Eshaq from comment #19)

I reproduced the state you were in a local build (by adding code to delete the send tab keys) - then I applied my fix. It didn't work right away, but once I signed out, and back into firefox I was able to send tabs again.

Let me know if signing out and back into your firefox account doesn't work for you and I can explore more!

Sending and receiving tabs works after I logged out of Sync and back in on my desktop computer.

But now that it's working again for me, I can't capture a useful sync log when reproducing the error. Is there any other information I can provide to help you debug?

Comment 23

[Tarik Eshaq [:teshaq]]

But now that it's working again for me, I can't capture a useful sync log when reproducing the error. Is there any other information I can provide to help you debug?

Happy that it works for you!! and nope, you've already helped us debug more than we could ever ask for, so thanks a ton!!! (and I've luckily been able to reproduce this, albeit by a hack, but it's good enough to help me understand it better)

we'll probably keep this bug open as we figure out a fix that doesn't involve users signing back in, but with the patch that's already landed, you should never (famous last words) end up in this state again

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:teshaq, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 25

[Tarik Eshaq [:teshaq]]

Attachment: Bug 1752609 - Adds autorecovery from outdated sendtab keys. r=markh

Comment 26

[9w3p2mtay]

I experienced the same and at first glance it seems that logging out and in on desktop did the trick
All apps are at the latest current versions for macOS and android

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6cad439567d8
Adds autorecovery from outdated sendtab keys. r=markh

Comment 28

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/6cad439567d8