1756222 - voice.google.com calls work in firefox 95 but broken in firefox 98 (unclear about 97) and nightly

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[gregrwm]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Launch nightly (currently 99.0a1)(2022-02-17)(64-bit), login to voice.google.com, make a call.

Actual results:

No sound heard during call, although the "hangup sound" plays normally.

Expected results:

Still works properly in firefox eg 95.0 (64-bit). Worked properly recently in nightly, within the last couple weeks at most, probably even more recently. Nightly99 lack of in-call sound and firefox95 working fine both verified both in updated manjaro install and in manjaro-kde-21.2rc1-minimal-211211-linux515.iso live system.

Comment 1

[gregrwm]

manjaro just updated to firefox 97 and that's broken too!

Still works if i bring up the manjaro-kde-21.2rc1-minimal-211211-linux515.iso live system and run the firefox 95 on there, but that's not very convenient! Now i wonder how to go about finding and installing firefox 95.

Comment 2

[Marco Castelluccio [:marco]]

You can use mozregression to pinpoint the change that caused this bug.

Comment 3

[gregrwm]

28:02.76 INFO: Running autoland build built on 2022-02-16 00:18:37.110000, revision a0a3a33a
28:22.55 INFO: Launching /tmp/tmp10s33mxt/firefox/firefox
28:22.55 INFO: Application command: /tmp/tmp10s33mxt/firefox/firefox -profile /tmp/tmpyui8m2vx.mozrunner
28:22.57 INFO: application_buildid: 20220216000426
28:22.57 INFO: application_changeset: a0a3a33a43e049eafeaa1046d90c250948685628
28:22.57 INFO: application_name: Firefox
28:22.57 INFO: application_repository: https://hg.mozilla.org/integration/autoland
28:22.57 INFO: application_version: 99.0a1
Was this integration build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry', 'back' or 'exit' and press Enter): b
29:27.73 INFO: Narrowed integration regression window from [3404388b, 2143556e] (4 builds) to [3404388b, a0a3a33a] (3 builds) (~1 steps left)
29:27.73 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3404388bdfa118a667a71a1d7293d4d4aa91416d&tochange=a0a3a33a43e049eafeaa1046d90c250948685628

29:27.73 INFO: Using local file: /tmp/tmpyq2bivy5/5e45ff1e2074-pgo--autoland--target.tar.bz2 (downloaded in background)
29:27.73 INFO: Running autoland build built on 2022-02-16 00:17:06.426000, revision 5e45ff1e
29:47.91 INFO: Launching /tmp/tmp_l9lsh7j/firefox/firefox
29:47.91 INFO: Application command: /tmp/tmp_l9lsh7j/firefox/firefox -profile /tmp/tmp8mc5409d.mozrunner
29:47.92 INFO: application_buildid: 20220215235135
29:47.92 INFO: application_changeset: 5e45ff1e2074b4d905e7dea2f7759d5d65f54627
29:47.92 INFO: application_name: Firefox
29:47.92 INFO: application_repository: https://hg.mozilla.org/integration/autoland
29:47.92 INFO: application_version: 99.0a1
Was this integration build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry', 'back' or 'exit' and press Enter): g
30:51.78 INFO: Narrowed integration regression window from [3404388b, a0a3a33a] (3 builds) to [5e45ff1e, a0a3a33a] (2 builds) (~1 steps left)
30:51.78 INFO: No more integration revisions, bisection finished.
30:51.78 INFO: Last good revision: 5e45ff1e2074b4d905e7dea2f7759d5d65f54627
30:51.78 INFO: First bad revision: a0a3a33a43e049eafeaa1046d90c250948685628
30:51.78 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5e45ff1e2074b4d905e7dea2f7759d5d65f54627&tochange=a0a3a33a43e049eafeaa1046d90c250948685628

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1754027

Comment 5

[Pascal Chevrel:pascalc (PTO until April 26)]

Tracking for 98 as this is a regression from an uplift that was fixing a Jitsi regression. We are building out Release Candidate today, is that bug something we need to fix in 98 before we ship? Backing out bug 1754027 would reintroduce the regression we had a fix for, which of the two regressions is impacting more users?

Comment 6

[Andreas Pehrson [:pehrsons]]

Comment 1 suggests 97 is broken too. If there is an intermittent factor at play here I would expect mozregression to not pinpoint something relevant so accurately. This seems odd, at least. Reporter, could you verify the state of 97 once again?

If 97 is broken in some way and bug 1754027 is part of this I wonder if we need the UnsetRemoteSSRC logic for AudioConduit in order to fix this properly.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I've confirmed the regression range on macOS. It points to bug 1754027 as the sole reason for me:

21:16.38 INFO: Narrowed integration regression window from [3404388b, a0a3a33a] (3 builds) to [5e45ff1e, a0a3a33a] (2 builds) (~1 steps left)
21:16.38 INFO: No more integration revisions, bisection finished.
21:16.38 INFO: Last good revision: 5e45ff1e2074b4d905e7dea2f7759d5d65f54627
21:16.38 INFO: First bad revision: a0a3a33a43e049eafeaa1046d90c250948685628
21:16.38 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5e45ff1e2074b4d905e7dea2f7759d5d65f54627&tochange=a0a3a33a43e049eafeaa1046d90c250948685628

97.0.1 (64-bit) worked fine the several times I tried it. So I suggest we proceed based on that until the reporter has reconfirmed comment 1.

No sound heard during call, although the "hangup sound" plays normally.

It's reception of sound on the desktop side that's broken. The phone side is receiving audio fine from the desktop mic (so no problem sending from Firefox). I.e. everything's totally quiet desktop side until the phone side hangs up and you get the (loud since I have the desktop cranked up just in case) toodle-loo hangout sound.

Byron, you reviewed bug 1754027. What do we do here? We should probably back this out no?

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

ccing Nils as well who wrote the patch.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Marking this as serious since it breaks audio calls in Google Voice with no apparent workaround

Comment 10

[Pascal Chevrel:pascalc (PTO until April 26)]

I am marking this bug as blocking 98 as a decision is needed here to start build our Release Candidate.

It seems to me that the Jitsi problem is less a problem for users than the Google Voice problem as the Jitsi ticket says "sometimes a random user would not receive audio and/or video from another random user" while the breakage for Google Voice seems to be more a consistent pattern. Furthermore we already shipped 97 with the Jitsi bug.

Jan-Ivar, can you confirm that 98 beta 7 (https://ftp.mozilla.org/pub/firefox/releases/98.0b7/) which doesn't have the patch from bug 1754027 works with Google Voice? Thanks

Comment 11

[Pascal Chevrel:pascalc (PTO until April 26)]

Removing the NI I had set to Jan-Ivar, we decided to back out bug 1754027 from beta to unlock the release.

Comment 12

[Pascal Chevrel:pascalc (PTO until April 26)]

https://hg.mozilla.org/releases/mozilla-release/rev/97dfb71c2282
Backed out changeset 974fb4e6468c (bug 1754027) for breaking Google Voice on beta (bug 1756222)

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Pascal Chevrel:pascalc from comment #10)

Jan-Ivar, can you confirm that 98 beta 7 (https://ftp.mozilla.org/pub/firefox/releases/98.0b7/) which doesn't have the patch from bug 1754027 works with Google Voice? Thanks

Confirmed. I've also confirmed that the issue affects windows as well and that 97 works there.

Comment 14

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: about webrtc SDP from google voice on 99

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: about webrtc SDP from google voice on 97 (working)

Comment 16

[Byron Campen [:bwc]]

Oh wow. No mid, no ssrc attributes. That's crazy. There's quite a few things we could do to detect the need for ssrc switching logic.

Comment 17

[Byron Campen [:bwc]]

Nils, if we tweak your fix to allow ssrc switching if no a=ssrc attributes are present in the SDP, will that allow your fix to work for jitsi?

Comment 18

[Byron Campen [:bwc]]

^

Comment 19

[Byron Campen [:bwc]]

Attachment: Bug 1756222: Do not disable SSRC switching if there were no a=ssrc attributes in the remote SDP. r?mjf

Comment 20

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/jobs?repo=try&revision=40448fd18b8be8770fffe1ffb4db96aed4b9d142
https://treeherder.mozilla.org/jobs?repo=try&revision=d4b8ebc5801861f4a5c5d0adaeeda36b1c9c78ea

Try looks good. Waiting on confirmation from drno.

Comment 21

[gregrwm]

Running my installed manjaro system:
Nightly(99.0a1 2022-03-05 64-bit) remains silent during calls, but plays the 'hangup sound' just fine.
Mozilla Firefox(97.0.1 64-bit) for Manjaro Linux Manjaro- 1.0 is strangely unable to play any sound at all.
Mozregression remains able to bring up both fully working builds, and builds that remain silent during calls, but play the hangup sound just fine.

Comment 22

[gregrwm]

ie perhaps you're focusing on beta, but nightly is still broken.

Comment 23

[Nils Ohlmeier [:drno]]

Sorry for breaking things and for the late reply.

Jitsi signaling should always have SSRCs in it. So that should work fine.
Is there already a build I can use for testing?

Comment 24

[Donal Meehan [:dmeehan]]

Is there anything preventing landing the patch for this?
Once ready, please request a Beta uplift. 99 is now in beta as this is marked as a release blocker.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9fc87b983c55
Do not disable SSRC switching if there were no a=ssrc attributes in the remote SDP. r=mjf

Comment 26

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9fc87b983c55

Comment 27

[gregrwm]

nightly works now thank you!

Comment 28

[Byron Campen [:bwc]]

Comment on attachment 9266348 [details]
Bug 1756222: Do not disable SSRC switching if there were no a=ssrc attributes in the remote SDP. r?mjf

Beta/Release Uplift Approval Request

• User impact if declined: voice.google.com will not work
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Simple fix, just moved some code inside a check.
• String changes made/needed: None

Comment 29

[Donal Meehan [:dmeehan]]

Comment on attachment 9266348 [details]
Bug 1756222: Do not disable SSRC switching if there were no a=ssrc attributes in the remote SDP. r?mjf

Approved for 99.0b4. Thanks.

Comment 30

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8e1d2359a41a

Comment 31

[Mihai Boldan, Desktop QA [:mboldan]]

It seems that this issue was fixed on all three builds, RC 98, Firefox 99.0b4 and Firefox 100.
Since we don't have a voice.google.com account, Greg could you please confirm that the issue is fixed on the builds mentioned above?

Comment 32

[gregrwm]

what's the easiest way to do that on manjaro, for example can mozregression be told to give me the builds you want checked?