1608505 - whereby.com and appr.tc broken, no audio/video, and browser responsiveness extremely poor

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Byron Campen [:bwc]]

I have only tested this on OS X, but it may be a problem on other platforms.

STR: Open two tabs to whereby.com (same room). Observe that neither end receives audio or video. Also observe that browser janks really badly (multi-second).

Edit STR: Seems to happen with any whereby call, whether tab-to-tab or with another browser.

Output from mozregression:

8:18.16 INFO: No more inbound revisions, bisection finished.
8:18.16 INFO: Last good revision: 6ad8baec4ca949e4ede0e79a0a2219cdc10f8bfd
8:18.16 INFO: First bad revision: cb00f09b36fc8ded65b2bf0ca1a86a669f8f0a4a
8:18.16 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=6ad8baec4ca949e4ede0e79a0a2219cdc10f8bfd&tochange=cb00f09b36fc8ded65b2bf0ca1a86a669f8f0a4a

Looks like a regression from bug 1586370, or perhaps bug 1542321.

Bug 1607914 is likely a dupe.

Needinfoing padenot, since pehrsons is out.

Comment 1

[Byron Campen [:bwc]]

Here's a gecko profile: https://perfht.ml/2R5uz88

Comment 2

[Byron Campen [:bwc]]

appr.tc seems to be similarly broken. Here's a profile for that: https://perfht.ml/30a4lpd

Comment 3

[Dan Minor [:dminor]]

This seems severe enough to be a P1, which needs an owner. Paul, once you've had a chance to investigate, please feel free to reset the priority if things aren't as bad as they look.

Comment 4

[Byron Campen [:bwc]]

Profile of a whereby call with Chrome (not tab-to-tab, so a real world use-case): https://perfht.ml/2QGtG6Z

Same symptoms.

Comment 5

[Dan Minor [:dminor]]

[Tracking Requested - why for this release]: Appears to be a significant performance regression affecting multiple sites.

Comment 6

[Paul Adenot (:padenot)]

I can't repro on OSX and Linux, this is as smooth as ever. Byron, do you have anything specific in you audio setup that would cause this?

Comment 7

[Byron Campen [:bwc]]

This testing was carried out with the mic/speakers that are integrated into the laptop. Also happens with a pristine profile, otherwise mozregression would not have found the problem.

I did install Krisp a while back to see if it helped with audio call noise, but I've had it disabled for a while now. Maybe it is getting in the way somehow? Let me try uninstalling that.

Comment 8

[Byron Campen [:bwc]]

I can no longer reproduce. I've also restarted my laptop since the last time I tried to test this, so maybe that's what made the difference, and Krisp was a red herring. Let me try reinstalling it and see.

Comment 9

[Byron Campen [:bwc]]

Reinstalling Krisp didn't reintroduce the problem. Maybe my laptop had gotten itself into a weird state.

Comment 10

[Byron Campen [:bwc]]

De-prioritizing, but leaving open since it may highlight some flaw.

Comment 11

[Byron Campen [:bwc]]

So I tried reproducing bug 1607914 (connecting to a whereby room that Safari is already connected to) just now, and saw the problem again. This is how I first noticed this problem.

Padenot, can you try loading a whereby room in Safari, and then loading that room in Firefox?

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p1 as the bug is tracked by a release manager for the current beta.
See What Do You Triage for more information

Comment 13

[Paul Adenot (:padenot)]

(In reply to Byron Campen [:bwc] from comment #11)

So I tried reproducing bug 1607914 (connecting to a whereby room that Safari is already connected to) just now, and saw the problem again. This is how I first noticed this problem.

Padenot, can you try loading a whereby room in Safari, and then loading that room in Firefox?

This works well, I'm using today's Firefox Nightly and Safari Version 13.0.4 (15608.4.9.1.3), current one for macOS 10.15 Mojave.

Joining the room first on Safari, waiting a few seconds, then joining the same room in Firefox Nightly.

Comment 14

[Paul Adenot (:padenot)]

Wait a second I think I reproduced but I don't know how.

Comment 15

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Attachment: aboutWebrtc-connectionIssuesInMozMediaMeeting.html

I think saw this issue in a meeting today. The symptoms where that I saw most other people in the room's video stuttering on and off. I'm provding my about:webrtc in case it would be helpful for debugging.

Edit: was connecting from Windows with most recent nightly.

Comment 16

[Paul Adenot (:padenot)]

Attachment: Bug 1608505 - Lock when setting a graph's current driver. r?achronop

Comment 17

[Paul Adenot (:padenot)]

Attachment: Bug 1608505 - Cap the audio output channel count to something the device can handle. r?achronop

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/96f574da3bbb
Cap the audio output channel count to something the device can handle. r=achronop

Comment 19

[Cristina Coroiu [:ccoroiu]]

Backed out changeset 96f574da3bbb (bug 1608505) for build bustage at build/src/dom/media/MediaTrackGraph.cpp

Backout: https://hg.mozilla.org/integration/autoland/rev/78229efdaad37dbdf101fd435ade9b908df53ba2

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=96f574da3bbbaa7c7d5f5000f1f99fd3efb11967

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=286177356&repo=autoland&lineNumber=30262

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/60a6baeddbff
Cap the audio output channel count to something the device can handle. r=achronop

Comment 21

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/60a6baeddbff

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Should this have gotten an approval request for 73? We're in RC week at this point, so please nominate for release approval if the answer is yes.

Comment 23

[Alex Chronopoulos [:achronop]]

Comment on attachment 9122116 [details]
Bug 1608505 - Cap the audio output channel count to something the device can handle. r?achronop

Beta/Release Uplift Approval Request

• User impact if declined: Unable to do webrtc calls tab-to-tab or with another browser. Neither end receives audio or video
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Open two tabs to whereby.com (same room). Observe that neither end receives audio or video.
  The same happens with any whereby call between firefox and another browser.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): I has been verified in Nightly
• String changes made/needed:

Comment 24

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9122116 [details]
Bug 1608505 - Cap the audio output channel count to something the device can handle. r?achronop

Fixes a pretty bad WebRTC regression for some users. Approved for 73.0RC1.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/2b0af2646bc152c852bbe8effda16188318d6aff

Comment 26

[Oana Botisan, Desktop QA]

I am not sure if I managed to reproduce the issue completely, but the only way I managed to reproduce it is by using a Nightly from 2020-01-10 and did the steps from comment 0 and using bluetooth headphones.
The difference was that because the two tabs were not getting any video or sound, the bluetooth was disconnected and then the video and sound worked perfectly (even without refreshing the page). It's an intermittent issue, but as far as I can tell is not reproducing on latest Nightly 74.0a1 or Firefox 73.0.

On the other hand, the video lags a few good seconds if you do the steps from comment 0 and at the same time you open the safe room in a Safari browser. This one reproduces even on latest Nightly 74.0a1 and 73.0.

Comment 27

[Byron Campen [:bwc]]

(In reply to Oana Botisan, Desktop Release QA from comment #26)

I am not sure if I managed to reproduce the issue completely, but the only way I managed to reproduce it is by using a Nightly from 2020-01-10 and did the steps from comment 0 and using bluetooth headphones.
The difference was that because the two tabs were not getting any video or sound, the bluetooth was disconnected and then the video and sound worked perfectly (even without refreshing the page). It's an intermittent issue, but as far as I can tell is not reproducing on latest Nightly 74.0a1 or Firefox 73.0.

Yeah, this one was tricky to repro.

On the other hand, the video lags a few good seconds if you do the steps from comment 0 and at the same time you open the safe room in a Safari browser. This one reproduces even on latest Nightly 74.0a1 and 73.0.

Sounds like a separate issue. Could you file a bug so we can track it?

Comment 28

[Oana Botisan, Desktop QA]

(In reply to Byron Campen [:bwc] (PTO until Feb 5) from comment #27)

(In reply to Oana Botisan, Desktop Release QA from comment #26)

I am not sure if I managed to reproduce the issue completely, but the only way I managed to reproduce it is by using a Nightly from 2020-01-10 and did the steps from comment 0 and using bluetooth headphones.
The difference was that because the two tabs were not getting any video or sound, the bluetooth was disconnected and then the video and sound worked perfectly (even without refreshing the page). It's an intermittent issue, but as far as I can tell is not reproducing on latest Nightly 74.0a1 or Firefox 73.0.

Yeah, this one was tricky to repro.

Did I manage to reproduce the issue by using those steps? Or do I need to investigate further? And if I do, can you please leave me some additional info that might help me reproduce the bug.
Otherwise I'd say that this issue is fixed, because as I said before, I can't seem to reproduce it on latest Nightly.

Comment 29

[Paul Adenot (:padenot)]

I'm pretty confident my patch fixed this, thanks for trying hard Oana. The video lag you observed is known, we're tracking/fixing this separately. Thanks!

Comment 30

[Oana Botisan, Desktop QA]

Byron,
Can you please verify if this issue is fixed? Because I am not sure 100% if I managed to reproduce/verify it.
Thank you.

Comment 31

[Byron Campen [:bwc]]

Nightly is fixed now for sure.

Comment 32

[Oana Botisan, Desktop QA]

Thank you, Byron!
According to comment 31, I will mark this bug accordingly.

Comment 33

[Oana Botisan, Desktop QA]

Byron, can you please verified the fix on firefox 73.0, too?
Thank you.

Comment 34

[Byron Campen [:bwc]]

I can't repro on 73.0, although I don't have a release binary to compare to because 72 was unaffected.