1759884 - WebEx meeting sometimes loses audio on nightly

Status: VERIFIED FIXED

(Core :: WebRTC, defect, P2)

Description

[Emilio Cobos Álvarez (:emilio)]

[Tracking Requested - why for this release]: Makes WebEx unusable from Firefox.

On last CSSWG meeting I lost audio from other participants multiple times on Nightly. This also happened last week, and it doesn't happen on release.

The CSSWG instance is hosted in mit.webex.com. I think we can get access to a room to debug if we ask.

This is on Linux with Wayland. Has anything landed over the last two weeks that could've broken this?

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Daniel, I wonder if you've seen something like this if you've been in the csswg calls last week or this one?

Comment 2

[Daniel Holbert [:dholbert]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)

Daniel, I wonder if you've seen something like this if you've been in the csswg calls last week or this one?

I've been using phone for audio, but I can try in-browser WebRTC audio for next week.

Comment 3

[Jim Mathies [:jimm]]

Matt, this is on Linux. Any chance the audioipc work caused this?

Comment 4

[Jim Mathies [:jimm]]

Also could be related to audio work Paul was doing.

Comment 5

[Jim Mathies [:jimm]]

Emilio, is this a new issue or have you been running into this for a while? If it's recent and possible a regression range might help.

Comment 6

[Emilio Cobos Álvarez (:emilio)]

I use Webex on a weekly basis, but not on the first week of the month because that has a different timezone. I've seen this on the last two meetings (yesterday's and the Tuesday before). I usually have an up to date nightly, on day old at most.

So the regression should have been introduced between the 23rd of February (last known good) and the 9th of March (first known bad).

Comment 7

[Paul Adenot (:padenot)]

Haven't landed much in that range myself, but there's audioipc v2 beeing reenabled on linux, but that got backed out, so depending on when the test was happening and how up-to-date the builds were, it might or might not be the cause.

Other things from us in this range:

https://bugzilla.mozilla.org/show_bug.cgi?id=1755318
https://bugzilla.mozilla.org/show_bug.cgi?id=1746466
https://bugzilla.mozilla.org/show_bug.cgi?id=1756077
https://bugzilla.mozilla.org/show_bug.cgi?id=1757862

https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2022-02-23&enddate=2022-03-09

Comment 8

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Comment deleted. Conflated bugs.

Comment 9

[Matthew Gregan [:kinetik]]

(In reply to Jim Mathies [:jimm] from comment #3)

Matt, this is on Linux. Any chance the audioipc work caused this?

Certainly possible. The buildid window for Linux AudioIPC v2 being enabled via pref flip (bug 1757862) in Nightly was 20220303190240 to 20220305092613, which overlaps with Emilio's guessed regression range in comment 6.

Comment 10

[Emilio Cobos Álvarez (:emilio)]

Matthew, I can try to either disable it locally for next meeting, or gather some diagnostic info or so... What do you prefer me to do and if the later, what can be useful?

Comment 11

[Paul Adenot (:padenot)]

It's disabled for now, bug 1757862 has been backed out 11 days ago.

Comment 12

[Emilio Cobos Álvarez (:emilio)]

Huh, then it can't be it... Is there any diagnostics / logging that would be useful to track this down? Or I can try to ask for a test room if that helps.

Comment 13

[Donal Meehan [:dmeehan]]

Rejecting tracking for 99 as there are not many duplicates/reports.
Setting to Won't Fix for 99 as it's late in the beta cycle.

Comment 14

[Emilio Cobos Álvarez (:emilio)]

Today it happened again. Only WebRTC warning on the console was:

WebRTC: onaddstream is deprecated! Use peerConnection.ontrack instead.

Comment 15

[Bob Hood [:bhood]]

Jan-Ivar, can we please get somebody assigned to this?

Comment 16

[Jim Mathies [:jimm]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #12)

Huh, then it can't be it... Is there any diagnostics / logging that would be useful to track this down? Or I can try to ask for a test room if that helps.

If you can grab a full about:webrtc log after it happens, that might help us diagnose.

Comment 17

[Emilio Cobos Álvarez (:emilio)]

Attachment: about:webrtc log while reproducing the bug.

Comment 18

[Byron Campen [:bwc]]

So that's the ICE logging, although I doubt that is where the problem lies here. Is there nothing else in the file generated using the "Save Page" button at the top?

Comment 19

[Daniel Holbert [:dholbert]]

I'm hitting it as well on today's call, BTW. I'm hearing audio ~just fine for a few folks, vs. getting zero audio from other folks.

I'm using Nightly 100.0a1 (2022-03-30) (64-bit).
Release 98.0.2 (64-bit) works just fine (I'm using the packaged version from Ubuntu.)

Comment 20

[Emilio Cobos Álvarez (:emilio)]

Attachment: aboutWebrtc-bad.html

Comment 21

[Emilio Cobos Álvarez (:emilio)]

Attachment: aboutWebrtc-good.html

Comment 22

[Daniel Holbert [:dholbert]]

Attachment: dholbert's aboutWebrtc.html

Here's what I got from "save page" while reproducing the bug.

(I clicked "start debug mode" partway through, too.)

Comment 23

[Emilio Cobos Álvarez (:emilio)]

I bisected this during the meeting, and this is what I came up with: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=923f64ad8d51a0ba1540c400bb6ee63cbe1233c4&tochange=11820f4551ab30b94ff80685ed0a4690eadbb0e8

So it's a UA version 100 thing... Karl do we have contacts at WebEx?

Comment 24

[Dennis Schubert [:denschub]]

I have reached out to them.

I'll also tentatively add this bug to bug 1762217, as we can ship a UA override that caps the version number to 99. I'll re-check this some time during the 100 Beta cycle, but I really hope they can fix that on their end before that.

Comment 25

[Daniel Holbert [:dholbert]]

Hi Dennis -- any chance you've heard back from WebEx folks?

Looks like beta100 has gone out (yesterday I think), so our beta users who use WebEx may start to be affected by this bug.

(I'll also test beta100 in the CSSWG meeting in 10 minutes, with the default UA string vs. modified to use version 99, to double-check that it's an issue there & confirm it's version-number-related.)

Comment 26

[Daniel Holbert [:dholbert]]

I was able to reproduce this in beta100 on macOS in today's CSSWG call, though it wasn't until ~halfway through the meeting that I noticed missing audio streams.

Interestingly, there were several people who's audio worked at some points in the call and didn't-work at other parts in the call.

Comment 27

[Daniel Holbert [:dholbert]]

Bad news; I was able to reproduce this after setting my UA string to v99 and reloading WebEx. My modified UA string (as visible in navigator.userAgent):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0

(I copypasted the default value from navigator.userAgent, changed 100 to 99, and pasted the result into about:config string called general.useragent.override, and restarted Firefox for good measure. I think this matches Emilio's STR that he used when confirming his bisection in comment 23. Not sure if there's anywhere else that they could be seeing the version number, but I think no?)

At this moment in time, I'm getting zero audio from astearns via my Firefox-100-with-v99-UA-string WebEx connection (for several minutes). I'm getting his audio just fine, over a separate phone-audio connection to the meeting.

So my testing suggests this is not version-number-100-related after all, and it might be a WebRTC regression as originally suspected.

Comment 28

[Daniel Holbert [:dholbert]]

(ni=emilio to see if he can separately confirm whether the UA-string-override does/doesn't help on his end, in the next CSSWG call, or if he knows why that might not be helping for me... Otherwise, if I'm not missing anything, I worry this may have just been a case where the regression was further back than comment 23 had indicated, and some purportedly "good" bisection builds were really "bad" but didn't get tested long enough to semi-randomly exhibit the issue.)

Comment 29

[Dennis Schubert [:denschub]]

any chance you've heard back from WebEx folks?

Not yet, unfortunately.

My current plan is to land a few additional UA overrides next week (or shortly after) and uplift them to Beta, so that we get as much stuff fixed in Beta as possible. I'll keep an eye on this bug - if this still is a UA thing, there will be a UA override in 100.

Comment 30

[Emilio Cobos Álvarez (:emilio)]

I confirmed that with a UA override now I was still getting missing audio :(

Comment 31

[Emilio Cobos Álvarez (:emilio)]

It also happens on macOS.

Comment 32

[Emilio Cobos Álvarez (:emilio)]

And release also works on macOS

Comment 33

[Daniel Holbert [:dholbert]]

I'm bisecting right now (using autoland builds, on macOS, w/ Intel chip).

My initial bisection today (without specifying --repo autoland) landed on the same regression range as emilio in comment 23, but it's hard to trust that given our results with UA spoofing, and given that it's hard to trust that any build is definitely "good" since it's possible we didn't test long enough to hear the issue (so the regression could always be a bit older than a regression run supposedly tells us).

Right now my autoland bisection has this regression range:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=90543529b51a9397efc90405544454ad8a45ef61&tochange=f0ded9bd2c10548d706f55999e1a345a140e25ba

This includes Bug 1757618 which is an audio-specific change, which feels like it'd be believable as being involved here?

(having said that: I'm currently testing 09dc0429a37d44f71ef41e25478ad94b23af8195 which is a bit newer than that^ audio-specific commit, and I haven't lost audio yet with a few minutes having passed...)

Comment 34

[Paul Adenot (:padenot)]

(In reply to Daniel Holbert [:dholbert] from comment #33)

This includes Bug 1757618 which is an audio-specific change, which feels like it'd be believable as being involved here?

(having said that: I'm currently testing 09dc0429a37d44f71ef41e25478ad94b23af8195 which is a bit newer than that^ audio-specific commit, and I haven't lost audio yet with a few minutes having passed...)

Sorry, this only used during non-real-time workloads, such as playing pre-recorded video streams, or even live streams with high latency (multiple seconds), but not during any video calls type scenarios.

Comment 35

[Daniel Holbert [:dholbert]]

Yeah, my mozregression-downloaded 09dc0429a37d44f71ef41e25478ad94b23af8195 build (newer than Bug 1757618) seemed likely-good -- I went many minutes without any issues, hearing at least 3-4 different folks speaking.

And then my next bisected commit only had a minute or so of testing before the end of the meeting, but I successfully heard 3-4 folks speaking there too, which leaves me with this pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=0005f913b5e96ff1cfc36108b97184da2fbef0d2&tochange=11820f4551ab30b94ff80685ed0a4690eadbb0e8

which is again about the same as the regression range that emilio found in comment 23.

So maybe there's something not-UA-string-sniffing-related that was triggered by our version bump?

Comment 36

[Dennis Schubert [:denschub]]

Given previous comments, this unfortunately does not seem to be an issue with UAs, so I'm removing the relation to bug 1762217 here.

Comment 37

[Daniel Holbert [:dholbert]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #12)

I can try to ask for a test room if that helps.

I've got a room for testing now (not sharing the link publicly, though ask me if you need it). It's not available all the time, but we can request to change its availability as-needed. I'll see if I can reproduce in this testing room, with a few different clients of my own all connected.

Comment 39

[Daniel Holbert [:dholbert]]

FWIW, I've verified our current findings so far with the test room.

I've got Chrome 100 on my MacBook connected with a mic hooked up, and then I've been testing whether that audio is audible in Firefox Nightly on my Ubuntu Thinkpad.

Broken builds seem to reliably play no WebEx audio at all in my setup, and I get the same regression range as comment 23. I also tested current Nightly, using https://addons.mozilla.org/en-US/firefox/addon/user-agent-string-switcher/ to set my UA string back to match my Nightly99 UA string, and I still hit the issue. (I verified in devtools on WebEx that navigator.userAgent showed 99.0.)

So: this still seems to be something not-sniffing-related (and not UA-string-related) that was somehow triggered by our internal version bump.

Comment 40

[Daniel Holbert [:dholbert]]

Also, for symmetry, I tested a v99 build[1] from just before the regression, with the UA String set to v100 (using the same User Agent String Switcher add-on); and the audio still worked just fine there.

So, further evidence that the UA string itself isn't involved -- "bad" UA strings don't break a good build (nor do "good" UA strings break a bad build).

[1] specifically https://hg.mozilla.org/integration/autoland/rev/281c1c1dadc103fe79acfa6a3c151b6d3ed4e8b4 launched via mozregression

Comment 41

[Daniel Holbert [:dholbert]]

Attachment: diagnostic patch/hackardound: revert version.txt

This patch reverts one part of the version-bump back to v99 -- https://searchfox.org/mozilla-central/source/browser/config/version.txt -- and that seems to be sufficient to avoid the issue (in e.g. current mozilla-central builds), based on my testing.

Comment 42

[Emilio Cobos Álvarez (:emilio)]

Daniel will write more details in a sec, but for the record this fixes it for me (he's confirming that it fixes it for him as well):

diff --git a/dom/media/webrtc/jsep/JsepSessionImpl.cpp b/dom/media/webrtc/jsep/JsepSessionImpl.cpp
index 01b98eadd704c..4b5772c3d7809 100644
--- a/dom/media/webrtc/jsep/JsepSessionImpl.cpp
+++ b/dom/media/webrtc/jsep/JsepSessionImpl.cpp
@@ -2028,7 +2028,7 @@ nsresult JsepSessionImpl::CreateGenericSDP(UniquePtr<Sdp>* sdpp) {
   //     entire o= line needs to be unique, but selecting a random number
   //     for <sess-id> is sufficient to accomplish this.
 
-  auto origin = SdpOrigin("mozilla...THIS_IS_SDPARTA-" MOZ_APP_UA_VERSION,
+  auto origin = SdpOrigin("mozilla...THIS_IS_SDPARTA-99.0a1" /* MOZ_APP_UA_VERSION */,
                           mSessionId, mSessionVersion, sdp::kIPv4, "0.0.0.0");
 
   UniquePtr<Sdp> sdp = MakeUnique<SipccSdp>(origin);

Most of the debugging merit is his ,btw. This was... more tricky to find than it should, probably? :-)

Comment 43

[Daniel Holbert [:dholbert]]

Attachment: Patch that fixes things: change version number in SdpOrigin call

Yeah, here's my local version of the patch that Emilio just posted. This "fixes" things for me on current mozilla-central trunk (reverting our version number to 99.0a1 in this specific place where it's hardcoded into some WebRTC session code).

So, our theory is that WebEx servers must be parsing this out and doing some sort of server-side string comparison on it (which they could & probably-are doing in a Firefox-specific way, given the "mozilla" in this string).

Maybe they're trying to gracefully handle some differing WebRTC behavior in old Firefox versions, e.g. less than version 70 (where 70 is a number I'm picking arbitrarily just as an example), and they're e.g. pulling out the first two digits of the version number and seeing if that's less than or greater than 70 and doing something different depending on the result?

Comment 44

[Daniel Holbert [:dholbert]]

ni=denschub to send another poke to WebEx, to let them know we've got a bit more info about precisely where the crucial version-number is here, where we get broken results from them if we increment that version to 100.

In the meantime if they can't/won't fix this in time for version 100 release, emilio had a good suggestion that maybe we can & should add some logic to this code, to let us configure the version number, and send an old one for a limited list of hosts (maybe just WebEx? though I wouldn't be surprised if other folks made the same mistake, assuming there's some reason that sites need to distinguish between old/new Firefox versions in this string).

Comment 45

[Daniel Holbert [:dholbert]]

I tried some hard-coding in some other version-numbers in this SdpOrigin() expression, and found:

• 67.0a1 and lower don't work (they get the same no-audio broken result as current Nightly gets)
• 68.0a1 and higher (up to 99.0a1) do work.
• Version 679.0a1 doesn't work, but version 680.0a1 does work.

So WebEx does indeed to be stripping off the first two digits from the version-number expression and checking whether that's > 67 (or >= 68), for some reason.

The good news is that things will just start working when we reach Firefox version 680. :)

Comment 46

[Daniel Holbert [:dholbert]]

It looks like WebEx's parsing/regex/etc. is specifically caring about the version number IF this string starts with "mozilla", BTW. (The rest of the string doesn't seem to matter)

So if I change the patch to use SdpOrigin("mozilla-100.0a1", I still get no audio; but if use SdpOrigin("mozill-100.0a1" (dropping the final "a" from "mozilla"), then it works just fine, i.e. they don't care about the string's version-number then.

Comment 47

[Daniel Holbert [:dholbert]]

I tested a bunch of SdpOrigin() strings (w/ current mozilla-central, substituted into the expression quoted in comment 42) to a get a better idea of what WebEx is actually looking for here.

It looks like they don't care about the ...THIS_IS_SDPARTA snippet (that can be removed or replaced with any text); what really matters is:

• "mozilla" at the beginning (this triggers the version-number-checking)
• later in the string, a hyphen followed by a version number
• if the first two digits of the version number are between 68 and 99, then all is well. Otherwise we get some broken (legacy?) behavior.

Here are the strings that I tested in this SdpOrigin() expression, annotated with whether or not they work (i.e. whether they receive any audio from a Chrome client attached to the same WebEx meeting on a different computer), and my rationalization for why I think they do/don't work, based on how WebEx is probably handling the string on the server side.

# Testing various syntaxes with 68 (which is "good") as my version number:
mozill68.0a1    # Works; WebEx can't tell what engine we are, and defaults to
                # some standard behavior that happens to work properly.
mozilla68.0a1   # BROKEN; WebEx recognizes us as Gecko but fails to parse
                # version number (due to lack of "-"), and apparently defaults
                # to some broken (legacy?) behavior.
mozilla-68.0a1  # Works; WebEx recognizes us as Gecko v68 which it likes.
mozillaANYTHING-68.0a1 # Works; for the same reason (& because WebEx allows any
                       # text to appear between "mozilla" and "-versionNum".)

# Testing historical/current Firefox version numbers:
mozilla-67.0a1  # BROKEN; WebEx recognizes us as Gecko v67 & gives legacy/broken behavior.
mozilla-68.0a1  # Works; WebEx recognizes us as Gecko v68 which it likes (as noted above).
mozilla-99.0a1  # Works; WebEx recognizes us as Gecko v99 which it likes.
mozilla-100.0a1 # BROKEN; WebEx recognizes us as Gecko v10 & gives legacy/broken behavior.

# Testing distant-future Firefox version numbers:
mozilla-679.0a1  # BROKEN; WebEx recognizes us as Gecko v10 & gives legacy/broken behavior.
mozilla-680.0a1  # Works; WebEx recognizes us as Gecko v68 which it likes.
mozilla-999.0a1  # Works; WebEx recognizes us as Gecko v99 which it likes.
mozilla-1000.0a1 # BROKEN; WebEx recognizes us as Gecko v10 & gives legacy/broken behavior.
mozilla-6800.0a1 # Works; WebEx recognizes us as Gecko v68 which it likes.

Comment 48

[Daniel Holbert [:dholbert]]

All of this is to say: I think we want to ask WebEx to check their WebRTC handling code for places where they screen for the string mozilla, and then parse a 2-digit version number, and check if that's > 67 (or >= 68). It seems like we only get working behavior if that comparison succeeds.

We need them to fix their parsing so that they can accommodate at least 3 digits there, so that 100 gets parsed in a way where it passes the comparison, instead of failing due to being mis-parsed as 10.

Hopefully this gives them enough information to be able to grep & find the check in question on their end...

Comment 49

[Daniel Holbert [:dholbert]]

Maybe we should just freeze this version-number at 99 (or something like that), and stop reporting the actual version number?

Looks like we added the version-number in Bug 1109160, and it doesn't sound like it was particularly important and it's probably less-important now. ("It would be helpful for debugging if we reported the current Firefox version in our SDP"; at that point this code was under more-active development, but I suspect it's relatively stable now?)

Comment 50

[Byron Campen [:bwc]]

This is still useful in practice, and we are actually getting close to replacing our SDP code with a rewrite done in rust. Maybe using the build id (and release channel?) would be less likely to be abused, and also more useful for debugging? It might break some services, but that seems tolerable if it is a one-time thing.

Comment 51

[Daniel Holbert [:dholbert]]

Hmm, switching to report the build ID likely would break WebEx, since they're expecting a number whose first two digits are >= 68. (Maybe/hopefully they'll fix or remove that check, but in the meantime it'd be nice if we could do something to fix this on our end as well.)

If we really need to report some legitimate version indicator as part of this string, maybe we could freeze the version number (so that sites that sniff for version number will hopefully continue to see a "good" version-number), and we could add the build ID as a suffix? i.e.:

auto origin = SdpOrigin("mozilla...THIS_IS_SDPARTA-99.0-" BUILD_ID

...or something along those lines?

Comment 52

[Daniel Holbert [:dholbert]]

(And if comment 51 sounds workable, maybe we could do the "freeze-version-number-at-99" part right now, as a clearly-safe-to-uplift-to-beta100 fix, and then we could consider introducing a new/meaningful version suffix in a followup bug, as a lower-priority fix, possibly as a first part of the reimplement-in-rust work?)

Comment 53

[Daniel Holbert [:dholbert]]

Attachment: Bug 1759884: Freeze our reported version number at 99.0 in WebRTC's SdpOrigin "username" string. r?bwc

It turns out sites (e.g. WebEx) were parsing this as a 2-digit number and
conditioning their behavior on the result. This causes problems if we
truthfully report version numbers of 100 or larger as part of this string.

Comment 54

[Daniel Holbert [:dholbert]]

Following up on comment 51-52: I've attached a proposed-patch to do the version-number-freezing right now, so that sites don't break in a few weeks when we ship version 100 to release.

This is just intended as a stopgap that preserves existing known-good behavior & avoids breakage.

I admittedly don't know much about SDP but I suspect the cost of lying about our version-number in this string is worth the benefit of not-breaking-sites (or having to beg sites like WebEx to fix their parsing code on a tight timeline before 100 release / for the benefit of current beta/Nightly100+ users). If (per comment 50) we want to report some incrementing version indicator, I'm hoping we can take a reasoned approach to that, without the time pressure of an impending release and potential crash-landing of new behavior on beta.

How does this sound?

Comment 55

[Byron Campen [:bwc]]

Putting 99 in the origin might be acceptable short-term, but I definitely do not want to put bogus stuff in here long-term just to placate services that engage in practices that have been "known bad" for decades.

The downside of using build ID is that it could make it easier to fingerprint, I guess.

Comment 56

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0513ddd45604
Freeze our reported version number at 99.0 in WebRTC's SdpOrigin "username" string. r=bwc

Comment 57

[Daniel Holbert [:dholbert]]

(In reply to Byron Campen [:bwc] from comment #55)

Putting 99 in the origin might be acceptable short-term, but I definitely do not want to put bogus stuff in here long-term just to placate services that engage in practices that have been "known bad" for decades.

Agreed. (Though we occasionally get stuck doing that; see e.g.Gecko/20100101 in our UA string.)

Hopefully the broken parsing code in question gets fixed, or better-still, removed (given that it's apparently trying to handle Firefox 67-and-lower which hopefully is an empty set of users at this point).

If we do add the version number back into this string somehow (e.g. in a not-really-it-is-actually-<real version> suffix as you suggested in phabricator), we should be mindful of running into this same issue in the future; though I suppose if <real-version> is always 3 digits in that string, then we'll be ~safe against these sorts of parsing bugs until Firefox 1000, which is ~70 years away (assuming our 4-week-release cycle continues for the next 900 versions).

The downside of using build ID is that it could make it easier to fingerprint, I guess.

Agreed; it also might prevent us from changing our build-ID format if we want to do that at some point, which would be unfortunate.

Comment 58

[Daniel Holbert [:dholbert]]

Anyway, I filed bug 1764970 to track the fact that (it sounds like) we'd like to add some additional suffix with the real >=100 version number (or something similar) going forward.

(I'm not planning on working on that, but I wanted to be sure that it was on-file, to fix at our convenience/need.)

Comment 59

[Daniel Holbert [:dholbert]]

Comment on attachment 9272511 [details]
Bug 1759884: Freeze our reported version number at 99.0 in WebRTC's SdpOrigin "username" string. r?bwc

Beta/Release Uplift Approval Request

• User impact if declined: WebEx teleconferences would be broken for Firefox 100 users, due to broken server-side parsing of the version-number that we provide (for diagnostic purposes) in a WebRTC client identification string
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This bug effectively just preserves existing behavior; it makes us continue to report 99.0 (as we did in the Firefox 99.0 release) in an internal client identification string, during WebRTC session setup.

(Up until now, this string included our actual version-number; this was as a minor diagnostic/debugging aid per bug 1109160 comment 0; but WebEx unfortunately seems to use this version-number and gives bad results if we show a 3-digit major version number of 100 or greater.)

• String changes made/needed: None (the string in question isn't a localized/human-facing string)

Comment 60

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #59)

• Has the fix been verified in Nightly?: No

Note/clarification: I did verify the fix locally in my own mozilla-central build. It just hasn't been in Nightly long enough to have been verified in actual-Nightly builds.

Comment 61

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/0513ddd45604

Comment 62

[Daniel Holbert [:dholbert]]

I verified the fix using autoland builds, too, BTW. (Trying to do as much verification today since my WebEx "test room" doesn't last forever[1].)

I compared these two commands:

# The first expected-good build (the build with the patch):
mozregression --repo autoland --launch 0513ddd45604
# The last expected-bad build (the previous commit):
mozregression --repo autoland --launch 0596d9e811e8

I ran each command a few times, testing repeated loads & reloads of the WebEx test meeting (with 10+ pageloads in both builds).

The expected-good build (with the patch) had working audio every time I entered the meeting. The expected-bad build (just before the patch) had broken audio about 2/3 of the time. (not too surprising that it occasionally worked; I observed that in "bad" builds earlier as well)

So, at least in autoland builds, we can consider this verified.

[1] actually: literally as I was typing this comment, my WebEx test-meeting booted my last client, apparently due to 24 hours having passed since the scheduled end-time. I guess it stays open for 24 hours if folks are still connected, but no longer than that. So, I can't do further testing at the moment, and I won't be able to verify the fix in Nightlies; but autoland-build-verification is essentially just-as-good, and we can get another test room next week if needed.

Comment 63

[Dianna Smith [:diannaS]]

Comment on attachment 9272511 [details]
Bug 1759884: Freeze our reported version number at 99.0 in WebRTC's SdpOrigin "username" string. r?bwc

Approved for 100.0b7

Comment 64

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/10d0a3d08e69

Comment 65

[Dennis Schubert [:denschub]]

Hi Dennis -- any chance you've heard back from WebEx folks?

I have not. But I have asked around and found another possible contact, to whom I just sent an email. I let them know that we're shipping the v99 cap but would still appreciate a "proper" fix on their end, and I'll comment here if I hear back.

Comment 66

[Bogdan Maris, Desktop QA]

Emilio or Daniel would it be to much to ask you to verify this fix on the latest Nightly 101 and Beta 100 builds when/if you have time?
I am not able to access webex meeting and I can't use the demo from this bug due to time zone differences.

Comment 67

[Emilio Cobos Álvarez (:emilio)]

Works on latest beta.

Comment 68

[Daniel Holbert [:dholbert]]

Yeah, I'm using 100.0b8 (64-bit) on macOS to dial into CSSWG meeting right now, and I haven't run into this bug yet, so this does indeed seem to be fixed there.

Once the test WebEx room's recurring-timeslice opens up (tomorrow), I'll spend some time doing more thorough verification, using my same process from comment 39 etc., comparing beta from before/after the fix.

Comment 69

[Daniel Holbert [:dholbert]]

tl;dr: verified fixed in 100 & 101.

I just tested builds of Firefox 100b6 vs b7 (where the latter has the patch), on Linux (during the currently-underway CSSWG meeting), in parallel. (I had the two builds running side-by-side on the same machine, with me periodically muting/unmuting each build's tab, to double-check which build was/wasn't actually receiving folks' audio.)

I verified that:
100.0b6 (64-bit) is bad; several people were intermittently inaudible (due to this bug).
100.0b7 (64-bit) is good; no one was inaudible while I was testing.

So, the fix is verified in 100b7.

Also marking as verified in 101 per comment 62.

Comment 70

[Daniel Holbert [:dholbert]]

(also canceling my ni=bwc from comment 52, which I ended up just morphing into the patch that landed.)

Comment 71

[Bogdan Maris, Desktop QA]

Thank you very much guys!

Comment 72

[Cullen Jennings]

I'm trying to track this down on the Webex side.

Comment 73

[Daniel Holbert [:dholbert]]

Thanks, Cullen! That's good to hear.

Comment 74

[Cullen Jennings]

I am shocked, shocked to find a version check bug in WebEx. Fix is likely to be in May 1 release which, depending on which version of webex you use, can be awhile before it is deployed everywhere. Some customer are on a release train called "slow train" that lags by multiple months.

Just to provide a bit of information on why we even look at this information, it had to do with checking for negotiation of symmetric Payload Type back in version 34 to 37.

Thank you to all the people that helped track this down and isolate the problem. I know that can be a ton of work and very much appreciated.

Comment 75

[Daniel Holbert [:dholbert]]

Thanks for the update and the fix!

Good to know about the possible "slow train" lag for some installations; fortunately the patch landed here (comment 56 / comment 64) should take care of things in the meantime.

Followup question for you... Is it possible that there are WebEx installations that aren't receiving updates at all who will have this bug ~indefinitely who we might need to account for? Or can we assume that effectively every installation is on some sort of update schedule and will receive this update within a few months?

(This will inform how we might proceed over in bug 1764970, where we'd like to eventually un-freeze our version number in this string, vs. possibly add a suffix with the real version number. Our approach over there may depend on whether we need to indefinitely allow for WebEx installations with the current parsing behavior.)

Comment 76

[Cullen Jennings]

I'm sorry I have not been able to get a good answer on when all version will get this bug fix. Sooner or later all version will get this, but I suspect the later is more like 12+ months not 2. Some customers freeze their version of webex and only periodically update so they do not need to update any training. Some of the current sanctions stop us from delivering updates to some countries. I'm not sure how long it will be to update all version but certainly longer than a few months.

Comment 77

[Daniel Holbert [:dholbert]]

Thanks! We'll keep our change in place for the indefinite future, then.

Comment 78

[BugBot [:suhaib / :marco/ :calixte]]

Based on comment #35, this bug contains a bisection range found by mozregression. However, the Regressed by field is still not filled.

:dholbert, if possible, could you fill the Regressed by field and investigate this regression?

For more information, please visit auto_nag documentation.

Comment 79

[Daniel Holbert [:dholbert]]

regressed-by isn't really meaningful here; this was regressed by the version-number bump as noted above (combined with some broken server-side parsing code on WebEx's side which wasn't expecting a 3-digit number)