2036757 - chatgpt.com - "Use voice" feature shows spinner stuck in a loading state

Status: VERIFIED FIXED

(Web Compatibility :: Site Reports, defect, P1)

Description

[Raul Bucata]

Environment:
Operating system: Windows 10
Firefox version: Firefox 150.0

Preconditions

• Microphone permission granted

Steps to reproduce:

1. Navigate to:https://chatgpt.com/
2. Perform account login
3. From the input field area, click on the "Use voice" icon and observe

Expected Behavior:
Voice mode is activated and functional

Actual Behavior:
Voice mode is stuck in a loading state

Notes:

• Reproduces regardless of the status of ETP
• Reproduces in firefox-nightly, and firefox-release
• Does not reproduce in chrome

Created from https://github.com/webcompat/web-bugs/issues/218981

Comment 1

[Raul Bucata]

Attachment: Chr vs ff

Comment 2

[BugBot [:suhaib / :marco]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 3

[Daniel Holbert [:dholbert]]

I can reproduce. Here's a performance profile, if it ends up being useful for diagnosis: https://share.firefox.dev/42evA2I
I'm not seeing any obviously-relevant diagnostic output in Web Console (from pageload up until the issue happens)...

Comment 4

[Daniel Holbert [:dholbert]]

Here's a perhaps-better[1] profile: https://share.firefox.dev/48I0R1A

[1] I included WebRTC logging (on a guess that WebRTC is involved in some way, because voice), and I kept the profiler running until ChatGPT explicitly showed a "Lost connection. Retry"" notification.

Comment 5

[Daniel Holbert [:dholbert]]

(moving to diagnosis-team:video-conferencing on my hunch that this is webrtc-adjacent I also gave a heads-up to some contacts at OpenAI; maybe/hopefully they can give us some clues if there's something for us to fix here.)

Comment 6

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #5)

I also gave a heads-up to some contacts at OpenAI; maybe/hopefully they can give us some clues if there's something for us to fix here.

--> Marking as webcompat:contact-in-progress

Comment 7

[Byron Campen [:bwc]]

I've looked at this from an ICE perspective, and that does not appear to be the primary problem. ICE connects quickly, DTLS handshake seems to go fine, and we get bi-directional SRTP flow containing silence. Same thing happens on ESR 140, so not a regression. Eventually bug 2037009 also happens, but it was busted before then. Additionally, this bug does not happen in the STR for bug 2037009 (https://bugzilla.mozilla.org/show_bug.cgi?id=2037009#c5).

Comment 8

[Byron Campen [:bwc]]

I don't think this is the issue either, but one odd thing I notice: their SDP does not have any a=extmap for the audio m-section, which is the one that's being used. That means the SDES MID extension will not be working, which could cause problems because they're using BUNDLE. They do have an a=extmap for SDES MID in the unused video m-section for some reason, but that's not going to help things on the audio stream.

Comment 9

[Byron Campen [:bwc]]

I don't see anything suspicious in the profile in comment 4. I've taken a profile myself, and I do see some bad jank (1.3s) right as ICE starts.

I should mention, when I tried again just now I did get audio from OpenAI. Maybe this issue is intermittent?

Comment 10

[Byron Campen [:bwc]]

Looking at the HTMLMediaElement logging, I'm seeing three consecutive calls to setSinkId on the same media element:

07.526949 [Child 12915: Main Thread]: I/HTMLMediaElement 10f515800, setSinkId(/JfIAZjAFjonCgt7Ytim3McqpA3ZJl/XGbpx4unoWzQ=)
07.528793 console.error: "error in gIndicatorWindow.updateIndicatorState(): gIndicatorWindow.updateIndicatorState is not a function"
07.529334 [Child 12915: Main Thread]: I/HTMLMediaElement 10f515800, setSinkId(/JfIAZjAFjonCgt7Ytim3McqpA3ZJl/XGbpx4unoWzQ=)
07.529448 [Child 12915: Main Thread]: I/HTMLMediaElement 10f515800, setSinkId(/JfIAZjAFjonCgt7Ytim3McqpA3ZJl/XGbpx4unoWzQ=)

That seems like a pretty odd usage pattern to me. Jan-Ivar, do you want to take this for a spin?

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment 1 repros for me as well, with a twist: ChatGPT responds to my voice just fine. It's a stuck spinner. Hard to see this being a Firefox bug.

Then about 30 seconds in it says "Lost connection: Retry", which is probably bug 2037009.

Comment 12

[Byron Campen [:bwc]]

So the initial greeting does not come through, but subsequent stuff does? Flubbing the setSinkId stuff seems like it could cause this, right?

Comment 13

[Daniel Holbert [:dholbert]]

FWIW I just retested today and confirmed I can still reproduce this.

Jeff checked in with an OpenAI contact yesterday (following up on comment 5) and they're looking into it on their end.

Comment 14

[Byron Campen [:bwc]]

It looks like bug 2037009 is fixed now, which is at least part of this.

Comment 15

[Jeff Muizelaar [:jrmuizel]]

Daniel, it looks like bug 2037009 was perhaps fixed after your restested. Can you check again?

Comment 16

[Daniel Holbert [:dholbert]]

I'm still seeing the issue; I get the results shown in comment 1 ("Connecting to voice", the voice waveform button turns into a "Cancel" button with a spinner that just keeps spinning, never getting to a good state. And after 20 seconds or so, it changes to say "Lost connection, Retry".)

I'm testing in Firefox 151 (release) and 153 Nightly on macOS. Tested on linux yesterday and saw the same bad-results there, too.
Chrome gives expected results on the same machines.

[While I'm here, I'm going to remove "even though it's working" from the bug summary, because AFAICT that wasn't accurate -- this never appears to reach a state where it's actually working, in my experience or in the comment 1 screencast. The reporter on GitHub did phrase the issue as if it was working for a bit -- their translated description was "it loses the connection after the third sentence" -- but I suspect they're seeing the same thing that I'm seeing, and they were just talking while it was in the 'loading' state and then they noticed it failing over to "Lost connection, Retry" after they talked for a bit.]

Comment 17

[Jeff Muizelaar [:jrmuizel]]

This was caused by an issue in pion that was fixed with https://github.com/pion/webrtc/commit/ec1dd730986df3d129c96f3691be04145d23259a

Jan-Ivar could we have more Chrome/WebKit compatible behaviour here?

Comment 18

[Daniel Holbert [:dholbert]]

I verified that this works on the live chatgpt.com site now. I was able to have a 1-2 minute voice conversation with ChatGPT, using the "Use Voice" button, in Firefox 151.

Comment 19

[Byron Campen [:bwc]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #17)

This was caused by an issue in pion that was fixed with https://github.com/pion/webrtc/commit/ec1dd730986df3d129c96f3691be04145d23259a

Jan-Ivar could we have more Chrome/WebKit compatible behaviour here?

I'm a little unsure what we're contemplating here. Are we talking about not using port 0 on bundle-only m-sections? Because the IETF specs require using port 0 for this (at a MUST level):

https://datatracker.ietf.org/doc/html/rfc9143#section-7.2-5

Now, the bundle, jsep, and webrtc-pc specs are kinda a disaster when it comes to bundle-only (after looking, I have at least one spec bug that needs filing), with places that don't line up or flat out contradict one another. JSEP now says not to use a=bundle-only on reoffers for m-sections that are, in fact, bundle only (which contradicts the bundle spec), but this openai bug did not involve any renegotiation so that's not the cause I think.

Comment 20

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

If Chrome/WebKit are not following https://datatracker.ietf.org/doc/html/rfc9143#section-7.2-5, I think the first step is file issues on their trackers for them to fix that. Byron, can you file those since you have most of the details loaded?

Comment 21

[Calin Tanase]

Verified, the issue no longer reproduces.

Tested with:

• Browser / Version: Firefox 153.0-candidate build 1
• Operating System: Windows 10