1938533 - chatgpt.com - "Voice mode" is missing

Status: RESOLVED WORKSFORME

(Web Compatibility :: Site Reports, defect, P1)

Description

[Raul Bucata]

Environment:
Operating system: Ubuntu
Firefox version: Firefox 133.0

Steps to reproduce:

1. Navigate to: https://chatgpt.com/
2. Perform account login
3. Inside the input field, hover over the "send" button
4. Observe

Expected Behavior:
Voice mode is triggered

Actual Behavior:
Voice mode is missing

Notes:

• Reproduces regardless of the status of ETP
• Reproduces in firefox-nightly, and firefox-release
• Does not reproduce in chrome
• If account login is not performed, chrome shows the same behavior

Created from https://github.com/webcompat/web-bugs/issues/145388

Comment 1

[Raul Bucata]

Attachment: Chr vs FF

Comment 2

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:denschub, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Relink to the current on-device speech recognition bug

Comment 4

[Jim Mathies [:jimm]]

A high priority for this bug should really only apply to mobile, IMHO. Desktop speech input for search is pretty corner case.

Comment 5

[u772726]

Workaround for Windows users: Use voice typing on your PC.

Comment 6

[Daniel Holbert [:dholbert]]

This seems to just be UA-string sniffing.

If I use Firefox Nightly with Chrome Mask activated for chatgpt.com, then the voice button shows up just as it does in Chrome. It works, too -- I just tested it out in Firefox Nightly 143 and release 141, spoofing as Chrome, and I was able to have a real-time voice chat, getting verbal answers to verbal questions.

So: this seems to just be needs-outreach to remove (or find out the reason for) the UA-sniffing block; and I don't think this is dependent on a platform-bug (bug 1856507) at all. --> removing references

(In reply to Jim Mathies [:jimm] from comment #4)

A high priority for this bug should really only apply to mobile, IMHO. Desktop speech input for search is pretty corner case.

This bug is desktop-specific -- at least, in my testing today, this "voice" ChatGPT button isn't shown on mobile at all (I tested Chrome and Firefox on Android, both logged-in and not-logged-in). There's a microphone button on mobile (without any UA-sniffing guard) but that's a bit different -- that's just for speech-to-text, to populate the chat bubble input with text.

Comment 7

[Daniel Holbert [:dholbert]]

freshness reached out to an OpenAI contact about this --> changing to contact-in-progress.

Comment 8

[Mark Richards [:freshness]]

OpenAI has asked for a sanitized HAR file that captures the issue while using the User Agent string
A comparison showing expected behavior in Firefox
and whether the issue occurs in both the standard Firefox app and a private/incognito window.

If those can be posted here, I can relay it to them directly.

Comment 9

[Daniel Holbert [:dholbert]]

Attachment: screencast showing that Firefox 141 doesn't get the "voice" button unless we use a Chrome UA-striing (with User Agent Switcher)

(In reply to Mark Richards [:freshness] from comment #8)

OpenAI has asked for a sanitized HAR file that captures the issue while using the User Agent string

Here's a saved HAR with Firefox 141 using the default user agent string, with a fresh load of https://chatgpt.com:
https://drive.google.com/file/d/1XR5UgqGixA4cvrv-C5aYCouLNdoc9fes/view?usp=drive_link
("bad": bottom-right of the "Ask anything" chat textbox shows a button with an uparrow in a circle, with id="composer-submit-button")

Here's a saved HAR with Firefox 141 using a Chrome user agent string
("good": bottom-right of the "Ask anything" chat textbox shows a button with the word "Voice", wrapped in a div "composer-speech-button-container")
https://drive.google.com/file/d/1rWPL8KzRNO-UpqaT_R6kREsXJoJoiBVW/view?usp=drive_link

A comparison showing expected behavior in Firefox

See attached screencast, with a fresh user profile in Firefox 141.

The screencast shows that Firefox doesn't get a "voice" button by default, but it shows up after I activate Chrome Mask to spoof a Chrome UA string.

(And as noted above, I verified that the "voice" button works just fine in Firefox, when we make it show up by pretending to be Chrome. After I log in [required for the feature], I can have a spoken verbal conversation with the chatbot. Not sure if you were saying they need a screencast showing that too? I might not be able to provide that at the moment; I burned through nearly all of my monthly free-ChatGPT-account "Voice chat" budget-of-minutes when testing this earlier.)

and whether the issue occurs in both the standard Firefox app and a private/incognito window.

Yes; it affects both standard Firefox and private-browsing windows in the same way.

Comment 10

[Bern]

Also applies to Sidebar functionality as the voice dictation does not detect mic in sidebar with any AI tools

Comment 11

[Mark Richards [:freshness]]

OpenAI is asking how many users this is impacting. Do we have a way to present an accurate number of impacted users?

Comment 12

[Daniel Holbert [:dholbert]]

It impacts all Firefox users; OpenAI appears to be simply blocking access to this feature based on whether your browser says it's Firefox vs. Chrome.

Comment 13

[Daniel Holbert [:dholbert]]

(I don't have concrete numbers for how many Firefox users visit ChatGPT, or would hypothetically click this Voice Mode button if it were presented to them, but perhaps comment 12 is enough to answer their question?)

Our ask for them here is just to remove the block, since as far as we can tell, the feature works just fine in Firefox (if we pretend to be Chrome in order to bypass the block).

Or if there's a reason for the block (e.g. if this feature is known to be broken in Firefox on some configuration), to share any details they've got on that brokenness.

Comment 14

[Mark Richards [:freshness]]

1. Feature is now enabled by OpenAI for Mozilla Firefox users. Feel free to test it!!
2. OpenAI's engineering team would like to check with us:

Firefox does not seem to be stable with AVM, seems to be an issue with Firefox and WebRTC causing instability and issues that we don't face with any other browser, can you please check if that is something you can fix so that the feature can work as needed?

Comment 15

[Daniel Holbert [:dholbert]]

(In reply to Mark Richards [:freshness] from comment #14)

1. Feature is now enabled by OpenAI for Mozilla Firefox users. Feel free to test it!!

Thanks! Confirmed, I see the button now.

2. OpenAI's engineering team would like to check with us:

Firefox does not seem to be stable with AVM, seems to be an issue with Firefox and WebRTC causing instability and issues that we don't face with any other browser, can you please check if that is something you can fix so that the feature can work as needed?

Hmm, I didn't hit any such instability locally when I briefly tested this last week (in Firefox-pretending-to-be-Chrome on Linux). But perhaps I didn't test long enough or from the right hardware. In general, WebRTC should just work in Firefox; I'm not an expert in it, but :jib [+CC] is. (:jib, do you have any theories about what might be involved here?)

Freshness, could you ask OpenAI folks, regarding the instability they're describing:

• Any particular OS/configuration that's known-to-be-affected/unaffected, and any tips on how to reproduce it?
• Did they see it directly, or have they just gotten reports/telemetry indicating badness from users?
• Can they share any data about the nature of the instability? (e.g. bad audio-quality, unexpected disconnections, JS exceptions being thrown, browser crashes, etc?)

Comment 16

[Mark Richards [:freshness]]

I've requested the additional information regarding the instability. Will update here with their response.

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(:jib, do you have any theories about what might be involved here?)

I've tested ChatGPT voice today in Nightly, and am observing an intermittent issue: after recording, the app sometimes fails to connect over WebRTC, and stops, saying "Connection Lost", with web console stating:

WebRTC: ICE failed, add a STUN server and see about:webrtc for more details 

If I re-record, it sometimes works and sometimes not.

Looking into this, the app (ChatGPT voice) seems to be relying on peer reflexive candidates. It isn’t giving Firefox any STUN/TURN servers in the RTCPeerConnection config which is a bit unusual:

{"bundlePolicy":"max-bundle","certificates":[{}]}

So Firefox only gathers host candidates. Success hinges on the remotes (:3478/udp) seeing Firefox's initial STUN binding requests so a peer-reflexive mapping can be discovered and nominated, which seems brittle: If these initial UDP packets are lost, no connection can be established.

On a good run: Firefox's checks (UDP) reach 4.151.200.38:3478 (and friends), it replies, Firefox forms prflx (10.128/10.130), nominate, done.

On a bad run: A bidirectional pcap shows Firefox sending the checks, but every check to those remote host candidates times out (UDP and ICE-TCP). No reply ⇒ no prflx ⇒ all pairs failed. My wifi router is unremarkable. I have no VPN installed that I'm aware of. Unsure where the packets went.

Workaround

Injecting the following into web console appears to fix it:

(() => {
  const Orig = window.RTCPeerConnection;
  window.RTCPeerConnection = function(cfg = {}) {
    const extra = [{ urls: 'stun:stun.l.google.com:19302' }];
    const merged = { ...cfg, iceServers: [...(cfg.iceServers || []), ...extra] };
    const pc = new Orig(merged);
    pc.addEventListener('icecandidateerror', e => console.warn('ICE error', e));
    pc.addEventListener('iceconnectionstatechange', () =>
      console.log('iceConnectionState', pc.iceConnectionState));
    return pc;
  };
  window.RTCPeerConnection.prototype = Orig.prototype;
})();

Comment 18

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: ice3478-bidir.pcap

sudo tcpdump -i pktap -k NP -s 0 \
  -w ~/Desktop/ice3478-bidir.pcap \
  'udp and port 3478 and (host 172.203.39.49 or host 172.214.226.198 or host 4.151.200.38)'

Comment 19

[Byron Campen [:bwc]]

Yeah, we're definitely sending STUN checks. Something's wrong outside of Firefox I think.

Comment 20

[Byron Campen [:bwc]]

It is possible that there's some firewall policy on the other end that requires it to be the first one to send an ICE check. Or maybe it won't respond to ICE checks before it has gotten a response to one of its requests (which is not possible with only host candidates)? Whatever the case, we're holding up our end here.

Comment 21

[Randell Jesup [:jesup] (needinfo me)]

Can we loop back with the OpenAI engineers to feed this analysis back to them?

Comment 22

[Mark Richards [:freshness]]

I've sent jib's analysis to OpenAI's engineers on August 28th. They have confirmed receipt but have not provided an update (or subsequent questions) yet. As soon as they do, I'll post their response here.

Comment 23

[Nico Grunbaum [:ng, @chew:mozilla.org]]

A user in bug 1986180 is reporting that the microphone works exactly once in chatgpt.com in a clean profile.

Comment 24

[Daniel Holbert [:dholbert]]

(In reply to Nico Grunbaum [:ng, @chew:mozilla.org] from comment #23)

A user in bug 1986180 is reporting that the microphone works exactly once in chatgpt.com in a clean profile.

That sounds like it might be a version of what jib diagnosed above ("after recording, the app sometimes fails to connect over WebRTC[...]").

Comment 25

[Jim Mathies [:jimm]]

Any updates here for outreach? The fix is apparently quite simple.

Comment 26

[Mark Richards [:freshness]]

I've followed up with OpenAI regarding the clean profile information and STR. Will provide any follow-up questions here.

Comment 27

[Jim Mathies [:jimm]]

Hey Thomas, wondering if we can use what Jib put together in https://bugzilla.mozilla.org/show_bug.cgi?id=1938533#c17 as a possible intervention?

Comment 28

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Any updates from OpenAI on this?

Comment 29

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1938533 - add a JS webcompat intervention for ChatGPT to fix voice mode; r?denschub

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d53792658299
https://hg.mozilla.org/integration/autoland/rev/5179644568f2
add a JS webcompat intervention for ChatGPT to fix voice mode; r=webcompat-reviewers,ksenia

Comment 31

[Mark Richards [:freshness]]

No updates from OpenAI on this.
I reached out to them last week, but they haven't responded yet.

Comment 32

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5179644568f2

Comment 33

[Justin Uberti]

Hey folks, Justin from OpenAI here. This just crossed my desk the other day.

I analyzed the original pcap and nothing seems off, but we can't find any record of this session in our logs, it's possible it aged out as it's months old. Can you generate a new pcap? Or should this reproduce immediately on a clean install?

Comment 34

[Daniel Holbert [:dholbert]]

Jib generated that last pcap, maybe he can address comment 33 (after disabling the intervention at about:compat)

Comment 35

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hi Justin, I'm no longer able to repro this intermittent (after turning off the intervention in about:compat). Should have double-checked on Friday before turning it on. The issue appears to have resolved itself.

Will do some more tests on release to be sure and check with Tom tomorrow to remove the intervention again assuming I don't find anything.

Unrelated: I am observing some "Lost connection. Retry" messages when I cancel with X, which I don't see in other browsers, and once it disabled the voice mode button with a tooltip saying "voicemode is not supported". But otherwise the functionality seems solid now!

Thanks for checking in!

Comment 36

[Justin Uberti]

Great to hear. We deployed a new RTC infrastructure over the past couple months so I imagine that's what triggered the original issue. Let me know what you find in your testing tomorrow and I can take a closer look if you tell me the ICE username that was used by the client.

Comment 37

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hi Justin, sorry for the delay. We're unable to reproduce the original symptom, so we're removing the intervention.

I ran into an issue that initially looked similar but turned out to be different: this was with ufrag c5nP2TLO/u18 but unlike the above symptom, never sent any STUN requests due to failing to generate local IP4 host candidates and a "Couldn't bind socket to address IP6:[xxx]:62835/UDP", leaving me with a single IP6 TCP candidate. This was a one-off maybe from trying to use a cold tab, and/or some local firewall UDP or firefox issue. We'll work on fixing that on our end if it happens again.

Comment 38

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1938533 - remove our webcompat intervention for chatgpt.com; r?denschub

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/01d500f3dd6e
https://hg.mozilla.org/integration/autoland/rev/65a297159af1
remove our webcompat intervention for chatgpt.com; r=webcompat-reviewers,dholbert

Comment 40

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/65a297159af1

Comment 41

[Justin Uberti]

Thanks for the update. The situation you mention does sound more like a client issue, but happy to take a closer look if you can get a reliable repro.

Comment 42

[Daniel Holbert [:dholbert]]

Justin: are you are of any remaining/ongoing stability issues with this Voice Mode feature and Firefox? We had a message that freshness relayed from OpenAI folks in comment 14, about some observed instability - I wonder if you've got any signal on whether that's still an issue or not. Based on recent comments here, I'm guessing/hoping that there aren't substantial remaining issues here.

If we're good on that front, then I think we're probably good to close this bug, given that...
A) the originally reported issue (the voice-mode button being missing) has been addressed -- the button's there now.
B) the issue that we temporarily ran into after the button was added (comment 17) seems to have gone away (hopefully?)
C) we deployed an intervention for (B) but removed it as no-longer-needed.

Comment 43

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

To me it seems likely that comment 17 and comment 14 were the same issue. Since it no longer repros I think we're fine to close this.

Comment 44

[Daniel Holbert [:dholbert]]

Thanks! I'll clear my needinfo for Justin, then -- but please don't hesitate to let us know about any Firefox-specific trouble on ChatGPT.