2028676 - [a11y] Answer to user's question not obvious to screen reader users

Status: VERIFIED FIXED

(Core :: Machine Learning: Frontend, defect)

Description

[John Northup]

Impacted Section/Element

Smart Window

Steps to Reproduce

In Advanced Preferences, set browser.smartwindow.enabled=true.

Open a content-rich page and open the Smart Window. While using a screen reader, enter a question in the combobox and hit Enter. If an answer is already present, trigger the "Retry" button.

Expected Behavior

The screen reader will begin by announcing the answer to the question, or at least guide the user to the answer.

Actual Behavior

If the user submitted a new question from the combobox, that text is cleared, focus remains on the combobox, screen reader announces its label again.

If the user triggered the "Retry" button, the answer text is replaced with a new answer and focus is lost. Hitting Tab sends focus to the web page link near the top of the Smart Window.

User Impact

Users relying on screen reader output will not know where to find the answer to their question.

WCAG 2.2 References

4.1.3 Status Messages

Recommendations

When the answer is injected, set focus to its container. The container should have tabindex="-1" to make it programmatically focusable.

Testing Environment

Firefox Beta 150.0b2 using "Flexible" Smart Window model

Assistive Technology Used

all screen readers

Severity

2-Significant

Comment 1

[Anna Yeddi [:ayeddi]]

Users relying on screen readers to access content of the page would not be aware that their question was, in fact, answered and would have to try to find the information earlier on the page themselves - but the unexpected focus behavior would make it even harder to complete.

While it is still technically possible to locate the answer, the current behavior makes the main feature of the Smart Window extremely cumbersome for users of assistive technology, thus marking it as access-S2

Comment 2

[:jlewis]

Attachment: Bug 2028676 - Answer to user's question not obvious to screen reader users r=#accessibility-frontend-reviewers

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4dbd66acb8ac
https://hg.mozilla.org/integration/autoland/rev/4230efb6573f
Answer to user's question not obvious to screen reader users r=accessibility-frontend-reviewers,ai-frontend-reviewers,Gijs,morgan,omarg

Comment 4

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4230efb6573f

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:jlewis, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox151 to wontfix.

For more information, please visit BugBot documentation.

Comment 6

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined/Reason for urgency: S2 accessibility requirement - users using screen reader or VO will not have the Smart Window's chat assistant responses read out to them. This patch notifies AT of incoming messages and gives them semantic meaning.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: no
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Open up Smart Window > enable VoiceOver > chat with SW assistant, here assistant's responses being read out
• Risk associated with taking this patch: medium
• Explanation of risk level: VoiceOver/AT integration adds some behind-the-scenes data flow, but should not affect any front-end facing features.
• String changes made/needed?: No
• Is Android affected?: no

Comment 7

[:jlewis]

Attachment: Bug 2028676 - Answer to user's question not obvious to screen reader users

Original Revision: https://phabricator.services.mozilla.com/D294494

Comment 8

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: S2 accessibility requirement - users using screen reader or VO will not have the Smart Window's chat assistant responses read out to them. This patch notifies AT of incoming messages and gives them semantic meaning.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: no
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Open up Smart Window > enable VoiceOver > chat with SW assistant, here assistant's responses being read out
• Risk associated with taking this patch: medium
• Explanation of risk level: VoiceOver/AT integration adds some behind-the-scenes data flow, but should not affect any front-end facing features.
• String changes made/needed?: No
• Is Android affected?: no

Comment 9

[:jlewis]

Attachment: Bug 2028676 - Answer to user's question not obvious to screen reader users

Original Revision: https://phabricator.services.mozilla.com/D294494

Comment 10

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/dc13e85e40d2
https://hg.mozilla.org/releases/mozilla-beta/rev/b69e6761a9ce

Comment 11

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/f9c1ccca3c9b
https://hg.mozilla.org/releases/mozilla-release/rev/76e45be6352e

Comment 12

[Rares Doghi, Desktop QA]

Attachment: Nvda.mp4

@jlewis, I encountered a few issues, Im using NVDA on Windows 11 and The response from the AI is not being read, after I hit submit to a prompt the NVDA screen reader will only read out loud Ask search or type a URL, the response is never read, and If we go back using Tab + Shift , it will reach a frame where it would start to read all the prompts and all the answers provided, I think it should just read the latest response and if the user goes back with Tab + shift it should read the other prompts and answers. please take a look at the screen recording.

@Anna can you take a look at this, how should this work ?

Comment 13

[:jlewis]

Filed a follow-up. Should also encompass issue that Rares flagged as well: https://bugzilla.mozilla.org/show_bug.cgi?id=2037648

Comment 14

[Rares Doghi, Desktop QA]

Hi, @jlewis, the issue I mentioned is not that its not reading it when its performing a search, its all the time, any prompt I submit it wont read the response, I have to go back using Shift plus Tab and when I reach that frame it would read all prompts from top to bottom instead of just the last response, what is the correct behavior ? what is implemented with this bug ?

Comment 15

[Rares Doghi, Desktop QA]

This is indeed Verified as fixed in all Firefox Versions from Release 151 to our latest Nightly 153.0a1 (2026-05-20) build but only for MacOS. Voice over reads the response without issues. James mentioned that there is a fix for Windows and Ubuntu, I will verify that when it lands as well.

Comment 16

[John Northup]

Resolved.