Last Comment Bug 718637 - [Mac] Firefox doesn't tell VoiceOver when a page has finished loading.
: [Mac] Firefox doesn't tell VoiceOver when a page has finished loading.
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Disability Access APIs (show other bugs)
: Trunk
: x86_64 Mac OS X
: P1 normal (vote)
: mozilla44
Assigned To: Frédéric Wang (:fredw)
:
Mentors:
Depends on:
Blocks: osxa11y
  Show dependency treegraph
 
Reported: 2012-01-17 06:34 PST by Marco Zehe (:MarcoZ)
Modified: 2015-10-05 13:04 PDT (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
fixed


Attachments
Notify that the document is loaded. r= (3.45 KB, patch)
2012-04-10 17:19 PDT, Hubert Figuiere [:hub]
mzehe: feedback-
Details | Diff | Splinter Review
Patch (3.37 KB, patch)
2015-09-28 08:37 PDT, Frédéric Wang (:fredw)
no flags Details | Diff | Splinter Review
Patch V2 (3.42 KB, patch)
2015-10-02 05:42 PDT, Frédéric Wang (:fredw)
surkov.alexander: review+
Details | Diff | Splinter Review
Modify Mac Bundle id to claim being Safari so that VoiceOver knows we are a web browser. (1.74 KB, patch)
2015-10-02 05:59 PDT, Frédéric Wang (:fredw)
no flags Details | Diff | Splinter Review

Description Marco Zehe (:MarcoZ) 2012-01-17 06:34:09 PST
STR:
1. Run accessibility-enabled build of Firefox on OS X with VoiceOver.
2. Open a page.
Expected: When the page finishes loading, you should hear an ascending three tone sound effect, and VoiceOver should start reading.
Actual: VoiceOver remains silent, and only by navigation do you notice that the new content is available.

Safari somehow notifies VoiceOver when it has finished loading the page, similar to events of doc load complete etc. on Windows.
Comment 1 Marco Zehe (:MarcoZ) 2012-01-18 07:01:32 PST
This is very important that we notify VoiceOver that it can update its view of things. Otherwise users will never know when a page is loaded.
Comment 2 Hubert Figuiere [:hub] 2012-04-10 17:19:46 PDT
Created attachment 613830 [details] [diff] [review]
Notify that the document is loaded. r=
Comment 3 Hubert Figuiere [:hub] 2012-04-10 17:21:01 PDT
Suggested patch. We get the notification from Gecko, but then there is no output in VoiceOver. Chrome doesn't either so I'm not sure what's appropriate here. Attaching for now.
Comment 4 Marco Zehe (:MarcoZ) 2012-04-11 02:01:33 PDT
Comment on attachment 613830 [details] [diff] [review]
Notify that the document is loaded. r=

I built with this patch locally, and am sorry to report that there's no difference to previous behavior. VoiceOver still doesn't get notified correctly that a new document loaded.
Comment 5 Hubert Figuiere [:hub] 2012-04-11 12:53:35 PDT
That's what I said. VoiceOver does nothing. Not sure of the reason. I'll ask the Mac a11y list.
Comment 6 Hubert Figuiere [:hub] 2012-04-16 15:44:23 PDT
According to this reply in the mac accessibility mailing list, the chime in Safari is actually performed by Safari by playing a sound back.

http://prod.lists.apple.com/archives/accessibility-dev/2012/Apr/msg00007.html
Comment 7 Marco Zehe (:MarcoZ) 2012-04-17 01:38:35 PDT
(In reply to Hub Figuiere [:hub] from comment #6)
> According to this reply in the mac accessibility mailing list, the chime in
> Safari is actually performed by Safari by playing a sound back.
> 
> http://prod.lists.apple.com/archives/accessibility-dev/2012/Apr/msg00007.html

Yes, but there are more things that happen once this sound is played. If set in the VoiceOver settings (on by default), VoiceOver then focuses the web area and automatically starts reading the page. Perhaps it takes the sound as a queue to do that, but I would think it would rather react to that notification postmessage....
Comment 8 Frédéric Wang (:fredw) 2015-09-28 08:37:12 PDT
Created attachment 8666747 [details] [diff] [review]
Patch

Just updating Hubert's patch.

It seems that notifications sent by WebKit has changed:
http://trac.webkit.org/browser/trunk/Source/WebCore/dom/Document.cpp#L2697
Comment 9 Frédéric Wang (:fredw) 2015-09-28 11:35:34 PDT
@Marco: can you please try this one (I added the AXLayoutComplete notification):

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/fred.wang@free.fr-e55a7bf751a3/try-macosx64/firefox-44.0a1.en-US.mac.dmg
Comment 10 Marco Zehe (:MarcoZ) 2015-09-28 11:57:57 PDT
I'm not seeing any difference in behavior with this try build unfortunately. :(
Comment 11 Frédéric Wang (:fredw) 2015-09-29 08:01:17 PDT
What about that one:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/fred.wang@free.fr-de96135319b4/try-macosx64/firefox-44.0a1.en-US.mac.dmg

I notify NSAccessibilityFocusedUIElementChangedNotification on the document root, hopefully that will help to get focus.

Other public notifications on https://developer.apple.com/library/prerelease/mac/documentation/AppKit/Reference/NSAccessibility_Protocol_Reference/index.html seem irrelevant or are not used in WebKit code.
Comment 12 Marco Zehe (:MarcoZ) 2015-09-29 08:40:59 PDT
Nope, this is only slightly better in that it now sets focus to the document, but when activating a link, I still don't hear when the new page has finished loading. What about that LayoutChangedNotification that you mentioned? Are we firing that or not?
Comment 13 Frédéric Wang (:fredw) 2015-09-29 08:56:28 PDT
(In reply to Marco Zehe (:MarcoZ) from comment #12)
> Nope, this is only slightly better in that it now sets focus to the
> document,

Yes, that's what I wanted to do with this latest change.

> but when activating a link, I still don't hear when the new page
> has finished loading.

IIUC the answer by Apple dev (comment 6), we'll have to play the sound ourselves rather than sending a notification to VoiceOver to ask it to play the sound ; I guess this is doable. However, I'm not sure how we can do the "automatically starts reading the page" mentioned in comment 7 without explicitly sending a notification to VoiceOver.

> What about that LayoutChangedNotification that you
> mentioned? Are we firing that or not?

This is what is fired in the latest version:
NSAccessibilityPostNotification(realSelf, @"AXLoadComplete");
NSAccessibilityPostNotification(realSelf, @"AXLayoutComplete");
NSAccessibilityPostNotification(realSelf, NSAccessibilityFocusedUIElementChangedNotification);
Comment 14 Marco Zehe (:MarcoZ) 2015-09-29 10:06:09 PDT
(In reply to Frédéric Wang (:fredw) from comment #13)
> > but when activating a link, I still don't hear when the new page
> > has finished loading.
> 
> IIUC the answer by Apple dev (comment 6), we'll have to play the sound
> ourselves rather than sending a notification to VoiceOver to ask it to play
> the sound ; I guess this is doable. However, I'm not sure how we can do the
> "automatically starts reading the page" mentioned in comment 7 without
> explicitly sending a notification to VoiceOver.

I have a feeling the answer in comment #6 isn't the whole story. Could we take a peek at what Chromium does, since that is a separate project and has different code? It does work, so we should be able to get it to work, too.
Comment 15 Frédéric Wang (:fredw) 2015-10-01 16:34:58 PDT
(In reply to Marco Zehe (:MarcoZ) from comment #14)
> I have a feeling the answer in comment #6 isn't the whole story. Could we
> take a peek at what Chromium does, since that is a separate project and has
> different code? It does work, so we should be able to get it to work, too.

Checking the Chromium code didn't help much. I essentially found the same notifications.

I just wrote a mail on the WebKit mailing list:
https://lists.webkit.org/pipermail/webkit-dev/2015-October/027703.html
Comment 16 Frédéric Wang (:fredw) 2015-10-02 04:32:19 PDT
So if I understand correctly the answer from Apple developers, this won't work until Firefox is white listed as a "web browser" by VoiceOver. So what about just taking attachment 8666747 [details] [diff] [review] now and test that later if more work is needed.

Apple developers only mentioned "AXLoadComplete", but I guess it is safe to send more notifications than needed. I propose the following order:

1) NSAccessibilityFocusedUIElementChangedNotification
2) "AXLoadComplete"
3) "AXLayoutComplete"

So that VoiceOver will first focus the document (after NSAccessibilityFocusedUIElementChangedNotification) and then we will tell it to play the sound and read the previously focused document (after "AXLoadComplete" and "AXLayoutComplete").
Comment 17 Frédéric Wang (:fredw) 2015-10-02 05:42:05 PDT
Created attachment 8668907 [details] [diff] [review]
Patch V2
Comment 18 Marco Zehe (:MarcoZ) 2015-10-02 05:49:45 PDT
Yup, let's do that!
Comment 19 Frédéric Wang (:fredw) 2015-10-02 05:59:53 PDT
Created attachment 8668915 [details] [diff] [review]
Modify Mac Bundle id to claim being Safari so that VoiceOver knows we are a web browser.
Comment 20 Marco Zehe (:MarcoZ) 2015-10-02 07:35:28 PDT
OK, so without the bundle modifications, the patch still doesn't cause VoiceOver to recognize DocumentLoadComplete, but I really suspect this might have something to do with the whitelisting. I'd say let's get this reviewed and landed.
Comment 21 alexander :surkov 2015-10-02 07:55:49 PDT
Comment on attachment 8668907 [details] [diff] [review]
Patch V2

Review of attachment 8668907 [details] [diff] [review]:
-----------------------------------------------------------------

::: accessible/mac/AccessibleWrap.mm
@@ +100,5 @@
>  
>    uint32_t eventType = aEvent->GetEventType();
>  
> +  // ignore everything but focus-changed, value-changed, caret, selection
> +  // and document load complete events for now.

eventually this comment will get huge :)
Comment 24 Wes Kocher (:KWierso) 2015-10-05 13:04:26 PDT
https://hg.mozilla.org/mozilla-central/rev/2a6d058f82d1

Note You need to log in before you can comment on or make changes to this bug.