718637 - [Mac] Firefox doesn't tell VoiceOver when a page has finished loading.

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P1)

Description

[Marco Zehe (:MarcoZ)]

STR:
1. Run accessibility-enabled build of Firefox on OS X with VoiceOver.
2. Open a page.
Expected: When the page finishes loading, you should hear an ascending three tone sound effect, and VoiceOver should start reading.
Actual: VoiceOver remains silent, and only by navigation do you notice that the new content is available.

Safari somehow notifies VoiceOver when it has finished loading the page, similar to events of doc load complete etc. on Windows.

Comment 1

[Marco Zehe (:MarcoZ)]

This is very important that we notify VoiceOver that it can update its view of things. Otherwise users will never know when a page is loaded.

Comment 2

[Hubert Figuiere [:hub]]

Attachment: Notify that the document is loaded. r=

Comment 3

[Hubert Figuiere [:hub]]

Suggested patch. We get the notification from Gecko, but then there is no output in VoiceOver. Chrome doesn't either so I'm not sure what's appropriate here. Attaching for now.

Comment 4

[Marco Zehe (:MarcoZ)]

Comment on attachment 613830 [details] [diff] [review]
Notify that the document is loaded. r=

I built with this patch locally, and am sorry to report that there's no difference to previous behavior. VoiceOver still doesn't get notified correctly that a new document loaded.

Comment 5

[Hubert Figuiere [:hub]]

That's what I said. VoiceOver does nothing. Not sure of the reason. I'll ask the Mac a11y list.

Comment 6

[Hubert Figuiere [:hub]]

According to this reply in the mac accessibility mailing list, the chime in Safari is actually performed by Safari by playing a sound back.

http://prod.lists.apple.com/archives/accessibility-dev/2012/Apr/msg00007.html

Comment 7

[Marco Zehe (:MarcoZ)]

(In reply to Hub Figuiere [:hub] from comment #6)
> According to this reply in the mac accessibility mailing list, the chime in
> Safari is actually performed by Safari by playing a sound back.
> 
> http://prod.lists.apple.com/archives/accessibility-dev/2012/Apr/msg00007.html

Yes, but there are more things that happen once this sound is played. If set in the VoiceOver settings (on by default), VoiceOver then focuses the web area and automatically starts reading the page. Perhaps it takes the sound as a queue to do that, but I would think it would rather react to that notification postmessage....

Comment 8

[Frédéric Wang (:fredw)]

Attachment: Patch

Just updating Hubert's patch.

It seems that notifications sent by WebKit has changed:
http://trac.webkit.org/browser/trunk/Source/WebCore/dom/Document.cpp#L2697

Comment 9

[Frédéric Wang (:fredw)]

@Marco: can you please try this one (I added the AXLayoutComplete notification):

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-e55a7bf751a3/try-macosx64/firefox-44.0a1.en-US.mac.dmg

Comment 10

[Marco Zehe (:MarcoZ)]

I'm not seeing any difference in behavior with this try build unfortunately. :(

Comment 11

[Frédéric Wang (:fredw)]

What about that one:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-de96135319b4/try-macosx64/firefox-44.0a1.en-US.mac.dmg

I notify NSAccessibilityFocusedUIElementChangedNotification on the document root, hopefully that will help to get focus.

Other public notifications on https://developer.apple.com/library/prerelease/mac/documentation/AppKit/Reference/NSAccessibility_Protocol_Reference/index.html seem irrelevant or are not used in WebKit code.

Comment 12

[Marco Zehe (:MarcoZ)]

Nope, this is only slightly better in that it now sets focus to the document, but when activating a link, I still don't hear when the new page has finished loading. What about that LayoutChangedNotification that you mentioned? Are we firing that or not?

Comment 13

[Frédéric Wang (:fredw)]

(In reply to Marco Zehe (:MarcoZ) from comment #12)
> Nope, this is only slightly better in that it now sets focus to the
> document,

Yes, that's what I wanted to do with this latest change.

> but when activating a link, I still don't hear when the new page
> has finished loading.

IIUC the answer by Apple dev (comment 6), we'll have to play the sound ourselves rather than sending a notification to VoiceOver to ask it to play the sound ; I guess this is doable. However, I'm not sure how we can do the "automatically starts reading the page" mentioned in comment 7 without explicitly sending a notification to VoiceOver.

> What about that LayoutChangedNotification that you
> mentioned? Are we firing that or not?

This is what is fired in the latest version:
NSAccessibilityPostNotification(realSelf, @"AXLoadComplete");
NSAccessibilityPostNotification(realSelf, @"AXLayoutComplete");
NSAccessibilityPostNotification(realSelf, NSAccessibilityFocusedUIElementChangedNotification);

Comment 14

[Marco Zehe (:MarcoZ)]

(In reply to Frédéric Wang (:fredw) from comment #13)
> > but when activating a link, I still don't hear when the new page
> > has finished loading.
> 
> IIUC the answer by Apple dev (comment 6), we'll have to play the sound
> ourselves rather than sending a notification to VoiceOver to ask it to play
> the sound ; I guess this is doable. However, I'm not sure how we can do the
> "automatically starts reading the page" mentioned in comment 7 without
> explicitly sending a notification to VoiceOver.

I have a feeling the answer in comment #6 isn't the whole story. Could we take a peek at what Chromium does, since that is a separate project and has different code? It does work, so we should be able to get it to work, too.

Comment 15

[Frédéric Wang (:fredw)]

(In reply to Marco Zehe (:MarcoZ) from comment #14)
> I have a feeling the answer in comment #6 isn't the whole story. Could we
> take a peek at what Chromium does, since that is a separate project and has
> different code? It does work, so we should be able to get it to work, too.

Checking the Chromium code didn't help much. I essentially found the same notifications.

I just wrote a mail on the WebKit mailing list:
https://lists.webkit.org/pipermail/webkit-dev/2015-October/027703.html

Comment 16

[Frédéric Wang (:fredw)]

So if I understand correctly the answer from Apple developers, this won't work until Firefox is white listed as a "web browser" by VoiceOver. So what about just taking attachment 8666747 [details] [diff] [review] now and test that later if more work is needed.

Apple developers only mentioned "AXLoadComplete", but I guess it is safe to send more notifications than needed. I propose the following order:

1) NSAccessibilityFocusedUIElementChangedNotification
2) "AXLoadComplete"
3) "AXLayoutComplete"

So that VoiceOver will first focus the document (after NSAccessibilityFocusedUIElementChangedNotification) and then we will tell it to play the sound and read the previously focused document (after "AXLoadComplete" and "AXLayoutComplete").

Comment 17

[Frédéric Wang (:fredw)]

Attachment: Patch V2

Comment 18

[Marco Zehe (:MarcoZ)]

Yup, let's do that!

Comment 19

[Frédéric Wang (:fredw)]

Attachment: Modify Mac Bundle id to claim being Safari so that VoiceOver knows we are a web browser.

Comment 20

[Marco Zehe (:MarcoZ)]

OK, so without the bundle modifications, the patch still doesn't cause VoiceOver to recognize DocumentLoadComplete, but I really suspect this might have something to do with the whitelisting. I'd say let's get this reviewed and landed.

Comment 21

[alexander :surkov (:asurkov)]

Comment on attachment 8668907 [details] [diff] [review]
Patch V2

Review of attachment 8668907 [details] [diff] [review]:
-----------------------------------------------------------------

::: accessible/mac/AccessibleWrap.mm
@@ +100,5 @@
>  
>    uint32_t eventType = aEvent->GetEventType();
>  
> +  // ignore everything but focus-changed, value-changed, caret, selection
> +  // and document load complete events for now.

eventually this comment will get huge :)

Comment 22

[Frédéric Wang (:fredw)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=39335b8bc2b8

Comment 23

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2a6d058f82d1

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2a6d058f82d1