1144516 - [ScreenReader] Wrong position of highlight box in content process' iframe member

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[Kevin Chen[:kechen] (UTC + 8)]

When the screen reader focuses on the member of iframe of a child process, the highlight box will be drawn at wrong position on the screen while the system can still read the right element.

Comment 1

[Kevin Chen[:kechen] (UTC + 8)]

When focus changes the EventManager.jsm will get an AccEvent and pass the position of nsIAccessiblePivot to AccessFu.jsm by sending a "Access:Present" message[1].

The AccessFu.jsm in parent process will adjust this position by adding the offset of window in parent process And set the bound to highlight box[2].

And as my observation, the position of nsIAccessiblePivot is already incorrect which can be got from Accessible::Bounds() [3].


[1] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#531
[3] https://dxr.mozilla.org/mozilla-central/source/accessible/generic/Accessible.cpp#670

Comment 2

[alexander :surkov (:asurkov)]

out of curiosity, what is AccessFu architecture for multiprocess?

Comment 3

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

(In reply to alexander :surkov from comment #2)
> out of curiosity, what is AccessFu architecture for multiprocess?

[cross-post from bug 1113494]
Current ScreenReader Flow
- [Gaia] use mozSettings “accessibility.screenreader” to start [1]
- [Gecko-b2g] AccessFu.jsm inject content-script.js into all content processes
- [Gecko-content] content-script.js initialize EventManager, ContentControl, and all required accessibility API

- [Gecko-content] ContentControl capture mousemove events and move visual caret by vc.moveToPoint() [2]
- [Gecko-content] handle moveToPoint, move visual cursor to corresponding element.
- [Gecko-content] EventManager catch visual cursor change event and trigger Presentation.pivotChanged callback [3]
     - [Gecko-content] VisualPresenter.pivotChanged will create a corresponding pivot context ('vc-change') for moving highlighting indicator [4] 
     - [Gecko-content] B2GPresenter.pivotChanged will create a corresponding pivot context (‘vc-change’) for speech synthesis [5]
     - [Gecko-content] EventManager send AccessFu:Present with both pivot contexts

- [Gecko-b2g] AccessFu.jsm received AccessFu:Present message
     - [Gecko-b2g] use Output.Visual() to adjust highlighting indicator [6]
     - [Gecko-b2g] use Output.B2G() to forward speech synthesis data via mozChromeEvent [7]
- [Gaia] capture mozChromeEvent(‘accessibility-action’) 'vc-change’ type.
- [Gaia] build utterance and speech synthesis. [8]

Text for speech synthesis are generated by UtteranceGenerator according to element types. You may customize the generation rules. [9]
AccessFu uses PointerAdapter.jsm to capture touch gesture which overrides the input behavior. You might want to disable it. [10]

[1] https://github.com/mozilla-b2g/gaia/blob/master/apps/settings/js/panels/accessibility_screenreader/confirm_dialog.js#L23
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/ContentControl.jsm#142
[3] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149
[4] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#180
[5] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#505
[6] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#531
[7] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#525
[8] https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/accessibility.js#L291
[9] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/OutputGenerator.jsm#304
[10] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#128

Comment 4

[Kevin Chen[:kechen] (UTC + 8)]

When pass the pivot position from content process to b2g process [1], it will be wrapped into an object 'PivotContext' defined in 'Utils.jsm'[2]. When getting the position from this object, it will be scaled with a value called 'getContentResolution'[3].

I have done a experiment which generate a div and iframe with the same size and position contains a text with offset 0 to the top and left, and let the screen reader focus on these texts separately.

In this case, I get the same value of position from Accessible::Bounds(), but the values go different when I get the position from the PivotContext in b2g[4]. This is caused by the different getContentResolution in PivotContext, in this case value is 0.32653 for div and 1 for the iframe. This might cause the wrong position of the highlight box, and I am tracking the meaning of this value now.



https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149 [1]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Utils.jsm#675 [2]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Utils.jsm#311 [3]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#552 [4]

Comment 5

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: Modify the resolution to fix the incorrect position of highlight box in iframe

In current patch, I use the resolution of content's root window for every windows. I am not sure if it makes sense, but it dose solve the problem in current cases.

I will keep tracing this, maybe there are some condition that I ignore.

Comment 6

[Kevin Chen[:kechen] (UTC + 8)]

There is another problem for Accessibility code.

When receiving the 'scroll' event, the highlight box will repaint on the scroll event's target window.[1][2]

However, if the current highlight box is drawn in an not scrollable iframe embedded in another. And once you scroll the window, the highlight box will repaint in outer iframe because the scroll event's target is the outer window.


[1] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#111
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#158

Comment 7

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: Fix the position of highlight box in iframe when scroll the window

In this patch, I save the target window when VIRTUALCURSOR_CHANGED event occurred.
So, when receive the scroll event, I use the saved target window above instead of scroll event's.

For now, there is no side effect in my consideration, but maybe there are some events or variables can describe this situation better.

Comment 8

[Kevin Chen[:kechen] (UTC + 8)]

Comment on attachment 8582338 [details] [diff] [review]
Fix the position of highlight box in iframe when scroll the window

This problem will be discussed at Bug 1148313

Comment 9

[Kevin Chen[:kechen] (UTC + 8)]

Comment on attachment 8579963 [details] [diff] [review]
Modify the resolution to fix the incorrect position of highlight box in iframe

In this patch, we can fix the highlight box's position in some cases(when the resolution of the iframe in mozapp is 1).

In other cases(when the resolution of the iframe in mozapp is not 1), we cannot verify the correctness of this patch, because, in current b2g code, this situation won't happen. The current resolution-changed functions(e.g, Pinch zoom, orientation) can only change the root layer's resolution.

I've try the function nsIDOMWindowUtils::SetResolutionAndScaleTo to change the resolution manually, however, it seems just scale the layout in compositor thread only but doesn't trigger the reflow. So the positions of the element don't really match the visual layout.

I am still trying how to set up the situation to verify this patch, but have no idea so far. Therefore, it may take some time.

Comment 10

[Eitan Isaacson [:eeejay]]

Attachment: Remove offset of mozbrowser iframe.

The problmen I see is that the rectangle is draw below the position. It looks like a recent change to the accessibility bounds made this. In the past, the accessible bounds inside a child OOP iframe were relative to the child document, now it seems like we get true screen coords, which is good. So this patch removes the offset that we added to OOP content.

Comment 11

[Eitan Isaacson [:eeejay]]

Attachment 8579963 [details] [diff] doesn't remedy the issue I see. Perhaps these are two separate bugs?

Comment 12

[Eitan Isaacson [:eeejay]]

Looks like this was introduced in bug 1075670.

Comment 13

[Kevin Chen[:kechen] (UTC + 8)]

(In reply to Eitan Isaacson [:eeejay] from comment #11)
> Attachment 8579963 [details] [diff] doesn't remedy the issue I see. Perhaps
> these are two separate bugs?

Thanks for the help.

Yes, I think these maybe can be considered as two separate bugs. The problem I state in this bug is more related to the resolution.

attachment 8592374 [details] [diff] [review] can fix the rectangle position shift cause by bug 1075670. But when browsing the frame with resolution is not 1, the rectangle is still in incorrect position.

But after bug 1075670, my solution cannot work anymore, and it seems kind of changing the act of the bug I state. I need some time to figure it out.

Comment 14

[Eitan Isaacson [:eeejay]]

Comment on attachment 8592374 [details] [diff] [review]
Remove offset of mozbrowser iframe.

This seems to be a fix we should land.

Comment 15

[Yura Zenevich [:yzen] (please use "needinfo")]

Comment on attachment 8592374 [details] [diff] [review]
Remove offset of mozbrowser iframe.

Review of attachment 8592374 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, tests pass too.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7e8f2f8cd18c

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7e8f2f8cd18c