[ScreenReader] Wrong position of highlight box in content process' iframe member

RESOLVED FIXED in Firefox 41



Disability Access APIs
3 years ago
3 years ago


(Reporter: kechen, Assigned: eeejay)


Gonk (Firefox OS)

Firefox Tracking Flags

(firefox41 fixed)



(2 attachments, 1 obsolete attachment)



3 years ago
When the screen reader focuses on the member of iframe of a child process, the highlight box will be drawn at wrong position on the screen while the system can still read the right element.

Comment 1

3 years ago
When focus changes the EventManager.jsm will get an AccEvent and pass the position of nsIAccessiblePivot to AccessFu.jsm by sending a "Access:Present" message[1].

The AccessFu.jsm in parent process will adjust this position by adding the offset of window in parent process And set the bound to highlight box[2].

And as my observation, the position of nsIAccessiblePivot is already incorrect which can be got from Accessible::Bounds() [3].

[1] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#531
[3] https://dxr.mozilla.org/mozilla-central/source/accessible/generic/Accessible.cpp#670

Comment 2

3 years ago
out of curiosity, what is AccessFu architecture for multiprocess?
(In reply to alexander :surkov from comment #2)
> out of curiosity, what is AccessFu architecture for multiprocess?

[cross-post from bug 1113494]
Current ScreenReader Flow
- [Gaia] use mozSettings “accessibility.screenreader” to start [1]
- [Gecko-b2g] AccessFu.jsm inject content-script.js into all content processes
- [Gecko-content] content-script.js initialize EventManager, ContentControl, and all required accessibility API

- [Gecko-content] ContentControl capture mousemove events and move visual caret by vc.moveToPoint() [2]
- [Gecko-content] handle moveToPoint, move visual cursor to corresponding element.
- [Gecko-content] EventManager catch visual cursor change event and trigger Presentation.pivotChanged callback [3]
     - [Gecko-content] VisualPresenter.pivotChanged will create a corresponding pivot context ('vc-change') for moving highlighting indicator [4] 
     - [Gecko-content] B2GPresenter.pivotChanged will create a corresponding pivot context (‘vc-change’) for speech synthesis [5]
     - [Gecko-content] EventManager send AccessFu:Present with both pivot contexts

- [Gecko-b2g] AccessFu.jsm received AccessFu:Present message
     - [Gecko-b2g] use Output.Visual() to adjust highlighting indicator [6]
     - [Gecko-b2g] use Output.B2G() to forward speech synthesis data via mozChromeEvent [7]
- [Gaia] capture mozChromeEvent(‘accessibility-action’) 'vc-change’ type.
- [Gaia] build utterance and speech synthesis. [8]

Text for speech synthesis are generated by UtteranceGenerator according to element types. You may customize the generation rules. [9]
AccessFu uses PointerAdapter.jsm to capture touch gesture which overrides the input behavior. You might want to disable it. [10]

[1] https://github.com/mozilla-b2g/gaia/blob/master/apps/settings/js/panels/accessibility_screenreader/confirm_dialog.js#L23
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/ContentControl.jsm#142
[3] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149
[4] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#180
[5] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#505
[6] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#531
[7] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#525
[8] https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/accessibility.js#L291
[9] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/OutputGenerator.jsm#304
[10] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#128

Comment 4

3 years ago
When pass the pivot position from content process to b2g process [1], it will be wrapped into an object 'PivotContext' defined in 'Utils.jsm'[2]. When getting the position from this object, it will be scaled with a value called 'getContentResolution'[3].

I have done a experiment which generate a div and iframe with the same size and position contains a text with offset 0 to the top and left, and let the screen reader focus on these texts separately.

In this case, I get the same value of position from Accessible::Bounds(), but the values go different when I get the position from the PivotContext in b2g[4]. This is caused by the different getContentResolution in PivotContext, in this case value is 0.32653 for div and 1 for the iframe. This might cause the wrong position of the highlight box, and I am tracking the meaning of this value now.

https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#149 [1]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Utils.jsm#675 [2]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Utils.jsm#311 [3]
https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/AccessFu.jsm#552 [4]

Comment 5

3 years ago
Created attachment 8579963 [details] [diff] [review]
Modify the resolution to fix the incorrect position of highlight box in iframe

In current patch, I use the resolution of content's root window for every windows. I am not sure if it makes sense, but it dose solve the problem in current cases.

I will keep tracing this, maybe there are some condition that I ignore.

Comment 6

3 years ago
There is another problem for Accessibility code.

When receiving the 'scroll' event, the highlight box will repaint on the scroll event's target window.[1][2]

However, if the current highlight box is drawn in an not scrollable iframe embedded in another. And once you scroll the window, the highlight box will repaint in outer iframe because the scroll event's target is the outer window.

[1] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/EventManager.jsm#111
[2] https://dxr.mozilla.org/mozilla-central/source/accessible/jsat/Presentation.jsm#158

Comment 7

3 years ago
Created attachment 8582338 [details] [diff] [review]
Fix the position of highlight box in iframe when scroll the window

In this patch, I save the target window when VIRTUALCURSOR_CHANGED event occurred.
So, when receive the scroll event, I use the saved target window above instead of scroll event's.

For now, there is no side effect in my consideration, but maybe there are some events or variables can describe this situation better.


3 years ago
Assignee: nobody → kechen

Comment 8

3 years ago
Comment on attachment 8582338 [details] [diff] [review]
Fix the position of highlight box in iframe when scroll the window

This problem will be discussed at Bug 1148313
Attachment #8582338 - Attachment is obsolete: true

Comment 9

3 years ago
Comment on attachment 8579963 [details] [diff] [review]
Modify the resolution to fix the incorrect position of highlight box in iframe

In this patch, we can fix the highlight box's position in some cases(when the resolution of the iframe in mozapp is 1).

In other cases(when the resolution of the iframe in mozapp is not 1), we cannot verify the correctness of this patch, because, in current b2g code, this situation won't happen. The current resolution-changed functions(e.g, Pinch zoom, orientation) can only change the root layer's resolution.

I've try the function nsIDOMWindowUtils::SetResolutionAndScaleTo to change the resolution manually, however, it seems just scale the layout in compositor thread only but doesn't trigger the reflow. So the positions of the element don't really match the visual layout.

I am still trying how to set up the situation to verify this patch, but have no idea so far. Therefore, it may take some time.
Flags: needinfo?(eitan)

Comment 10

3 years ago
Created attachment 8592374 [details] [diff] [review]
Remove offset of mozbrowser iframe.

The problmen I see is that the rectangle is draw below the position. It looks like a recent change to the accessibility bounds made this. In the past, the accessible bounds inside a child OOP iframe were relative to the child document, now it seems like we get true screen coords, which is good. So this patch removes the offset that we added to OOP content.

Comment 11

3 years ago
Attachment 8579963 [details] [diff] doesn't remedy the issue I see. Perhaps these are two separate bugs?
Flags: needinfo?(eitan)

Comment 12

3 years ago
Looks like this was introduced in bug 1075670.

Comment 13

3 years ago
(In reply to Eitan Isaacson [:eeejay] from comment #11)
> Attachment 8579963 [details] [diff] doesn't remedy the issue I see. Perhaps
> these are two separate bugs?

Thanks for the help.

Yes, I think these maybe can be considered as two separate bugs. The problem I state in this bug is more related to the resolution.

attachment 8592374 [details] [diff] [review] can fix the rectangle position shift cause by bug 1075670. But when browsing the frame with resolution is not 1, the rectangle is still in incorrect position.

But after bug 1075670, my solution cannot work anymore, and it seems kind of changing the act of the bug I state. I need some time to figure it out.

Comment 14

3 years ago
Comment on attachment 8592374 [details] [diff] [review]
Remove offset of mozbrowser iframe.

This seems to be a fix we should land.
Attachment #8592374 - Flags: review?(yzenevich)


3 years ago
Assignee: kechen → eitan
Comment on attachment 8592374 [details] [diff] [review]
Remove offset of mozbrowser iframe.

Review of attachment 8592374 [details] [diff] [review]:

Looks good, tests pass too.
Attachment #8592374 - Flags: review?(yzenevich) → review+
Last Resolved: 3 years ago
status-firefox41: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
You need to log in before you can comment on or make changes to this bug.