Windows 10 instantiates accessibility

RESOLVED INVALID

Status

()

defect
RESOLVED INVALID
4 years ago
3 years ago

People

(Reporter: davidb, Unassigned)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Tested the public beta today:
FF Release, FF Nightly (40.0a1 20150508030204)
With and without an external keyboard.
about:support shows accessibility activated.
Force disabling accessibility doesn't seem to break anything so far but I don't know expected Windows 10 behaviour yet so me missing anomalies is likely.

Accessibility instantiation will have non-zero performance cost. It will also incur the cost of how we land e10s+a11y (possible annoying doorhanger opt out etc).
Device is a tablet: Samsung 700T (?)
Flags: needinfo?(vdjeric)
David: do you have any ideas about why e10s is being instantiated?
Aaron: do you know if there is anything unique about Windows 10 and accessibility?
Flags: needinfo?(vdjeric)
Flags: needinfo?(dbolter)
Flags: needinfo?(aklotz)
(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #2)
> Aaron: do you know if there is anything unique about Windows 10 and
> accessibility?

Not off the top of my head.
Flags: needinfo?(aklotz)
A popular case is that it seems windows touch devices have a virtual keyboard that makes a11y API calls (thereby instantiated our a11y logic). (Related: bug 1092525, bug 1108607)

By the way, Vladan have you seen or heard of accessibility showing up in any significant way during profiling (lately)?
Flags: needinfo?(dbolter) → needinfo?(vdjeric)
(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #2)
> David: do you have any ideas about why e10s is being instantiated?
Oh and note: *e10s* wasn't being 'instantiated' or at least that isn't what I'm claiming in this bug. My point about "It will also incur the cost of how we land e10s+a11y" was about the fact that we may well find ourselves in a tough spot shipping e10s when a11y support is not ready because even more users will be faced with something like a doorhanger explaining e10s is auto-disabled because of an accessibility tool.

We may also be forced to look at something like having two (or multi) stage instantiation of accessibility API, starting with just lightweight stuff that many tools like virtual keyboards need, and the full monty for read accessibility technology (ATs). We'd need to use heuristics based on specific API use...
> David: do you have any ideas about why e10s is being instantiated?

I typed the wrong thing, I meant why is a11y being instantiated on Windows 10. Virtual keyboards seems like a likely explanation

> By the way, Vladan have you seen or heard of accessibility showing up in any significant way during profiling (lately)?

No, but I haven't looked at many profiles recently either

I'll keep myself needinfo'd as a reminder to revisit this topic later
(In reply to David Bolter [:davidb] from comment #0)
> Tested the public beta today:
> FF Release, FF Nightly (40.0a1 20150508030204)
> With and without an external keyboard.
> about:support shows accessibility activated.
> Force disabling accessibility doesn't seem to break anything so far but I
> don't know expected Windows 10 behaviour yet so me missing anomalies is
> likely.
> 
> Accessibility instantiation will have non-zero performance cost. It will
> also incur the cost of how we land e10s+a11y (possible annoying doorhanger
> opt out etc).

Sounds no different from win8 behavior. FWIW we just landed some work that should keep e10s on for this so people shouldn't get a prompt to disable.
So what's the current situation with e10s + a11y + touch screen devices?

Is this correct:
- e10s still doesn't support a11y as I understand it -- roughly when will support be added?
- e10s will not be disabled by win 8 & 10 tablets activating accessibility, right?
- win 8 & win 10 tablets will still work properly with e10s Firefox despite the lack of a11y support, correct?
Flags: needinfo?(vdjeric)
Flags: needinfo?(jmathies)
Flags: needinfo?(dbolter)
(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #8)
> So what's the current situation with e10s + a11y + touch screen devices?
> 
> Is this correct:
> - e10s still doesn't support a11y as I understand it -- roughly when will
> support be added?

We prompt to disable e10s if we detect a known a11y client in our process space, otherwise they both run together. This is the case on nightly, in 41 a11y is disabled with e10s. I don't anticipate this changing, 42 is our current roll out target.

> - e10s will not be disabled by win 8 & 10 tablets activating accessibility,
> right?

Correct.

> - win 8 & win 10 tablets will still work properly with e10s Firefox despite
> the lack of a11y support, correct?

Correct.
Flags: needinfo?(jmathies)
(In reply to Jim Mathies [:jimm] from comment #9)
> (In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #8)

> > - win 8 & win 10 tablets will still work properly with e10s Firefox despite
> > the lack of a11y support, correct?
> 
> Correct.

One thing I haven't tested in a while is on-screen keyboard input.
Flags: needinfo?(dbolter)
> One thing I haven't tested in a while is on-screen keyboard input.

Avi, can you test this quickly on your T100 with Windows 10?
Flags: needinfo?(avihpit)
(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #11)
> > One thing I haven't tested in a while is on-screen keyboard input.
> 
> Avi, can you test this quickly on your T100 with Windows 10?

I've read all the comments on this bug, but I'm not sure what you want me to test, i.e. what's the scenario and which kinds of outcomes should I be observing?

Meanwhile, I'll give some info on my systems, both with Windows 10:

Asus T100 (tablet/hybrid), with both Nightly (43.0a1 2015-08-18) and Aurora (42.0a2 2015-08-18):

("troubleshooting") Accessibility Activated: true, Prevent accessibility: 0, e10s enabled at the options panel (no warning), no a11y related prefs modified manually.

On both browsers focusing the URL bar with the mouse or with touch doesn't pop up the on screen keyboard (OSK).

HP Pavilion 15" laptop with touch screen, Nightly 2015-08-02:

Same as with the T100 hybrid.


I do recall that on one of those systems focusing the URL bar with the _mouse_ popped the OSK, but it doesn't happen anymore. I don't know if that's a bug or not.

(off topic?) I'm not sure I understand what's this bug about. Is it some unexpected behavior (what's unexpected and what's expected instead?) is it about existing or expected performance regressions when $something happens so we should brace ourselves for the impact? etc...
Flags: needinfo?(avihpit)
So vladan clarified that he wanted to test focusing text fields (URL bar or others) using touch, with and without e10s.

Setup: Asus T100 tablet/hybrid, win10, Firefox Nightly 2015-08-19

"Troubleshooting" info: Accessibility Activated: true, Prevent accessibility: 0

The On Screen Keyboard never popped when focusing:
- The URL bar or the search box at addons.mozilla.com
- Using touch or using the mouse,
- With e10s enabled or disabled (was enabled by default on a clean profile).

Overall, the OSK never popped in any combination which I tried.

If there are more things you want tested on this system, let me know.
Avi: did the on-screen keyboard pop up when editing a text field inside a webpage (such as GMail)

David: are these results expected? is this a regression from Windows 10?
Flags: needinfo?(dbolter)
For comment 13, I only tested the URL bar and the http://addons.mozilla.org search box.

Now I also tested gmail (compose a new email), switched focus using touch between the subject/content fields, with e10s both enabled and disabled, and the OSK never popped.
(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #14)

> David: are these results expected? is this a regression from Windows 10?

I don't recall ever seeing the OSK work with FF on Windows 10 (which is a presumably bug for sure). I'm curious if Jim has the same experience.
Flags: needinfo?(dbolter) → needinfo?(jmathies)
should be working, someone added support in bug 1007063. out of the box windows didn't support this for us, we had to add something.
Flags: needinfo?(jmathies)
I filed bug 1232745 for collecting telemetry on our API usage, and hope to learn more about what Windows 8 and 10 tablets/touch screens require.
I have a Surface Book and this is what I see:

a) e10s is disabled "Enable multi-process Nightly (disabled: An accessibility tool is or was active. See bug 1198459.)"
b) no OSK with Firefox for the URL bar or text fields AFAICT.

All tested with Nightly (32-bit) on a very new stock Surface Book (has a detachable touch screen) running Windows 10.
Alex, in the meantime could you check a windows tablet locally to see what API is used? (cc jimm, because he may have done some of this in a related bug)
Flags: needinfo?(surkov.alexander)
Flags: needinfo?(jmathies)
(In reply to David Bolter [:davidb] from comment #20)
> Alex, in the meantime could you check a windows tablet locally to see what
> API is used? (cc jimm, because he may have done some of this in a related
> bug)

Can you think of any practices I can run for a check? Injected dlls, their names?
Flags: needinfo?(surkov.alexander)
(In reply to David Bolter [:davidb] from comment #20)
> Alex, in the meantime could you check a windows tablet locally to see what
> API is used? (cc jimm, because he may have done some of this in a related
> bug)

I have not looked at this.

(In reply to Andrew Overholt [:overholt] from comment #19)
> I have a Surface Book and this is what I see:
> 
> a) e10s is disabled "Enable multi-process Nightly (disabled: An
> accessibility tool is or was active. See bug 1198459.)"
> b) no OSK with Firefox for the URL bar or text fields AFAICT.
> 
> All tested with Nightly (32-bit) on a very new stock Surface Book (has a
> detachable touch screen) running Windows 10.

Not sure about the osk, that should probably get filed and will likely get duped to an existing bug. For e10s + a11y, bug 1198459 and friends just landed which do the following:

1) record when a11y is run in prefs
2) prevent e10s from loading if a11y was in use over the last seven days or in the last session
3) disabled a11y in content until a restart if e10s is running and a11y starts up.
Flags: needinfo?(jmathies)
(In reply to alexander :surkov from comment #21)
> (In reply to David Bolter [:davidb] from comment #20)
> > Alex, in the meantime could you check a windows tablet locally to see what
> > API is used? (cc jimm, because he may have done some of this in a related
> > bug)
> 
> Can you think of any practices I can run for a check? Injected dlls, their
> names?

I was thinking, for Windows 10, it would be very interesting to see what API usage happens on a stock device with default settings. It would probably require instrumenting our external API with some kind of counter variable, and dumping it to the console or something.
(In reply to David Bolter [:davidb] from comment #23)
> (In reply to alexander :surkov from comment #21)
> > (In reply to David Bolter [:davidb] from comment #20)
> > > Alex, in the meantime could you check a windows tablet locally to see what
> > > API is used? (cc jimm, because he may have done some of this in a related
> > > bug)
> > 
> > Can you think of any practices I can run for a check? Injected dlls, their
> > names?
> 
> I was thinking, for Windows 10, it would be very interesting to see what API
> usage happens on a stock device with default settings.

do you mean to record what interfaces (like MSAA, IAccessible2, UIA ones) were requested? Isn't it covered by telemetry yet? I recall, at least we had something for ISimpleDOM.
(In reply to alexander :surkov from comment #24)
> (In reply to David Bolter [:davidb] from comment #23)
> > (In reply to alexander :surkov from comment #21)
> > > (In reply to David Bolter [:davidb] from comment #20)
> > > > Alex, in the meantime could you check a windows tablet locally to see what
> > > > API is used? (cc jimm, because he may have done some of this in a related
> > > > bug)
> > > 
> > > Can you think of any practices I can run for a check? Injected dlls, their
> > > names?
> > 
> > I was thinking, for Windows 10, it would be very interesting to see what API
> > usage happens on a stock device with default settings.
> 
> do you mean to record what interfaces (like MSAA, IAccessible2, UIA ones)
> were requested? Isn't it covered by telemetry yet? I recall, at least we had
> something for ISimpleDOM.

Not quite, as we don't capture the granularity. I'm talking about actual API method calls, e.g. get_groupPosition. Probably easiest to test locally and dump to stderr, as per jimm (in bug 1232745 comment 2)
(In reply to David Bolter [:davidb] from comment #25)

> Not quite, as we don't capture the granularity. I'm talking about actual API
> method calls, e.g. get_groupPosition. Probably easiest to test locally and
> dump to stderr, as per jimm (in bug 1232745 comment 2)

curious, why is it a matter of interest?

if I understand right then win 10 has to use UIA, which is not implemented yet in Firefox, so it should proxy the calls to MSAA methods. So dumping should let us know which MSAA methods are triggered. How do you want to use that?
(In reply to alexander :surkov from comment #26)
> (In reply to David Bolter [:davidb] from comment #25)
> 
> > Not quite, as we don't capture the granularity. I'm talking about actual API
> > method calls, e.g. get_groupPosition. Probably easiest to test locally and
> > dump to stderr, as per jimm (in bug 1232745 comment 2)
> 
> curious, why is it a matter of interest?
> 
> if I understand right then win 10 has to use UIA, which is not implemented
> yet in Firefox, so it should proxy the calls to MSAA methods. So dumping
> should let us know which MSAA methods are triggered. How do you want to use
> that?

The thinking is that if all Windows 10+ laptops instantiate accessibility it might be worth figuring out a good 'mode' for that which doesn't have the same churn as our full accessibility solution. (It is vaguely related to discussions we've had in the past around (but not in) bug 527003 and another bug/thread I can't find, where we discussed future API design involving a client handshake where it registers what info it is interested in, and/or different a11y APIs, like a light-weight one and a heavy one.)
(In reply to David Bolter [:davidb] from comment #27)

> The thinking is that if all Windows 10+ laptops instantiate accessibility it
> might be worth figuring out a good 'mode' for that which doesn't have the
> same churn as our full accessibility solution. (It is vaguely related to
> discussions we've had in the past around (but not in) bug 527003 and another
> bug/thread I can't find, where we discussed future API design involving a
> client handshake where it registers what info it is interested in, and/or
> different a11y APIs, like a light-weight one and a heavy one.)

Windows expect UIA implemented, so if you do have a MSAA-only light-weight implementation, then this is an interim solution. As an alternative approach, do we break the user experience if accessibility is disabled by default when no known AT detected? As another alternative, the accessibility engine may be broken by API features, for example, I think that Windows doesn't need to work actively with an accessible tree as NVDA does, so we may turn off an automatic tree creation, which should help with perfromance a lot.

Updated

4 years ago
Blocks: 1232416
(In reply to alexander :surkov from comment #28)
> As another alternative, the accessibility
> engine may be broken by API features, for example, I think that Windows
> doesn't need to work actively with an accessible tree as NVDA does, so we
> may turn off an automatic tree creation, which should help with perfromance
> a lot.

Perhaps this is a way we could 'lazily instantiate' automatic tree creation based on API usage?
This is unactionable. We understand why this happens, the fix involves getting e10s and a11y to work together. That work is covered by other bugs.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INVALID
(In reply to Jim Mathies [:jimm] from comment #30)
> That work is covered by other bugs.

See bug 1258839.
See Also: → 1258839
You need to log in before you can comment on or make changes to this bug.