368204 - Provide original event handler for mutation events, to enable accessibility of live changes

Status: RESOLVED FIXED

(Core :: Disability Access APIs, enhancement)

Description

[Aaron Leventhal]

Problem:
It's impossible for a screen reader or screen magnifier to know when and how much to interrupt a user's context to speak changes in a web page. This is becoming a very big problem. Screen readers especially do not have enough information about changes to determine whether to speak them, and possibly annoy the user, or not speak them, where the user will possibly miss important information.

Screen readers currently receive DOMNodeInserted (EVENT_SHOW) and DOMNodeRemoved (EVENT_HIDE) events. They can collect these and analyze where they the changes were on the page. They can collect information about how often, and what kinds of changes occured, and the size of the changes. However, this is still not enough to know if the change is important, because a small or medium sized change that happens every 10-60 seconds near the user could have occurred for any reason -- important or trival. It's too easy to annoy the user but what if it's very important?

Currently available solutions requires author cooperation:
Today's ARIA roadmap requires that authors will mark important regions with aaa:live and optionally aaa;atomic and aaa:relevant. See http://www.w3.org/TR/aria-state/
Obviously not all (or many) authors will do this. If they do, it will be great information for the screen reader. Assistive technology support is coming in Fire Vox, and should be pushed to other assistive technologies.

More global solution:
However, a very reasonable solution is possibly even in pages that have no author cooperation, by providing more information as part of the change events. 

If we provide the original cause for the change, that goes a huge way to helping screen readers and screen magnifiers.
The change originally comes from some event handler:
A timer, click, keypress, focus, mouseover, load, etc.
If an AT knows this information, it can determine whether the event was actually caused by the user. That's a huge indicator of whether the change should be spoken. Screen reader users usually don't want to be interrupted (if they're doing something), by changes caused by a timer, unless that timer was short and caused by an event handler from a user action. Ignoring non-user generated changes includes returning XMLHttpRequests that were initialized by a timer.
We'd need to provide this information via accessibility APIs or to privileged scripts.

While I understand what we need from the AT end, implementing this in Firefox is outside of what I can quickly understand. It would be nice to get help.

Perhaps knowing the original cause for something  as similar to popup blocking logic. E.g. you want popups from onclick but not not onload. However, I don't know if we can reuuse any of that code.

The JS code knows about incoming event handlers here. The original handler for a stack could be stored on script contexts (perhaps here):
http://lxr.mozilla.org/seamonkey/source/dom/src/events/nsJSEventListener.cpp#110
Timers and XMLHttpRequests could store the original handler as well, and pass that onto the handlers for timer and incoming requests rather than starting with nothing.

Comment 1

[Aaron Leventhal]

CC'ing some folks who work on popup blocking, in the hopes of getting some advice on how to do this.


Just a note:
I was told to look at what the Web Security Context group is doing in this area: http://www.w3.org/2006/WSC/

Comment 2

[Boris Zbarsky [:bzbarsky]]

Note that at any point you can ask for the current popup blocking state.  Would that do what you want?

Comment 3

[Aaron Leventhal]

I'd need to know more about what that reflects.

Is it basically the same thing as knowing whether an explicit user action led to the current code path?

For example, if a change occurred from a click, keypress, focus etc. that's an explicit user action. It shouldn't be fooled by timeouts or xmlhttprequest callbacks.

Comment 4

[Boris Zbarsky [:bzbarsky]]

> I'd need to know more about what that reflects.

See the code?

> Is it basically the same thing as knowing whether an explicit user action

That's the goal, more or less, yes.  Note that it's not a binary value: it's a multistate.  See <http://lxr.mozilla.org/seamonkey/source/dom/public/base/nsPIDOMWindow.h#55>.

> For example, if a change occurred from a click, keypress, focus etc.

See nsDOMEvent::GetEventPopupControlState.

> It shouldn't be fooled by timeouts or xmlhttprequest callbacks.

Of course it's not.  If it were, it wouldn't be much use as a popup blocker.

Comment 5

[Aaron Leventhal]

Thanks! Since we're not using DOM events for mutations anymore, I'll have to see . We get informed about mutations in 2 places.

1) nsCSSFrameConstructor tells us if it's just a frame change but not a DOM Mutation
2) We're an nsIMutationObserver

I think we might just want ::IsHandlingUserInput(). Still looking.

Comment 6

[Boris Zbarsky [:bzbarsky]]

nsIMutationObserver would work fine, but frame changes can happen asynchronously off restyle events.  ::IsHandlingUserInput() will be false when that happens.

Comment 7

[Aaron Leventhal]

Boris, this works really really well so far -- thanks a ton.

It does not consider mouse hover events to be input, and I can see why. That should be alright (and maybe even good) for our screen reader scenarios. It's possibly bad for some screen magnifier scenarios.

That's really a tiny nit compared to the big wins we can get from this.

Comment 8

[Aaron Leventhal]

This was fixed by bug 390692.