480347 - AccessibleComponent_getAccessibleAtPoint does not work if menu is displayed

Status: NEW

(Core :: Disability Access APIs, defect)

Description

[Samuel B. Quiring]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2009012320 Minefield/3.0.5

I'm trying to workaround the fact that Minefield delivers a useless "redundant object" to the Gtk:GtkWidget:button-press-event event listener.  By the way, is there a better event to be listening for?

As a user I click on the "Help" menu, causing the menu to be displayed -- my event hander correctly figures out that Menu(Help) is the GUI item being clicked.  

As a user I next click on the "About Minefield" menu item.  I get the correct mouse coordinates and use an algorithm involving AccessibleComponent_getAccessibleAtPoint() to find the GUI item under the mouse.  Unfortunately, this routine does not appear to be aware of the Help drop-down menu that is being displayed on top of the web page -- so it finds the GUI item in the web page under the "About Minefield" menu item.

I'm not sure how to work around this.


Reproducible: Always

Steps to Reproduce:
1. Set up an event handler for Gtk:GtkWidget:button-press-event
2. When an event comes in, use AccessibleComponent_getAccessibleAtPoint to drill to the topmost GUI item.
3. as a user click on "Help", causing the Help Menu to appear
4. as a user click on the "About Minefield" menu item.

Actual Results:  
The topmost GUI item will be from the webpage underneath the "About Minefield" menu item, instead of the Menu item.



Expected Results:  
AccessibleComponent_getAccessibleAtPoint() should know that a menu window is being displayed and that this menu window is above the web page (Z-order), and therefore higher priority.

What event AT-SPI handler(s) should I set up so that I can figure out what menu item the user clicked on?  I'm very open to suggestions.

Comment 1

[Samuel B. Quiring]

The email on this should be [e-mail], not [e-mail].  If I knew how to change it myself, I would.

Comment 2

[Samuel B. Quiring]

I tried this on FF 3.0.1 and FF 3.0.6 and I am seeing the same problems there.

Comment 3

[David Bolter [:davidb] (NeedInfo me for attention)]

Ginn, any ideas?

Comment 4

[Ginn Chen]

Currently, from a11y API, we don't know which window/widget is above, and which window/widget is behind.
Even if the window is minimized, the state of a11y object is still showing/visible.

I don't know if it is expected result.

For your case, can you use "focus event" and "focused state"?

Comment 5

[David Bolter [:davidb] (NeedInfo me for attention)]

Thanks Ginn, how does the at-spi fill out the button-press-event information? If we expose the z-order of the FF gui somehow could you fix this in at-spi?

Comment 6

[Ginn Chen]

The problem of getAccessibleAtPoint is on all platforms.
The code is in
http://mxr.mozilla.org/mozilla-central/ident?i=GetDeepestChildAtPoint

I think at-spi or IA2 doesn't have button-press-event.
So basically AT client should not listen to it, at least for Firefox.

Comment 7

[Samuel B. Quiring]

Gtk made up their own event: Gtk:GtkWidget:button-press-event.  Mozilla is at liberty to do this as well - AT-SPI supports vendor-specific events.  If you go this route, I urge you to have an event, button-click, representing the fact that the user both pushed and released the mouse button on the same object.

In the meantime, what event or sequence of events should the FF/Minefield AT client listen for so it can determine unambiguously:

1) the user has clicked a GUI item (left mouse push, left mouse release)
2) the GUI item clicked by the user
?

I do not think 'focus event' nor 'focused state' will work.  When the menu comes up (after clicking on 'Help'), the user is going to slide his mouse over each item to get to the bottommost, "About Mozilla Firefox".  As he does this, each item gets the focus.  So, unless I am misunderstanding what you mean, I do not think you can use focus to say "the user clicked on a menu item".

Comment 8

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #0)

> As a user I next click on the "About Minefield" menu item.  I get the correct
> mouse coordinates and use an algorithm involving
> AccessibleComponent_getAccessibleAtPoint() to find the GUI item under the
> mouse.

...

> Actual Results:  
> The topmost GUI item will be from the webpage underneath the "About Minefield"
> menu item, instead of the Menu item.

Before I look into this one, are you sure the menu (and the menu item) are still shown when you call AccessibleComponent_getAccessibleAtPoint?

Comment 9

[Samuel B. Quiring]

Am I sure?  Pretty sure.  Here's the experiment I just ran to be sure:

My key event handler (as previously documented) is for Gtk:GtkWidget:button-press-event.  Since it triggers on the "press" and not the "click", I can press the mouse button but not release it and get my code to execute while keeping the menu open.  Here are the steps of my experiment:

1. start listening for events
2. click on Firefox's Help menu
3. The Help menu drops down
4. move mouse down to "About Mozilla Firefox"
5. see all the focus events for each item in the menu
6. press left mouse down on "About Mozilla Firefox" but do not let it up, just sit there.
7. Move mouse (left button still pressed) off menu, then release left button.
8. Help Menu is still being displayed - it never went away

In the log I can see that Gtk:GtkWidget:button-press-event was delivered.  My code begins with the Firefox application (because it contains the mouse) and uses AccessibleComponent_getAccessibleAtPoint to find the gui item under the mouse.  I found the web page item (a form) that was under the "About Mozilla Firefox" menu item.  The menu never showed up.

That is as sure as I can be.

Comment 10

[David Bolter [:davidb] (NeedInfo me for attention)]

I spoke briefly with Neil Deakin and it looks like we'll need to look at popups first using the nsXULPopupManager, like is done in PresShell::HandleEvent. (I have recreated the bug on Linux via Accerciser's console).

Comment 11

[Ginn Chen]

(In reply to comment #6)
> I think at-spi or IA2 doesn't have button-press-event.
Actually, at-spi has mouse:button:* events, but the only information is pointer x,y coordinate.

David,
It sounds good for looking at popups first.
It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.

But it may not solve the use case.

We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component accessible, x, y.
How can we find the right "component accessible" to query?

If we have 2 browser window and have overlaps and we got a screen (x,y), how do we know which window should be asked for AccessibleComponent_getAccessibleAtPoint?
We still need get Z-order work?

Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which is produced be gail, we can do nothing about it, it's out of our control.

Comment 12

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #11)
> (In reply to comment #6)
> > I think at-spi or IA2 doesn't have button-press-event.
> Actually, at-spi has mouse:button:* events, but the only information is pointer
> x,y coordinate.
> 
> David,
> It sounds good for looking at popups first.
> It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.
> 
> But it may not solve the use case.
> 
> We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component
> accessible, x, y.
> How can we find the right "component accessible" to query?
> 
> If we have 2 browser window and have overlaps and we got a screen (x,y), how do
> we know which window should be asked for
> AccessibleComponent_getAccessibleAtPoint?
> We still need get Z-order work?
> 
> Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which
> is produced be gail, we can do nothing about it, it's out of our control.

Yeah, hmmm, Ginn let's we treat these as separate bugs?

Do you the popup work (this bug) is probably needed anyways?

Comment 13

[David Bolter [:davidb] (NeedInfo me for attention)]

Attachment: WIP - doesn't link :(

Comment 14

[David Bolter [:davidb] (NeedInfo me for attention)]

Attachment: cleaner WIP - with link problem

Comment 15

[David Bolter [:davidb] (NeedInfo me for attention)]

Here's the plan:

davidb: Enn: what do you recommend?
Enn: probably to add a new interface or modify nsIMenuRollup or something
Enn: when does this need to be called?
Enn: do you have access to a presshell?
davidb: Enn: anytime, by AT (indirectly)
davidb: Enn: yep
Enn: davidb: if you have a presshell, I'd just add a method there
davidb: Enn: ok i'll give that a shot thanks
Enn: davidb: and perhaps refactor it into the existing code which checks popups

Comment 16

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #11)
> (In reply to comment #6)

> David,
> It sounds good for looking at popups first.
> It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.
> 
> But it may not solve the use case.

I think it will because XUL popups are not their own windows AFAIK.

> 
> We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component
> accessible, x, y.
> How can we find the right "component accessible" to query?

I'm not sure, what are you passing in for this Sam?

> 
> If we have 2 browser window and have overlaps and we got a screen (x,y), how do
> we know which window should be asked for
> AccessibleComponent_getAccessibleAtPoint?
> We still need get Z-order work?

Probably, if so, then that's a spin-off piece to this.

> 
> Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which
> is produced be gail, we can do nothing about it, it's out of our control.

What does gail use to get the accessible?

Comment 17

[David Bolter [:davidb] (NeedInfo me for attention)]

Attachment: WIP (for safe keeping)

I spent some time on this today, but I'd like to know what those redundant objects Sam describes are all about before spending much more.

(BTW I'm not endorsing the approach I took in this WIP, we might what an nsPresShell GetTopFrameFor(PRInt32 screenx, PRInt32 screeny) instead, or something else entirely.)

Comment 18

[Samuel B. Quiring]

I call AccessibleComponent_getAccessibleAtPoint() with each direct child of the
Firefox Application node that meets the requirements below (if I'm doing this
wrong, let me know).

1. role == SPI_ROLE_FRAME, SPI_ROLE_DIALOG, or SPI_ROLE_ALERT
2. Accessible_isComponent(child)
3. AccessibleComponent_contains(childComponent, mouse_x, mouse_y)

The general algorithm builds a list of these, but for Firefox I've never seen
more than one direct child, the "frame".  Here's an example log:

app 19/19 (n:Firefox,d:,r:application)[subs: Application] (children: 1)
  child 1/1 (n:Alan Turing - Google Search - Mozilla Firefox,d:,r:frame)
      Box[407,49,979,642][subs: Component Document] (children: 15)
  child 1/1   - attr 1/3 event-from-input:false
  child 1/1   - attr 2/3 tag:window
  child 1/1   - attr 3/3 id:main-window

Firefox is the 19th child of the 19 children of the desktop.  The application
node has one child with role name "frame" and attributes as shown.  That
"frame" is the Accessible * object passed to ..._getAccessibleAtPoint().  The
(x, y) coordinates are the mouse coordinates.

If it were possible to deliver the correct Accessible *object via the
Gtk:GtkWidget:button-press-event that would be excellent for me.

Comment 19

[David Bolter [:davidb] (NeedInfo me for attention)]

Attachment: WIP for Sam

Sam are you able to build FF with this patch applied to test it out?

Comment 20

[Samuel B. Quiring]

I have never built Firefox before.  I do not have the source downloaded nor do I know how to download it.  I do not know how to apply patches.  I'll gladly download a binary from somewhere and try it out.

Comment 21

[David Bolter [:davidb] (NeedInfo me for attention)]

Firing up a tryserver build (sorry for delay).

Comment 22

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #20)
> I'll gladly
> download a binary from somewhere and try it out.

Should be here now/soon: https://build.mozilla.org/tryserver-builds/2009-05-14_17:[e-mail]-480347/

Comment 23

[David Bolter [:davidb] (NeedInfo me for attention)]

Sam how do this build work for you?

Comment 24

[Samuel B. Quiring]

I am not working on this at the moment, which means it is not easy to find time to download and test.  I will try it in the next 2-3 days.

Comment 25

[alexander :surkov (:asurkov)]

Fernando, can you check if this is still issue?

Comment 26

[Fernando Herrera]

It is.

I can check any element coordinate with this python script:

def callback(event):
        global count
        if event is None:
                return
        if type(event) == pyatspi.event.DeviceEvent:
                return
        print event

reg = pyatspi.Registry
reg.registerEventListener(callback, "mouse:abs")
reg.start()

So mouse over any ff element, I get the screen absolute coords. With them, I got to accerciser and do:

a = acc.queryComponent().getAccessibleAtPoint(x,y,0)
while a is not None:
  a = a.queryComponent().getAccessibleAtPoint(x,y,0)
  print a

So I get the final child contained a t that point.

It works fine for some elements, but sometimes it enters a loop returning itself as a child. So probably that is another bug.

Checking for menus, I can confirm that that it gives you the element below the menu.

Comment 27

[alexander :surkov (:asurkov)]

let's get it fixed for fx5?