Closed Bug 763613 Opened 7 years ago Closed 7 years ago

Support hover motion events

Categories

(Firefox for Android :: General, defect)

ARM
Android
defect
Not set

Tracking

()

RESOLVED FIXED
Firefox 16

People

(Reporter: eeejay, Assigned: eeejay)

Details

Attachments

(1 file, 1 obsolete file)

I am implementing an explore-by-touch accessibility feature. This utilizes the Android ACTION_HOVER_* events. In a non-accessibility context this would be similar to mouse motion events with no button depressed. In an accessibility context, Android has a system setting (in ICS, under Accessibility, "explore by touch") that morphs touch events into hover events. This allows a user to move their finger on the display and only send hover events to the underlying views which in turn dispatch a11y events that describe the object under the finger. There are three options for implementing this (with the priority being accessibility).

1. Translate hover events to mousemove DOM events: This is probably the most correct translation. Besides the accessibility usecase, it would allow onmouseover events to work correctly when a bluetooth mouse is connected. The only issue I see is that mouse events seemed to have been abolished with the introduction of MOZ_ONLY_TOUCH_EVENTS. So to do this more holistically would mean to support MotionEvent.TOOL_TYPE_MOUSE in a better way and send DOM mouse events for those Android events.

2. Morph hover events into touch events: This is tricky on a few fronts.
 (a) Only morph when accessibility explore-by-touch is toggled. Otherwise a bluetooth mouse would be perma-pressed during movement. So this would not benefit mouse usage.
 (b) On the gecko chrome end we would need to interpret touch events and prevent defaults in order to capture them and emulate Android's explore-by-touch.
 (c) Android's explore-by-touch implementation (com.android.server.accessibility.TouchExplorer) stupidly morphs two finger pans into one finger pans with no way for us to figure that out. So we would have no way to emulate the two finger scroll in ebt mode.

3. Send specialized messages to gecko from LayerView kind of like Gesture:Tap etc. Maybe something like Accessibility:ExploreByTouch with the coordinates in json. This is the least intrusive approach, but the downside is that it does not bring support for mice.

That's it. I am thinking that in a future B2G feature that would be similar to this, we might want to take the hover route as well (ie. mousemove with buttons=0), so that give more points to option 1.

Also, on the note of accessibility, alternative pointer access besides touch is definitely an accessibility feature as it provides multimodal input. So I think it makes sense to support mice. So there is another point for option 1.

I would love input on this. Sorry about the verbose braindump.
(In reply to Eitan Isaacson [:eeejay] from comment #0)
 
> 1. Translate hover events to mousemove DOM events: This is probably the most
> correct translation. Besides the accessibility usecase, it would allow
> onmouseover events to work correctly when a bluetooth mouse is connected.
> The only issue I see is that mouse events seemed to have been abolished with
> the introduction of MOZ_ONLY_TOUCH_EVENTS. So to do this more holistically
> would mean to support MotionEvent.TOOL_TYPE_MOUSE in a better way and send
> DOM mouse events for those Android events.

Can you elaborate on this issue and the proposed solution?
I'm very much in favor of solution #1 and would be happy to implement (or guide someone who is implmenting). We can pick up the mouse events in our java layer and send them down, probably by extending our current touch code with a different pointer type, from there is should be extremely easy to fire off mouse events to gecko.

There's some caveots there with making sure we're not sending mouse events twice to builds not using MOZ_ONLY_TOUCH_EVENTS (i.e. XUL Fennec).
(In reply to Wesley Johnston (:wesj) from comment #2)
> I'm very much in favor of solution #1 and would be happy to implement (or
> guide someone who is implmenting). We can pick up the mouse events in our
> java layer and send them down, probably by extending our current touch code
> with a different pointer type, from there is should be extremely easy to
> fire off mouse events to gecko.
> 
> There's some caveots there with making sure we're not sending mouse events
> twice to builds not using MOZ_ONLY_TOUCH_EVENTS (i.e. XUL Fennec).

Thanks, Wesley. Since you wrote the original multitouch patch, you are the one to approve of this. I already have a half-baked solution here. I'll finish it up and post it here for your feedback.
This seems right to me. Let's get some feedback first!
Attachment #632402 - Flags: feedback?(wjohnston)
Comment on attachment 632402 [details] [diff] [review]
Make hover events mousemove events.

Review of attachment 632402 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me from a distance. Yay to see it didn't require rewriting the world!

::: mobile/android/base/gfx/TouchEventHandler.java
@@ +233,5 @@
>      }
>  
>      private boolean isDownEvent(MotionEvent event) {
>          int action = (event.getAction() & MotionEvent.ACTION_MASK);
> +        return (action == MotionEvent.ACTION_DOWN || action == MotionEvent.ACTION_POINTER_DOWN || action == MotionEvent.ACTION_HOVER_ENTER);

TouchEventHandler has a lot of specialized code to handle the touch events spec. Its fragile. I'd like to avoid it entirely here if we can and send the events directly from LayerView to Gecko. Is there some reason we need to send these events here?

::: widget/android/nsWindow.cpp
@@ +845,2 @@
>                      if (!preventDefaultActions && ae->Count() < 2)
>                          target->OnMotionEvent(ae);

While you're here, we might as well rename this to onMouseEvent (which I think is all it handles now?)

@@ +1308,5 @@
> +
> +        case AndroidMotionEvent::ACTION_HOVER_MOVE:
> +            msg = NS_MOUSE_MOVE;
> +            buttons = 0;
> +            break;

Will this give us mousedown and mouseup events somehow? Can we pull them in to this as well.
Attachment #632402 - Flags: feedback?(wjohnston) → feedback+
(In reply to Wesley Johnston (:wesj) from comment #5)
> Comment on attachment 632402 [details] [diff] [review]
> Make hover events mousemove events.
> 
> Review of attachment 632402 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This looks good to me from a distance. Yay to see it didn't require
> rewriting the world!
> 
> ::: mobile/android/base/gfx/TouchEventHandler.java
> @@ +233,5 @@
> >      }
> >  
> >      private boolean isDownEvent(MotionEvent event) {
> >          int action = (event.getAction() & MotionEvent.ACTION_MASK);
> > +        return (action == MotionEvent.ACTION_DOWN || action == MotionEvent.ACTION_POINTER_DOWN || action == MotionEvent.ACTION_HOVER_ENTER);
> 
> TouchEventHandler has a lot of specialized code to handle the touch events
> spec. Its fragile. I'd like to avoid it entirely here if we can and send the
> events directly from LayerView to Gecko. Is there some reason we need to
> send these events here?
> 

Good idea. It will also make it more responsive, I assume? Since we won't be bunching events.

> ::: widget/android/nsWindow.cpp
> @@ +845,2 @@
> >                      if (!preventDefaultActions && ae->Count() < 2)
> >                          target->OnMotionEvent(ae);
> 
> While you're here, we might as well rename this to onMouseEvent (which I
> think is all it handles now?)
> 
> @@ +1308,5 @@
> > +
> > +        case AndroidMotionEvent::ACTION_HOVER_MOVE:
> > +            msg = NS_MOUSE_MOVE;
> > +            buttons = 0;
> > +            break;
> 
> Will this give us mousedown and mouseup events somehow? Can we pull them in
> to this as well.

Isn't mousedown and mouseup for buttons? Since hover by definition does not include buttons pressed, it should not have mousedown and mouseup. Right?
Attachment #632402 - Attachment is obsolete: true
Attachment #632803 - Flags: review?(wjohnston)
Attachment #632803 - Flags: review?(wjohnston) → review+
http://hg.mozilla.org/integration/mozilla-inbound/rev/460056e3df00
Assignee: nobody → eitan
Target Milestone: --- → Firefox 16
https://hg.mozilla.org/mozilla-central/rev/460056e3df00
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.