894086 - [AccessFu] stop using doAction altogether and simply synthesize tap

Status: RESOLVED WONTFIX

(Core :: Disability Access APIs, defect)

Description

[Eitan Isaacson [:eeejay]]

We are not going to fix the internet any time soon. If mobile web apps want to use touch events for activation we need to synthesize the sequence of events that the UA dispatches: touchstart, mousedown, touchend, mouseup, click. Never mind looking at roles or connected listeners.

This should be bomb proof and answer all contingencies.

Comment 1

[Eitan Isaacson [:eeejay]]

Attachment: Synthesize touch events along with mouse events when activating items.

Comment 2

[alexander :surkov (:asurkov)]

One day you will reinvent nsIAccessible. It's sort of opposite to what I'd wanted from jsAT.

Comment 3

[Marco Zehe (:MarcoZ)]

Alex, i understand your point of view and your desire to keep as much common code in nsAccessible as possible.

However, what Eitan does here looks to me like a specialized form of activation that is only applicable to mobile platforms, and thus would not warrant inclusion in the platform-independent code. I think of JSAT as being our nsAccessibleWrap for mobile, only that it is no longer written in C++, but in JS instead. Considering that parts of Firefox for Android are written in Java, and GAIA for Firefox OS is completely written in web languages, I think this is perfectly fine.

Comment 4

[alexander :surkov (:asurkov)]

(In reply to Marco Zehe (:MarcoZ) from comment #3)

> However, what Eitan does here looks to me like a specialized form of
> activation that is only applicable to mobile platforms, and thus would not
> warrant inclusion in the platform-independent code. 

We can have mobile layer if needed.

> I think of JSAT as being
> our nsAccessibleWrap for mobile, only that it is no longer written in C++,
> but in JS instead.

Do I understand right that jsAT is not a screen reader anymore, it's an engine to run screen readers on mobile?

Comment 5

[Marco Zehe (:MarcoZ)]

Comment on attachment 775985 [details] [diff] [review]
Synthesize touch events along with mouse events when activating items.

Here is some feedback from this attachment:

It seems to be able to activate more things than doAction, however it falls over in some circumstances. For one, it does not fix bug 793795. The items in the shutdown dialog still don't activate.

Second, this patch breaks focusability of the password field in a WIFI network authentication screen:
1. Go to Settings, WiFi, and select a protected WIFI network.
2. On the next screen, there is a list with 3 items, first shows the name, then the signal strength, and the last item is the password field. Swipe to it and double-tap.

Result: If I build without this patch, the keyboard activates. If I build with this patch, the keyboard does not activate. Note that in both cases bug 793153 is present, or the keyboard would not work for me in any case.

In other instances, text fields activate. For example when downloading the Wikipedia app from MarketPlace, the search field at the top of the screen can be double-tapped and activates the keyboard either with or without this patch applied.

Note also that in the WIFI authentication screen, the checkbox for showing the password can be toggled without problems with and without this patch. So the problem seems to be specific to the text field, and as mentioned above, to other touchable items of lists such as the ones in the shutdown dialog. Maybe something wrong with the bounds in these cases?

Comment 6

[Marco Zehe (:MarcoZ)]

(In reply to alexander :surkov from comment #4)
> We can have mobile layer if needed.

Yes, and I think it is necessary. However, does it need to be separate from JSAT, and does it need to be written in C++? JSAT is already written in the language of the web, and taking advantage of JavaScript language features.

> > I think of JSAT as being
> > our nsAccessibleWrap for mobile, only that it is no longer written in C++,
> > but in JS instead.
> Do I understand right that jsAT is not a screen reader anymore, it's an
> engine to run screen readers on mobile?

I think it has strong properties of both. Because we need to integrate so tightly with the hardware in supporting touch events, and because we need to do the synthesis of stuff to speak for both Android and Firefox OS, it definitely has strong screen reader properties, but for a very specific purpose. In order to accomplish that goal, like we see here, it needs to tightly integrate with techniques used on mobile devices. I don't think that web authors will suddenly start requiring touch events on desktop sites, since that would break all mouse users. But because the vast majority of mobile devices no longer has a mouse, and touch events are available, simulating them in JSAT does not seem the wrong thing to do here.

I think the question is really one of "do we separate the interaction into two modules, one being JSAT making the request, and one being nsAccessibleWrap for mobile, overriding doAction to simulate the event sequence shown in the patch for this bug?" versus "Do we keep stuff in our already present touch adapters where we process gestures and translate them for the mobile platforms anyway?".

Comment 7

[alexander :surkov (:asurkov)]

Ok, just there's something completely wrong to ignore doAction logic of nsIAccessible. I'm pretty sure it will be reused for mobile Windows 8.

Comment 8

[Marco Zehe (:MarcoZ)]

Comment on attachment 775985 [details] [diff] [review]
Synthesize touch events along with mouse events when activating items.

Further finding: This patch solves the problem of items not activating on mobile.twitter.com, like tweets not showing details, profiles not opening etc. So Twitter does indeed react to Touch events, and now that we provide them, things start to work.

Comment 9

[Eitan Isaacson [:eeejay]]

I agree with Alex, this should be in core. It is easier to experiment in jsat and change things than in core, so I proposed this here first, but I'll spin off another bug/patch soon. It might help with the inconsistencies that Marco sees too.

Comment 10

[Eitan Isaacson [:eeejay]]

We will be doing this in bug #894485.

Comment 11

[David Bolter [:davidb] (NeedInfo me for attention)]

Comment on attachment 775985 [details] [diff] [review]
Synthesize touch events along with mouse events when activating items.

Review of attachment 775985 [details] [diff] [review]:
-----------------------------------------------------------------

r- to reflect consensus. (and to get rid of auto-nag emails)