Consider using a separate accessibility thread for e10s accessibility

RESOLVED INVALID

Status

()

Core
Disability Access APIs
RESOLVED INVALID
2 years ago
2 years ago

People

(Reporter: billm, Unassigned)

Tracking

(Blocks: 1 bug)

Trunk
Points:
---

Firefox Tracking Flags

(e10s+, firefox45 affected)

Details

We're running into a lot of problems with our current approach of using synchronous IPC messages to answer a11y queries in e10s. Bug 1219350 is probably the most serious case--we don't know how to fix it without adding tons of complexity to IPC. We've discussed using a caching approach for accessibility in e10s, but that would be a lot of work.

Fundamentally it seems unfortunate that we can't defer responding to a11y queries until we're ready. The screen reader software is just blocked, waiting for us--there's no reason not to make it wait a little longer. However, the way a11y is implemented in Windows makes this difficult.

It seems that all a11y queries start by sending a WM_GETOBJECT message to the browser window's window procedure. We're supposed to return, via LresultFromObject, a COM object that supports the IAccessible interface.

I wonder if we could create a separate accessibility thread that would have its own separate single-threaded apartment. We would create a singleton root IAccessible object for each browser window in this apartment. When we got a WM_GETOBJECT call on the main thread, we would just return this object. Our main thread would be in a different STA, but I think it's allowed to return objects from other apartments. (This is crucial to the approach, so we should test it soon to see if the idea is feasible.)

One we returned the singleton IAccessible object, all COM method calls on it from the screen reader would happen on the a11y thread (which would have to pump the Windows message loop). When the thread got a method call, it would post a runnable to the main thread that would send an async message over PContent asking for the requested data. Upon receiving the reply, the result would be returned to the a11y thread, which would give it to the screen reader.

Basically, this mechanism would allow us to process accessibility queries asynchronously. I don't know if it's feasible, but I think it's worth a try. I talked to Aaron Klotz and he said it seems reasonable. I think the crucial piece is the ability to return an IAccessible from a different apartment in the WM_GETOBJECT message handler.
(In reply to Bill McCloskey (:billm) from comment #0)
> We're running into a lot of problems with our current approach of using
> synchronous IPC messages to answer a11y queries in e10s. Bug 1219350 is
> probably the most serious case--we don't know how to fix it without adding
> tons of complexity to IPC. We've discussed using a caching approach for
> accessibility in e10s, but that would be a lot of work.
> 
> Fundamentally it seems unfortunate that we can't defer responding to a11y
> queries until we're ready. The screen reader software is just blocked,
> waiting for us--there's no reason not to make it wait a little longer.

performance can matter, but the point stands.  (on windows I believe this particularly an issue because there screen readers want to slurp in a whole ton of information on document load).

> However, the way a11y is implemented in Windows makes this difficult.
> 
> It seems that all a11y queries start by sending a WM_GETOBJECT message to
> the browser window's window procedure. We're supposed to return, via
> LresultFromObject, a COM object that supports the IAccessible interface.

That's more or less correct, though its worth noting in our case normal the object implements a number of other interfaces that client software may use.

> I wonder if we could create a separate accessibility thread that would have
> its own separate single-threaded apartment. We would create a singleton root
> IAccessible object for each browser window in this apartment. When we got a

I don't think you want it to be a singleton.  Once the client has one IAccessible object they can call methods on it that will return other IAccessible objects.  But that's basically a detail :-)


> WM_GETOBJECT call on the main thread, we would just return this object. Our
> main thread would be in a different STA, but I think it's allowed to return
> objects from other apartments. (This is crucial to the approach, so we
> should test it soon to see if the idea is feasible.)

Actually if you can't do that we might be able to setup a fake HWND on that thread whose only purpose is to handle WM_GETOBJECT.  We'd need to trick screen readers into only caring about that window somehow, I'm not sure how screen readers decide what windows to care about, but jamie probably does.  (a11y already has some hacks for older screen readers using fake HWND which may be useful here).

> One we returned the singleton IAccessible object, all COM method calls on it
> from the screen reader would happen on the a11y thread (which would have to
> pump the Windows message loop). When the thread got a method call, it would
> post a runnable to the main thread that would send an async message over
> PContent asking for the requested data. Upon receiving the reply, the result
> would be returned to the a11y thread, which would give it to the screen
> reader.

I thought we did some work so other threads could use ipc? could we just skip the main thread all together?

I guess having sync calls from 2 threads might be "fun", but in the worst case we can just require holding a mutex to make a sync call and that shouldn't be any worse than the runnable thing?

> Basically, this mechanism would allow us to process accessibility queries
> asynchronously. I don't know if it's feasible, but I think it's worth a try.
> I talked to Aaron Klotz and he said it seems reasonable. I think the crucial
> piece is the ability to return an IAccessible from a different apartment in
> the WM_GETOBJECT message handler.

its kind of crazy ;) but I'm tempted to try it.

I think we'd probably need to handle requests for  chrome objects on the thread handling content a11y requests.  However that should be doable,  the code to proxy content should mostly work fine with chrome and xul we'd just need to make it happen since mac and linux don't need it.

Jamie any other thoughts?
Flags: needinfo?(jamie)

Comment 2

2 years ago
(In reply to Trevor Saunders (:tbsaunde) from comment #1)
> > Fundamentally it seems unfortunate that we can't defer responding to a11y
> > queries until we're ready. The screen reader software is just blocked,
> > waiting for us--there's no reason not to make it wait a little longer.
> performance can matter, but the point stands.  (on windows I believe this
> particularly an issue because there screen readers want to slurp in a whole
> ton of information on document load).
The perf hit is the fact that you now proxy a11y queries to the content process. As long as you can avoid an extra level of indirection, it shouldn't get any worse.

> > I wonder if we could create a separate accessibility thread that would have
> > its own separate single-threaded apartment. We would create a singleton root
> > IAccessible object for each browser window in this apartment. When we got a
> > WM_GETOBJECT call on the main thread, we would just return this object. Our
> > main thread would be in a different STA, but I think it's allowed to return
> > objects from other apartments. (This is crucial to the approach, so we
> > should test it soon to see if the idea is feasible.)
I don't think you'll be able to use a COM object from a different apartment without marshalling it first. I suspect (but am not sure) calling LResultFromObject in your main thread with an object from your a11y thread will throw RPC_E_WRONG_THREAD. However, I *think* you might be able to call LResultFromObject in the a11y thread and then return *this* value in WM_GETOBJECT. That raises the question of how you're going to make that call in the a11y thread, though.

> Actually if you can't do that we might be able to setup a fake HWND on that
> thread whose only purpose is to handle WM_GETOBJECT.  We'd need to trick
> screen readers into only caring about that window somehow, I'm not sure how
> screen readers decide what windows to care about, but jamie probably does.
I considered this too, but I have two concerns with this idea:
1. If a user uses the screen reader to walk through the top level windows, it'll hit the main HWND. It may even skip the a11y window if it's invisible or the like.
2. We sometimes care about GetGuiThraedInfo(...).hwndFocus; e.g. when needing to fetch the focus on-demand. That'll be your main HWND, not the a11y HWND, so getting the focus on-demand won't work.
Flags: needinfo?(jamie)
(In reply to James Teh [:Jamie] from comment #2)
> (In reply to Trevor Saunders (:tbsaunde) from comment #1)
> > > Fundamentally it seems unfortunate that we can't defer responding to a11y
> > > queries until we're ready. The screen reader software is just blocked,
> > > waiting for us--there's no reason not to make it wait a little longer.
> > performance can matter, but the point stands.  (on windows I believe this
> > particularly an issue because there screen readers want to slurp in a whole
> > ton of information on document load).
> The perf hit is the fact that you now proxy a11y queries to the content
> process. As long as you can avoid an extra level of indirection, it
> shouldn't get any worse.

yes, I was thinking if you added indirection by going to the main thread that would hurt more.  On the other hand if you never touch the main thread I could believe things get slightly better since there's less contending with your messages and stuff.

> > > I wonder if we could create a separate accessibility thread that would have
> > > its own separate single-threaded apartment. We would create a singleton root
> > > IAccessible object for each browser window in this apartment. When we got a
> > > WM_GETOBJECT call on the main thread, we would just return this object. Our
> > > main thread would be in a different STA, but I think it's allowed to return
> > > objects from other apartments. (This is crucial to the approach, so we
> > > should test it soon to see if the idea is feasible.)
> I don't think you'll be able to use a COM object from a different apartment
> without marshalling it first. I suspect (but am not sure) calling
> LResultFromObject in your main thread with an object from your a11y thread
> will throw RPC_E_WRONG_THREAD. However, I *think* you might be able to call
> LResultFromObject in the a11y thread and then return *this* value in
> WM_GETOBJECT. That raises the question of how you're going to make that call
> in the a11y thread, though.

that would make sense.  I would think you could call it once when you start the thread, and hold a reference to it for the life of the process?  Then you'd just have to share that result between threads.

> > Actually if you can't do that we might be able to setup a fake HWND on that
> > thread whose only purpose is to handle WM_GETOBJECT.  We'd need to trick
> > screen readers into only caring about that window somehow, I'm not sure how
> > screen readers decide what windows to care about, but jamie probably does.
> I considered this too, but I have two concerns with this idea:
> 1. If a user uses the screen reader to walk through the top level windows,
> it'll hit the main HWND. It may even skip the a11y window if it's invisible
> or the like.

I believe we explicitly make the fake HWNDs we create now visible and with correct bounds for one of the proprietary screen readers.

> 2. We sometimes care about GetGuiThraedInfo(...).hwndFocus; e.g. when
> needing to fetch the focus on-demand. That'll be your main HWND, not the
> a11y HWND, so getting the focus on-demand won't work.

yeah, I don't think we can do anything about that.

Comment 4

2 years ago
(In reply to Trevor Saunders (:tbsaunde) from comment #3)
> > The perf hit is the fact that you now proxy a11y queries to the content
> > process. As long as you can avoid an extra level of indirection, it
> > shouldn't get any worse.
> yes, I was thinking if you added indirection by going to the main thread
> that would hurt more.
Absolutely. However, if you do that, you may hit different incarnations of some of your existing problems because the main thread will still get interrupted/re-entered by a11y queries.

> > However, I *think* you might be able to call
> > LResultFromObject in the a11y thread and then return *this* value in
> > WM_GETOBJECT.
> that would make sense.  I would think you could call it once when you start
> the thread, and hold a reference to it for the life of the process?  Then
> you'd just have to share that result between threads.
Afraid not. I'm pretty sure LResultFromObject is single use only. Once a client uses it, that LResult is gone. You might be able to acquire a pool of them and have the a11y thread periodically re-stock the pool, though. Or I guess you can send WM_GETOBJECT to an a11y HWND and make sure that is really fast.

> > 1. If a user uses the screen reader to walk through the top level windows,
> > it'll hit the main HWND. It may even skip the a11y window if it's invisible
> > or the like.
> I believe we explicitly make the fake HWNDs we create now visible and with
> correct bounds for one of the proprietary screen readers.
Doesn't solve the problem of it hitting the main HWND, though.

> > 2. We sometimes care about GetGuiThraedInfo(...).hwndFocus; e.g. when
> > needing to fetch the focus on-demand. That'll be your main HWND, not the
> > a11y HWND, so getting the focus on-demand won't work.
> yeah, I don't think we can do anything about that.
That'll almost certainly be a problem.
(In reply to James Teh [:Jamie] from comment #2)
> I don't think you'll be able to use a COM object from a different apartment
> without marshalling it first. I suspect (but am not sure) calling
> LResultFromObject in your main thread with an object from your a11y thread
> will throw RPC_E_WRONG_THREAD. However, I *think* you might be able to call
> LResultFromObject in the a11y thread and then return *this* value in
> WM_GETOBJECT. That raises the question of how you're going to make that call
> in the a11y thread, though.

Thanks Jamie. This is definitely what I was most worried about.

I looked at the Wine implemenation of LResultFromObject. It marshals the object to a stream, copies the stream to a shared memory region, creates a string containing the HANDLE for the shared memory region as an integer, and adds that string to the global atom table. The index of this entry in the global atom table is the LRESULT. It's possible that actual Windows does nothing like this of course.

I think the important question is what CoMarshalInterface does if you give it a proxy object that references another thread's object. Will it try to ask the other thread's object to marshal itself? Or will it try to marshal the proxy, so that calls always go through the proxy first? I tried to read the Wine code, but it's pretty complicated and I can't figure out what happens.

Anyway, it seems like sending WM_GETOBJECT to the a11y thread would probably work. Since it has nothing to do but answer a11y queries, it won't be doing anything else at the time. And I'm guessing that screen readers will cache the root object once they get it, so this should be an uncommon operation.
(In reply to James Teh [:Jamie] from comment #2)
> I don't think you'll be able to use a COM object from a different apartment
> without marshalling it first. I suspect (but am not sure) calling
> LResultFromObject in your main thread with an object from your a11y thread
> will throw RPC_E_WRONG_THREAD. However, I *think* you might be able to call
> LResultFromObject in the a11y thread and then return *this* value in
> WM_GETOBJECT. That raises the question of how you're going to make that call
> in the a11y thread, though.

Looking at how we currently implement and instantiate IAccessibles, I agree that we would need to implement some marshaling to make this happen. I've been reading up on the COM utility functions that are necessary to do this. After wading into this for a bit, now that I grok the details, I've concluded that it is pretty straightforward to do.

(In reply to Bill McCloskey (:billm) from comment #5)
> I think the important question is what CoMarshalInterface does if you give
> it a proxy object that references another thread's object. Will it try to
> ask the other thread's object to marshal itself? Or will it try to marshal
> the proxy, so that calls always go through the proxy first? I tried to read
> the Wine code, but it's pretty complicated and I can't figure out what
> happens.

From what I've read, the COM runtime is able to recognize when it's being asked to marshal a proxy and automagically do the Right Thing (TM). In fact, Raymond Chen just did a series on this topic just a couple of weeks ago:

http://blogs.msdn.com/b/oldnewthing/archive/2015/10/20/10648886.aspx
http://blogs.msdn.com/b/oldnewthing/archive/2015/10/21/10649190.aspx
http://blogs.msdn.com/b/oldnewthing/archive/2015/10/22/10649480.aspx
http://blogs.msdn.com/b/oldnewthing/archive/2015/10/23/10649707.aspx

Personally I am inclined to implement this via CoMarshalInterface and CoUnmarshalInterface to ensure maximum compatibility with various usage scenarios, as well as friendliness with smart pointers.
Thinking about this has also made me wonder: since COM proxies can marshal across process boundaries, is it possible and desirable in the context of a11y to do some of that outside of our own IPC? ie have IAccessibles in the chrome process that directly proxy to IAccessibles in the content process?

I admit that I am not sufficiently familiar with our a11y implementation to know whether or not there is any use for that idea in this case...

Comment 8

2 years ago
(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #6)
> > I think the important question is what CoMarshalInterface does if you give
> > it a proxy object that references another thread's object. Will it try to
> > ask the other thread's object to marshal itself? Or will it try to marshal
> > the proxy, so that calls always go through the proxy first?
We've spent a fair bit of time in the past pondering this question too. :)

> From what I've read, the COM runtime is able to recognize when it's being
> asked to marshal a proxy and automagically do the Right Thing (TM).
We have this horrible hack where, for insane privilege reasons, we need to call GetActiveObject in a helper process with lower integrity and then get that object to NVDA. We do this by using LresultFromObject in the helper process, passing this via stdout and then calling ObjectFromLresult in NVDA. As long as we do the ObjectFromLresult call before the helper exits, this works just fine and continues to work once the helper exits. Assuming other marshalling works the same way (and it'd be insane if it didn't), that would suggest COM is smart enough to short circuit the indirection.

(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #7)
> Thinking about this has also made me wonder: since COM proxies can marshal
> across process boundaries, is it possible and desirable in the context of
> a11y to do some of that outside of our own IPC? ie have IAccessibles in the
> chrome process that directly proxy to IAccessibles in the content process?
My understanding was that the content sandbox didn't allow COM through. Otherwise, there'd be no need to proxy anything; you'd just pass a marshalled COM pointer when WM_GETOBJECT was called. That'd certainly make life a hell of a lot easier and more performant if that were possible. FWIW, I'm pretty sure this is done by at least one other product with a sandboxed content process. I believe that trick only works on Vista and above for some reason; I'm not quite sure why.

Comment 9

2 years ago
(In reply to Bill McCloskey (:billm) from comment #0)
> Basically, this mechanism would allow us to process accessibility queries
> asynchronously. I don't know if it's feasible, but I think it's worth a try.
> I talked to Aaron Klotz and he said it seems reasonable. I think the crucial
> piece is the ability to return an IAccessible from a different apartment in
> the WM_GETOBJECT message handler.

Help me here, I don't see how this solves the problem of sync calls to content for a11y responses. You can decouple incoming a11y calls by using the separate thread and COM marshaling between the new thread and the browser main thread (I think your goal here is to solve the problem of a11y calls coming in during sync ipc calls with a11y on the stack?), but once that a11y call reaching the main thread through COM, you still need sync access to the content process. Or am I missing something here? :)

Comment 10

2 years ago
One other item we should think about, I believe a11y clients often muck with our hwnds and make gui related api calls associated with our windows. This would mean the main event queues are attached. So when the main thread of the a11y client is bound up waiting on something, the event processing on our main thread might stop as well. Maybe Jamie can offer some insight here. We should prototype this up independently to look for issues. I'd be happy to help with that.
(In reply to Jim Mathies [:jimm] from comment #9)
> (In reply to Bill McCloskey (:billm) from comment #0)
> > Basically, this mechanism would allow us to process accessibility queries
> > asynchronously. I don't know if it's feasible, but I think it's worth a try.
> > I talked to Aaron Klotz and he said it seems reasonable. I think the crucial
> > piece is the ability to return an IAccessible from a different apartment in
> > the WM_GETOBJECT message handler.
> 
> Help me here, I don't see how this solves the problem of sync calls to
> content for a11y responses. You can decouple incoming a11y calls by using
> the separate thread and COM marshaling between the new thread and the

no, you'd use something other than COM to communicate between the a11y thread and the main thread (or ideally just send ipc messages directly to the child).

> browser main thread (I think your goal here is to solve the problem of a11y
> calls coming in during sync ipc calls with a11y on the stack?), but once

well, a11y calls when we are already handling one a11y is one case, but another is that there's an a11y call while waiting for a cpow to respond.

> that a11y call reaching the main thread through COM, you still need sync
> access to the content process. Or am I missing something here? :)

well, because you talk to the main thread with something other than COM you can just block dealing with it until we aren't already blocking on a sync ipc message.  So you could say dispatch a nsIRunnable to the main thread, and then wait on a condvar until that runnable is run at which point it'll write the result somewhere and signal the condvar.


(In reply to Jim Mathies [:jimm] from comment #10)
> One other item we should think about, I believe a11y clients often muck with
> our hwnds and make gui related api calls associated with our windows. This
> would mean the main event queues are attached. So when the main thread of
> the a11y client is bound up waiting on something, the event processing on
> our main thread might stop as well. Maybe Jamie can offer some insight here.

honestly I don't understand windows message queues well enough to say something useful.  My uninformed expectation is that the main thread can't block on random windows things, but that's probably bogus :)

This does make me realize the rather obvious fact that while changing the thread IAccessibles are accessed on is probably fine for out of process clients using COM marsheling it could well screw with things that inject themselves into our process, but maybe not?  I still really don't understand windows threading :(

> We should prototype this up independently to look for issues. I'd be happy
> to help with that.

yeah, we should definitely test it before spending too much time on it.

Comment 12

2 years ago
(In reply to Bill McCloskey (:billm) from comment #5)
> I think the important question is what CoMarshalInterface does if you give
> it a proxy object that references another thread's object. Will it try to
> ask the other thread's object to marshal itself? Or will it try to marshal
> the proxy, so that calls always go through the proxy first? I tried to read
> the Wine code, but it's pretty complicated and I can't figure out what
> happens.

Ok thanks, I see why this is so critical now.

Comment 13

2 years ago
> (In reply to Jim Mathies [:jimm] from comment #10)
> > One other item we should think about, I believe a11y clients often muck with
> > our hwnds and make gui related api calls associated with our windows. This
> > would mean the main event queues are attached. So when the main thread of
> > the a11y client is bound up waiting on something, the event processing on
> > our main thread might stop as well. Maybe Jamie can offer some insight here.
> 
> honestly I don't understand windows message queues well enough to say
> something useful.  My uninformed expectation is that the main thread can't
> block on random windows things, but that's probably bogus :)

Yeah it can. If the two queues are attached, and one stops processing events, the other stops seeing its events because events have to be delivered in order for both threads. This applies to attached queues that are in separate processes as well. I'm concerned that if we have a client that is blocked on a sync a11y request we could lock up too. I don't know how a11y clients handle their requests though, do they do those on a background htread, or do they make sync calls on the IAccessible proxy they have on the main thread? I'm guessing main thread.

> This does make me realize the rather obvious fact that while changing the
> thread IAccessibles are accessed on is probably fine for out of process
> clients using COM marsheling it could well screw with things that inject
> themselves into our process, but maybe not?  I still really don't understand
> windows threading :(

What do they do when they inject into our process? Do they instantiate our a11y library directly and make accessibility calls?

Comment 14

2 years ago
(In reply to Jim Mathies [:jimm] from comment #10)
> One other item we should think about, I believe a11y clients often muck with
> our hwnds and make gui related api calls associated with our windows. This
> would mean the main event queues are attached.
What sort of "mucking" are you thinking? Do you mean just AttachThreadInput or something else? We don't ever use AttachThreadInput in NVDA (except in one insane case in Microsoft Word where Word refuses to cooperate with us otherwise). I can't speak for other ATs, though.

> So when the main thread of
> the a11y client is bound up waiting on something, the event processing on
> our main thread might stop as well.
I can't think of any case where NVDA would cause that. Again, can't speak for other ATs.

(In reply to Trevor Saunders (:tbsaunde) from comment #11)
> This does make me realize the rather obvious fact that while changing the
> thread IAccessibles are accessed on is probably fine for out of process
> clients using COM marsheling it could well screw with things that inject
> themselves into our process
It should still work, but in-process could definitely be a bit slower (even slower than it already is with proxies) because of the cross-thread marshaling. That *could* be ugly with buffers.

(In reply to Jim Mathies [:jimm] from comment #13)
> What do they do when they inject into our process? Do they instantiate our
> a11y library directly and make accessibility calls?
We use the same a11y calls as we do out-proc. However, because of the way Windows works, there are two key differences:
1. WM_GETOBJECT will still hit the WindowProc, but not via the message queue; I believe in-context messages go directly to the WindowProc.
2. Win events are synchronous, not asynchronous, because in-context event callbacks are called directly from NotifyWinEvent. ATs sometimes rely on this assumption; i.e. that the tree will remain unchanged during the event callback. NVDA rarely does this, though it does (has no choice in fact) for live regions. This makes me realise live regions are probably going to be very broken by e10s, but that's a separate issue...
Jamie, does NVDA run in the Firefox process or in a separate process?

Comment 16

2 years ago
(In reply to James Teh [:Jamie] from comment #14)
> (In reply to Jim Mathies [:jimm] from comment #10)
> > One other item we should think about, I believe a11y clients often muck with
> > our hwnds and make gui related api calls associated with our windows. This
> > would mean the main event queues are attached.
> What sort of "mucking" are you thinking? Do you mean just AttachThreadInput
> or something else? We don't ever use AttachThreadInput in NVDA (except in
> one insane case in Microsoft Word where Word refuses to cooperate with us
> otherwise). I can't speak for other ATs, though.

Any win32 gui apis will automatically attach input queues, so calls like Set/GetFocus, GetWindowText, etc.. Anything that takes an HWND parameter of a window owned by another thread or process.

Comment 17

2 years ago
(In reply to Bill McCloskey (:billm) from comment #15)
> Jamie, does NVDA run in the Firefox process or in a separate process?
Both:
1. We do most interaction out-proc; e.g. reporting focus, property changes, object navigation, etc.
2. In most browsers (including Firefox), for performance reasons, we build our own complete representation of the document in-proc. We then query this representation remotely, but queries are answered on our own thread.
3. We have to do live regions in-proc because they require synchronous events.

(In reply to Jim Mathies [:jimm] from comment #16)
> Any win32 gui apis will automatically attach input queues, so calls like
> Set/GetFocus, GetWindowText, etc.. Anything that takes an HWND parameter of
> a window owned by another thread or process.
With fear of questioning someone who probably knows more about this than I do, are you absolutely certain of this? It seems pretty insane that a single call would cause the queues to be attached forever; we'd see a lot of freezes all over the place if that were true. And if you just meant they're attached for the duration of the call, I don't follow why that would matter because we can't make a11y calls while we're also making a Win32 call. With particular reference to GetWindowText, MSDN is pretty clear that this doesn't even go cross-proc:

> If the target window is owned by the current process, GetWindowText causes a WM_GETTEXT message to be sent to the specified window or control. If the target window is owned by another process and has a caption, GetWindowText retrieves the window caption text. If the window does not have a caption, the return value is a null string. This behavior is by design. It allows applications to call GetWindowText without becoming unresponsive if the process that owns the target window is not responding.
(https://msdn.microsoft.com/en-us/library/windows/desktop/ms633520%28v=vs.85%29.aspx)

Finally, SetFocus doesn't work cross-thread/proc *unless* you AttachThreadInput first. MSDN confirms this:

> By using the AttachThreadInput function, a thread can attach its input processing to another thread. This allows a thread to call SetFocus to set the keyboard focus to a window attached to another thread's message queue.
(https://msdn.microsoft.com/en-us/library/windows/desktop/ms646312%28v=vs.85%29.aspx)

Comment 18

2 years ago
Btw, I believe some other ATs are almost entirely in-proc, but obviously I can't be certain of this. NVDA is probably more out-proc than most.
Raymond Chen's blog [1, under "Bonus Reminder"] seems to say that you can only get accidentally attached input queues from two windows that are a parent/child or owner/owned relationship. It sounds like NVDA doesn't do this.

[1] http://blogs.msdn.com/b/oldnewthing/archive/2013/06/07/10424279.aspx

Comment 20

2 years ago
(In reply to Bill McCloskey (:billm) from comment #19)
> Raymond Chen's blog [1, under "Bonus Reminder"] seems to say that you can
> only get accidentally attached input queues from two windows that are a
> parent/child or owner/owned relationship. It sounds like NVDA doesn't do
> this.
> 
> [1] http://blogs.msdn.com/b/oldnewthing/archive/2013/06/07/10424279.aspx

This is good news, I wasn't aware that the auto attachment was restricted to this relationship.
Now I'm worried this isn't going to work. If the screen reader injects code that runs on our main thread, then the main thread is going to be busy while we're handling the accessibility query. Even if we communicate directly from the a11y thread to the content process, it's still possible that the content process will need to synchronously message the main process in order to handle the a11y query. This could happen if the a11y query changes focus, since we need to synchronously message the main process for some IME stuff when that happens.
(In reply to Bill McCloskey (:billm) from comment #21)
> Now I'm worried this isn't going to work. If the screen reader injects code
> that runs on our main thread, then the main thread is going to be busy while

I don't entirely know where their code runs, but some of it at least probably runs on the main thread.

> we're handling the accessibility query. Even if we communicate directly from
> the a11y thread to the content process, it's still possible that the content
> process will need to synchronously message the main process in order to
> handle the a11y query. This could happen if the a11y query changes focus,
> since we need to synchronously message the main process for some IME stuff
> when that happens.

I think we can require that the a11y messsages don't trigger sending sync messages to the parent main thread.  In fact we already require that to not dead lock.

a11y sends a few async messsages to the child to do things like focus for this exact reason.  That probably isn't quiet to spec, but its probably fine :/

Comment 23

2 years ago
(In reply to Trevor Saunders (:tbsaunde) from comment #22)
> I don't entirely know where their code runs, but some of it at least
> probably runs on the main thread.
Some will run on the thread that owns the HWND. Some will run on the thread that calls NotifyWinEvent. These should generally be the same. This raises an interesting point: I don't think we had considered that NotifyWinEvent really should be called in the same thread that owns the HWND. Otherwise, ATs will be switching apartments unexpectedly which can cause problems if they happen to hold objects from previous calls.
(In reply to James Teh [:Jamie] from comment #23)
> (In reply to Trevor Saunders (:tbsaunde) from comment #22)
> > I don't entirely know where their code runs, but some of it at least
> > probably runs on the main thread.
> Some will run on the thread that owns the HWND. Some will run on the thread
> that calls NotifyWinEvent. These should generally be the same. This raises
> an interesting point: I don't think we had considered that NotifyWinEvent
> really should be called in the same thread that owns the HWND. Otherwise,
> ATs will be switching apartments unexpectedly which can cause problems if
> they happen to hold objects from previous calls.

It should be pretty straight forward to forward a11y events back to the main thread, or to fire them on the a11y thread if we used a11y specific HWND.

That said I wonder if we're getting to the point that we'd be better off just going to caching everything now.  A prototype test would be really usefu, but I think really if we can make something work we really should try that, I don't really see how we could get all the caching done in less than "many months" :/
tracking-e10s: --- → +
Ok, so what do people think here?  DOes this stand some chance of working, and does anyone think they could try it?
I'm going to close it. I don't think it will work, given the issues with DLL injection on our main thread.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.