508898 - Make plugin IME support more flexible on OS X

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Plug-ins, defect)

Description

[Steven Michaud [:smichaud] (Retired)]

A number of Adobe Flash developers have been working to add support
for inline IME to the Flash plugin.  But they've found this is
impossible with current Mozilla.org code, and have asked us to make a
couple of changes:

1) Use the application target instead of the event dispatcher target
   in NS_InstallPluginKeyEventsHandler() (in ChildView.mm) when
   installing the Carbon event handlers for plugin-based IME.

   (It appears that the Flash developers want to use the event
   dispatcher target themselves, to pre-empt the browsers event
   handlers.)

2) Stop manipulating the active TSM document on every keyDown event
   when a plugin is active.

As best I can tell, the first change won't cause any trouble.  But the
second change is (for various reasons) completely impractical -- at
least if we want to maintain compatibility with current versions of
the Flash plugin.

The Flash developers haven't told us much about how they're trying to
implement inline IME.  But (as I understand it) they aren't yet using
the new NPAPI Cocoa event model (which as it happens doesn't yet
support inline IME).  Clearly everyone's long-term goal should be to
get inline IME working in the NPAPI Cocoa event model, and use that.
But in the short/medium term I can understand the motivation to at
least try to support inline IME using the older NPAPI Carbon event
model.

With that in mind, I've created a patch that will (I hope) allow the
Flash developers to continue their work.  This patch (which I'll post
in my next comment), though fairly minimal, is still somewhat risky.
So I think it's too late to target the 1.9.2 branch.  But if we can
come up with a patch that 1) allows the Flash developers to implement
inline IME using the NPAPI Carbon event model and 2) doesn't cause any
regressions in current plugin support, it might be reasonable to try
to get it into the 1.9.3 branch.

Comment 1

[Steven Michaud [:smichaud] (Retired)]

No support of any kind for plugin IME was available on OS X before my
patch for bug 357670 landed.  When writing that patch I found that we
need to do considerable tweaking of TSM docs to allow plugin IME to
work properly.  There are two main reasons for this:

1) Each plugin's ChildView has its own TSM document, which (unless we
   intervene) is always made active whenever that ChildView gets the
   keyboard focus.  But (for reasons that I still don't fully
   understand) these TSM documents aren't compatible with the
   TSMProcessRawKeyEvent() hack that both we and WebKit use to
   implement support for non-inline plugin IME.

   So we must either create our own TSM document or use the one
   provided by the plugin (if it does provide one).

2) The TSM document created by current versions of the Flash plugin
   doesn't support the UnicodeDocument interface type.  So if we use
   it (instead of creating our own TSM document) we encounter problems
   like those described in bug 357670 comment #40 and bug 357670
   comment #51.

Neither of these two problems happen using current versions of the
Flash plugin in Safari.  Digging deep into Apple's system code, I find
that an additional (presumably special) TSM document gets created when
doing Flash IME in Safari that doesn't get created when doing Flash
IME in Firefox.  I haven't (yet) been able to figure out why this only
happens in Safari.  But I suspect Apple somehow special-cases IME
handling when running in a WebKit browser.  And I also suspect this
"extra" TSM document is what makes things work in WebKit that don't
work in Firefox.

I may eventually find the missing key to this behavior.  But until I
do (or someone else does), TSM document handling will inevitably be
more complex in Firefox than it is in WebKit browsers (at least until
we both transition to the NSAPI Cocoa event model).

(This comment is already long enough.  Now I'll post my patch in my
*next* comment.)

Comment 2

[Steven Michaud [:smichaud] (Retired)]

Attachment: Provisional patch

Current code (as of my patch for bug 357670) creates its own TSM
document and always uses it when doing text input in any plugin.  This
was how I got around problem 2 from comment #1.

But, though we definitely can't use the TSM document created by
current versions of the Flash plugin (because it uses the "input
window" and doesn't support the UnicodeDocument interface type), we
might be able to use a plugin-provided TSM document that *does*
support the UnicodeDocument interface type, or that *doesn't* use the
input window.  So this patch's code looks for a plugin-provided TSM
doc, and tests it if it finds it.  If these tests pass my patch makes
the plugin-owned TSM document active on every keyDown event.
Otherwise it creates its own TSM document and makes that active.

Michael, please test this patch and pass it along to your colleagues.
Let us know if it suits your purposes -- if you're able to get
Flash-based inline IME working with it.

A tryserver build will follow eventually.  But the tryservers are
jammed, so don't expect to see one before tomorrow morning.

Comment 3

[Steven Michaud [:smichaud] (Retired)]

Here's a tryserver build made with my patch from comment #2:

https://build.mozilla.org/tryserver-builds/[e-mail]-bugzilla508898/bugzilla508898-macosx.dmg

Comment 4

[Jeff Mott]

I removed Michael because he's no longer with Adobe.  I'll look at this next week, thanks for continuing to work on it.  Inline IME is a big deal for the next player release (player 10.1).

Comment 5

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

If Flash Player would notify us of proper position to show candidate window like Windows, we could improve the UX. However, I'm not sure how long do we keep supporting Flash Player...

FYI: Currently, composition string is shown under bottom edge of plugin.