Closed Bug 398013 Opened 17 years ago Closed 15 years ago

Common on-screen dictionary lookup programs can't retrieve text under mouse cursor

Categories

(Core :: Graphics, defect)

Product:
Core
Component:
Graphics
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: yaoziyuan, Unassigned)

References

Details

(Keywords: regression)

Attachments

(2 files)

Firefox 2.0.0.x with StarDict: What's Desired
65.55 KB, image/png
Details
Minefield 3 with StarDict: What's NOT Desired
53.07 KB, image/png
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a9pre) Gecko/2007092904 Minefield/3.0a9pre

Since Minefield, dictionary programs NO LONGER can retrieve the word in the browser right under the user's mouse thus no longer can show translation results for the on-screen word.

This bug does not exist in Firefox 2.0.0.x.

Such dictionary programs (such as StarDict, Kingsoft XDict) usually intentionally draw a spot at the mouse position, invalidating the GUI region that contains this spot, and then use Windows API hooks to intercept the upcoming TextOut() call (which redraws the text at that position) in order to get the text at the mouse position. Other dictionary programs, such as Babylon 5, which use OCR algorithms to recognize on-screen text right under the user's mouse, will not be affected.

Screen readers and other accessibility software, such as Windows XP Narrator, which usually listen to Microsoft Accessibility (MSAA) notifications and messages generated by the target application (Firefox) in order to retrieve major screen events such as a menu popups, do not retrieve web page text right under the mouse, so they are not concerned in this bug report.


Reproducible: Always

Steps to Reproduce:
1. Download and install StarDict, a free, open source and cross-platform dictionary program from http://stardict.sourceforge.net , or directly from http://stardict.sourceforge.net/download/stardict-3.0.1-beta.exe

2. Start Minefield (preferably the latest thunk)
3. Start StarDict. Move your mouse onto any English word on a web page loaded in Minefield.
Actual Results:  
StarDict would not correctly recognize the word under your mouse but just return a random string. StarDict's on-screen word lookup function would work with other normal Windows applications.

Expected Results:  
StarDict should correctly recognize the under-your-mouse word on a Minefield web page and give Chinese translations for that word.
Keywords: regression
Version: unspecified → Trunk
A screenshot of Firefox 2.0.0.x with StarDict correctly retrieving the word under mouse.
A screenshot of Minefield with StarDict incorrectly retrieves the word under mouse.
Are you sure it uses MSAA for this? MSAA cannot be used to find a word at the mouse pointer -- it will return an object via AccessibleObjectFromPoint(), but that could e an entire paragraph.

What MSAA call is no longer returning the same results?

We need more information to resolve this bug.
MSAA has NOTHING TO DO with this whole thing. I mentioned MSAA in the original report just to PREVENT you from taking MSAA into consideration...

What goes wrong is Minefield's TextOut() calls (if they ever exist in Minefield) are no longer intercepted by dictionary programs (see Paragraph 3 in original report).

You can download StarDict's source code from http://sourceforge.net/projects/stardict/ and see how these dictionary programs do the hook job.


(In reply to comment #3)
> Are you sure it uses MSAA for this? MSAA cannot be used to find a word at the
> mouse pointer -- it will return an object via AccessibleObjectFromPoint(), but
> that could e an entire paragraph.
> 
> What MSAA call is no longer returning the same results?
> 
> We need more information to resolve this bug.
> 


OCR-based dictionary programs mentioned in Paragraph 3 of original report and MSAA-based accessibility programs mentioned in Paragraph 4 of original report have nothing to do with the bug here. The bug solely applies to TextOut()-hook-based dictionary programs that want to get the word on a web page in Minefield right under your mouse.
Obviously it's not disability access API bug.
Exactly.
Assignee: aaronleventhal → nobody
Component: Disability Access APIs → Layout
QA Contact: accessibility-apis → layout
StarDict's author, Hu Zheng, is my friend and is on my MSN buddy list. His MSN is huzheng_001@hotmail.com. You can contact him for questions.
Note that this bug does not only affect StarDict. StarDict represents a class of dictionary programs that use the same on-screen word retrieval technique. The most widely used dictionary program in China, Kingsoft XDict, is also of this class.
Another free dictionary program for Windows, Lingoes (http://www.lingoes.net) can also illustrate this bug.
I don't think that Firefox doesn't use TextOut() anymore to draw text on the screen, so this trick wouldn't work anymore.
So either Firefox 3 should add a dummy (transparent? font size 0? invisible canvas? etc.) TextOut() call to look as if it still uses TextOut() to draw text and notify dictionary programs what's printed there, or dictionary programs must include a plugin which is a Firefox 3 extension that communicates with Firefox...

I post here only to confirm this bug.
In my experience I've always used babylon with any browser and no problems occurred.

I've just installed F3 and now when I click (with the central mouse button) on a word to translate it it seems as if I've just aimed a few spaces AFTER.

I've also noticed that the problem disappear when I try to use the IE tab extension.

I really hope this bug will be fixed soon :)
Comment 20 is probably unrelated: comment 0 said that this specific problem doesn't affect the way that Babylon reads text from the screen. However, Babylon does also have problems in Firefox 3 which they say they are working to fix on their side -- see bug 410535 comment 3.
(In reply to comment #12)
> I don't think that Firefox doesn't use TextOut() anymore to draw text on the
> screen, so this trick wouldn't work anymore.

To be precise, Firefox uses ExtTextOut with the ETO_GLYPH_INDEX flag to draw text by glyph index rather than by character.
Component: Layout → GFX: Thebes
QA Contact: layout → thebes
I post this comment to confirm that this bug actually exists in the release version of the Firefox. After updating the Firefox from 2 to 3 the renowned dictionary program ABBYY Lingvo 12 can no longer show translation of a word under a mouse cursor. Safari 3.1 affects the Lingvo in the same way, just FYI. 


Pop up translation doesn't work with Firefox 3. Abby Lingvo 12 no longer showing translation under the mouse cursor. It must be Firefox 3 uses the same method as Firefox 2, there is a bug...
Already common knowledge, I guess. Just putting another name to the list- 

Firefox 2 read babyon 7 and 6, firefox 3 doesn't. Garbles something or reads a few words ahead.No problem double clicking on the word or dragging the mouse to copy. Doesn't matter what font size. IE 7 doesn't have that problem (I'd opt Firefox, which I can't use now when reading a foreign language book). Thanks in advance.
I am using Abbyy Lingvo 12's french-russian translator with FF 3.0.1, and of course have the same problem.
 
I don't know if it might be of any help, but I've notice that it DO works (a little window opens with the world translated) in some parts of a page, for example on the "suivant" (next) blue button on this page :

http://www.lemonde.fr/europe/portfolio/2008/07/22/liesse-a-sarajevo_1075803_3214.html

Maybe it might gives clues to debuggers ?


(In reply to comment #26)
> I don't know if it might be of any help, but I've notice that it DO works (a
> little window opens with the world translated) in some parts of a page, for
> example on the "suivant" (next) blue button on this page :
> 
> http://www.lemonde.fr/europe/portfolio/2008/07/22/liesse-
>a-sarajevo_1075803_3214.html

True, it works. But it works from an Adobe Flash object that probably uses their own rendering mechanism.


Babylon 5.0.5 have the same bug in Firefox 3. In Firefox 2 its work correctly. This is very annoying.
Issues with Babylon were not the same as this bug, and in any case are fixed in the latest version, see bug 410535 (or just read earlier comments).
StarDict has a patch with a workaround, which tries to fix this (it just adds 29 to all bytes in the string, which was retrieved from Firefox), but it works only for ANSII latin symbols (patch was rejected). It seems that non-ANSII symbols are returned in the unicode (4 bytes), but all bytes are corrupted too, so they are unreadable.

https://sourceforge.net/tracker/?func=detail&aid=2927658&group_id=80679&atid=560634
(In reply to comment #0)
> Such dictionary programs (such as StarDict, Kingsoft XDict) usually
> intentionally draw a spot at the mouse position, invalidating the GUI region
> that contains this spot, and then use Windows API hooks to intercept the
> upcoming TextOut() call (which redraws the text at that position) in order to
> get the text at the mouse position.

While this is a neat hack, it's not in any way an application requirement to draw text in this way.  Since version 3.0, Firefox uses ExtTextOut with ETO_GLYPH_INDEX to draw text -- that is, it draws specific glyph indices, not characters, at specific locations.  It's possible for an application to go the reverse direction, from glyph indices back to unicode characters, but it's not trivial and will not be 100% exact as there isn't a 1:1 relationship.  (A character can result in multiple glyphs, or multiple characters can map to one glyph... or different character sequences might also map to the same glyphs.)  The "add 29" approach seems to work for ACSII because fonts often have the ASCII glyphs at specific locations, but there's no guarantee.

At some point in the near future, we may not use GDI to draw text at all.  I would guess that as apps rely less and less on GDI that this problem will come up more and more; does the dictionary app work with apps that use Direct2D for rendering, for example?  Or any WPF/XAML apps?

Regardless, this is WONTFIX on our end, unfortunately.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → WONTFIX
I don't think that's right thing to wontfix bugs about Firefox integration with OS. I believe Firefox must work with Dictionary, must support CMD+E on OS X because that's the way how OS X application are about to work. If there is no way to fix this bug let's keep it open until things are changed.
This isn't OS integration -- the OS does not ask for this functionality, nor is it a standard expected behaviour (like using Growl on the Mac for notification).
(In reply to comment #34)
> This isn't OS integration -- the OS does not ask for this functionality, nor is
> it a standard expected behaviour (like using Growl on the Mac for
> notification).

Possibly I used wrong term but nevertheless OS X applications (like Safary, XCode and etc) works with Dictionary. I believe the fact Firefox isn't compatible with Dictionary makes Firefox users unhappy.
This bug is a filed for Windows XP. Stardict and Kingsoft XDict are windows applications, they use an (unsupported) GDI trick.
(In reply to comment #36)
> This bug is a filed for Windows XP. Stardict and Kingsoft XDict are windows
> applications, they use an (unsupported) GDI trick.

Sorry, I thought this bug is about OS X as well, bug on OS X was marked as duplicate but it wasn't.
(In reply to comment #32)
> (In reply to comment #0)
> > Such dictionary programs (such as StarDict, Kingsoft XDict) usually
> > intentionally draw a spot at the mouse position, invalidating the GUI region
> > that contains this spot, and then use Windows API hooks to intercept the
> > upcoming TextOut() call (which redraws the text at that position) in order to
> > get the text at the mouse position.
> 
> While this is a neat hack, it's not in any way an application requirement to
> draw text in this way.  Since version 3.0, Firefox uses ExtTextOut with
> ETO_GLYPH_INDEX to draw text -- that is, it draws specific glyph indices, not
> characters, at specific locations.  It's possible for an application to go the
> reverse direction, from glyph indices back to unicode characters, but it's not
> trivial and will not be 100% exact as there isn't a 1:1 relationship.  (A
> character can result in multiple glyphs, or multiple characters can map to one
> glyph... or different character sequences might also map to the same glyphs.) 
> The "add 29" approach seems to work for ACSII because fonts often have the
> ASCII glyphs at specific locations, but there's no guarantee.
> 
> At some point in the near future, we may not use GDI to draw text at all.  I
> would guess that as apps rely less and less on GDI that this problem will come
> up more and more; does the dictionary app work with apps that use Direct2D for
> rendering, for example?  Or any WPF/XAML apps?
> 
> Regardless, this is WONTFIX on our end, unfortunately.


Can you tell me how to go the reverse direction to convert glyph indices to unicode characters? Or can you show me an example code?
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: