218887 - (uniscribe) Use Uniscribe APIs in GFX:Win and layout/editor

Status: RESOLVED DUPLICATE of bug 322938

(Core Graveyard :: GFX: Win32, defect)

Description

[Jungshik Shin]

This is a Windows counterpart of bug  215219, bug 214715(Linux) and bug
121540(MacOS). At the moment, I'm filing this bug as a place holder to offer a
forum for discussion. 

Currently, Mozilla relies on ExtTextOutW and friends for text drawing. That more
or less works even for complex scripts as far as rendering is concerned, but
that doesn't work for selection(copy'n'pasting) and editing (see bug 79286, bug
100173, bug 122552, bug 157534). Moreover, even rendering doesn't work on Win
9x/ME (which is why I'm settign the platform to  Windows ME. see bug 166520, for
instance). With custom font encoding support, some of this problem has been
fixed for Tamil, Korean, Thai and Devanagari (the last one only with CTL
enabled), but that solution is not general enough.  

Given all these, it's pretty obvious that we need to use Uniscribe APIs (as is
done by MS IE) (in the long run).

Comment 1

[Rajkumar S]

> Given all these, it's pretty obvious that we need to use Uniscribe APIs (as is
> done by MS IE) (in the long run).

I feel that mozilla should support OS like win98 and win2k as they are going to
be popular for a long time to come. Here in India very few people are going to
upgrade to XP or heigher. Even the current XP do not have Malayalam support. So
we may not want to force people to upgrade their OS to get local language suport. 

Is it possible to get pango ported in Windows? I am no expert in it, but just a
question, and use that if Uniscribe is not available?

raj

Comment 2

[Jungshik Shin]

I'm afraid you missed my point of filing this bug. Using Uniscribe APIs would
enable Mozilla to support complex scripts on Win 9x/ME as well as on Win 2k/XP.
(it's not Win 9x/ME/2k vs Win XP but Win9x/ME vs Win 2k/XP. Win 2k and XP are
basically the same OS which is very different from Win 9x/ME. It's easy to be
fooled by MS' strange versioning of its OS'). 

Well, the complex script support of Uniscribe is available on Win 9x/ME _only_
if MS IE 6 is installed. So, we have to tell users to install MS IE6 to enable
Mozilla to render complex scripts on Win 9x/ME. This is a downside of using
Uniscribe. We can do a run-time detection of the presence of Uniscribe, but that
wouldn't help us with cutting down the code size, which is another objective of
using Uniscribe APIs.   

As for Pango, Win32 port is available, but it's implemented with Uniscribe APIs
internally. Even if it's not, I guess making the default build of Mozilla-Win
dependent on Pango would be out of question.

Comment 3

[Mike Kaply [:mkaply]]

I don't think requiring the users to get the Uniscribe DLL for Win9X/ME is a bad
thing if Mozilla supported Uniscribe.

Does anyone have a sense for how big this effort would be?

We (IBM) might be interested in doing some work on this.

Comment 4

[Jungshik Shin]

I have little idea. It could be a pretty extensive work touching both gfx-win
and layout/content. gfx-win part should be easier than fixing nsText*.cpp.
We may phase in changes a little by little.

Another possibility in complex script support is using SIL Graphite. Frank Tang
has worked on it. See http://sila.mozdev.org. 

BTW, Uniscribe's Korean support is worse than Mozilla's current support based on
custom-encoded fonts.  

Another BTW, we have to be careful not to hurt MathML support (that relies on
custom font encoding). On Linux, going all the way to Pango could 'sacrifice'
MathML support.

Comment 5

[Jungshik Shin]

After working on bug 215219 (Gfx-Gtk2 counterpart of this bug using Pango), I
realized that a similar approach can be applied to this bug because Uniscribe
and Pango have something common in their APIs.
(I'll take a look at ATSUI APIs on Mac OS X as well)  When I filed this bug, I
intended to use Uniscribe for text selection and cursor movement as well as for
text-drawing/width measurement. However, now I plan to use Uniscribe only for
the text drawing and width measurement. For text selection and cursor movement,
an XP-solution (bug 229896) can be made to work more or less.  

This may not be the best approach, but seems to be the path with the least
'resistance'.

Comment 6

[Jungshik Shin]

Just like Xft has an API to use glyph indices instead of Unicode characters for
text drawing, ExTextOutW on Win32 has a flag to indicate glyphs are being passed
over (ETO_GLYPH_INDEX).

With this approach (using Uniscribe to only a limited extent), we can keep
MathML and custom-encoded font support (necessary because Uniscribe's Korean
handling is worse than Mozilla's : bug 176315. also handy in rare cases where
Uniscribe is not available.) intact while takind advantage of Uniscribe's
capability to render complex scripts. 

Comment 7

[tor]

Attachment: experiment using uniscribe in win gfx

A quick experiment to hook up uniscribe to the windows font code, complete
with debug spew.  The attempt to bridge the difference between layout's
idea of character spacing and uniscribe glyph based spacing is a complete
hack and doesn't work right.

Comment 8

[tor]

Attachment: updated patch

Spacing now somewhat working - spaced strings don't match the length
layout is expecting (since it tries adding space based on characters),
but seems like the right amount is being added.

Comment 9

[Jungshik Shin]

tor, great ! attachment 138332 [details] has a test case for justified text rendering.
I've just built a firebird with your patch applied and it seems like your patch
is at about the same status as my patch for Xft with pango was around last
September (patch v 0.3 to bug 215219). Anyway, it's a good start. 

BTW, although it's not necessary at this stage, we later have to add a test for
the presence of Uniscribe and have to fall back if it's not present. 

Comment 10

[Jungshik Shin]

Attachment: Malayalam test case

You need to install Thoolika opentype font to test this page.

http://www.supersoftweb.com/Unicode.htm

Comment 11

[Jungshik Shin]

I found that MS IE's (or perhaps Uniscribe's) notion of 'grapheme cluster' is
different from that of Pango (I relied on in bug 215219). MS IE renders the
second and the third columns of attachment 140515 [details] differently in some cases. My
patched version of Xft+gtk2 on Linux renders them identically because Pango's
notion of grapheme clusters in Indic scripts is the same as mine. However, this
is not a matter of who is right  and who is wrong because even the Unicode
Technical committee  gave up defining what constitutes 'atomic' units  in Indic
scripts. 

Comment 12

[tor]

Attachment: rendering of http://bugzilla.mozilla.org/attachment.cgi?id=138332&action=view

Rendering of the pango spacing testcase with my current uniscribe work
(later than the attached patch).  The justification case looks a little
cramped, but seems to reflect the narrow interword spacing that layout
is passing down to gfx.

Comment 13

[Jungshik Shin]

It's working ! Unicode characters making up a single grapheme cluster are held
together correctly regardless of the letter spacing specified. I have some doubt
about 'justified' case in the screenshot, but that's minor. Why don't you
install Thoolika font for Malayalam and test attachment 140515 [details] as well? The font
is available at http://www.supersoftweb.com/Unicode.htm

Comment 14

[tor]

Attachment: uniscribe rendering of Malayalam test

After fixing a couple problems it gets close to the correct rendering,
missing only testcases four and five.  I'll investigate further
tomorrow, unless you happen to know offhand of a bug in the Thoolika
font causing this.

Comment 15

[tor]

Attachment: another experimental patch

Comment 16

[tor]

*** Bug 234781 has been marked as a duplicate of this bug. ***

Comment 17

[tor]

Attachment: checkpoint

Different spacing conversion than the pango bridge patch, though I don't
think either really create the spacing that would result if we could
get text frame to work with graphemes.

Comment 18

[tor]

jshin, could you try the latest patch and let me know if you see problems?
Thanks.

Comment 19

[Jungshik Shin]

I'm building now on Windows (having just one machine with two OS' makes it
rather hard ..).  BTW, I'm afraid your current patch allocates too often. Why
don't you take a look at nsAutoBuffer and use it if/where possible? 

Comment 20

[tor]

Right now I'm more interested in getting the functionality right before
doing code cleanup.

Comment 21

[Jungshik Shin]

Comment on attachment 142786 [details] [diff] [review]
checkpoint

I've just tested a few test cases and it seems to work well. Thanks !

>+DEFINES	+= -DSTRICT -DDEBUG_tor

I guess you're gonna clean this up later, but even now, I wonder why you have
to hard-code this. If your Windows login name is 'tor', DEBUG_tor is
automatically defined for debug builds, isn't it? 

>+        int val = -1;
>+        for (int j=0; j<itemLength; j++) {
>+          if (val != clusters[j]) {
>+            val = clusters[j];
>+            spacing[j] = aSpacing[items[i].iCharPos + j];

We can do a little better than this, can't we? Of course, once bug 229896 is
fixed and layout makes use of new APIs (made in the fix), we don't have to
worry less about this.

Comment 22

[tor]

Attachment: use different layout/gfx space conversion

Comment 23

[tor]

I don't suppose anyone lurking in this bug happens to know about font
substitution and uniscribe, do they?  On the site
http://tdil.mit.gov.in/hindi_site/homepage.htm the unisribe code ends
up in problems because ScriptShape can't find a glyph it wants in the
chosen font (though it seems the right fonts are installed because
IE and a mozilla with tree font code look fine) and returns an error.

Comment 24

[Greg K.]

As with bug 121540 for Mac, this bug may depend on bug 157967.

Comment 25

[Sukh]

Any update on this?  I've been waiting for quite a while to see when the issues
with Indic text on Windows will be fixed.  If anyone has any updates please let
me know.

Thanks!

Comment 26

[Jungshik Shin]

*** Bug 294579 has been marked as a duplicate of this bug. ***

Comment 27

[Satish]

Telugu font rendering not proper with Firefox 1.0.4 under Windows XP. The
combination of letters (especially Vattulu) breaks.

Here is a page for seeing some Telugu text.

http://nosaku.blogspot.com

The text is fine with "Pango enabled Firefox 1.0.4 build" on Linux. But not
proper in the Firefox 1.0.4 on Windows XP (and 2000 too).

Any work going on on this?

Satish.

Comment 28

[H P Nadig]

https://bugzilla.mozilla.org/show_bug.cgi?id=377345 is the same as this.