Last Comment Bug 60546 - [BiDi] Unicode Hebrew/Yiddish Diacritics do not correctly align in some fonts.
: [BiDi] Unicode Hebrew/Yiddish Diacritics do not correctly align in some fonts.
Status: RESOLVED FIXED
: intl
Product: Core
Classification: Components
Component: Layout: Text (show other bugs)
: Trunk
: All All
: P3 normal with 19 votes (vote)
: ---
Assigned To: Mike Kaply [:mkaply]
:
:
Mentors:
http://www.mechon-mamre.org/i/t/t2801...
: 123218 127050 140788 184910 196453 210122 212808 214801 221399 257756 279490 279925 349532 387987 438558 (view as bug list)
Depends on: 214715 CTL2 uniscribe 333659
Blocks: 115713 bidi_relnotes 209468 240501
  Show dependency treegraph
 
Reported: 2000-11-17 19:35 PST by Arthur Barrett
Modified: 2008-08-03 09:26 PDT (History)
50 users (show)
bzbarsky: in‑testsuite?
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
The Font Frankruehl displayed correctly rendering this page in IE, wrong in Bidi-Mozilla (36.91 KB, image/gif)
2000-11-17 19:40 PST, Arthur Barrett
no flags Details
http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm in Mozilla font Tahoma (from Office 2000) (168.36 KB, image/jpeg)
2001-01-23 02:54 PST, Arthur Barrett
no flags Details
http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm in IE5.5 using Font Tahoma (from Office 2000) (164.61 KB, image/jpeg)
2001-01-23 02:56 PST, Arthur Barrett
no flags Details
Looks fine in Build#2002053012 (43.36 KB, image/png)
2002-06-28 00:17 PDT, Sagie Maoz
no flags Details
a png image which demonstrates incorrect rendering in Fx 2.0.0.4 (49.03 KB, image/png)
2007-06-13 09:05 PDT, Amir Aharoni
no flags Details
a png image which demonstrates incorrect rendering in Fx 3.a alpha 5 (47.12 KB, image/png)
2007-06-13 09:06 PDT, Amir Aharoni
no flags Details
a png image which demonstrates correct rendering in IE6 (62.17 KB, image/png)
2007-06-13 09:08 PDT, Amir Aharoni
no flags Details
Hebrew Characters - Bad Placement of Vowel Points - with font Miriam Fixed Transparent (12.76 KB, image/jpeg)
2007-06-13 10:02 PDT, Mark H. David
no flags Details
Hebrew Characters - Good Placement of Vowel Points - with font Miriam (13.87 KB, image/jpeg)
2007-06-13 10:03 PDT, Mark H. David
no flags Details

Description Arthur Barrett 2000-11-17 19:35:23 PST
When Unicode web pages containing Hebrew/Yiddish Diacritic marks are displayed 
using the BiDi version of mozilla 
(ftp://ftp.software.ibm.com/ps/products/warpzilla/bidi-win32-rel-11-13.zip) 
they appear BESIDE the character that they should appear BENEATH.

When the font is altered in preferences (and force font) to Lucidia Sans 
Unicode, then they appear correctly.

Fonts that work in IE5 but not Mozilla include:
Tahoma (tested)
Frankruehl (reported by 3rd party)

Some explanation as to why some fonts work and others do not would be useful, 
but ultimately it would be prefferred if they all could work.
Comment 1 Arthur Barrett 2000-11-17 19:40:44 PST
Created attachment 19429 [details]
The Font Frankruehl displayed correctly rendering this page in IE, wrong in Bidi-Mozilla
Comment 2 Arthur Barrett 2000-11-18 04:33:17 PST
I got these comments from Mark David, from Yiddish Information Processing 
(uyip.org).

Oh, I see.  Someone got this to work with the Lucida font, but not the
normal Windows Hebrew fonts.

The Lucida font is broken for Hebrew.  My Microsoft sources say this
is known, but cannot be changed for compatibility with existing
applications.  This font has been around since Windows NT 3.51, around
1994.  

In fact, Internet Explorer with Hebrew extensions has not been fixed
to compensate for Lucida's problems, and only works for the standard
Hebrew fonts, and displays nikud wrong ONLY with Lucida.

It's very nice to have "special" code that makes it display Hebrew
correctly, but the display code should also handle correctly constructed
fonts, in particular the standard Hebrew fonts supplied with Hebrew-enabled
Windows 98/NT/2000 and with Hebrew extensions for Internet Explorer 5.  
E.g., Arial, Times-Roman, Tahoma, Frankruehl, et al.

Sure enough I could reproduce this with a plain text page

http://uyip.org/rotshd-cp1255.txt

I simply had to explicitly say the encoding is Hebrew Windows Code Page 1255
under View -> Character Coding.
Comment 3 nhottanscp 2000-11-20 10:18:11 PST
Reassign to mkaply, cc erik.
Comment 4 Keyser Sose 2001-01-04 16:33:19 PST
Arthur is this still a problem in the latest nightlies?
Comment 5 Arthur Barrett 2001-01-05 11:47:38 PST
I will check ASAP, however we just relocated our office and so this may take me 
a few days.
Comment 6 Keyser Sose 2001-01-12 20:17:16 PST
Arthur any luck reproducing?
Comment 7 Fabian Guisset 2001-01-16 05:29:45 PST
new bidi code should be checked in by mkaply soon pending review of his changes.
Look for the bug out there.
Comment 8 Arthur Barrett 2001-01-18 04:42:11 PST
Tested with a WinNT 4.0 compiled copy of the latest (9/Jan/01) BiDI source code.
 Still does exactly the same thing.  This is not good ...

No-one has yet offered an explanation as to why this is happening either...

I will upload a binary copy of the WinNT build some time over the weekend and
post a URL when available.

The source code is from ftp://ftp.software.ibm.com/ps/products/warpzilla
Comment 9 Keyser Sose 2001-01-18 10:16:25 PST
Marking NEW as per comments.
Comment 10 Arthur Barrett 2001-01-18 16:48:18 PST
URL of binary for Win NT is:
http://march-hare.com/downloads/bidimoz.zip

It is a debug build weighing in at 48Mb.
Comment 11 Arthur Barrett 2001-01-18 18:30:20 PST
To reproduce with teh text file, you need to Choose the character coding (View
-> Character Coding -> More -> Middle Eastern ) Hebrew (Windows 1255).

http://uyip.org/rotshd-cp1255.txt

And set the font (Preferences, Font, Hebrew) to Tahoma.

To reproduce with the Unicode HTML page (See attachment), set the character set
to Unicode UTF-8 and set the preferences font for Unicode to Tahoma.

So I guess this means that it is not Unicode specific, since it also reproduces
with CP1255 ?



Comment 12 Arthur Barrett 2001-01-23 02:47:36 PST
After some e-mails from Lina I thought I should clarify:

I am using standard Windows NT 4.0 + SP4.  I've never heard of "Hebrew Enabled" 
versions, and do not need "Hebrew Enabled" for these same pages to work fine in 
IE5 (even with Tahoma).  I have heard of the Microsoft localised Hebrew Windows 
NT, but not Hebrew Enabled....

I will now attach bitmaps of Lina's page in both bidi Mozilla, and ie5, on the 
same machine, same font etc etc.

(http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm)
Comment 13 Arthur Barrett 2001-01-23 02:54:47 PST
Created attachment 23252 [details]
http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm in Mozilla font Tahoma (from Office 2000)
Comment 14 Arthur Barrett 2001-01-23 02:56:55 PST
Created attachment 23253 [details]
http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm in IE5.5 using Font Tahoma (from Office 2000)
Comment 15 Simon Montagu :smontagu 2001-01-23 05:03:45 PST
After investigating a number of different fonts from different sources, I have
come to the conclusion that Hebrew nikud is broken in all Microsoft Hebrew
fonts: Lucida Sans Unicode is broken in one way, and other fonts such as Tahoma
are broken in another way. Microsoft's own programs with Hebrew support (whether
Bidi enabled NT or IE5) are capable of displaying text in Hebrew script with
nikud correctly, but the standard text-output APIs are not.
Finding a workaround for this problem is not trivial: we will probably have to
detect text with nikud and send it to special output methods, while also
sniffing the current font to work out in what way it is broken and how to
compensate (or whether it is a compliant third party font).

Comment 16 Teruko Kobayashi 2001-03-01 13:26:09 PST
Changed QA contact to andreasb@netscape.com.
Comment 17 Arthur Barrett 2001-05-17 20:42:58 PDT
smontagu@il.ibm.com (Simon Montagu) on the 
netscape.public.mozilla.i18n newsgroup posted this

>
>= http://bugzilla.mozilla.org/show_bug.cgi?id=60546, right?
>

I dont personally agree, I think from the comments on 60546 that it is windows 
specific wheras this bug is on Sun/Solaris.

Comment 18 Arthur Barrett 2001-05-17 20:56:34 PDT
Oops - posted that last comment the wrong way around.

smontagu@il.ibm.com (Simon Montagu) on the 
netscape.public.mozilla.i18n newsgroup has suggested that
http://bugzilla.mozilla.org/show_bug.cgi?id=81367
and this bug (60546) are related.

I dont personally agree, I thought 60546
is windows specific wheras 81367 is Sun/Solaris.

If they are the same then does that help resolve it ?
Comment 19 Rann Bar-On 2001-07-02 14:50:59 PDT
confirming on win95, 2001063003
Comment 20 Andreas Becker 2001-07-02 17:59:00 PDT
Switching QA contact to giladehven@hotmail.com.
Comment 21 Simon Montagu :smontagu 2001-07-04 00:27:22 PDT
Copied from n.p.m.i18n:

In spite of their otherwise good BIDI support, Mozilla 0.9.1 and 0.9.2
as well as the nightly build as of yesterday fail to display Hebrew
diacritics in correct positions above letters if the CSS property
TEXT-ALIGN is set to the value JUSTIFY, whether internally or
externally, for an inline element containing the combination of
Hebrew letters and diacritics. It took me a while to pin down this
as the environment of the buggy display.

----------------------------------------------------------------
Tsuguya Sasaki, PhD
E-mail: tsuguya@gol.com
WWW: http://www2.gol.com/users/tsuguya/
----------------------------------------------------------------
Comment 22 Arthur Barrett 2001-07-04 00:48:10 PDT
I'd attatch a sample of what Tsuguya is talking about, but Bugzilla currently 
wont let me.  

You can see it at http://mail.hlmm.com.au/diacritics.html

If you have problems with this URL - contact me direct.

The original example URL (as above) uses a stylesheet:
http://www2.gol.com/users/tsuguya/ts.css

This uses justify for the <body> tag - so it looks like this is the bug.

Comment 23 Zach Lipton [:zach] 2001-08-25 18:12:12 PDT
Mass-move all BiDi Hebrew and Arabic qa to me, zach@zachlipton.com. 
Thank you Gilad for your service to this component, and best of luck to you 
in the future.

Sholom. 
Comment 24 Arthur Barrett 2001-11-12 12:53:45 PST
Confirming this is still present in 0.9.5 for Linux.

The user reports:
diacritics are placed after, not with, corresponding letters.  My font setting 
is for Hebrew misc-fixed-iso8859-8; for Unicode (all styles) misc-fixed-iso-
10646-1.  In fact, I am certainly not seeing my Yiddish text in the latter 
font, even though Mozilla claims that it is using Unicode encoding.  I may be 
seeing it in the former font, but I'm not sure.  

The URL I am looking at is 
http://www.cs.uky.edu/~raphael/bavebter/numer.1.3/sholem.redak.utf.html 
Comment 25 Simon Montagu :smontagu 2002-02-21 14:17:04 PST
*** Bug 127050 has been marked as a duplicate of this bug. ***
Comment 26 Gyula Zsigri 2002-03-16 08:45:45 PST
This may have something to do with OpenType Layout Tables which allow to display
spacing diacritics as if they were properly aligned nonspacing diacritics. But
this is just a guess. You can get some info about Visual OpenType at

http://www.microsoft.com/typography/developers/volt/

and

http://communities.msn.com/MicrosoftVOLTuserscommunity/

Gyula
Comment 27 Gyula Zsigri 2002-03-29 13:24:35 PST
It has nothing to do with OpenType Layout Tables.
Comment 28 Alon Altman 2002-04-15 18:26:02 PDT
Mozilla 0.9.9: Works under Windows, fails under Linux.
Comment 29 Gyula Zsigri 2002-04-29 03:17:03 PDT
The bug is still there in Mozilla 1.0 Release Candidate 1.
I am using Windows 98 original edition.
Comment 30 Simon Montagu :smontagu 2002-04-29 10:39:17 PDT
*** Bug 140788 has been marked as a duplicate of this bug. ***
Comment 31 Gyula Zsigri 2002-05-23 09:08:11 PDT
No improvement in Version 1.0 RC2.
Comment 32 Sagie Maoz 2002-06-28 00:17:34 PDT
Created attachment 89520 [details]
Looks fine in Build#2002053012

The diacritics seems to render almost perfect on my build. Only problem I can
see is with the 'Holam' (?) sign, which is followed by an unnecessary space
(see second word, 'Ovadia', in http://www.qsm.co.il/Hebrew/HebrewTest/test5.htm
).
Of course this might be fixed in lately builds, I'm not sure.
Comment 33 Alon Altman 2002-06-28 03:40:28 PDT
Under Linux this is still a problem? Is there a bug for Linux or should I change
this bug to All OS?
Comment 34 Simon Montagu :smontagu 2002-06-28 10:02:57 PDT
re comment 32: I think the Holam problem is a bug in specific fonts. I see the
extra space after the Holam in Arial, Arial Unicode MS and Tahoma, not only in
Mozilla but also in Wordpad and IE. In other fonts, the extra space disappears.

re comment 33: we have bug 81367 for the corresponding problem on Unix systems,
and bug 144157 for a more limited problem with diacritics on Mac.
Comment 35 Shoshannah Forbes 2002-06-29 12:09:30 PDT
as of build 2002061603, the page still looks bad on osx,
Comment 36 Gyula Zsigri 2002-08-16 06:20:09 PDT
Where can I get the build compiled by Sagie Maoz and when
will his corrections be added to the official build?
Comment 37 Sagie Maoz 2002-08-16 06:31:35 PDT
You must be confusing me with somebody else, I don't do patches :)
Comment 38 Al Merkrebs 2002-08-16 11:51:24 PDT
There is no available BiDi build yet that properly displays Yiddish.
Comment 39 Gyula Zsigri 2002-08-17 03:22:46 PDT
AbiWord suffers from a similar bug:
http://bugzilla.abisource.com/show_bug.cgi?id=3768
Could the two teams join forces?

More bugs with screenshots that show the same wrong alignment: Bug 123218 and 
Bug 144157
Comment 40 Christopher Hoess (gone) 2002-12-11 16:16:33 PST
*** Bug 123218 has been marked as a duplicate of this bug. ***
Comment 41 Christopher Hoess (gone) 2002-12-11 16:16:48 PST
*** Bug 184910 has been marked as a duplicate of this bug. ***
Comment 42 Simon Montagu :smontagu 2003-03-10 09:46:46 PST
*** Bug 196453 has been marked as a duplicate of this bug. ***
Comment 43 William Astle 2003-03-20 19:49:33 PST
There seems to be a general problem with the diacritcs rendering after the base
character; I'm seeing the problem on Win98 with mozilla 1.3 except I'm not using
Hebrew; simply trying to combine a ^ with a z for example.
Comment 44 jan hofer 2003-03-26 20:06:31 PST
in addition there is something funny:
if I write: Hachodesch (&#1492;&#1495;&#1493;&#1491;&#1513;) hebrew is wrong
direction. it is right direction in internetexplorer.
if I write: Hachodesch (<bdo
dir="rtl">&#1492;&#1495;&#1493;&#1491;&#1513;</bdo>) it does not change
it I write: Hachodesch (<bdo
dir="ltr">&#1492;&#1495;&#1493;&#1491;&#1513;</bdo>) it displays correctly. But
wrong with internetexplorer.

this sample is from www.synagogenverein.at/bethaus.asp

this is a terrible thing :'(. not even unicodes but also "<bdo dir" tags are
misinterpreted!

please fix.

johannes
Comment 45 Simon Montagu :smontagu 2003-03-26 20:44:40 PST
Comment 44 is completely unrelated to this bug. Can you please file a separate
report?
Comment 46 Alon Bar-Lev 2003-06-13 02:57:44 PDT
Not only for Yiddish, same for hebrew:
http://www.mechon-mamre.org/i/t/t0101.htm

Please fix so we can stop use IE.
Comment 47 Leston Buell 2003-06-16 17:32:50 PDT
Sometimes when the diacritics display incorrectly in the way described here, the
line of text goes beyond the margin, but sometimes it does not. Is this behavior
just a side-effect of this bug, or does it need its own bug? Such a bug has been
filed as bug 209468.
Comment 48 Boris Zbarsky [:bz] (still a bit busy) 2003-06-20 11:07:53 PDT
*** Bug 210122 has been marked as a duplicate of this bug. ***
Comment 49 Peter Kirk 2003-07-16 16:21:10 PDT
See also bug 212808 which is partly a duplicate of this one. The behaviour of a
page of fully cantillated biblical Hebrew,
http://www.mechon-mamre.org/c/ct/c0101.htm (in Mozilla 1.4 on Windows 2000),
with font Ezra SIL (OpenType, designed specially for biblical Hebrew, from
www.sil.org) set as the default depends on the version of Uniscribe installed,
but even with the latest version of Uniscribe set for release with Office 11
there is a diacritic placement problem, not found with IE6.

Actually this is not really this bug at all as originally reported. It looks to
me as if this bug started with incorrect behaviour of some pre-OpenType
Microsoft etc fonts which would be the same in IE as in Mozilla. The problem I
am reporting has similar symptoms but is Mozilla specific. It seems to be a
matter of Mozilla failing to use Uniscribe and OpenType tables properly. Should
I open a new bug or continue this as bug 212808?

But comment 46 here may be related to what I am reporting, although
http://www.mechon-mamre.org/i/t/t0101.htm, on the same site as my problem page,
has no cantillation marks and is encoded in CP1255 rather than UTF-8.
Comment 50 Simon Montagu :smontagu 2003-07-24 16:29:57 PDT
*** Bug 212808 has been marked as a duplicate of this bug. ***
Comment 51 Simon Montagu :smontagu 2003-08-01 13:34:12 PDT
*** Bug 214801 has been marked as a duplicate of this bug. ***
Comment 52 Simon Montagu :smontagu 2003-10-07 16:53:50 PDT
*** Bug 221399 has been marked as a duplicate of this bug. ***
Comment 53 Ilya Tsindlekht 2004-01-10 02:12:33 PST
Is anybody working on this bug now? I'm trying to look into it. Is use of Pango
to display text in mozilla out of question? (Pango is a library used by GNOME
for displaying text, it seems to handle Hebrew diacritics successfully.)
Comment 54 Jungshik Shin 2004-01-10 02:34:21 PST
As for using Pango on Linux, see bug 215219. My patch there works well for justified as  
well as unjustified text. However, I'm not dealing with Hebrew and Arabic in the patch 
because I'm not sure what Mozilla's internal Hebrew/Arabic routine does.  
  
On Windows, we have to use Uniscribe APIs instead of standard Windows text drawing APIs 
to render justified text correctly (see bug  218887) .   
 
BTW, is there any freely available opentype font for Hebrew supporting Hebrew diacritic 
marks? 
  
Comment 55 Peter Kirk 2004-01-10 05:32:26 PST
Re comment 54, see Ezra SIL from http://www.sil.org/computing/fonts/silhebruni/.
See also http://www.sbl-site.org/Resources/Resources_BiblicalFonts.aspx; the
long promised new Unicode Hebrew font will be very good, from what I have seen
of the beta, but unfortunately it is has not yet been released.

There is also an initiative for using the SIL Graphite rendering system within
Mozilla, see http://sila.mozdev.org/. This might be useful for rendering Hebrew
and Arabic. Unfortunately SIL's Hebrew font does not work with SIL's rendering
system, because it does not include a Graphite table.
Comment 56 Jungshik Shin 2004-01-10 07:17:19 PST
Thanks for the info. about SIL opentype fonts for Hebrew. With them installed,
Mozilla on Win 2k works well for _unjustified_ Hebrew with diacritics (vowel
marks). That's expected because ExTextOutW (Win32 API) used by Mozilla  can take
care of complex script rendering given a 'enough' context on Win2k/XP (and
Hebrew/Arabic Windows 9x/ME for Hebrew/Arabic and Thai Win 9x/ME for Thai)
However, it doesn't work for 'justified' text. For that, we have to use
Uniscribe (bug 218887). Now, I'll see how Pango works on Linux. 

As for SILA, I'm aware of that, but as you pointed out, without Graphite fonts
for Biblical Hebrew, it's not useful. 

Comment 57 Prognathous 2004-02-16 08:42:00 PST
The URL returns 404, so I replaced it with the one from Bug 144157. It should be
sufficient for testing this bug.

Prog.
Comment 58 Simon Montagu :smontagu 2004-08-22 11:22:32 PDT
Dov Grobgeld has submitted a patch to pango to use GPOS and GSUB tables in
Hebrew fonts. See
http://article.gmane.org/gmane.linux.region.israel.ivrix.discuss/921 and
http://bugzilla.gnome.org/show_bug.cgi?id=150785.
Comment 59 Jungshik Shin 2004-08-22 17:18:02 PDT
blizzard recently added nsFontMetricsPango to mozilla trunk. (the relevant bug
is bug 214715, but he just did it off-line) To test it, you need to install
pango 1.5 (plus the patch for GPOS/GSUB support in Hebrew mentioned ) and build
mozilla with 'enable-pango'. 
Comment 60 Dov Grobgeld 2004-08-22 23:27:20 PDT
This is my first bugzilla comment in Mozilla, so please excuse me if I'm not too
familiar with how the rendering works.

From the discussion above, I don't understand if you are going all the way to
using pango. If you are the rest of this comment is irrelevant, as you will
receive the below functionality from pango.

If not you may want to consider to copy my heuristics that is encoded in the
Hebrew module. This heuristics does a reasonable job of nikkud placement for
fonts without GPOS and GSUB tables by simple rules such as "PATAH should be
centered below the character", "SHIN DOT should be right aligned", etc. The
result is imho really good enough.
Comment 61 Peter Kirk 2004-08-26 02:57:08 PDT
(In reply to comment #60)
... 
> If not you may want to consider to copy my heuristics that is encoded in the
> Hebrew module. This heuristics does a reasonable job of nikkud placement for
> fonts without GPOS and GSUB tables by simple rules such as "PATAH should be
> centered below the character", "SHIN DOT should be right aligned", etc. The
> result is imho really good enough.

This result may be good enough for basic pointed Hebrew. But it will not be good
enough for fully pointed and accented Hebrew as use e.g. in the Bible text,
which is rendered successfully by pango with Dov's patch (see his samples).
There is a significant demand for reading the Hebrew Bible online, and several
sites at which it is available e.g. http://mechon-mamre.org/c/ct/c0.htm and
http://users.ntplx.net/~kimball/Tanach/Tanach.xml. These sites already work well
with IE, and in fact with Mozilla 1.7 on Windows XP. I hope Mozilla won't be
satisfied long term with a partial solution but will implement full pango
rendering of Hebrew - and I hope Dov's patch will make it into the pango trunk.
Comment 62 Dov Grobgeld 2004-08-26 03:22:07 PDT
FYI, I got permission to and applied the (modified) patch to the main pango
trunk yesterday.

Regarding heuristics, it is true that the current heuristics that is in the
pango Hebrew renderer does not handle fully pointed and accented Hebrew. But I
am convinced that it may be changed to do that. The rendering heuristics has
access to complete bounding box information as well as the coding of the glyphs.
That is enough to do make sure that the various marks do not collide. Don't take
the current implementation in pango (which ignores accents) as an indication
that is not possible!

The problem with using opentype tables is pushed to the font designer who has to
create tables for several thousand potential combinations. One idea I have is to
use e.g. the online tenach to automatically generate all possible combinations,
and then use some automatic layout routine (e.g. involving distance transforms)
to generate all the different GPOS combinations... But now I am getting carried
away.

Thus my question remains. Is Mozilla moving over to using pango?
Comment 63 Peter Kirk 2004-08-26 03:58:33 PDT
(In reply to comment #62)
> FYI, I got permission to and applied the (modified) patch to the main pango
> trunk yesterday.
> 
Great!
...
> The problem with using opentype tables is pushed to the font designer who has to
> create tables for several thousand potential combinations. One idea I have is to
> use e.g. the online tenach to automatically generate all possible combinations,
> and then use some automatic layout routine (e.g. involving distance transforms)
> to generate all the different GPOS combinations...

Sounds good for use with a font which doesn't have all the GPOS etc information
set up. But when you have a font like SBL Hebrew which has been designed
carefully so that all (or nearly all) of those several thousand combinations are
supported, surely it is better to use that information rather than try to do
better with heuristics.
Comment 64 Mike Kaply [:mkaply] 2004-08-26 05:29:10 PDT
Blizzard - this question is for you.

> Thus my question remains. Is Mozilla moving over to using pango?
Comment 65 Simon Montagu :smontagu 2004-09-05 10:50:09 PDT
*** Bug 257756 has been marked as a duplicate of this bug. ***
Comment 66 Christopher Blizzard (:blizzard) 2004-09-07 12:59:00 PDT
I don't think there's any official mozilla.org policy on this.  I will say,
however, that I've added support for pango in Mozilla.  I think we (Red Hat)
intends to ship this in some upcoming version of Fedora.  It does do a better
job with a lot of scripts including arabic and hebrew.  It's worth checking out.
Comment 67 kobi zamir 2005-01-23 14:04:22 PST
I have just compiled firefox cvs with --enable-pango on debian unstable, 
I can see diacritic in the correct palce using culmus fonts.
(http://benyehuda.org/bialik/bia001.html)

but:
1. when using text-align justify the text gets all mixed up:
(http://www.mechon-mamre.org/c/ct/c0101.htm)
2. printing hebrew is reversed.
Comment 68 Simon Montagu :smontagu 2005-01-24 03:53:38 PST
*** Bug 279490 has been marked as a duplicate of this bug. ***
Comment 69 Simon Montagu :smontagu 2005-01-26 23:31:54 PST
*** Bug 279925 has been marked as a duplicate of this bug. ***
Comment 70 hcy 2005-02-24 23:26:12 PST
Hi, I'm wondering when this will finally be fixed! For me, it's quite a serious
problem, not having the vowels and cantilation marks placed correctly. I have
the Ezra SIL SR font installed and am trying to look at the Bible with vowels
and cantillation marks at http://www.mechon-mamre.org/c/ct/c0101.htm .  

Some observations:

Aside from mozilla, the following applications seem to have the same problems:
epiphany, openoffice (both in linux and windows), scribus (and IIRC
mozilla/firefox in windows as well!)

The following apps do NOT have the problem:
MS IE, MS WORD, and Konqueror (I run it in gnome).
                    ^^^^^^^^^

So another linux app has managed to fix this!  Could we please fix it in Mozilla
soon?  Please?

Thanks,

HCY.
Comment 71 Peter Kirk 2005-02-25 02:56:58 PST
(In reply to comment #70)
> ... the Bible with vowels
> and cantillation marks at http://www.mechon-mamre.org/c/ct/c0101.htm .  
> 
> Some observations:
> 
> Aside from mozilla, the following applications seem to have the same problems:
> epiphany, openoffice (both in linux and windows), scribus (and IIRC
> mozilla/firefox in windows as well!)

I can confirm that this problem is still found with this page in Mozilla 1.7 on
Windows, and this continues when I edit the page in Composer and change the font
to SBL Hebrew. But the problem may be just with this page. The pages
http://www.cvkimball.com/Tanach/Genesis.DH.xml and
http://www.anastesontai.com/b-cantilee/en-cant.asp display the same text
correctly, with SBL Hebrew. Is there a subtle setting in the stylesheets causing
the difference?
Comment 72 hcy 2005-02-25 04:24:33 PST
I get the same problems with those sites as well.

HCY.
Comment 73 Simon Montagu :smontagu 2005-02-25 04:52:54 PST
I'm fairly sure I've said this before, but if so I'll say it again.

On Windows the diacritics and cantillation marks are displayed correctly (via
Uniscribe), except where there is "align=justify" or the equivalent in CSS.

On Linux and other platforms, the problem exists with or without justification.
The Pango support which is currently in development (bug 214715, marked as
blocking this one) should bring Linux at least into line with Windows. The other
blocker, bug 218887 should cover the justify issue on Windows. I am adding bug
215219 as blocker, which is the Linux equivalent of bug 218887.
Comment 74 Simon Montagu :smontagu 2005-03-13 23:10:56 PST
(In reply to comment #73)
> The Pango support which is currently in development (bug 214715, marked as
> blocking this one) should bring Linux at least into line with Windows.

It turns out that non-justified text in Linux builds with Pango enabled is
already at least on a par with Windows. See attachment 176897 [details] in bug 285007.

Comment 75 Raanan Barzel 2005-04-02 09:02:10 PST
(In reply to comment #73)
> I'm fairly sure I've said this before, but if so I'll say it again.
> 
> On Windows the diacritics and cantillation marks are displayed correctly (via
> Uniscribe), except where there is "align=justify" or the equivalent in CSS.
> 
> On Linux and other platforms, the problem exists with or without justification.
> The Pango support which is currently in development (bug 214715, marked as
> blocking this one) should bring Linux at least into line with Windows. The other
> blocker, bug 218887 should cover the justify issue on Windows. I am adding bug
> 215219 as blocker, which is the Linux equivalent of bug 218887.

Is Uniscribe connected to Firefox?
Comment 76 Stephen Blackheath 2005-05-30 02:09:07 PDT
(In reply to comment #73)
> On Windows the diacritics and cantillation marks are displayed correctly (via
> Uniscribe), except where there is "align=justify" or the equivalent in CSS.
> 
> On Linux and other platforms, the problem exists with or without justification.

I have been working on fixing these issues in Firefox - see my web page
http://blacksapphire.com/firefox-rtl/ for my latest work on these bugs.

For rendering Hebrew with vowels+cantillation marks, there are FIVE bugs present
as of Firefox-1.0.4:

1. (GTK-based Unix, including Linux). As previously described, the default
renderer does not render Hebrew properly, while Pango does.  Since some
languages render better in Pango and some in Mozilla default, it would be nice
to have it automatically select between the two for each language.

2. (All platforms).  In layout/html/base/src/nsTextFrame.cpp, PaintTextSlowly is
called when "align=justify" is used.  PaintTextSlowly does not render Hebrew
properly even with pango enabled.  I fixed it by making it always use
PaintUnicodeText/PaintAsciiText when bidi is enabled.  See the patch on my site.

3. (All platforms).  Justified text (with "align=justify") looks very strange,
because it treats whole sentences as a single word.  I fixed this in
layout/base/src/nsBidiPresUtils.cpp by changing CalculateCharType so it starts a
new text run when it sees a white space.  The Mozilla people might have a better
way to fix this.

4. (All platforms).  The Hebrew Biblical texts have the occasional single letter
that is larger or smaller than usual (e.g. Genesis 1:1 and Genesis 2:4).  When
"align=justify" is used, along with HTML to give a different-sized character (as
in the Mechon-Mamre texts), the odd character ends up as a different "run".  In
some situations (certain font sizes on Windows), the justify algorithm puts a
gap in between.  Justify really should only add extra space where there is
*actual whitespace*.  I haven't fixed this yet.

5. (All platforms).  Printing is all wrong with one or two words per line and
big gaps.  I have not tracked this one down.
Comment 77 fantasai 2005-06-11 14:21:06 PDT
Stephen, make sure your work and smontagu's aren't conflicting; he's working on
related issues in bug 297074.
Comment 78 Antony Gelberg 2005-07-28 08:14:20 PDT
I have been working with Stephen to test and submit his patches.  Upon trying
Deer Park, he finds that much of his code (against 1.0.4) is in the trunk.
I quote his email to me below:

>I tried the DP Alpha 2 release on Windows 2000.  Result:
>
>* Hebrew Today - same.
>* TanakhML - all over the place, same as Linux.
>* Mechon-Mamre - niqqud is all skew as in original Firefox 1.0.6.  My
>windows patch will fix this.
>
>I also noticed printing (at least the preview) seems to be fixed!!
>
>Priority 1: Prepare patch to fix Mechon-Mamre for you to submit.
>
>Priority 2: Fix TanakhML page.
>
>Priority 3: Think about how to get Pango enabled automatically per
>language, so it can be on by default in the build.

I think that this should be a separate bug.  I have filed this as
https://bugzilla.mozilla.org/show_bug.cgi?id=302514

>Priority 4: Test actual printing.
>
>Priority 5: Fix wobble when selecting in Pango patch on M-M text.  I
>know what the problem is: The pango renderer is ignoring the spacing
>chosen by the higher layers, which is used when align="justify" is on.

I hope this provides a reasonable update on status.
Comment 79 Michael Grant 2006-03-23 11:52:19 PST
I've just upgraded to Firefox 1.5 (Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051202 Fedora/1.5-0.fc4 Firefox/1.5) on Linux (Red Hat Fedora Core 2).  Previously I'd been using Mozilla 1.7.3, and, by using a Mozilla build with Pango support and setting MOZ_ENABLE_PANGO as described upthread, I'd got Hebrew vowels to display correctly, i.e. within and around the preceding characters.

According to what I've seen, Firefox 1.5 is supposed to have Pango support built in.  However, it's rendering the Hebrew vowels between the characters, which is pretty much unreadable, and setting MOZ_ENABLE_PANGO makes no difference.

Is there a configuration issue of some kind I'm missing, or is this a new bug in Firefox?
Comment 80 Niro 2006-08-21 09:03:07 PDT
*** Bug 349532 has been marked as a duplicate of this bug. ***
Comment 81 John Clark 2006-12-03 05:57:47 PST
Looks like this bug has been around for six years now, without anyone taking it too seriously.

The suggestion of using a Tanakh (Bible) website to genetrate all possible conmbinations of consonants and vowels (= nikodot or nikodos = diacritical marks) will probably work for Hebrew, but not for Yiddish.  The diacritical marks in Yiddish are different from those in Hebrew: there is only partial overlap.  Furthermore, while Hebrew can be written without these marks (and read by proficient speakers), Yiddish cannot.

In fact there is only a small number of letter/mark combinations in Yiddish, I think fewer than in, say, French.  Fixing this problem in Yiddish is therefore not likely to be rocket science.
Comment 82 Prognathous 2006-12-03 10:09:32 PST
(In reply to John Clark, comment #81)
> Looks like this bug has been around for six years now, without anyone taking it
> too seriously.

It's likely that none of the developers who volunteer to fix bugs have found this one interesting enough. If you feel it's important and you wish to improve the chances of it being fixed, then you might want to start a bounty.

Prog.
Comment 83 Ilya Tsindlekht 2006-12-03 10:41:16 PST
It seems to work correctly with firefox 1.5.0.7. Was there switch to PANGO for text rendering?
Comment 84 Simon Montagu :smontagu 2006-12-03 21:57:11 PST
This will be fixed when the new textframe code lands, or shortly afterwards.
Comment 85 Ran 2007-03-01 12:06:47 PST
I find that, at least on Windows XP Media Center Edition and Windows XP Professional, Hebrew diacritics work properly when I have "Install files for complex script and right-to-left languages (including Thai)" checked in the Regional and Language Options section of the Control Panel, and do not work properly when I do not.
Comment 86 John Clark 2007-06-03 04:31:06 PDT
Comment #85 worked for me.  Problem solved.  Thank you very much.  John Clark.
Comment 87 Amir Aharoni 2007-06-13 09:05:03 PDT
Created attachment 268240 [details]
a png image which demonstrates incorrect rendering in Fx 2.0.0.4

the website can be accessed at http://haharoni.wordpress.com/2007/06/12/meuhedet-si/
Comment 88 Amir Aharoni 2007-06-13 09:06:08 PDT
Created attachment 268241 [details]
a png image which demonstrates incorrect rendering in Fx 3.a alpha 5

the website can be accessed at http://haharoni.wordpress.com/2007/06/12/meuhedet-si/
Comment 89 Amir Aharoni 2007-06-13 09:08:50 PDT
Created attachment 268242 [details]
a png image which demonstrates correct rendering in IE6

the website can be accessed at http://haharoni.wordpress.com/2007/06/12/meuhedet-si/
Comment 90 Amir Aharoni 2007-06-13 09:10:52 PDT
I still see this bug in 2.0.0.4.

What's more interesting is that i see a similar bug in the latest released binary Gran Paradiso alpha 5. But in Gran Paradiso it is slightly different.

In Firefox 2 the vowel dot is somewhere in between its consonant and the next letter.

In Firefox 3 alpha 5 the vowel dot is rendered correctly in relation to its consonant, but after the consonant there's a space which shouldn't be there.

I attached three images which demonstrate rendering in Fx 2, Fx 3 and IE6. The words with vowel dots are accented with red.
Comment 91 Mark H. David 2007-06-13 09:51:57 PDT
The thing I think I figured out the other day: if a font doesn't itself have nikudes (nonspacing marks for vowel points, etc.), then Firefox tries to get them from other fonts.  It does an admirable job finding them; however, it applies them in the wrong way, i.e., devoting a whole new character cell to them.  You can see an example of this by selecting most of the "transparent" fonts on Windows, e.g., Miriam fixed transparent.  So, using this approach, I could reproduce the problem of having "nonspacing" marks become spacing and consume their own character cell on all platforms (Linux, Macintosh, Windows).
Comment 92 Mark H. David 2007-06-13 10:02:00 PDT
Created attachment 268243 [details]
Hebrew Characters - Bad Placement of Vowel Points - with font Miriam Fixed Transparent

This illustrates, along with the next image uploaded, the point in my most recent post: Comment #91, Mark H. David 2007-06-13 09:51:57 PDT
Comment 93 Mark H. David 2007-06-13 10:03:35 PDT
Created attachment 268244 [details]
Hebrew Characters - Good Placement of Vowel Points - with font Miriam

Illustrates comment #91 by Mark H. David 2007-06-13 09:51:57 PDT
Comment 94 Mark H. David 2007-06-13 10:08:35 PDT
Note: I've created image files and uploaded them as attachments (id=268244) and (id=268243).

Note 2: Here's verion info for what I used create these attachments:
Mozilla Firefox for Windows
Firefox 2.0.0.3
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Comment 95 Simon Montagu :smontagu 2007-06-13 11:19:49 PDT
(In reply to comment #90)
> I still see this bug in 2.0.0.4.

Yes, you would. As I said in comment 84, this will be fixed when the new textframe lands, which should be before Gran Paradiso Alpha 6. Checking "Install files for
complex script and right-to-left languages (including Thai)" works for many cases on 2.0.0.4, but not for justified text, as used in http://haharoni.wordpress.com/2007/06/12/meuhedet-si/

When using fonts without vowel points on trunk with new textframe, the vowel points simply don't appear. I'll file a new bug about that.
Comment 96 Jesse Ruderman 2007-06-23 16:57:18 PDT
Is this fixed now that new textframe is turned on?
Comment 97 Simon Montagu :smontagu 2007-06-23 20:00:12 PDT
It's fixed on Windows. I don't know what the story is on Mac, and on Linux it isn't 100% perfect, but it will be better to file new bugs for remaining issues.
Comment 98 Mark H. David 2007-06-23 20:52:20 PDT
(In reply to comment #97)
> It's fixed on Windows. I don't know what the story is on Mac, and on Linux it
> isn't 100% perfect, but it will be better to file new bugs for remaining
> issues.

Simon, have you filed a bug for vowel points not showing up if missing
from a given font (i.e., not being substituted from another font), which
you reported and said you'd file a bug for in comment #95?  Thanks. 
Comment 99 Simon Montagu :smontagu 2007-06-23 23:13:47 PDT
Thanks for the reminder: filed bug 385622.
Comment 100 Simon Montagu :smontagu 2007-07-15 00:58:12 PDT
*** Bug 387987 has been marked as a duplicate of this bug. ***
Comment 101 Eyal Rozenberg 2008-01-25 23:57:17 PST
How can a fixed bug depend on an unresolved bug? Please consider removing dependence on bug 215219.
Comment 102 Simon Montagu :smontagu 2008-06-15 00:43:50 PDT
*** Bug 438558 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.