446496 - Hebrew diacritics ("nikud") not rendered properly in FF3 (non-spacing chars are spaced)

Status: RESOLVED INCOMPLETE

(Core :: Layout: Text and Fonts, defect)

Description

[Doron Shikmoni]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

I have just installed FF3.0.1 to replace FF 2.0.11. It appears as if Hebrew points ("nikud") rendering has broken between 2 to 3, in the following sense: The Hebrew point characters are non-spacing; when typed after a Hebrew character, they do not space, but rather are "superpositioned" over it.

In FF3, some of these diacritics are rendered with spacing, so that the overall string appears garbled.

Note: although my "steps to repro" indicate "type text", this doesn't have to do with input, but rather with rendering (i.e. no input needs to be involved)


Reproducible: Always

Steps to Reproduce:
1. Type a Hebrew string with diacritics, into a form, dialog box etc. (e.g. gmail compose)
2.
3.
Actual Results:  
Hebrew diacritics are positioned shifted from the characters to which they "belong"

Expected Results:  
Hebrew diacritics should be positioned right over (below, under...) the characters to which they "belong"

Comment 1

[Hasham]

Can you please attach a screenshot of this issue? Thanks.

Comment 2

[Doron Shikmoni]

Attachment: Screenshot of text with diacritics

Sure - should have attach one in the 1st place (sorry it's a jpeg). I tried to zoom in a little, so the problem is more visible.

You can see that the diacritics are "off" (to the left) of their proper position. Visible with the SEGOL, PATACH and QAMATZ.

Comment 3

[Hasham]

I'm not positive, but I have seen some issues about text rendering because of ClearType in Windows. Can you try to disable it and check if this is still a problem?

Steps to disable ClearType:
1. Go to Control Panel - > Display
2. In the dialog that opens, click the 'Appearance' tab.
3. Click the 'Effects...' button
4. Check 'Use the following method to smooth edges of screen fonts'
5. Select 'Standard' from the drop down (if ClearType is selected).

Comment 4

[Doron Shikmoni]

Thanks. So, the problem was reported on a Windows 2000 machine, where ClearType is less relevant.

Following your comment, I tried it on a different machine running WinXP, and I could NOT reproduce. Out of curiosity, I tried both with and without ClearType, and in both cases, could NOT reproduce.

So the problem appears to be limited to Win2K. However (a) it does not happen in, say, notepad or anywhere else, and (b) it started when I moved from FF2 to FF3.

Comment 5

[Joshua Solomon]

This problem is not limited to Win2K.  I experience it on my iBook G4 running 10.5.3.  The diacritics are all out of sorts on some Wikipedia pages (for example, Leviathan and Behemoth), and it's especially apparent in the Yiddish version of Google (which needs better rendering for kometz alef, fey (pey with a dash above it), and zwei (double) yud).

Comment 6

[Prognathous]

Is this report related to Bug 209468? I can see similar behavior when viewing the following attachment in Firefox 2.0.17 (Gecko 1.8x) under WinXP:

https://bug209468.bugzilla.mozilla.org/attachment.cgi?id=125839

It looks fine in Firefox 3.0.5 (Gecko 1.9x)

Prog.

Comment 7

[Simon Montagu :smontagu]

This is reported as working in Gecko 1.8x but broken (on Win2K only) in 1.9x, so it isn't the same as bug 209468.

I don't have a Win2K environment to test in, unfortunately.

Comment 8

[Doron Shikmoni]

Indeed. Seems to be broken still, with FF 3.0.6 / Win2K. Apparently works fine under WinXP.

Comment 9

[Mark H. David]

Still broken with

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20

I'll upload a picture.

Comment 10

[Mark H. David]

Attachment: image showing bad Yiddish/Hebrew nikud (diacritic) placement

The source of this is today's (24FEB09) Yiddish Forward, http://yiddish.forward.com.  (Note: the problems reported here are common to both Hebrew and Yiddish languages, which share the same Hebrew script.)

Comment 11

[Simon Montagu :smontagu]

(In reply to comment #9)
> Still broken with
> 
> Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217
> Firefox/2.0.0.20

Can you test with Firefox 3? http://yiddish.forward.com/ displays with the nikud correctly placed for me.

Comment 12

[Mark H. David]

(In reply to comment #11)
> (In reply to comment #9)
> > Still broken with
> > 
> > Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217
> > Firefox/2.0.0.20
> 
> Can you test with Firefox 3? http://yiddish.forward.com/ displays with the
> nikud correctly placed for me.

I'll try to soon.  I think it will win.  I tried Gimp, which has pango --
the good stuff -- on the same box, and it did fine with the same text
when I copy/pasted it in.

I also just tried on a Mac upgrading to Firefox 3.x, and it worked great.

It will take me some time because apparently FC3 is a bear to deal with
for installing Firefox 3.x.  I tried all the many and quite different
how-to instructions for installing it in Linux, and none worked, and then
I asked for help from an admin, but he said it's unsupported -- does not
play nice with FC3.  Someone else told me I'm best off switching to Ubuntu,
so that's what I'm planning to do.  Not today, but some day soon.

Thanks.

Comment 13

[Mark H. David]

(In reply to comment #11)
> (In reply to comment #9)
> > Still broken with
> > 
> > Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217
> > Firefox/2.0.0.20
> 
> Can you test with Firefox 3? http://yiddish.forward.com/ displays with the
> nikud correctly placed for me.

OK, I tested with fresh Mozilla on fresh Ubuntu --

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.6) Gecko/2009020911 Ubuntu/8.10 (intrepid) Firefox/3.0.6

Out of the box, with the font selected, the placement of nikudes was wildly and unpredictably off.  The nikudes were not occupying a cell next to their base character in a predictable fashion, and subsequent base characters were not placed off to the left of the previous base character's nikudes.  Rather, each base character was placed properly, but then the nikudes were placed off somewhere, usually to the left.  In some cases, far to the left, as with komets and pasekh -- about 1.5 character cells to the left, and in some cases just a smidge to the left, as with RAFE (over a PEH base character, i.e., forming in Yiddish orthography a "Fey").  So, this looks like just a really very, very bad font problem.

What the heck is this font?  Why is this the default?  Is this Firefox's or Ubuntu's problem?  Or some other system's?  Or can this otherwise reasonably looking font be fixed?

Now, here's what I did to kind of fix things:

Change

  Firefox Preferences -> Content (tab) -> Fonts & Colors (section) -> Advanced... -> Fonts for: -> Hebrew -> Serif

from 

  serif

to

  Bitstream Charter

What happens then?

Well, it then uses a *sans serif* font, which to me looks just like Arial Hebrew. Anyhow, in this case, the nikudes are placed quite reasonably well.  Another reason I know it's not Bitstream Charter: if I change this to Century Schoolbook, URW Bookman L, or ori1uni, it always used the same font, but just changed the spacing between words.

Comment 14

[Mark H. David]

Oh, one more thing: if you try the foregoing workaround with some sites, namely http://yiddish.forward.com, you have to turn off letting pages choose their own fonts in preferences, i.e., check on

    Firefox Preferences -> Content (tab) -> Fonts & Colors (section) ->
Advanced... -> [x] Allow pages to choose their own fonts, instead of my selections above

Comment 15

[Mark H. David]

(In reply to comment #14)
Sorry, CORRECTION:

Check OFF

    Firefox Preferences -> Content (tab) -> Fonts & Colors (section) ->
Advanced... -> [ ] Allow pages to choose their own fonts, instead of my selections above

Comment 16

[Simon Montagu :smontagu]

(In reply to comment #13)

> What the heck is this font?  Why is this the default?  Is this Firefox's or
> Ubuntu's problem?  Or some other system's?  Or can this otherwise reasonably
> looking font be fixed?

What is the output of 

fc-match serif:lang=he

on your system?

Bitstream Charter (at least the version I have) does not support Hebrew, so when you select that font, we are falling back to some other font.

Comment 17

[Mark H. David]

(In reply to comment #16)
> (In reply to comment #13)
> 
> > What the heck is this font?  Why is this the default?  Is this Firefox's or
> > Ubuntu's problem?  Or some other system's?  Or can this otherwise reasonably
> > looking font be fixed?
> 
> What is the output of 
> 
> fc-match serif:lang=he
> 
> on your system?
> 
> Bitstream Charter (at least the version I have) does not support Hebrew, so
> when you select that font, we are falling back to some other font.

OK, seems you're right.  I was seeing substitution.

fc-match serif:lang=he
=> FreeSerif.ttf: "FreeSerif" "Medium"

fc-match serif:lang=yi
FreeSerif.ttf: "FreeSerif" "Medium"

By the way, off topic: I just noticed that an HTML page encoded with the old Yiddish language code JI is not being recognized as being in the Hebrew script.  I also noticed

fc-match serif:lang=ji
=> DejaVuSerif.ttf: "DejaVu Serif" "Book"

Same result for language code IW (old language code for Hebrew).

I.e., fc-match gets a different for IW vs. HE and JI vs. YI.  That's bad.  I'd say it's a bug in both Firefox/mozilla and fc-match.  I plan to file a bug on Mozilla about that.  A workaround may be to set the fonts for "Other Languages" to your desired fonts for Hebrew, the script.  Example page: http://www.cs.uky.edu/~raphael/yiddish/hayzele.utf8.html

Also, by the way, why does it say "Other Languages". Should be "Other Scripts". Is Cyrillic a language?

Comment 18

[Mark H. David]

(In reply to comment #11)
> (In reply to comment #9)
> > Still broken with
> > 
> > Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217
> > Firefox/2.0.0.20
> 
> Can you test with Firefox 3? http://yiddish.forward.com/ displays with the
> nikud correctly placed for me.

Yes, I've gotten Firefox 3 installed, downloaded three Hebrew-capable fonts (EzraSIL, Cardo, and Everson Mono), and figured out how to use Preferences sufficiently to control the Hebrew font, and can confirm, yes, nikud placement with all three fonts is working well in my new version of Firefox.  Here's my version string:

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.6) Gecko/2009020911 Ubuntu/8.10 (intrepid) Firefox/3.0.6

So, I consider this issue resolved. Thanks!

Comment 19

[Doron Shikmoni]

Circling back to the original environment where it was reported - Win2k / FF3 - where it seems to still be broken.

Comment 20

[Mark H. David]

(In reply to comment #19)
> Circling back to the original environment where it was reported - Win2k / FF3 -
> where it seems to still be broken.

Good point, and sorry I sort of hijacked this bug. :)

I have a slightly different, I think later, version string:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)

I tried to reproduce your results, but I could not.  I'll attach a screen shot showing good nikud placement in a gmail email composition form.  It's got a little of your text at the bottom, and it does not go astray at the point it did in your example screen shot.  The rest of the text is copy/pasted from an Israeli news site in "Easy Hebrew" (hebrewtoday.com).

Comment 21

[Mark H. David]

Attachment: image showing test of hebrew diacritics - working OK

This is a screen shot of Hebrew in a gmail mail composition form. It shows Hebrew diacritics being placed correctly, not reproducing the buggy placement shown in originally submitted screen shot.  The bottom line is part of the text shown in the original screen shot, but it gets past the point where diacritics (nikud) were misplaced in the original. The rest of the text is copied from a random news story at hebrewtoday.com. It's full of nikud, place reasonably nicely.
I'm using Firefox on Windows XP with the following version string:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)

Comment 22

[Doron Shikmoni]

Mark, I believe your testing is under Windows XP. The report is re Win2K (aka NT 5.0). As you can see in Comment #8 above, things work fine in WinXP.

Comment 23

[Mark H. David]

(In reply to comment #22)
> Mark, I believe your testing is under Windows XP. The report is re Win2K (aka
> NT 5.0). As you can see in Comment #8 above, things work fine in WinXP.

Sorry, you are right.  I cannot account for Win2K.  Sorry I missed that qualification.

Comment 24

[Tyler Downer [He/Him]]

This is a mass search for bugs which are in the Firefox General component, are
UNCO, have not been changed for 500 days and have an unspecified version. 

Reporter, can you please update to Firefox 3.6.10 or later, create a fresh profile, http://support.mozilla.com/en-US/kb/managing+profiles, and test again. If you still see the issue, please update this bug. If the issue is gone, please set the status to RESOLVED > WORKSFORME.

Comment 25

[Doron Shikmoni]

Just did that. 3.6.10, on Win2k. Problem still there, as reported.

Comment 26

[dror3go]

Attachment: testcase for the bug

I can reproduce this bug in 3.6.15 and 4.0b13pre (2011-03-03) under XP.
I created a minimal testcase for this.

Comment 27

[dror3go]

Sorry, this seems to be unrelated to this bug.

Comment 28

[Jonathan Kew [:jfkthame]]

Looks like this was some kind of font/shaping issue on Win2k.... no longer relevant to any current code or products.