609649 - Complex scripts (Persian, Arabic, Gujarati) not rendered correctly on Fennec

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[(no longer active)]

I tested this on Jeff's Samsung Galaxy S.  Fennec can't render Persian text, and it will just render a box for each Persian glyph.  This can be seen on http://fa.wikipedia.org for example.

I also tried to create a demo page which uses a web font to display some Persian text <http://people.mozilla.com/~eakhgari/terafik/demo.html>, and the same thin happened again, so I don't think this is a font issue.

Comment 1

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Attachment: Nexus One screenshot

Not sure what's wrong, I wonder if it is device's related.

Your example works on a custom desktop build and also on my N1 (see attached screenshot) even if the ligature looks bad

Comment 2

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

I forgot to says that the ligature are fine on deskop

Comment 3

[Jonathan Kew [:jfkthame]]

(In reply to comment #1)

> Your example works on a custom desktop build and also on my N1 (see attached
> screenshot) even if the ligature looks bad

For a very limited definition of "works".... although it renders glyphs, it has no shaping at all and as such is not really "correct" or readable Persian.

Ehsan, I notice that your test page uses a CFF-flavored OpenType font (in WOFF format) - maybe the device doesn't support CFF, only TrueType? What happens if you put a TrueType version first in the source list?

Comment 4

[(no longer active)]

(In reply to comment #3)
> (In reply to comment #1)
> 
> > Your example works on a custom desktop build and also on my N1 (see attached
> > screenshot) even if the ligature looks bad
> 
> For a very limited definition of "works".... although it renders glyphs, it has
> no shaping at all and as such is not really "correct" or readable Persian.

Exactly!  But I didn't even get the glyphs on Galaxy S.

> Ehsan, I notice that your test page uses a CFF-flavored OpenType font (in WOFF
> format) - maybe the device doesn't support CFF, only TrueType? What happens if
> you put a TrueType version first in the source list?

No idea, need to borrow Jeff's device Monday when we're both at the office...

Comment 5

[Doug Turner (:dougt)]

stuart, blocking-fennec?

Comment 6

[(no longer active)]

(In reply to comment #3)
> (In reply to comment #1)
> 
> > Your example works on a custom desktop build and also on my N1 (see attached
> > screenshot) even if the ligature looks bad
> 
> For a very limited definition of "works".... although it renders glyphs, it has
> no shaping at all and as such is not really "correct" or readable Persian.
> 
> Ehsan, I notice that your test page uses a CFF-flavored OpenType font (in WOFF
> format) - maybe the device doesn't support CFF, only TrueType? What happens if
> you put a TrueType version first in the source list?

I did that in <http://people.mozilla.com/~eakhgari/terafik/demo-tt.html>, and the result is exactly the same.

Comment 7

[Jonathan Kew [:jfkthame]]

Just to clarify, is there actually a Persian-specific issue here, or should the bug be retitled "Complex scripts (Arabic, Indic, etc) are not rendered correctly on Fennec"? I'm assuming that Persian is just one example of the more general issue that we don't have any complex-script support at present.

Comment 8

[(no longer active)]

(In reply to comment #7)
> Just to clarify, is there actually a Persian-specific issue here, or should the
> bug be retitled "Complex scripts (Arabic, Indic, etc) are not rendered
> correctly on Fennec"? I'm assuming that Persian is just one example of the more
> general issue that we don't have any complex-script support at present.

I only tested this with Persian.  Arabic is similarly affected, but I didn't test any other complex script.  I can resummarize this bug if you want.

Comment 9

[Matt Brubeck (:mbrubeck)]

Perhaps the font defaults from bug 568700 need to be changed for these scripts.

Comment 10

[Matt Brubeck (:mbrubeck)]

According to https://support.mozilla.com/en-US/questions/764640, Gujarati has the same problem.

Comment 11

[Matt Brubeck (:mbrubeck)]

Attachment: screenshot: fa.wikipedia.org on G2

In a current Fennec trunk build, I do see actual glyphs for Arabic and Persian on my Nexus One and T-Mobile G2 (both running Android 2.2).  For Gujarati, I see gray boxes.

The Persian and Arabic text does not appear to be rendered correctly; there are no ligatures.  (I'm not sure, because I can't read either of these languages.)  Ehsan, can you look at this screenshot and say whether it is broken?  If it is, we should open a separate bug about the layout.

The stock browser behaves exactly the same in all of the above cases, so I suspect this depends partly on what fonts are available on the device.  Anyone seeing this issue: Does Fennec behave differently from the stock browser on your device?

Comment 12

[(no longer active)]

(In reply to comment #11)
> Created attachment 490590 [details]
>   --> https://bugzilla.mozilla.org/attachment.cgi?id=490590
> screenshot: fa.wikipedia.org on G2
> 
> In a current Fennec trunk build, I do see actual glyphs for Arabic and Persian
> on my Nexus One and T-Mobile G2 (both running Android 2.2).  For Gujarati, I
> see gray boxes.
> 
> The Persian and Arabic text does not appear to be rendered correctly; there are
> no ligatures.  (I'm not sure, because I can't read either of these languages.) 
> Ehsan, can you look at this screenshot and say whether it is broken?

Yes, it is broken.  No Arabic shaping is happening, so each letter is displayed in its disjoint form.  But the layout (RTL-ness) is correct.

> If it is,
> we should open a separate bug about the layout.

Why?  This bug was not only about glyph selection, which is almost the only thing which is working correctly in this screenshot.

> The stock browser behaves exactly the same in all of the above cases, so I
> suspect this depends partly on what fonts are available on the device.  Anyone
> seeing this issue: Does Fennec behave differently from the stock browser on
> your device?

It doesn't seem to have anything to do with font/glyph selection.  It is the shaping which is missing here.

Comment 13

[Matt Brubeck (:mbrubeck)]

(In reply to comment #12)
> > If it is, we should open a separate bug about the layout.
> 
> Why?  This bug was not only about glyph selection, which is almost the only
> thing which is working correctly in this screenshot.

We have two different behaviors.  On some devices/scripts, we see gray boxes instead of glyphs.  On other devices, the same scripts appear with correct glyphs but without shaping.

These seem to be two different bugs, because they affect different script/device combinations.  The gray box problem may be related to font selection or availability, because it varies from one device to another, and appears in other Android apps too.

Here are results from testing current Fennec trunk builds with Arabic, Persian, and Gujarati on the Nexus One, G2, and Galaxy S:

    N1 G2 GS
fa   b  b  x    b = no shaping
ar   b  b  x    x = no glyphs (rectangles only)
gu   x  x  x

In all nine tests, Fennec and WebKit had the same rendering problems.

Comment 14

[(no longer active)]

Hmm, I'll let Jonathan or John comment on whether these are both manifestations of the same problem, or if we're in fact seeing two underlying problems...

Comment 15

[Matt Brubeck (:mbrubeck)]

Opera Mobile beta on T-Mobile G2 has the same lack of shaping as Firefox and WebKit (even version 10.1.1011151731, which they claim "fixed Japanese/Arabic/Hebrew/Thai fonts support").

Comment 16

[(no longer active)]

(In reply to comment #15)
> Opera Mobile beta on T-Mobile G2 has the same lack of shaping as Firefox and
> WebKit (even version 10.1.1011151731, which they claim "fixed
> Japanese/Arabic/Hebrew/Thai fonts support").

Why is that significant?  They don't use Gecko, IINM.

Comment 17

[(no longer active)]

Jonathan, with bug 603879 landed on trunk, do we witch to Harfbuzz for Arabic shaping on Android?  If yes, I'd be willing to test the latest nightly on a device borrowed from someone at the office.  :-)

Comment 18

[Jonathan Kew [:jfkthame]]

(In reply to comment #17)
> Jonathan, with bug 603879 landed on trunk, do we witch to Harfbuzz for Arabic
> shaping on Android?  If yes, I'd be willing to test the latest nightly on a
> device borrowed from someone at the office.  :-)

No, bug 603879 doesn't affect that - it updated the version of harfbuzz we're using, but not where we are using it.

Once we get bug 569770 (and its dependencies) landed, I think we'll be a lot closer to being able to hook up harfbuzz on Android as well, and that would provide Arabic-script shaping support. Should be pretty soon....

Comment 19

[(no longer active)]

Good to hear that, I'm adding a dependency on that bug for now.

Comment 20

[Jonathan Kew [:jfkthame]]

Attachment: part 1 - refactor gfxFT2Font code in preparation for connecting to harfbuzz

Comment 21

[Jonathan Kew [:jfkthame]]

Attachment: part 2 - support harfbuzz shaper in the FT2 font backend

These patches apply on top of the patches in bug 569770.

With these patches, the FT2 backend as used in --disable-pango builds will work with harfbuzz shaping, and thus supports Arabic script.

Comment 22

[Jonathan Kew [:jfkthame]]

There's a tryserver build with these patches at http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-ffec80a523d2. If anyone cares to test, I'd be curious to know whether it (a) works at all; and (b) actually does anything useful with Arabic script.

Comment 23

[Matt Brubeck (:mbrubeck)]

Attachment: screenshot with jfkthame's patches

Here's a screenshot using the tryserver build from comment 22.

Comment 24

[Karl Tomlinson (:karlt)]

Comment on attachment 495886 [details] [diff] [review]
part 1 - refactor gfxFT2Font code in preparation for connecting to harfbuzz

This feels good.

Looks like gfxOS2FontGroup is now the only reason why gfxFontGroup::MakeTextRun
are virtual.

Comment 25

[Karl Tomlinson (:karlt)]

Comment on attachment 495891 [details] [diff] [review]
part 2 - support harfbuzz shaper in the FT2 font backend

I notice mCharGlyphCache is the remaining sore thumb, but hopefully we can lose that by ditching gfxFT2Font::AddRange in the near future.

Comment 26

[Jonathan Kew [:jfkthame]]

Pushed to trunk:
http://hg.mozilla.org/mozilla-central/rev/ebdf5ff5e2dd (part 1)
http://hg.mozilla.org/mozilla-central/rev/88c1ac7b4d13 (part 2)

With this, Arabic & Persian should render properly provided the device has an opentype font with appropriate character coverage.

Gujarati and other Indic scripts will *not* work (even if fonts are available) because Indic shaping is not yet supported in harfbuzz.

Comment 27

[(no longer active)]

(In reply to comment #26)
> Pushed to trunk:
> http://hg.mozilla.org/mozilla-central/rev/ebdf5ff5e2dd (part 1)
> http://hg.mozilla.org/mozilla-central/rev/88c1ac7b4d13 (part 2)
> 
> With this, Arabic & Persian should render properly provided the device has an
> opentype font with appropriate character coverage.
> 
> Gujarati and other Indic scripts will *not* work (even if fonts are available)
> because Indic shaping is not yet supported in harfbuzz.

I've had little luck in rooting my device, which seems to be necessary in order to install fonts on it.  Can we force using a web font as the default font?

Comment 28

[(no longer active)]

I also tried the page in comment 0 in today's Fennec nightly and I still got a rendering including small gray boxes...

Comment 29

[Jonathan Kew [:jfkthame]]

Closing this, as we have the harfbuzz shaper hooked up on Fennec now. This means that Arabic script should work, provided a suitable font is available.

Filed followup bug 619521 re providing Arabic (etc) fonts for devices that don't have them, and bug 619524 to track adding support for more scripts to harfbuzz.

Comment 30

[(no longer active)]

I finally managed to install the proper fonts on my device, and Persian content seems to be rendered perfectly in Fennec!  The core OS doesn't support shaping, so the fact that we're actually providing this on a platform which doesn't support it is just awesome!  :-)

Comment 31

[Kevin Brosnan [Ex-Mozilla]]

v. Mozilla/5.0 (Android; Linux armv7l; rv:2.0b13pre) Gecko/20110314 Firefox/4.0b13pre Fennec/4.0b6pre ID:20110314043938

Comment 32

[(no longer active)]

(In reply to comment #31)
> v. Mozilla/5.0 (Android; Linux armv7l; rv:2.0b13pre) Gecko/20110314
> Firefox/4.0b13pre Fennec/4.0b6pre ID:20110314043938

I've also been enjoying this on my Android 2.2 device (which comes with a crappy looking Arabic font by default which also gets used for Persian) for quite a while.  Arabic shaping seems to be something that no Android 2.2 native app (including the stock browser) can do, so this is pretty cool!  :-)

Comment 33

[Ahmed Soliman]

Attachment: Arabic shaping (ligature) is broken

I've this problem on HTC Desire HD running Gingerbread (2.3) using Cyanogen 7.0.2-DesireHD using Firefox Nightly 7.0a1

Comment 34

[Pierros Papadeas [:pierros]]

Same problem here like comment 33 Shall we reopen it?

Comment 35

[Jonathan Kew [:jfkthame]]

Does this happen on all pages, or is it specific to certain websites? Does a desktop Nightly build show the same problem on those pages, or is it only on mobile?

Comment 36

[Pierros Papadeas [:pierros]]

Happens on all pages on this device (Comment 33) and it is only for the mobile version

Comment 37

[Jonathan Kew [:jfkthame]]

WFM on an HTC Desire HD running Android 2.2, using both Firefox 5.0 and Nightly 7.0a1. Tried both http://www.unicode.org/standard/translations/arabic.html and http://www.bbc.co.uk/arabic/, in both cases the Arabic text is shaping correctly. However, I notice that the BBC site is rendered using my default font, not the BBC's downloadable font, which seems like a problem...

Comment 38

[Jonathan Kew [:jfkthame]]

Updated my device to Android 2.3, and I still don't see this issue. I also checked the Masrawy site (as in comment #33) and the text renders fine - but in a completely different font than the screenshot shows.

Comment 39

[Ahmed Soliman]

maybe it's font's problem, which ROM are you using for gingerbread?