629386 - Square Boxes Appears in Many Text Labels and Webpages (RTL Versions of FireFox)

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect, P1)

Description

[Ahmed Harthi]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; rv:2.0b10) Gecko/20100101 Firefox/4.0b10
Build Identifier: Mozilla/5.0 (Windows NT 6.1; rv:2.0b10) Gecko/20100101 Firefox/4.0b10 ID:20110121161358

In many places of FireFox4b10 Arabic version, many text labels are showing small squares instead of letters.

These are shown in the About box, some menu items, and in many webpages (usually in titles that use <hx> tag).

I do not know if this affects other locales, I'll see and reply.

Reproducible: Always

Steps to Reproduce:
1. Download & Install Arabic version of FireFox4b10.
2. Launch the browser.
3. Open FireFox Menu, or go to the About Dialog.
Actual Results:  
In many labels there are small squares instead of letters.

Comment 1

[Ahmed Harthi]

Attachment: Main window + Main menu

Comment 2

[Ahmed Harthi]

Attachment: About Box

Instead of showing build info, little squares are replacing letters.

Comment 3

[Jonathan Kew [:jfkthame]]

Please also post the contents of about:support on your system.

It looks like the problem occurs where text is styled as italic. However, I'm not seeing this problem on my system; the italic text displays fine for me.

Comment 4

[Ahmed Harthi]

about:support
-----------------------
  أساسيات التطبيق

        الاسم
        Firefox

        النسخة
        4.0b10

        عميل المستخدم
        Mozilla/5.0 (Windows NT 6.1; rv:2.0b10) Gecko/20100101 Firefox/4.0b10

        مجلد الملف الشخصي

          افتح المجلد المحتوي

        الملحقات المفعّلة

          about:plugins

        إعدادات البناء

          about:buildconfig

  الامتدادات

        الاسم

        النسخة

        مفعّل

        المعرّف

        DigitalPersona Extension
        5.0.0.3790
        false
        [e-mail]

        Move Media Player
        7
        true
        [e-mail]

        McAfee SiteAdvisor
        3.2
        true
        {B7082FAA-CB62-4872-9106-E42DD88EDE45}

        RealPlayer Browser Record Plugin
        14.0.1
        true
        {ABDE892B-13A8-4d1b-88E6-365A6E755758}

        DivX Plus Web Player HTML5 <video>
        2.1.0.900
        false
        {23fcfd51-4958-4f00-80a3-ae97e717ed8b}

        DivX HiQ
        2.1.0.900
        false
        {6904342A-8307-11DF-A508-4AE2DFD72085}

        Feedback
        1.0.4
        true
        [e-mail]

        Java Console
        6.0.22
        true
        {CAFEEFAC-0016-0000-0022-ABCDEFFEDCBA}

        Java Console
        6.0.23
        true
        {CAFEEFAC-0016-0000-0023-ABCDEFFEDCBA}

        Add-on Builder Helper
        0.0.18
        true
        jid0-t3eeRQgGANLCH9c50lPqcTDuNng@jetpack

        Nightly Tester Tools
        3.1
        true
        {8620c15f-30dc-4dba-a131-7c5d20cf4a29}

  التفضيلات المعدلة

      الاسم

      القيمة

        browser.places.smartBookmarksVersion
        2

        browser.startup.homepage_override.buildID
        20110121161358

        browser.startup.homepage_override.mstone
        rv:2.0b10

        extensions.lastAppVersion
        4.0b10

        network.cookie.prefsMigrated
        true

        places.history.expiration.transient_current_max_pages
        96568

        privacy.sanitize.migrateFx3Prefs
        true

  الرسوميات

        وصف المولّف
        ATI Mobility Radeon HD 3400 Series

        معرّف المنتِج
        1002

        معرّف الجهاز
        95c4

        ذاكرة المولّف العشوائية
        256

        مشغلات المولّف
        aticfx32 aticfx32 atiumdag atidxx32 atiumdva

        نسخة المشغل
        8.801.0.0

        تاريخ المشغل

        Direct2D مفعّل
        true

        DirectWrite مفعّل
        true (6.1.7600.20781)

        مُصَيّر WebGL
        TransGaming Inc. -- ANGLE -- OpenGL ES 2.0 (git-devel Jan 21 2011 16:50:39)

        النوافذ المسرَّعة رسوميًا
        1/1 Direct3D 10
-----------------------

Comment 5

[Ahmed Harthi]

English Translation of about:support

----------------------


  Application Basics

        Name
        Firefox

        Version
        4.0b10

        User Agent
        Mozilla/5.0 (Windows NT 6.1; rv:2.0b10) Gecko/20100101 Firefox/4.0b10

        Profile Directory

          Open Containing Folder

        Enabled Plugins

          about:plugins

        Build Configuration

          about:buildconfig

  Extensions

        Name

        Version

        Enabled

        ID

        DigitalPersona Extension
        5.0.0.3790
        false
        [e-mail]

        Move Media Player
        7
        true
        [e-mail]

        McAfee SiteAdvisor
        3.2
        true
        {B7082FAA-CB62-4872-9106-E42DD88EDE45}

        RealPlayer Browser Record Plugin
        14.0.1
        true
        {ABDE892B-13A8-4d1b-88E6-365A6E755758}

        DivX Plus Web Player HTML5 <video>
        2.1.0.900
        false
        {23fcfd51-4958-4f00-80a3-ae97e717ed8b}

        DivX HiQ
        2.1.0.900
        false
        {6904342A-8307-11DF-A508-4AE2DFD72085}

        Feedback
        1.0.4
        true
        [e-mail]

        Java Console
        6.0.22
        true
        {CAFEEFAC-0016-0000-0022-ABCDEFFEDCBA}

        Java Console
        6.0.23
        true
        {CAFEEFAC-0016-0000-0023-ABCDEFFEDCBA}

        Add-on Builder Helper
        0.0.18
        true
        jid0-t3eeRQgGANLCH9c50lPqcTDuNng@jetpack

        Nightly Tester Tools
        3.1
        true
        {8620c15f-30dc-4dba-a131-7c5d20cf4a29}

  Modified Preferences

      Name

      Value

        browser.places.smartBookmarksVersion
        2

        browser.startup.homepage_override.buildID
        20110121161358

        browser.startup.homepage_override.mstone
        rv:2.0b10

        extensions.lastAppVersion
        4.0b10

        network.cookie.prefsMigrated
        true

        places.history.expiration.transient_current_max_pages
        96568

        privacy.sanitize.migrateFx3Prefs
        true

  Graphics

        Adapter Description
        ATI Mobility Radeon HD 3400 Series

        Vendor ID
        1002

        Device ID
        95c4

        Adapter RAM
        256

        Adapter Drivers
        aticfx32 aticfx32 atiumdag atidxx32 atiumdva

        Driver Version
        8.801.0.0

        Drivr Date

        Direct2D Enabled
        true

        DirectWrite Enabled
        true (6.1.7600.20781)

        WebGL Renderer
        TransGaming Inc. -- ANGLE -- OpenGL ES 2.0 (git-devel Jan 21 2011 16:50:39)

        GPU Accelerated Windows
        1/1 Direct3D 10
----------------------

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Jonathan, if you can't reproduce this or get enough information, we can unblock.

Comment 7

[Ahmed Harthi]

Robert, you can reproduce this with the latest nightly, just go to http://www.mozilla.com/ar/firefox/ and see.

you do not need an RTL version of FireFox to reproduce.

Comment 8

[Robert O'Callahan (:roc) (email my personal email if necessary)]

That page looks fine to me, so it's not that simple to reproduce.

Comment 9

[Jonathan Kew [:jfkthame]]

(In reply to comment #7)
> Robert, you can reproduce this with the latest nightly, just go to
> http://www.mozilla.com/ar/firefox/ and see.
> 
> you do not need an RTL version of FireFox to reproduce.

I don't think this depends on the localized version of Firefox, but it does probably depend on your OS and versions of installed fonts.

Ahmed, do you see the same problem in a minimal testcase such as this? Try:

data:text/html,<p>&#x627;&#x644;&#x639;&#x631;&#x628;&#x64A; <i>&#x627;&#x644;&#x639;&#x631;&#x628;&#x64A;</i></p>

(just copy & paste into the URL bar, and press Return). I'm guessing the second instance of العربي may show up as boxes for you. Please confirm whether that reproduces the issue on your system...

Comment 10

[Ahmed Harthi]

Yes I confirm !

Square boxes in the other instance of العربي

But this was not the case 1 or 2 weeks ago !!

Comment 11

[Jonathan Kew [:jfkthame]]

OK, could you please follow the steps from bug 628091 comment 4 to create a log file, using the latest nightly build? Please _don't_ browse to lots of pages while logging, though; just loading the data: testcase from comment 9 above will be sufficient.

Don't forget to delete the logging environment variables afterwards.

Then attach the fonts.log file here so I can see what's happening on your system, and try to reproduce it. Thanks.

Comment 12

[Ahmed Harthi]

Attachment: fonts log

I did it carefully.. exactly as you said.

Comment 13

[Jonathan Kew [:jfkthame]]

Thank you. It seems that for some reason, you are seeing different behavior with the Times font than what I see here, and I'm trying to figure out what could be causing this.

Please check in Control Panel / Fonts / Times New Roman, and let me know the versions of the Times New Roman fonts (both the Regular and Italic faces) that are present on your system. If you could confirm the file name and exact file size, and the version (shown in the Details tab of the Properties dialog), that would be helpful.

Comment 14

[Ahmed Harthi]

versions of all faces of Times New Roman are 5.05

I didn't manage to get the file size, Properties are not available for fonts in Windows7.

I don't think that the problem is the font, Times New Roman works fine everywhere else. 

My OS Region and Language settings are as follows:
Formats Tab:
Format: 'English (United States)'

Location Tab:
Current Location: 'Saudi Arabia'

Administrative Tab:
Language for non-Unicode programs: 'Arabic (Saudi Arabia)'


I'll try to reproduce this bug on other machines with the same locale settings.

Comment 15

[Jonathan Kew [:jfkthame]]

The font version matches what I have on Win7. What I don't understand at the moment is why Firefox is choosing to use Times New Roman Italic for the italic-styled Arabic text on your system, as the Italic face does not actually have any Arabic characters (and that's why you get boxes). On my machine, it correctly ignores Times New Roman Italic, and falls back to an alternative font instead.

Comment 16

[Jonathan Kew [:jfkthame]]

Aha - I can reproduce this now. The key was to set my system locale to Arabic. OK, now there's some hope for debugging the issue. Thanks for your patience!

Comment 17

[Jonathan Kew [:jfkthame]]

This is a regression from bug 602792, part 3. This can be demonstrated by setting the option gfx.font_rendering.directwrite.use_gdi_table_loading to FALSE in about:config. then you will no longer see boxes for the italicized Arabic text in Times New Roman; instead the browser will correctly fall back to another font.

It seems that when the system locale is Arabic, the GDI font system is "pretending" that in italics, the Times font can render the same repertoire as is supported by the Regular face; when we call GetFontData to read the 'cmap' table for Times New Roman Italic, converted via a GDI LOGFONT structure, it actually returns a copy of the Regular font's cmap. If we also draw through the GDI font system, this works ok, as it uses slanted versions of the Regular glyphs when true italics are not available. But when we try to render the Arabic characters via DirectWrite, which relies on the true italic face, they are not supported.

It's likely that this may affect other fonts and other locales as well; it looks like a fundamental problem with mixing GDI and DirectWrite font APIs, as they may not always handle fonts in a completely compatible way.

I think we need to back out bug 602792 part 3, unless we can find a reliable way to resolve this discrepancy.

This needs to hard-block, IMO; it's a fundamental flaw that will affect users throughout the Arabic world (and possibly other locales).

Comment 18

[John Daggett (:jtd)]

We're manually slanting italic rather than using the dwrite obliqued version.

For bug 602792 I did test to see if there were differences in the cmap between returned by either GDI or DirectWrite, but I didn't do it with the system locale set to Arabic.  I suspect this is not a widespread problem, that it only affects a few fonts in a few specific locales.  Better to workaround those than to undo the GDI table reading fix which affects are much larger set of users.

The test program code on bug 602792 includes the ability to sniff for cmap differences:

  FontEnumeration.exe -a -gc -v

where -a checks all fonts, -gc checks dwrite vs. gdi cmaps and -v dumps full results.  I think it reads all faces, you may need to tweak that if not.

Not sure if this makes a difference but we're manually skewing the regular face in the synthetic italic case for dwrite, I think it might make more sense to use the dwrite-generated oblique face in that case.  DirectWrite is sort of weird about this face, it always includes a fake italic face which we currently exclude.  Might be interesting to try if including that face resolves this problem.  I suspect it won't but it would be interesting to try anyways.  As part of 'FindStyleVariations', check to see if there are any real italic faces.  If none exist, use the dwrite fake oblique faces.

I agree that reading font tables via GDI is not ideal at all but the key thing it avoids is *massive* I/O when reading in cmaps for all fonts when the non-font-cache dwrite path is hit (e.g 300MB at startup).  Once we cache those (bug 600713), I think it wouldn't be a big impact to remove the table-reading-via-GDI code.

Comment 19

[John Daggett (:jtd)]

(In reply to comment #18)
>   FontEnumeration.exe -a -gc -v

Change that to:

  FontEnumeration.exe -a -c -gc -v

Comment 20

[John Daggett (:jtd)]

Attachment: test app output, dwrite vs. gdi cmap differences

Ran the test app attached to bug 602792 with the command-line flags noted above with the system local set to various locales.

Control Panels > Regional Settings > Admin > Non-Unicode system locale

Attached are the results for the languages below, running on Windows 7 JA as the base system:

Arabic
Bengali
Dutch
Greek
Hebrew
Japanese
Russian

The font 'Kredit' showed differences across locales.  Otherwise, only under Arabic was I able to find differences:

  Arial Italic cmapLen:6962 different!
  Arial Bold Italic cmapLen:6962 different!
  Courier New Italic cmapLen:6834 different!
  Courier New Bold Italic cmapLen:6834 different!
  Times New Roman Italic cmapLen:6894 different!
  Times New Roman Bold Italic cmapLen:6894 different!
  Kredit Regular cmapLen:916 different!

Comment 21

[Jonathan Kew [:jfkthame]]

Hmm. It seems Arabic-script locales (including non-Arabic-language ones such as Urdu) suffer particularly from this issue, because several common system fonts exhibit differing behavior. (It's not limited to MS system fonts, though; I also see a discrepancy with Osaka-Mono, which I happen to have installed locally. So it's possible this could happen to other third-party fonts that users have installed.)

We could mitigate the problem by avoiding the use of GDI table access for italic fonts (which seem the likeliest to exhibit this kind of issue), and for symbol-encoded fonts (which shouldn't normally be used anyway, but better to play it safe). This should resolve things for the majority of cases that actually affect users, though I am not confident that it is truly reliable; for example, similar issues might occur if the character coverage of a Bold font differs from its Regular counterpart, and GDI returns the Regular cmap for the Bold face (as if synthetic bolding were to be used), but DW uses the true Bold face. I think the risk of that occurring in the wild is probably small, though.

(In reply to comment #18)
> We're manually slanting italic rather than using the dwrite obliqued version.

This isn't relevant to the issue here; the font families where we're seeing this problem are ones that have a true Italic face.

Comment 22

[Jonathan Kew [:jfkthame]]

Attachment: patch, avoid GDI table loading for italic faces and symbol fonts

Workaround for the cmap-difference problem caused by loading tables via GDI when we're going to render via DWrite. This patch mitigates the issue by avoiding the GDI path for italic faces and symbol fonts, which seem the most likely to exhibit problems, while allowing the GDI path for other fonts (which should usually include most of those needed early in the startup process).

Comment 23

[Jonathan Kew [:jfkthame]]

Ahmed, if you'd like to try a test build with this patch, see http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-e43c2bd24d5a/try-w32/. I believe this should resolve the "boxes" problem in web content -- it's not a localized build so you won't see Arabic text in the UI with this version.

Comment 24

[John Daggett (:jtd)]

Comment on attachment 508372 [details] [diff] [review]
patch, avoid GDI table loading for italic faces and symbol fonts

Doing this for all italic fonts on all systems is going to result in lots of I/O when cmaps are read.  

Instead of:
  !mItalic
use:
  !(mItalic && PRIMARYLANGID(GetSystemDefaultLangID()) == LANG_ARABIC)

This will at least limit the i/o bump to users of Arabic, which seem to be the primary users affected by this.  Once MS ships a fix we can remove the GDI table reading logic, or at least only use it for older versions of dwrite.

Comment 25

[Jonathan Kew [:jfkthame]]

(In reply to comment #24)
> Doing this for all italic fonts on all systems is going to result in lots of
> I/O when cmaps are read.

Possibly, but most of that should be off the startup path, shouldn't it? Bug 602792 was about startup, and I would think we're unlikely to hit a large number of italic fonts during startup in most cases.

> Instead of:
>   !mItalic
> use:
>   !(mItalic && PRIMARYLANGID(GetSystemDefaultLangID()) == LANG_ARABIC)

If we're going to do that, we need to test for quite a number of languages: at least LANG_DARI, LANG_PASHTO, LANG_PERSIAN, LANG_SINDHI, and LANG_URDU come to mind as likely problem cases, and there might be one or two more I've overlooked.

I'm still a bit concerned that there could be other situations (other locales and/or other font configurations) where the GDI path gives us results that are incompatible with DWrite rendering, but perhaps the risk is small enough that we can ignore it for now.

Comment 26

[Jonathan Kew [:jfkthame]]

(In reply to comment #25)
> If we're going to do that, we need to test for quite a number of languages: at
> least LANG_DARI, LANG_PASHTO, LANG_PERSIAN, LANG_SINDHI, and LANG_URDU come to
> mind as likely problem cases, and there might be one or two more I've
> overlooked.

Also LANG_UIGHUR.

Comment 27

[Jonathan Kew [:jfkthame]]

Attachment: patch v2 - limit the workaround to arabic-script locales

This version limits the impact on italic fonts to Arabic-script locales. This should fix the specific cases we've seen; hope that's enough!

Comment 28

[John Daggett (:jtd)]

Comment on attachment 509224 [details] [diff] [review]
patch v2 - limit the workaround to arabic-script locales

Looks good.

Comment 29

[Jonathan Kew [:jfkthame]]

http://hg.mozilla.org/mozilla-central/rev/f911fba0ea0b