226758 - In some fonts, ‘ is rendered as a box / square []

Status: RESOLVED WONTFIX

(Core :: Internationalization, defect)

Description

[Michiel van Leeuwen (email: mvl+moz@)]

In some fonts (it seems to be the MS core fonts) ‘ is rendered as a box,
like []. In other fonts it works. For me, ’ works, but i have heard on
#mozillazine that that is broken sometimes too.

testcase: see url
expected results: show `name' three times, in three different fonts
actual restuls: []name' on the first two lines, `name' on the third.

This is a regression. (but an old one). The best regression window I could get
is somewhere between 2003070107 (works) and 2003072322 (fails). (all linux,
regular mozilla.org nightlies).

Comment 1

[Simon Montagu :smontagu]

This may be a dupe of bug 108231. Can you set the environment variable
NS_FONT_DEBUG to 5 and load the testcase, with output redirected to a file, and
attach the file here?

Comment 2

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: log from NS_FONT_DEBUG

attached the results of:
./mozilla -P testuser http://yellow.exedo.nl/~michiel/usit.html > fontlog.txt

bug 108231 is an old bug, from before this regression occured. This used to
work in old builds. So i don't think it is a dup.

Comment 3

[Jungshik Shin]

> 2003070107 (works) and 2003072322

  Nothing was checked into nsFontMetricsGTK.cpp  between those dates. It's
strange. Anyway, for Helvetica, iso10646-1 is used, but for other two fonts, it
uses 'iso8859-7'. Why? Because iso8859-7 includes U+2018 and U+2019. Apparently,
Windows fonts you installed on Linux (Arial and Vernada) don't cover U+2018 but
are still presented as covering iso8859-7 because the number of missing glyphs
are within the tolerance range set by 'mkfontscale' (or whatever tool you used
to generate fonts.scale/fonts.dir file.) You can check it out with 

  xfd -fn -monotype-arial-medium-r-normal--14-*-0-0-p-*-iso8859-7
(U+2018 and U+2019 should be at 0xa1 and 0xa2.

The patch for bug 229760 I just landed will solve your problem because
microsoft-cp1252 will be used for U+2018 and other characters present in
Windows-1252 but not in ISO-8859-1 before trying iso8859-7 and iso10646-1.


However, the root cause of this bug (that triggered regression between
2003-07-01 and 2003-07-23) is not yet known 

Comment 4

[Dave Jarvis]

Attachment: Shows that this bug is outstanding -- squares appear.

My original bug (http://bugzilla.mozilla.org/show_bug.cgi?id=242438) appears to
be a duplicate of this one.  As you can see from the screen shot, this looks
like an outstanding problem.

Again, please note that RC1 does not seem to work as a download from
mozilla.org (the screen shot also shows the version I _just_ installed after
wiping clean my older version).

Comment 5

[Dave Jarvis]

*** Bug 242438 has been marked as a duplicate of this bug. ***

Comment 6

[Simon Montagu :smontagu]

(In reply to comment #3)
>   Nothing was checked into nsFontMetricsGTK.cpp  between those dates. It's
> strange. Anyway, for Helvetica, iso10646-1 is used, but for other two fonts, it
> uses 'iso8859-7'. Why? Because iso8859-7 includes U+2018 and U+2019. 

It looks like this was me, as bz points out in bug 232026 (a dupe of this):
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=smontagu%25netscape.com&whotype=regexp&sortby=Date&hours=2&date=explicit&mindate=2003-07-08+00%3A00%3A00&maxdate=2003-07-09+00%3A00%3A00&cvsroot=%2Fcvsroot

That check in updated the mapping tables for ISO-8859-7 and added some new
codepoints including U+2018 and U+2019. There is obviously a font on the system
that still uses the old mapping, so doesn't include the glyphs for these
codepoints even though we expect it to.

Comment 7

[Simon Montagu :smontagu]

*** Bug 223244 has been marked as a duplicate of this bug. ***

Comment 8

[Simon Montagu :smontagu]

*** Bug 232026 has been marked as a duplicate of this bug. ***

Comment 9

[Boris Zbarsky [:bzbarsky]]

Simon, anything we can do about this for 1.7?

Comment 10

[Simon Montagu :smontagu]

I see a few possibilities:

1) ban the antique iso-8859-7 fonts

2) backout the patch, i.e. downgrade from ISO 8859-7:2003 to ISO 8859-7:1987. We
might do this selectively so as not to lose the Euro and Drachma characters
which are probably needed, and just replace the single quotation marks mappings
like this:

-0xA1   0x2018  #       LEFT SINGLE QUOTATION MARK
-0xA2   0x2019  #       RIGHT SINGLE QUOTATION MARK
+0xA1   0x02BD  #       MODIFIER LETTER REVERSED COMMA
+0xA2   0x02BC  #       MODIFIER LETTER APOSTROPHE

3) have a separate ISO-8859-7 encoder for the X fonts, which would solve this
bug without impacting Save As Charset.

My own inclination is towards option 2, unless option 3 would be a lot simpler
to implement than I think.

Comment 11

[Roland Mainz]

smontagu:
See bug 235391 comment #1 - Sun solved the issue via having a seperate XLFD
encoding for the newer ISO8859-7 variants.

Comment 12

[Jungshik Shin]

(In reply to comment #10)

> 3) have a separate ISO-8859-7 encoder for the X fonts, which would solve this
> bug without impacting Save As Charset.
> 
> My own inclination is towards option 2, unless option 3 would be a lot simpler
> to implement than I think.

option 3 is not that complex, but may be more complex than you want it to be
:-). A new encoder should be compiled in only on X11 (as is the case for several
X11-only encoders).  I have a reservation about option 2 because we have to
'sacrifice' the compliance to the 2003 version of ISO-8859-7. 

Comment 13

[Simon Montagu :smontagu]

What worries me about option 3 is that it assumes that *all* iso-8859-7 X fonts
use the old version of iso-8859-7. If this isn't the case we are likely to cause
regressions for users with up-to-date Greek fonts.

Comment 14

[Asa Dotzler [:asa]]

Simon, time is very short and we need a solution today. If backing out is the
safest thing, let's just do that. 

Comment 15

[Asa Dotzler [:asa]]

Maybe reverting the change at
http://bonsai.mozilla.org/cvslog.cgi?file=mozilla/intl/uconv/ucvlatin/cp1256.uf
for the 1.7 branch and working on a better solution for the trunk would be the
right thing to do, that is, if this "fixed" state is negatively affecting more
pages than the old version. 

Comment 16

[Simon Montagu :smontagu]

Attachment: partial backout as described in comment 10 option 2

This changes the right and left single quotation marks back to reversed comma
and apostrophe (which look much the same in most fonts: ʽlike thisʼ instead
of ‘like this’) without damaging the Euro and the Drachma.

Comment 17

[Simon Montagu :smontagu]

(In reply to comment #14)
> Simon, time is very short and we need a solution today. If backing out is the
> safest thing, let's just do that. 

It's almost midnight local time so I may not be awake long enough to check in. I
also can't test since I don't have a system which displays the bug. Can someone
else drive this in?

Comment 18

[Jungshik Shin]

Comment on attachment 150465 [details] [diff] [review]
partial backout as described in comment 10 option 2

r=jshin

Comment 19

[Jungshik Shin]

(In reply to comment #17)

> It's almost midnight local time so I may not be awake long enough to check in. I
> also can't test since I don't have a system which displays the bug. Can someone
> else drive this in?

I'm sorry I forgot to note that I'm gonna do that before I went to bed. Now that
I was up again, I'll take care of it as soon as a1.7 is granted. 

Comment 20

[Asa Dotzler [:asa]]

Comment on attachment 150465 [details] [diff] [review]
partial backout as described in comment 10 option 2

a=asa (on behalf of drivers) for checkin to 1.7

Comment 21

[Jungshik Shin]

fix was checked into 1.7 branch by leaf.

Comment 22

[Simon Montagu :smontagu]

Thank you for checkins, leaf and scott. Can anyone confirm that the bug no
longer appears on branch builds?

Comment 23

[Wily Yuen]

Just downloaded the latest-1.7 branch nightly build:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040611

Yes, the partial backout patch works and the bug is (partially) fixed for now;
you can double-check it with another page which triggers this bug as well:

http://www.mozilla.org/get-involved.html

Comment 24

[Jacek Piskozub]

Now that the 1.7 branch fix is in, do we really need the blocking1.7+ flag?

Comment 25

[Wily Yuen]

As I am on Mozilla 1.8 a3, this bug has made a comeback. Again you can test it
with this page:
 
http://www.mozilla.org/get-involved.html

Comment 26

[Simon Montagu :smontagu]

See comment 15. The fix was only ever checked in on the 1.7 branch.

Comment 27

[Simon Montagu :smontagu]

This was a bug with core X fonts, and is irrelevant to the current codebase.