423571 - U+FFFD replacement character is rendered as a hexbox instead of a question mark in a diamond

Status: RESOLVED FIXED

(Core :: Graphics, defect, P4)

Description

[Jesse Ruderman]

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b5pre) Gecko/2008031713 Minefield/3.0b5pre

To reproduce, load one of:
  data:text/html;charset=utf-8,<p>&#xfffd;</p>
  data:text/html;charset=utf-8,<p>%ee</p>

Result: A hexbox containing the digits 'FFFD'.

Expected: A black diamond containing a white question mark.

This happens with most fonts, but not all.  "font: -moz-field;" doesn't have this problem, but I don't know what font that is.

Hexboxes (added in bug 372629) are appropriate for "this is a Telegu character and I don't know how to render it", but I expect question-mark-in-diamond for "the page is marked as UTF-8 but contains lone bytes with the high bit set".

Comment 1

[John Daggett (:jtd)]

This was added by the fix for bug 396732, we explicitly add U+FFFD to a list of codepoints that have no glyph, thus causing the hexbox.

http://mxr.mozilla.org/mozilla/source/gfx/thebes/src/gfxQuartzFontCache.mm#793

This was added purely to improve performance of the binary-as-text situation, where lots of FFFD's show up.  If we don't blacklist this, we spend a lot of time in the font matching code searching for this glyph.

Definitely would rather have the diamond character, those hexboxes are ugly as sin.  We could special case this within the font matching code to choose a font explicitly, since Lucida Grande is one of the few fonts that has a glyph for this and it's a system font, it's always available.

Comment 2

[Damon Sicore (:damons)]

I don't think this should block.  IM'd jdaggett and he seems to agree.  Will take a patch.

Comment 3

[John Daggett (:jtd)]

Attachment: patch, v.0.1, hardwire Lucida Grande for U+FFFD

Rather than a hexbox, in the system font fallback case use Lucida Grande explicitly.  This font is a system font, guaranteed to be around, and is one of the few fonts that contain a glyph for the undefined character codepoint U+FFFD.  In situations like displaying binary-as-text or pages with lots of encoding errors this helps us avoid a lot of time in system font fallback but still displays a glyph rather than just a hexbox.

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 313555 [details] [diff] [review]
patch, v.0.1, hardwire Lucida Grande for U+FFFD

Looks good to me

Comment 5

[John Daggett (:jtd)]

checked in

Comment 6

[Egor Pelevin]

What about Windows? I see these FFFD hexboxes from time to time

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9pre) Gecko/2008041206 Minefield/3.0pre ID:2008041206

Comment 7

[Jesse Ruderman]

Egor, this bug ended up with a Mac-specific patch, so please file a new bug (with a testcase) for maybe doing something similar on Windows.

Comment 8

[Egor Pelevin]

Filed bug 428773