1834316 - Arabic question mark incorrectly falls back to system font

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[Richard Ishida]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0

Steps to reproduce:

This issue is applicable to Adlam and N'Ko.

When text contains a punctuation mark such as an Arabic question mark, the glyph displayed should be that contained in the Adlam or N'Ko font, if it contains one.

Actual results:

Various browsers on desktop and mobile render the Arabic question mark using a fallback font despite the font containing the appropriate glyph.

Example in N'Ko: https://github.com/w3c/afrlreq/issues/19

It may be related to a script itemization issue where Adlam or N'Ko is not properly recognized. A request was made to the Unicode Consortium to fix the Script Extensions table. This appears to have been done.

More:
GitHub issue https://github.com/w3c/afrlreq/issues/18

Expected results:

Interactive test: If an N'Ko font contains an Arabic question mark, the glyph rendered should be from that font, and not fall back to an another font. https://github.com/w3c/character_phrase_tests/issues/54

This was tested with the Noto Sans NKo font. In Gecko and Blink the question mark falls back to a different font.

Is this failing because something needs to be implemented in the browser to use the correct script extensions? Or is there some other browser implementation issue?

Comment 1

[Jonathan Kew [:jfkthame]]

Trying the testcase linked from https://github.com/w3c/character_phrase_tests/issues/54 on macOS, I see the problem with Firefox 113, but I get the "expected" question mark (rendered from Noto Sans Nko) with Firefox Nightly (115).

Comment 2

[Richard Ishida]

Thanks. That sounds promising. Any idea what was fixed? (So that i can suggest it when i raise bug reports for the other browsers.)

Comment 3

[Jonathan Kew [:jfkthame]]

OK, this is weird..... I wasn't able to reproduce the "correct" behavior in mozregression, and now after restarting my Nightly browser, I can't reproduce it there either; I'm consistently getting the Arabic fallback. But I don't know what changed to cause this. Some more digging seems in order...

Comment 4

[Jonathan Kew [:jfkthame]]

OK, I see what the issue is here. We have code (currently on macOS and Android only, I think) that "masks off" the Unicode blocks related to complex-shaped scripts from the character map of installed fonts, if it determines that the font doesn't actually support shaping the script in question. This is to avoid getting "garbage" rendering from fonts (such as some large East Asian fonts) that have character code coverage for lots of these Unicode blocks, but only provide nominal-form glyphs and don't have the OpenType tables and alternate glyphs needed for proper rendering.

However, this means that in the case of Noto Sans NKo, we detect that it doesn't have Arabic-script shaping, and so we mask out the Arabic block -- and thus "lose" the punctuation characters from that block that NKo actually does want to use. So the issue here would affect the Arabic comma and semicolon characters, as well as the question mark.

It's not clear to me exactly what I did that initially made things work locally, but I guess it must have involved loading the font as a webfont (via Google Fonts?) rather than just accessing a locally-installed one. (We don't apply this "masking" to webfont resources; it's assumed that the site knows what it is doing.)

Anyhow, I think the immediate fix here is to exclude these punctuation characters from the Arabic block that we mask off if shaping support is absent, because they don't actually need shaping, and may be required for N'Ko etc.

Comment 5

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1834316 - Don't mask Arabic comma/semicolon/question-mark from the font's cmap when 'arab' script support is absent, because they may be used with other scripts such as N'Ko. r=#layout-reviewers

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/757be353a1d4
Don't mask Arabic comma/semicolon/question-mark from the font's cmap when 'arab' script support is absent, because they may be used with other scripts such as N'Ko. r=emilio

Comment 7

[Richard Ishida]

I think this is bigger than just the Arabic punctuation in N'Ko. (I've been having a lot of trouble displaying certain Noto fonts lately, especially for some of the more long tail scripts, and that may or may not be related, but working with Limbu seems to throw up a similar problem to the one described here.)

If you go to https://r12a.github.io/pickers/limb-lif/index.html?text=%E1%A4%86%E1%A4%A5%E1%A4%BA%E1%A4%B0%E1%A4%95%E1%A4%A2%E1%A4%B6%E1%A4%93%E1%A4%A5%E1%A4%92%E1%A4%A0%20%E1%A4%8F%E1%A4%A2%20%E1%A4%90%E1%A4%A7%E1%A4%B6%E1%A4%92%E1%A4%A7%E1%A4%B6%E1%A4%92%E1%A4%A0%20%E1%A4%94%E1%A4%A7%E1%A4%98%E1%A4%A0%E1%A4%B9%20%E0%A5%A5 and set the Namdhinggo SIL webfont (set by default) everything looks fine, but if you change to the local font on the system the danda is displayed using (in my case, at least) Devanagari MT (which is somewhat larger). These changes can be seen in the inspector.

I imagine that if i test other scripts i may find similar results.

This doesn't occur with Chrome, but does appear to occur in Safari – although Safari's inspector and their approach to (not) handling non-system fonts makes it a little difficult to figure out what's really happening there.

Comment 8

[Jonathan Kew [:jfkthame]]

The Limbu/Devanagari case sounds like the same type of issue, yes -- the Devanagari danda (and presumably double-danda) is "borrowed" by various other scripts, so we should treat it similarly and not require the font to "support" Devanagari shaping. Could you file a followup (it can just briefly refer back to the above comment) for this, so that it gets tracked appropriately? Thanks.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/757be353a1d4

Comment 10

[Richard Ishida]

Raised a follow-on bug for other borrowed characters in various orthographies at https://bugzilla.mozilla.org/show_bug.cgi?id=1836024