1500531 - many combining underlines and diacritical marks rendered wrong, especially in monospace fonts

Status: UNCONFIRMED

(Core :: Graphics: Text, defect, P3)

Description

[John Denker]

Attachment: png: diacritical marks rendered wrong

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

1) The .html file to reproduce the attached screenshot will be uploaded soon.  Firefox 62.0.3 64-bit ubuntu.

2) Also the .gnumeric file to produce similar but not identical results will be uploaded soon.

3) The same phenomena can be reproduced in thunderbird, but it is more laborious.  I suggest you play with the thunderbird font preferences, then copy-and-paste from firefox (see item 1) to a thunderbird compose window.


Actual results:

See attached screenshot.  Many combining diacritical marks that should have attached to a particular character show up displaced to the right, sometimes half a space, sometimes a full space, as indicated by the comments "off by half" or "off by one".

Some fonts are mistreated differently from others.  Sometimes even within a single font, letters are OK but digits and punctuation are mistreated, as indicated by the comment "erratic".

This affects firefox, thunderbird, and gnumeric, so I suspect it is a core issue, not specific to any one application.  It does not affect all applications in exactly the same way, so there may be additional application-specific bugs.

It particularly affects monospace fonts, but also some others.  It affects a great many fonts, and gnumeric renders correctly one font that is mistreated by other applications, so I suspect it can't be blamed on the font metrics.

I can provide gnumeric and thunderbird screenshots if anybody has trouble reproducing those parts of the problem.


Expected results:

See the attached screenshot.  The top row shows how things are supposed to look.

All these diacritical marks look correct on Safari and on the Apple email platform.  I can provide screenshots if anybody is interested.

Being able to underline stuff is important.  It's really bad news if the sender underlines one character and then the recipient sees some *other* character underlined.

Comment 1

[John Denker]

Attachment: html: diacritical marks rendered wrong

As promised: html source to reproduce the problems.

Comment 2

[John Denker]

Attachment: gnumeric: diacritical marks rendered wrong

As promised:  source to reproduce problems.

Comment 3

[John Denker]

Additional observations:

0) Unicode provides quite a few combining diacritical marks.  Reference: https://www.fileformat.info/info/unicode/block/combining_diacritical_marks/utf8test.htm

1) The attached diacritical.html example looks "almost" correct under google-chrome.  All the named fonts in that example are handled correctly /except/ for dejavu-sans-mono, which is erratic.  I can send a screenshot if anybody has the least bit of trouble reproducing this.

2) Combining diacritical marks seem to work fine in my xterm window.  I haven't investigated all permutations and combinations, but I observe good results using font='-misc-fixed-medium-r-normal--20-200-75-75-c-100-iso10646-1'

For these reasons among others, I suspect we have a mozilla issue at least in part (as opposed to a deeper X issue).

3) xterm and chrome provide valuable workarounds.  When I receive something I have ways of seeing it correctly rendered.

4) FWIW there is a completely different set of symptoms when viewing diacritical.html using the openoffice.org editor.  I don't know what to make of this.  In many cases it appears to substitute a conspicuously different font when trying to render combining diacritical marks.  Weird.

Comment 4

[Jonathan Kew [:jfkthame]]

Basically, this is just the problem of trying to use Unicode characters (the combining diacritics) that aren't supported by the primary font being used for the text. So then fallback kicks in, and chooses some other font for the diacritic; and rendering a diacritic from one font on a base character from another hardly ever leads to good results.

Things that would mitigate the issue, to some extent, include:

(a) Apply NFC normalization before attempting to render the text. This would solve the problem for cases where there's a precomposed Unicode character for the base+diacritic combination, and that precomposed form is supported by the font even though the combining diacritic itself isn't. This is a fairly common scenario, as most of the commonly-used accented letters are provided in precomposed form, so we should perhaps consider doing it -- although it could regress rendering for some edge cases in more sophisticated fonts.

(b) Consider the entire grapheme cluster (base+diacritics) when doing font selection, so that fallback applies to the entire cluster rather than just the diacritic. This is bug 543200.

The only totally reliable "solution" is to choose a font that actually supports all the characters present in the content.

Comment 5

[Denis Bakharev]

Hi, I have a very bad results with this test on Windows 7.
Check the results for 'courier' and 'default' fonts: https://imgur.com/a/94k7tXI
You can notice that after first '0' almost all diacritical marks are rendered completely wrong.

Comment 6

[Chas Belov]

m̀ǹḿń displays correctly for me when I paste it into this comment, for the computed CSS font-family specification of:

sans-serif, "FiraGO"

The first combination does not display properly for the following computed font-family in CSS:

-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif

in which there is additional space added between the first m and the first n and the accent appears there. That is, it doesn't combine.

Happens also in Chrome. Doesn't happen in Safari.

(Actually, on the first m, the accent appears over the last curve of the m while on the second m, it appears correctly over the middle vertical stroke, so still slightly off.)

These character combinations are needed for Yale Cantonese transcription. There are no corresponding Unicode characters, so far as I know which could be used instead.

Comment 7

[Chas Belov]

Attachment: Showing m followed by combining falling accent not working in Firefox.

Comment 8

[Chas Belov]

Attachment: Showing m followed by combining falling accent working in Safari.

Comment 9

[Jonathan Kew [:jfkthame]]

As discussed above, this is a question of whether the chosen font fully supports the combining diacritics being used. If it doesn't, and fallback comes into effect -- either fallback positioning, if the font has the diacritic but doesn't provide positioning information, or fallback to a different font if the diacritic is completely absent -- the results will be inferior (and probably inconsistent between different systems/programs).

To get a more consistent result, you need to choose a font that fully supports these character combinations. For these characters, I see better results with Arial, for example.