Open Bug 1481545 Opened 6 years ago Updated 7 months ago

Poor kerning of fully hinted fonts on Linux (affects Roboto -> Gmail)

Categories

(Core :: Layout: Text and Fonts, defect)

Product:
Core
Component:
Layout: Text and Fonts
Type:
defect

Tracking

()

Status:
REOPENED

People

(Reporter: grayshade, Unassigned)

References

Details

Attachments

(11 files, 3 obsolete files)

about_support.txt
27.40 KB, text/plain
Details
Bug 1481545 - Prefer subpixel positioning over full hinting.
48 bytes, text/x-phabricator-request
Details | Review
56-kubuntu-noto.conf
29.96 KB, application/octet-stream
Details
fullhint_twitter.png
66.80 KB, image/png
Details
nohint_twitter.png
80.00 KB, image/png
Details
slighthint_twitter.png
75.74 KB, image/png
Details
fullhint_gmail.png
37.51 KB, image/png
Details
nohint_gmail.png
52.88 KB, image/png
Details
slighthint_gmail.png
46.74 KB, image/png
Details
56-kubuntu-noto-proposed.conf
31.81 KB, application/octet-stream
Details
56-kubuntu-noto.conf overrides default browser Fonts
98.35 KB, image/png
Details
Attached file about_support.txt — 
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Build ID: 20180805231147



Actual results:

See https://imgur.com/a/1lHu01S, https://www.reddit.com/r/firefox/comments/9577iv/does_anyone_else_experience_bad_kerning_with_most/ and https://www.reddit.com/r/firefox/comments/8nldcm/firefox_shows_wrong_fontkerning_in_gmail/. That font is Roboto.

Kerning is fine in Chrome, but not in Firefox. I'm on Arch Linux (rolling release) with Intel graphics running Gnome Shell and Wayland. WebRender is enabled in this instance, but that has no influence on this issue. I'm using slight font hinting with subpixel AA.
Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Component: Untriaged → Layout: Text
Product: Firefox → Core
Attached image broken_kerning.png (obsolete) —

I think the duping here was incorrect, or at least it's not the full story and this issue persists. I ran into it after upgrading an Ubuntu 18.04 LTS system to 20.04. Many sites, especially Twitter and Gmail have extremely hard to read text in several places. Devtools revealed that in all cases the hard to read font is Roboto.

I attached a screenshot to show just how off the kerning is. Left is Chrome, right is Firefox. I have some other sites where letters end up running into another and text becomes unreadable.

After experimenting with various tries related to WebRender etc, I stumbled upon: https://askubuntu.com/a/1289601/761268

I can confirm this is correct: Ubuntu changes the default (or even user set) hinting for Roboto to "hintfull", which is very unusual (the default is "slight", which only applies minor vertical hinting) and that is what causes the kerning to be completely off. Changing it to the default fixes that.

The interesting question is why Chrome(ium) is not affected here. These are wild guesses:

Maybe Google has some non-upstreamed changes in FreeType that we're missing:
https://bugs.chromium.org/p/chromium/issues/detail?id=740385
https://bugs.chromium.org/p/chromium/issues/detail?id=726631

Do they bundle a different Roboto?

We regressed bug 939132 somehow, and changing the hinting settings is a red herring? (This seems unlikely as I see the bad kerning outside Gmail)

I'm going to mark this S2. Although a workaround is available, it's hard to discover, and difficult for most users, plus the usage of Roboto on Google properties (Gmail!) means that "Bugs that could reasonably be expected to cause a Firefox user to switch browsers, either because the severity is bad enough, or the frequency of occurrence is high enough" seems to qualify. "Chrome renders text better" is a really bad thing to have. In fact, many people run into this:

https://www.reddit.com/r/firefox/comments/9577iv/does_anyone_else_experience_bad_kerning_with_most/
https://www.reddit.com/r/firefox/comments/8nldcm/firefox_shows_wrong_fontkerning_in_gmail/

I also found this:
https://www.reddit.com/r/firefox/comments/7xezu1/firefox_linux_font_kerning_is_off/

Notice how he also mentions Noto and Arimo. And indeed:
/etc/fonts/conf.avail/56-kubuntu-noto.conf
turns on full hinting for these exact fonts!

Severity: normal → S2
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: DUPLICATE → ---
Summary: Poor kerning of Roboto font (Google sites) on Linux → Poor kerning of fully hinted fonts on Linux (affects Roboto -> Gmail)
Version: 63 Branch → Trunk
Flags: needinfo?(jfkthame)

glandium provided a hint here: although we have freetype in the tree, we seem to use the system provided freetype library. (verified with readelf -d). Chrome does NOT seem to use the system freetype, at least I don't see the dependency on the lib. So it's possible the Ubuntu 20.04 freetype is broken with full font hinting.

Roboto and Noto are used all over Google properties, so even applying the workaround significantly changes how a lot of popular sites look :-(

Right, we use the in-tree freetype on Android, but on Linux we build with system freetype.

I suspect (but have not confirmed) that Chrome may not respect the fontconfig rule that's asking for "hintfull" rendering; they may ignore this and just stay with whatever rendering they've decided is best.

It's hard to tell for sure from the limited sample in the screenshot, but it looks to me like the Chrome rendering has subpixel glyph positioning, while the Firefox version doesn't; it has glyph positions rounded to integer pixels. This may be a result of the "hintfull" choice, and will tend to result in poorer spacing because of the stronger quantization of positions, if nothing else.

In addition, we do have a known issue with letter-spacing, because it gets applied after gfxHarfBuzzShaper::SetGlyphsFromRun has tried its best to deal nicely with rounding of glyph positions. That's what bug 939132 was about, which I see was closed as "worksforme" with most platforms now using subpixel glyph positioning in most cases, but I believe there are still cases -- such as this one -- where it remains a problem.

Flags: needinfo?(jfkthame)

https://bugs.chromium.org/p/chromium/issues/detail?id=824153#c7
https://chromium.googlesource.com/chromium/src/+/master/ui/gfx/font_render_params_linux.cc#229

It looks like Chrome may just disable hinting if subpixel positioning is enabled.

I don't know if it's a coincidence or not, but I use zooming on many sites where this is visible, and the problems aren't as obvious at all zoom factors. (I have pretty bad eyesight and use custom zoom factors almost everywhere)

(I don't think letter-spacing is the main issue - I see the problem on sites where that CSS property isn't used)

Ironically, the configuration files in Ubuntu that enable the full hinting come from ChromeOS, and as we see, that doesn't actually use the standard behavior of FreeType. This is facepalm territory.

Switching to the Chrome behavior would conflict with bug 1585931, although it may fix the same thing in another manner, I guess.

Attached image Differences on Twitter (obsolete) —

Hi Lee, see the past discussion for the background. The tl;dr is that many Linux distros, including stable Ubuntu, ship with some broken ChromeOS font overrides that force full hinting on Roboto and Noto, which are used all over Google properties. Combined with the patch in bug 1585931, this causes us to have some atrocious kerning even when we would be able to do subpixel positioning. Chrome completely disables hinting if it detects subpixel positioning is possible (essentially the reverse logic as what was added in bug 1585931). Because of the prevalence of the broken config, this causes Google properties to look substantially better in Chrome.

I attached a patch that changes gfx.text.subpixel-position.force-enabled into gfx.text.subpixel-position.prefer-enabled and defaults it on. If it is enabled, we ignore full hinting and always use subpixel positioning instead.

Two possible alternatives are: to disable slight hinting too, or to clamp full hinting to slight hinting if subpixel positioning is enabled. I don't feel strongly for any solution, as although there's slight differences in rendering with slight hinting, they aren't remotely as disturbing as what happens with full hinting.

The best solution would be to alert the distros to stop shipping the broken ChromeOS files, but I'm afraid this means we'll be stuck with bad rendering on Google properties for years.

Flags: needinfo?(lsalzman)

FWIW, both the left-hand and right-hand screenshots in comment 10 still look really rough to me; could you clarify what it's showing?

Left is subpixel positioning (and no hinting, as this is Roboto and I'm on an Ubuntu with the broken config), right is no subpixel positioning and full hinting (what you get now).

The kerning problem doesn't seem quite as bad here as on Gmail, but it's still a popular site where the difference is easily visible.

This shows how clamping to slight hinting would look. Perhaps it's a good compromise between not having too fuzzy outlines and keeping reasonable kerning. Note that slight hinting would actually be the Ubuntu default if it wasn't for the Roboto/Noto overrides!

I am confused looking at these screenshots. As Jonathan Kew noted, the kerning looks bad in all the cases in comment 10. Could you please separate out to one screenshot per case, and clearly label each one?

Currently, we go to great lengths to do what the Fontconfig setups request. One of the things our users have generally appreciated is that in contrast to Chrome, we actually honor user configurations. What you ask for is what you get, for the most part. If the user asks for full hinting, sometimes that's exactly what they want and they consider it a bug if they don't get it. If the problem is a couple broken configs, I'd almost rather somehow detect those broken configs rather than just disallow all configs.

Flags: needinfo?(lsalzman)

Also, can you attach Roboto/Noto fontconfig files (or whatever rule is actually targeting them) so we have an example of what the distro is doing?

I'll get you the config file. Interestingly it's named Kubuntu, so I wonder if a regular Ubuntu install has it or if it is somehow specific to Kubuntu (that would be great as less people would be affected). There's really a lot of reports about this though, those people just aren't finding their way to Bugzilla:
https://www.kubuntuforums.net/showthread.php/77528-Firefox-hint-issue-when-editing-text-in-Roboto-font-Had-to-fix-56-kubuntu-noto-conf
https://bugs.launchpad.net/ubuntu/+source/kubuntu-settings/+bug/1872492

Attached file 56-kubuntu-noto.conf —

Attaching the suspect fontconfig file.

I wonder if a regular Ubuntu install has it or if it is somehow specific to Kubuntu
I can confirm the problem is specific to KUbuntu; the files above are installed by kubuntu-settings-desktop:

$ dpkg -S /etc/fonts/conf.d/56-kubuntu-noto.conf /etc/fonts/conf.avail/56-kubuntu-noto.conf
kubuntu-settings-desktop: /etc/fonts/conf.d/56-kubuntu-noto.conf
kubuntu-settings-desktop: /etc/fonts/conf.avail/56-kubuntu-noto.conf

which, if you ask me, is the wrongest possible place where to put system-wide font configuration overrides for fonts used by a ton of unrelated applications (and causes several other problems). Removing the link from conf.d completely (and setting up a dpkg exclude for it to avoid re-creating it on package upgrade) fixed the issue.

Attached image fullhint_twitter.png —
Attachment #9199629 - Attachment is obsolete: true
Attachment #9200367 - Attachment is obsolete: true
Attachment #9200372 - Attachment is obsolete: true

Could you please separate out to one screenshot per case, and clearly label each one?

Attached. To clarify:

"fullhint" is what users on Kubuntu 20.04 (and later) currently get, regardless of what they configured.
"nohint" is what users on Kubuntu and Ubuntu (AFAIK) get in Chrome/Chromium, regardless of what they configured.
"slighthint" is what users SHOULD be getting, based on the default Ubuntu and Kubuntu configuration. I believe this is also what users on the regular Ubuntu actually get.

Lowering severity as this is Kubuntu specific. I don't know how the usage of the distro relates, but there's probably less Kubuntu users than regular Ubuntu users.

Severity: S2 → S3

Which in turn comes from ChromiumOS https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+log/master/media-libs/fontconfig/files/local.conf; Harald Sitter suggests:

https://invent.kde.org/neon/neon/settings/-/commit/665a672030a048087f9239c1d4f6e3f8692f26e7
"this does not take any action towards imporving font hinting except for what chrome has already (i.e. this is not adding infinality style settings)."

Seem strange given that it seems that Chrome essentially ignores them, and this ends up overruling the user settings.

that Firefox may be affected by a similar bug that was fixed in Chromium here: https://chromium.googlesource.com/chromium/src/+/2ff93446c3405557b6024d4e65eb5677195456cd

I'm not an expert (probably jfkthame or lsalzman can comment) but the overrule in the file is specific for the fonts we're seeing the problem with, and we determined above Chrome is not affected by that patch simply because it seems to be completely ignoring the users' or system hinting settings anyway (https://chromium.googlesource.com/chromium/src/+/master/ui/gfx/font_render_params_linux.cc#241). So I'm not sure that's the issue here.

Matteo - the room you linked to in "Harald Sitter suggests" is set to show history to new members only as far back as the time they joined; new people joining that room can't see comments further back. If there's more information at the end of that link that might be relevant to this but, I ask that you paste it into a comment here.

Uh sorry, I didn't notice; there's not much else, but I'll copy the transcript here:

cvtsi2sd: Hello, I'm tracking a problem in font hinting in Kubuntu + Firefox; it's described here https://askubuntu.com/q/1289600/208527, and info about it is scattered through various issues on Mozilla/Ubuntu bugtrackers

I tracked it down to this commit in kubuntu-packaging https://git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/kubuntu-settings/commit/desktop/etc/fonts/conf.avail/56-kubuntu-noto.conf?h=kubuntu_hirsute_archive&id=9ca4c58573bebf3d43634732bf56d06ed1737e48, which is a cherry pick of this file from Neon https://invent.kde.org/neon/neon/settings/-/commits/Neon/unstable/etc/fonts/conf.avail/56-neon-noto.conf. Removing it completely fixes the issues described in those bugs, without particular downsides (I'd say that actually the rendering quality gets better in general).
What was the point of overriding the global hinting/fallback settings? I suspect sitter can shed some light about it.

sitter: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/2b54932e1a59671d220483734a17835c0513e16f

cvtsi2sd: how deep the rabbit hole goes...

https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+log/master/media-libs/fontconfig/files/local.conf

sitter: it may be that this is in fact no longer necessary anyway. fontconfig at some point started to default to BCI I believe. one would have to try I suppose

cvtsi2sd: subjectively I only saw rendering quality getting better since I deleted the file

in general I think that on a "regular" distribution such configuration is a bit heavy-handed... it's very confusing seeing hinting broken, and changes in the font options in the system settings dialog not affecting it at all

sitter: firefox probably has a similar problem as was fixed here https://chromium.googlesource.com/chromium/src/+/2ff93446c3405557b6024d4e65eb5677195456cd

neon isn't "regular"

cvtsi2sd: IDK, from the site I'd say it's "Ubuntu stable + latest KDE stuff", but that's not for me to say, I'm a Kubuntu user

surely it shouldn't get in Kubuntu (and especially not in the kubuntu-settings-desktop package), but I'm talking to the Kubuntu maintainers about that

Thanks Matteo. The description here is correct: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/2b54932e1a59671d220483734a17835c0513e16f
i.e. you will not get BCI hinting if hintstyle is slight. But this is fully intentional by Fontconfing, and explicitly documented as such. Slight hinting exactly implies not too aggressive pixel snapping horizontally. It looks like Google wanted this explicitly for ChromeOS, but I don't see why we'd want to overrule the user's settings (probably not expected on ChromeOS), nor why we'd want to force full hinting on a KDE desktop. (As said, Chrome seems to just disable it entirely, and regular Ubuntu doesn't do this either...)

I see you saying you're talking to the Kubuntu maintainers. I think if necessary I can raise this issue to our contacts at Canonical as well.

Jonathan, Lee, any input here? It looks like we may have enough traction to get this fixed upstream, so we don't need a workaround in Firefox.

(In reply to Gian-Carlo Pascutto [:gcp] from comment #34)

Thanks Matteo. The description here is correct: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/2b54932e1a59671d220483734a17835c0513e16f
i.e. you will not get BCI hinting if hintstyle is slight. But this is fully intentional by Fontconfing, and explicitly documented as such. Slight hinting exactly implies not too aggressive pixel snapping horizontally. It looks like Google wanted this explicitly for ChromeOS, but I don't see why we'd want to overrule the user's settings (probably not expected on ChromeOS), nor why we'd want to force full hinting on a KDE desktop. (As said, Chrome seems to just disable it entirely, and regular Ubuntu doesn't do this either...)

I see you saying you're talking to the Kubuntu maintainers. I think if necessary I can raise this issue to our contacts at Canonical as well.

Jonathan, Lee, any input here? It looks like we may have enough traction to get this fixed upstream, so we don't need a workaround in Firefox.

If this is something that can just be fixed upstream, that's my vote. I think I would prefer to have us not go the Chrome route of ignoring explicit user settings.

Okay, I'll try to get a hold of upstream and ask if they can just remove the font hinting overrides in that configuration file. (Quite possibly the entire file could just go, but then I'm not sure of the side effects)

What's upstream exactly in this case? Chromium OS, the source of the original file? KDE Neon, which modified it and introduced it as some kind of KDE-specific customization? Kubuntu, that cherry-picked it into kubuntu-settings-desktop? Personally I think that the worst damage (in terms of user base) is in Kubuntu; I tried yesterday to contact the person who cherry-picked it (Rik Mills) yesterday on #kubuntu-devel, but they didn't seem particularly interested.

[11:01:53] <cvtsi2sd> Hello, is there anyone who knows how the 56-kubuntu-noto.conf fontconfig settings file got into kubuntu-settings-desktop in first place? IOW, what problem it was meant to solve?
[11:02:33] <cvtsi2sd> Because it's causing considerable damage, especially on Firefox
[11:03:19] <cvtsi2sd> see https://askubuntu.com/q/1289600/208527, https://bugzilla.mozilla.org/show_bug.cgi?id=1645681, https://bugzilla.mozilla.org/show_bug.cgi?id=1481545, https://bugs.launchpad.net/ubuntu/+source/kubuntu-settings/+bug/1872800, https://bugs.launchpad.net/ubuntu/+source/kubuntu-settings/+bug/1872492
[11:05:01] <cvtsi2sd> I'm trying to draw attention to the issue, as unfortunately it's the classic problem where it's difficult to find/correlate information, because it isn't immediately obvious that "font rendering glitches in Firefox", "all-around uglyness on many fonts in UI" and "kubuntu-settings-desktop being installed" are immediately related
[11:21:45] <RikMills> !testers
[11:21:45] <ubottu> Help is needed in #kubuntu-devel. Please ping RikMills, soee, BluesKaj, mamarley, mparillo, wxl, DarinMiller, tsimonq2, crydotsnake-M for more information.
[11:22:01] <RikMills> New spin of 20.04.2 :/
[13:07:53] <cvtsi2sd> hmm, from https://git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/kubuntu-settings/commit/desktop/etc/fonts/conf.avail/56-kubuntu-noto.conf?h=kubuntu_hirsute_archive&id=9ca4c58573bebf3d43634732bf56d06ed1737e48 it looks like RikMills is the man I'm looking for
[13:16:10] <RikMills> sadly no time to look. there are urgent release things at the moment
[13:17:49] <cvtsi2sd> given this breaks fonts for the last three releases IMO the prudent thing would be to just stop shipping the file immediately, and then maybe do a more thorough analysis; it's not like the rest of Ubuntu doesn't have fonts working without that file...

Hope you have better luck.

but then I'm not sure of the side effects

I don't think that's going to cause particular damage, given that this is pulled in only in Kubuntu through the kubuntu-settings-desktop package... all other Ubuntu variants do just fine without it. Probably it will impact the rendering of the default KDE theme (as Noto Sans became the "new" KDE/Plasma default font), but IMO only for the better.

Probably it will impact the rendering of the default KDE theme (as Noto Sans became the "new" KDE/Plasma default font), but IMO only for the better.

Yeah, my concern was that removing the entire file might have side effects there. Removing the font hinting override would be much less risky, but I'll test that too.

Chromium OS, the source of the original file? KDE Neon, which modified it and introduced it as some kind of KDE-specific customization? Kubuntu, that cherry-picked it into kubuntu-settings-desktop? Personally I think that the worst damage (in terms of user base) is in Kubuntu

ChromeOS might have reasons for forcing full hinting. The first batch of those devices shipped with really low end screens, on which full hinting may be a reasonable default. As for KDE Neon, from your previous conversation it seems the hinting change may not have been intended there, so it sounded like that upstream might be willing to fix it. I agree with your assessment that fixing it in Kubuntu seems the most urgent.

See Also: → 939132

Here's a 56-kubuntu-noto that disables the forced font hinting overrides for "firefox-bin" (only). I noticed https://invent.kde.org/neon/neon/settings/-/commit/338d2e1ae2f1a40f1d6ecce940eec119b1dbb4a3 was matching on just "firefox". Not sure which one is correct? Seems to require "firefox-bin" for matching on my Nightly.

I have my default fonts on the desktop set to Ubuntu, not Noto, though, so I'm not a good person to check whether this leaves the rest of the desktop unaffected.

So I just realized 56-kubuntu-noto.conf causes another problem: Even if the user changes the default Kubuntu fonts to something else than Noto, we will still think we should default to Noto.

Contrast the System Settings panel from KDE to the Preferences->Language and Appearance Panel in Firefox.

This does raise one question to me: We seem to be using the right font in the UI. I'm guessing System Settings is correctly configuring the default GTK fonts (without being hit by that override) and we just use that?

What does fc-match return as the default?

What does fc-match return as the default?

NotoSans-Regular.ttf: "Noto Sans" "Regular"

OK, so that's what we use. I guess System Settings here sets things at some other level, but doesn't actually touch the fontconfig setup.

I'm the original reporter here, but I haven't noticed any egregious kerning or positioning issues lately. I've switched to Fedora recently so I can't test on Arch. gcp, do you still have still problem, or should we mark it as resolved?

Flags: needinfo?(gpascutto)

I fixed the local font config on my Ubuntu installation. As far as I know, Kubuntu users will still get the broken font rendering.

Flags: needinfo?(gpascutto)

I'm having a difficulty with the color of the text. I'm unable to select real color for text.
https://news.ycombinator.com/item?id=31528605 coreball

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: