1743170 - "Hiragino Kaku Gothic ProN" font is not available when Firefox is built with macOS 11 SDK

Status: RESOLVED FIXED

(Core :: Graphics: Text, defect)

Description

[rinsuki]

Attachment: hiragino-check.html

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0

Steps to reproduce:

If Firefox are built with macOS 11 SDK (All arm64 binaries, and/or x86_64 binaries after https://hg.mozilla.org/integration/autoland/rev/572b175efb0960eb15cc814d9c8f8f65afecae52 ) , "Hiragino Kaku Gothic ProN" font is not available.

Actual results:

Even "Hiragino Kaku Gothic ProN" are included in font-family, Firefox skips "Hiragino Kaku Gothic ProN" and use other fonts to render.

Expected results:

If "Hiragino Kaku Gothic ProN" are included in font-family, Firefox uses "Hiragino Kaku Gothic ProN" to render.

Comment 1

[rinsuki]

Attachment: hiragino-check.html Screenshot in Intel Mac.

Left side is Firefox Nightly (2021-11-26) (wrong font), Right side is Firefox 94.0.2 (correct font).

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[rinsuki]

It seems CTFontManagerCopyAvailableFontFamilyNames (used by gfx/thebes/gfxMacPlatformFontList.mm) wont returns Document Fonts if app is built with newer SDK, and I think this behaviour starts with macOS 10.15 (Chromium has also have same issue on macOS 10.15 previously, but it was fixed on Chromium 80).

CTFontManagerCopyAvailablePostScriptNames still returns document fonts including "Hiragino Kaku Gothic ProN", but it returns PostScript name like "HiraKakuProN-W3" instead of normal name like "Hiragino Kaku Gothic ProN", and they also returns all weights (e.g. HiraKakuProN-W3 and HiraKakuProN-W6) even not needed.

Comment 4

[Markus Stange [:mstange]]

Jonathan, can you evaluate whether this regression is serious enough that it needs to be addressed before we switch to the macOS 11 SDK for Intel builds?

Comment 5

[Jonathan Kew [:jfkthame]]

Ugh... this is really confusing. We get the list of available fonts (families) from CTFontManagerCopyAvailableFontFamilyNames, but as rinsuki says in comment 3, the behavior appears to vary depending on the SDK used to build.

The Hiragino Kaku Gothic Pro family is not the only one affected, but it's hard to tell exactly how widespread the issue will be. Some of the "Document-support fonts" listed on Apple's page Fonts included with macOS Monterey don't seem to be available at all on my system, while others do work in Safari if requested by name even though they don't show up in Font Book's All Fonts list. The ones that work seem to be those found in the /System/Library/Fonts/Supplemental/ directory -- but then, some of the /Supplemental/ fonts are already in the base list of "Fonts installed and enabled automatically", so that's not the key distinction.

One weird thing is that there are cases like /System/Library/Fonts/NotoSansMyanmar.ttc, a TrueType Collection file that contains both the "Noto Sans Myanmar" family and the "Noto Sans Zawgyi" family, all in a single collection (they're essentially different encodings for the same collection of glyphs, I believe). But while "Noto Sans Myanmar" makes it into the "standard" list that is always available, the Zawgyi version is a "Document-support font" and is hidden from the available family names list under the new SDK. I don't see any clue in the font resources themselves as to how this distinction is made; I suspect Core Text must have some kind of internal table of which fonts to include in the "available" list.

Using CTFontManagerCopyAvailablePostScriptNames to try and get a more complete list looks like a painful way forward, given that for font-family purposes we need family names, and there's no reliable way to map PS names to family names. So the only way to get them would be to access each font ourselves via its PS name and extract the family name(s) from it ourselves. That'd get expensive.

And there's more .... AFAICS, the Hiragino Kaku Gothic Pro(N) family still wouldn't appear, because it's actually an alternate name for certain weights (300, 600) of the Hiragino Sans font! I'm not finding any concrete font resource that has the family name "Hiragino Kaku Gothic Pro", but the CFF font inside the /System/Library/Fonts/ヒラギノ角ゴシック W*.ttc files carries the Kaku name, e.g. <CFFFont name="HiraKakuProN-W3">, although the OpenType name table insists that its family-name is "Hiragino Sans" (or "ヒラギノ角ゴシック").

So I suspect Hiragino Kaku Gothic Pro is considered a "legacy" name, superseded by Hiragino Sans, but for backward compatibility Core Text (and presumably Cocoa) font managers map the legacy names to specific faces of the new font. But that's not exposed in the available families list (when built against the new SDK).

I'm not sure what the best way forward here will be. Perhaps we need to create a substitution mechanism of some kind. (We should also find out what Chromium did about this in v.80 per comment 3; maybe we can take a similar approach.)

Meanwhile, I am concerned about the regression that Japanese users (and probably some others) will see if we update the SDK without solving this.... Hiragino Kaku Gothic Pro has been widely recommended for Japanese content in the past. It's one thing if it doesn't work on people's new M1 machines, and they get some other fallback; it never has worked there, and they'll presumably ascribe this to the new machine, and figure out what alternative defaults they'd like. But it's worse if a Firefox update for users on Intel-based Macs suddenly changes the behavior such that Japanese sites render differently after the update. I think we could expect to hear bug reports about that as it would be a visible regression, probably for quite a number of Japanese pages.

Comment 6

[Hiroyuki Ikezoe (:hiro)]

Maybe, https://bugs.chromium.org/p/chromium/issues/detail?id=1000542 is the Chromium bug?

Comment 7

[Jonathan Kew [:jfkthame]]

Thanks, yes, that looks like the issue. One interesting piece of info there:

On Mac OS 10.15 Catalina, NSFontManager::availableFamilies does not list
"Hiragino Kaku Gothic ProN" any more but only lists it as "Hiragino
Sans". However, NSFontManager::availableFontsForFamily still shows
results when searching for this name.

though as we're using the Core Text APIs rather than Cocoa (we switched away from the NSFontManager version some years ago), we may need to adjust the approach a bit.

Comment 8

[Jonathan Kew [:jfkthame]]

(In reply to Markus Stange [:mstange] from comment #4)

Jonathan, can you evaluate whether this regression is serious enough that it needs to be addressed before we switch to the macOS 11 SDK for Intel builds?

In my local build, what I'm seeing is that it's not just a question of which SDK is used, but also the deployment-target setting. If I build with the 11.0 SDK, but with MACOSX_DEPLOYMENT_TARGET=10.14, the "deprecated" font family names remain available in the list; but when I build with the same 11.0 SDK but with MACOSX_DEPLOYMENT_TARGET=10.15, they're absent.

So what's our current schedule, if any, with regard to supporting/sunsetting older macOS versions? Could we consider moving Intel builds to a newer SDK, but keeping MACOSX_DEPLOYMENT_TARGET low enough that the deprecated font family names remain available for now? That could buy us a bit of time here, perhaps.

Comment 9

[Stephen A Pohl [:spohl]]

I'm going to take a look at this bug and will answer the questions in comment 8.

Comment 10

[Jonathan Kew [:jfkthame]]

Another thing we could consider is to explicitly add the "deprecated" font families back into our list; this would avoid regressing behavior for users who are accustomed to having them available. I've got an experimental patch to do this, currently building on try.

Comment 11

[Markus Stange [:mstange]]

(In reply to Jonathan Kew (:jfkthame) from comment #8)

In my local build, what I'm seeing is that it's not just a question of which SDK is used, but also the deployment-target setting.

Huh, that's surprising! When we briefly switched to the 11 SDK, we did not intend to make any changes to the deployment target.

So what's our current schedule, if any, with regard to supporting/sunsetting older macOS versions? Could we consider moving Intel builds to a newer SDK, but keeping MACOSX_DEPLOYMENT_TARGET low enough that the deprecated font family names remain available for now?

That was the plan, we are not planning to change the deployment target for now.

Comment 12

[Mike Hommey [:glandium]]

our current deployment target is 10.12 on x86_64, though...

Comment 13

[Jonathan Kew [:jfkthame]]

Curious; I wonder if there are other factors that also play into this. I was building locally on 12.3 (Intel) using a minimal mozconfig (pointing at the MacOSX11 SDK). Doing:

MACOSX_DEPLOYMENT_TARGET=10.14 ./mach build

I get a build that recognizes the deprecated fonts, while [clobbering and then] doing:

MACOSX_DEPLOYMENT_TARGET=10.15 ./mach build

gives me a build that hides them.

The try build at https://treeherder.mozilla.org/jobs?repo=try&revision=cd65d18b15de64142e84bb94f9084f1d433ddd59 has a patch that should make the deprecated fonts visible again even when built with a new SDK and deployment version; I will double-check on an Apple Silicon machine once the build is ready.

Comment 14

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1743170 - Include deprecated macOS font family names in the font list, if available at runtime. r=#gfx-reviewers

Comment 15

[Jonathan Kew [:jfkthame]]

The above patch seems to work as expected for me, across both Intel and M1 systems, though we should re-check the behavior when updating the SDK to make sure the build-time condition is behaving correctly.

(The hard-coded list here is ugly, but I haven't found any usable APIs that would let us avoid it for now. Exactly which "deprecated" fonts are/aren't visible in the list returned by the OS seems to vary, so I've included the most complete list I was able to come up with, and the code will drop any that are duplicates of what the OS already exposes to us.)

Comment 16

[Greg Tatum [:gregtatum] (REO Fx132)]

Is there anything blocking landing this code? This bug (and Bug 1696504) is blocking some product feature work (Bug 1759504) I now have up for review.

Comment 17

[Jonathan Kew [:jfkthame]]

OK, I've lando'd it just now. (Originally, I was waiting for try to confirm the test actually passed in CI...)

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a74b19849df9
Include deprecated macOS font family names in the font list, if available at runtime. r=gfx-reviewers,lsalzman

Comment 19

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/a74b19849df9