1340351 - Broken Fonts on macOS due to content process sandbox denying access to fonts synced by Adobe Creative Cloud

Status: RESOLVED FIXED

(Core :: Graphics: Text, defect)

Description

[Stephen Horlander [:shorlander]]

Attachment: Broken-Fonts.png

For the last few weeks on Nightly I have been getting a bunch of missing characters.

Attaching a screenshot from this URL: https://health.graphics/quantum/countdown

The calculated font in DevTools is listed a .LastResort

Everything seems to work normally in Release Firefox. Also I can't reproduce on another machine running the same version of macOS.

Comment 1

[Jonathan Kew [:jfkthame]]

That's.... disturbing. Have you tried Safe Mode? Fresh profile?

Does it affect only certain sites, or is it everywhere? 

Is the Open Sans font family installed on the problem machine?

A regression range would be really helpful here.

Comment 2

[Stephen Horlander [:shorlander]]

Attachment: 2016-12-14 vs. 2016-12-15

I have tried Safe Mode and Fresh Profiles.

I think I have found the conflict, it seems to be fonts that I have installed with Adobe Creative Cloud. But oddly only on some pages that reference those fonts?

I did find a regression range.

The last version that worked is this one: https://ftp.mozilla.org/pub/firefox/nightly/2016/12/2016-12-14-03-02-31-mozilla-central/firefox-53.0a1.en-US.mac.dmg

Starts breaking here: https://ftp.mozilla.org/pub/firefox/nightly/2016/12/2016-12-15-06-12-12-mozilla-central/firefox-53.0a1.en-US.mac.dmg

Comment 3

[Jonathan Kew [:jfkthame]]

Odd. I wonder if this is related to bug 1320665, though I don't really see why it should be.

Could you run mozregression to try and get a more exact regression range, please?

Comment 4

[Stephen Horlander [:shorlander]]

Mozregression says:

4:55.07 INFO: Looks like the following bug has the changes which introduced the regression:
https://bugzilla.mozilla.org/show_bug.cgi?id=1320665

Comment 5

[Jonathan Kew [:jfkthame]]

Can you selectively enable/disable the Adobe fonts you mentioned (comment 2), to try and pinpoint if there's one particular font whose presence is leading to this problem?

Can you reproduce the problem with a minimal testcase such as

  data:text/html,<div style="font-family: Open Sans, sans-serif">Hello world

or does it only occur on certain pages with more complex styling involved?

(While I recognize that bug 1320665 is related to fonts, character coverage, etc., I don't yet understand how it could result in the symptoms you're seeing. Hence wanting to narrow down the key factors that are involved here.)

Comment 6

[Stephen Horlander [:shorlander]]

Attachment: Reduced Test Case Screenshot

I disabled them all and re-enabled them one at a time to the same result: see attached example (top Nightly, bottom Release)

The ? glyph seems to be trigger any time Firefox tries to access one of the fonts installed by Creative Cloud locally. If the site is loading the font remotely as a Web Font then things work as expected.


It might be worth noting that it isn't clear to me how the Creative Cloud font syncing works. The synced fonts aren't in any of the normal font locations (/Library/Fonts, ~/Library/Fonts, etc.), they also don't show up in Font Book, yet they *are* available to other macOS apps.

Comment 7

[Jonathan Kew [:jfkthame]]

Hmmmm.

In Font Book, did you check the whole of the All Fonts list? (I wouldn't be entirely surprised if they showed up at the end instead of in the expected alphabetical position.)

OK, let's look at one more thing, please. First, ensure e10s is *disabled* and restart Nightly.

Then enable one of your local Creative Cloud fonts, and load a testcase that triggers the problem.

And then, go to Preferences / Content and look in the font popup menu. Does the CC font appear in that list?

Comment 8

[Stephen Horlander [:shorlander]]

If I *disable* e10s and restart the problem goes away and the font *is* listed in the Content font popup menu.

Comment 9

[Jonathan Kew [:jfkthame]]

Ahhhhh... interesting. And with e10s *enabled* (and so the problem is occurring again), does the font appear in the popup? (I'm guessing not, but please confirm.)

Comment 10

[Stephen Horlander [:shorlander]]

It still shows up with e10s enabled. I can even pick it. It just turns a lot of the web into question marks ;)

Comment 11

[Jonathan Kew [:jfkthame]]

OK, another thing to try. With e10s enabled, and a web page showing question marks -- so one of the CC fonts is supposedly in use, but not working for us -- go into Font Book and disable one of your locally-installed font families (i.e. something that's installed in ~/Library/Fonts, *not* a Creative Cloud font).

This should cause Firefox to update its list of available fonts (you could check that whatever you disabled no longer appears in the Content font menu).

Does that also cause the content process to start using the CC font properly, or does it still remain broken?

Comment 12

[Stephen Horlander [:shorlander]]

That has no effect, fonts remain broken.

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

Jonathan - any next-steps?  regression-window?

Comment 14

[Jonathan Kew [:jfkthame]]

We have a regression window that points at bug 1320665 (comment 4). What I don't currently understand is how that bug can be having that effect...

This may need to be debugged by someone who can reproduce it, which appears to mean someone with the Adobe Creative Cloud suite installed (see comment 2).

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

shorlander - can you coordinate with Jonathan to debug this?

Comment 16

[Stephen Horlander [:shorlander]]

I sent an email to Jonathan. I am at the end of my ability debug this further.

Comment 17

[Tim Nguyen :ntim]

I think this is reproducible with Google web fonts' version of Fira Sans.

Comment 18

[Astley Chen (inactive)]

I can reproduce on nightly 55 now after installing Adobe CC desktop app and syncing Google Open Sans font into local system. However, the symptom is unable to reproduce on Aurora 54 and Beta 53 which included patch from bug 1320665. Need more insight to figure out the differences among different versions.

STR on Mac:
1. Install Adobe CC from [1]
2. Sign up or log in with your Adobe ID.
3. In Adobe CC / Assets Tab / Sync Fonts from Typekit
4. Pick Open Sans font into your sync list on Typekit website.
5. You should see the selected font in synced font list.
6. Visit https://health.graphics/quantum/countdown with nightly 55.

[1] https://creative.adobe.com/products/download/creative-cloud

Comment 19

[Astley Chen (inactive)]

shorlander, could you help to confirm if it's still reproducible on aurora 54 and beta 53 ? Thanks.

Comment 20

[Tim Nguyen :ntim]

(In reply to Astley Chen [:astley] (UTC+8) from comment #18)
> I can reproduce on nightly 55 now after installing Adobe CC desktop app and
> syncing Google Open Sans font into local system. However, the symptom is
> unable to reproduce on Aurora 54 and Beta 53 which included patch from bug
> 1320665. Need more insight to figure out the differences among different
> versions.
Might be due to some pref (multi content process,...) that you don't have enabled on aurora 54/beta 53.

Comment 21

[Astley Chen (inactive)]

Before we got feedback from shortlander, I'm inclined to say this is a nightly only bug. Update status flags.

Jonathan, given the clues in comment 18, do you have any insight to continue the investigation ?

Comment 22

[Stephen Horlander [:shorlander]]

I can't reproduce on Beta or Dev Edition.

Comment 23

[Jonathan Kew [:jfkthame]]

I'm about to see if I can reproduce this using a "trial download" of AdobeCC; if so, I may have a chance of debugging. Leaving the ni? flag on myself for now, pending further news....

Comment 24

[Jonathan Kew [:jfkthame]]

OK, I can reproduce something like this if I sync Fira Sans or Open Sans from typekit, *if* I have set gfx.downloadable_fonts.enabled to false in about:config. But that's not the default setting, and comment 2 said that it happened even with a fresh profile, which I find puzzling.

Anyway, even with d/l fonts disabled, we shouldn't end up failing like this, so that's one place to start.

Comment 25

[Jonathan Kew [:jfkthame]]

I don't fully understand what's going on here yet, but a large part of the problem is that the mechanism Adobe CC is using to "activate" the synced fonts appears to be incompatible with e10s content processes.

Regardless of the recent change in bug 1320665, AFAICT such fonts have -never- worked properly in e10s content; it was just a lot less obvious previously, because a page like https://health.graphics/quantum/countdown would fall back to the browser's default font (Helvetica, usually), which doesn't look -so- different from Fira Sans or Open Sans, and so nobody complained.

To demonstrate this, try syncing a more distinctive face from the typekit library (I used "Birra 2" as an example, from https://typekit.com/fonts/birra) to the local machine, and then using Firefox Beta or Dev Edition, look at a testcase such as

  data:text/html,<div style="font: 36px 'Birra 2', sans-serif">Hello world

With e10s -disabled-, this renders "Hello world" in the Birra 2 font, as expected. But with e10s -enabled-, the text falls back to the default sans-serif face. Nevertheless, checking the font list in Preferences / Content still shows Birra 2 as being available.

I think the Adobe CC-synced fonts are being activated in a way that makes them available to the chrome process -- hence, they show up in the font list -- but not to the content process. The content process -thinks- they're available, because it was given the platform font list from chrome via IPC, but they don't actually work there. Since bug 1320665 landed, that failure manifests as missing-glyph symbols, making us sad; prior to that, it manifested as fallback to Helvetica (or similar), and may have gone unnoticed.

Comment 26

[Jonathan Kew [:jfkthame]]

Argh... now, things don't seem to be behaving as I just described for beta/dev-ed after all. Investigation continues....

Comment 27

[Jonathan Kew [:jfkthame]]

OK, I'm confused. I can reproduce the problem described in comment 25 (i.e. Adobe-CC-synced fonts not working in the content process) using mozregression to run mozilla-central builds from as far back as 2016-08-01, yet I can't seem to reproduce it with mozilla-aurora builds from the same timeframe, even up to present day.

I can't go further back with mozilla-central, because when I ask mozregression to go back to an earlier release, the builds don't launch successfully at all on my system. :(

So this aspect of the problem really does seem to be Nightly-specific, rather than linked to a particular version or changeset. But currently I have no idea why.

Comment 28

[Stephen Horlander [:shorlander]]

Is it possible that it's caused by a build time error? Are we building Beta and/or Developer Edition in a different environment than Nightly? Maybe using different SDKs or Libraries?

Comment 29

[Jonathan Kew [:jfkthame]]

Not AFAICT ... the central vs aurora difference persists in local builds for me, although I'm building both in very much the same environment.

Comment 30

[Jonathan Kew [:jfkthame]]

Ah, finally some progress.... it turns out the key Nightly-only setting is security.sandbox.content.level. This defaults to 2 in nightly builds, but 1 elsewhere; and when it is 2, it breaks the AdobeCC-activated fonts in the content process(es).

I'm not sure yet exactly how the font activation is happening, and why the sandbox blocks those fonts -- or what we can do about it.

Stephen, could you confirm that setting security.sandbox.content.level to 1 resolves the problem for you? Thanks.

Comment 31

[Stephen Horlander [:shorlander]]

Yes, changing that setting to one fixes this on Nightly for me.

Comment 32

[Jonathan Kew [:jfkthame]]

Ah, found some relevant messages in the Console, for example:

SandboxViolation: plugin-container(78961) deny file-read-data /Users/jkew/Library/Application Support/Adobe/CoreSync/plugins/livetype/.r/.22494.otf

Looks like Adobe CC installs its "synced" fonts into that folder under ~/Library/Application Support/..., and one of the things that sandbox level 2 specifically blocks is access to files under ~/Library, with a few exceptions.

So for those fonts to work with the sandboxed content process, we need to punch an additional hole in the sandbox for that path.

Comment 33

[Jonathan Kew [:jfkthame]]

Attachment: Allow sandboxed content process to access fonts synced by Adobe Creative Cloud

So a patch like this should work, afaics. :shorlander, I've started a try build at https://treeherder.mozilla.org/#/jobs?repo=try&revision=75f92e2c53735a2dd8b44bb08ce2fe1836154be1, if you'd like to check this works for you once the build is ready.

Comment 34

[Bob Owen (:bobowen)]

Comment on attachment 8847785 [details] [diff] [review]
Allow sandboxed content process to access fonts synced by Adobe Creative Cloud

I'm going to pass this over to haik, as he is the main person dealing with the OS X sandbox.

I don't know too much about this, but my main concern here is that we're having to add a third-party specific path to the sandbox, so I'm not sure this is a great long term solution.
What if other plugins require similar rules.

Comment 35

[Jonathan Kew [:jfkthame]]

(In reply to Bob Owen (:bobowen) from comment #34)
> I don't know too much about this, but my main concern here is that we're
> having to add a third-party specific path to the sandbox, so I'm not sure
> this is a great long term solution.

True, but for now I don't know of an alternative approach; happy to hear if there is one!

> What if other plugins require similar rules.

We'd have to decide case by case which things we believe should work in the sandboxed process. IMO, allowing the Adobe-activated fonts to work is a good candidate.

Comment 36

[Haik Aftandilian [:haik]]

(In reply to Jonathan Kew (:jfkthame) from comment #35)
> (In reply to Bob Owen (:bobowen) from comment #34)
> > I don't know too much about this, but my main concern here is that we're
> > having to add a third-party specific path to the sandbox, so I'm not sure
> > this is a great long term solution.
> 
> True, but for now I don't know of an alternative approach; happy to hear if
> there is one!

If we can't automatically add font directories to the whitelist, adding the directory this way makes sense for now because we need these fonts to work and this bug shouldn't block Bug 1332190.

A couple of questions/comments though:

1) Does the user get to choose where to install the fonts during install of the CreateCloud suite?

2) If the Chrome process knows about the font directories that Content needs to be able to read from before Content is started, it can pass the directories to Content and they can be whitelisted in the Content sandbox rules dynamically. Does Chrome know which directories that Content will need early enough for this?

i.e., we would find all the font directories that are in locations the content process can not read from and pass the list of directories to content to whitelist them. We would want to make sure this doesn't include $HOME or other sensitive directories.

This is how the profile directory works. The Chrome process passes the profile directory path to Content before the sandbox is enabled (via the command line in this case) and Content uses that directory to whitelist $PROFILE/extensions.

There might be other font directories and we just haven't hit the problems yet.

3) Another alternative is to proxy these reads through the parent, but that only applies if we control the code reading these files and it's not does by an OS or 3rd party library.

> > What if other plugins require similar rules.
> 
> We'd have to decide case by case which things we believe should work in the
> sandboxed process. IMO, allowing the Adobe-activated fonts to work is a good
> candidate.

Is this plugin-related? Or just custom fonts being installed in the user's home directory?

Comment 37

[Jonathan Kew [:jfkthame]]

(In reply to Haik Aftandilian [:haik] from comment #36)
> (In reply to Jonathan Kew (:jfkthame) from comment #35)
> > (In reply to Bob Owen (:bobowen) from comment #34)
> > > I don't know too much about this, but my main concern here is that we're
> > > having to add a third-party specific path to the sandbox, so I'm not sure
> > > this is a great long term solution.
> > 
> > True, but for now I don't know of an alternative approach; happy to hear if
> > there is one!
> 
> If we can't automatically add font directories to the whitelist, adding the
> directory this way makes sense for now because we need these fonts to work
> and this bug shouldn't block Bug 1332190.
> 
> A couple of questions/comments though:
> 
> 1) Does the user get to choose where to install the fonts during install of
> the CreateCloud suite?

Not AFAIK - it's a black-box as far as the user is concerned.

The user just selects fonts on the typekit web site and asks it to "sync" them, and the CreativeCloud desktop application apparently implements this by stashing the font files in this location under its ~/Library/Application Support/ folder, and then calls a Core Text or Core Graphics API to activate them. (I expect it's probably calling CTFont​Manager​Register​Fonts​For​URL or similar.)

I didn't see any user options to control how/where the fonts are installed.

> 
> 2) If the Chrome process knows about the font directories that Content needs
> to be able to read from before Content is started, it can pass the
> directories to Content and they can be whitelisted in the Content sandbox
> rules dynamically. Does Chrome know which directories that Content will need
> early enough for this?

No; they might not even exist at the time the process is started, the user might only run CC and activate fonts after the browser is already running.

> 3) Another alternative is to proxy these reads through the parent, but that
> only applies if we control the code reading these files and it's not does by
> an OS or 3rd party library.

They're not read directly by our code; we just ask Core Graphics to use a given font, and it is responsible for accessing the file as necessary.

> Is this plugin-related? Or just custom fonts being installed in the user's
> home directory?

As far as we're concerned, this is just about custom fonts being installed in a path under $HOME. (Other than the standard ~/Library/Fonts path that we already allow.) The "plugin" in the path here is probably because the font-sync feature is regarded as a Creative Cloud plugin, or something like that. Or it can function as a plugin for other Adobe desktop apps.

Comment 38

[Haik Aftandilian [:haik]]

(In reply to Jonathan Kew (:jfkthame) from comment #37)
> (In reply to Haik Aftandilian [:haik] from comment #36)
> > (In reply to Jonathan Kew (:jfkthame) from comment #35)
> > > (In reply to Bob Owen (:bobowen) from comment #34)
> > > > I don't know too much about this, but my main concern here is that we're
> > > > having to add a third-party specific path to the sandbox, so I'm not sure
> > > > this is a great long term solution.
> > > 
> > > True, but for now I don't know of an alternative approach; happy to hear if
> > > there is one!
> > 
> > If we can't automatically add font directories to the whitelist, adding the
> > directory this way makes sense for now because we need these fonts to work
> > and this bug shouldn't block Bug 1332190.
> > 
> > A couple of questions/comments though:
> > 
> > 1) Does the user get to choose where to install the fonts during install of
> > the CreateCloud suite?
> 
> Not AFAIK - it's a black-box as far as the user is concerned.
> 
> The user just selects fonts on the typekit web site and asks it to "sync"
> them, and the CreativeCloud desktop application apparently implements this
> by stashing the font files in this location under its ~/Library/Application
> Support/ folder, and then calls a Core Text or Core Graphics API to activate
> them. (I expect it's probably calling CTFont​Manager​Register​Fonts​For​URL
> or similar.)
> 
> I didn't see any user options to control how/where the fonts are installed.
> 
> > 
> > 2) If the Chrome process knows about the font directories that Content needs
> > to be able to read from before Content is started, it can pass the
> > directories to Content and they can be whitelisted in the Content sandbox
> > rules dynamically. Does Chrome know which directories that Content will need
> > early enough for this?
> 
> No; they might not even exist at the time the process is started, the user
> might only run CC and activate fonts after the browser is already running.

And that would mean the fonts wouldn't work until the browser was restarted (if we relied on the parent to pass the list of font directories to the child.) Which might be OK in this scenario.

Trying to come up with a solution that doesn't require whitelisting doesn't seem worth it right now. This directory should be safe to add and we're already whitelisting ~/Library font directories. And not requiring browser restart is a better user experience.

Thanks for the explanations. The patch looks good to me.

Comment 39

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d1e870c4540505fadf47c979792ba3d2a27b2d68
Bug 1340351 - Allow sandboxed content process on macOS to access fonts synced by Adobe Creative Cloud. r=haik

Comment 40

[Astley Chen (inactive)]

Jonathan, thank you for working on this.

Comment 41

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/d1e870c45405