617266 - Memory mapped font files consume large amounts of virtual address space

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect)

Description

[zhentar]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101206 Firefox/4.0b8pre
Build Identifier: 

Firefox 4 is memory mapping what appears to be all of the fonts on my system. This represents a regression of over 550MB of virtual address space as compared to Firefox 3.6 on my system (and I haven't installed any extra fonts!)

Given that this is over a quarter of the available space to processes on 32-bit windows, it's likely a major contributing factor to OOM crashes.

Reproducible: Always

Comment 1

[Boris Zbarsky [:bzbarsky]]

Are _we_ mmapping them, or is dwrite doing it?

Comment 2

[(dormant account)]

(In reply to comment #1)
> Are _we_ mmapping them, or is dwrite doing it?

dwrite. See https://bugzilla.mozilla.org/show_bug.cgi?id=602792#c61 . I'm guessing this a side-effect of dwrite doing the stupid clientside fallback.

Comment 3

[John Daggett (:jtd)]

Josh, why did you switch this to XP?  The bug was reported against Windows 7.

Comment 4

[John Daggett (:jtd)]

(In reply to comment #0)
> Firefox 4 is memory mapping what appears to be all of the fonts on my system.
> This represents a regression of over 550MB of virtual address space as compared
> to Firefox 3.6 on my system (and I haven't installed any extra fonts!)

I'm sure you're right but how are you measuring this?

If possible, could you try running with the tryserver build on bug 602792, comment 59.  I'd be really curious to hear if that reduces the VM usage you're seeing.

Comment 5

[zhentar]

I looked at the process using VMMap ( http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx )

I'll give the tryserver build a go.

Comment 6

[zhentar]

Or not. The link is dead.

Comment 7

[zhentar]

I must not have been firing on all pistons when I filed this last night, because I missed/screwed up a fairly significant detail. It's not all fonts being memory mapped, it's a small set of fonts being mapped multiple times. For example, msmincho.ttc is mapped 14 times currently. arial.ttf 6 times, arialbd.ttf 7, and so on. If each of the fonts were mapped only once, it would be a fairly reasonable amount of address space consumed.

Comment 8

[Benjamin Smedberg]

We know that VM usage has been a big problem on trunk/Windows, and this looks like one likely culprit. Joe, who's the best person to take this?

Comment 9

[Joe Drew (not getting mail)]

I don't like blocking on unconfirmed bugs, but I guess we're working on the assumption that this is real.

John, can you generate a new try build?

Comment 10

[Benjamin Smedberg]

I don't see the behavior mentioned with d2d off: just a single fairly large mapping of c:\Windows\Fonts\StaticCache.dat

With d2d on, I can see the problem as described: many of the common fonts are mapped multiple times into the address space.

Comment 11

[John Daggett (:jtd)]

Grr, tryserver builds are supposed to be around for 10 days...

A more recent build:

  http://bit.ly/hU9D4u

This creates metrics more lazily and uses GDI code for table reads.

If this is dwrite, then this bug is basically another manifestation of bug 602792.

Comment 12

[Benjamin Smedberg]

Tryserver builds are kept for four days now.

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #12)
> Tryserver builds are kept for four days now.

Tryserver builds are kept for 10 days at http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/{your_email_changeset} and then are moved to ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/old/{your_email_changeset} for 4 additional days until they are deleted permanently.  Bug 615963 has a request for symlinking those.

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

I apologize for being incorrect with my information.  Here is the correct information:

Tryserver builds are kept for 4 days at http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/{your_email_changeset}

and then are moved to
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/old/{your_email_changeset}
 for 10 additional days until they are deleted permanently.

Comment 15

[zhentar]

tryserver build is a big improvement, but doesn't totally eliminate it. Fonts are still mapped, but most of them are only mapped once. It takes the regression down to ~80MB, with about 25MB of that being duplicate mappings.

Comment 16

[John Daggett (:jtd)]

Some comparison stats using VMMap:

trunk (20101208):
total          size 517M total-ws  96M
mapped files   size 203M total-ws   9M (half fonts, half profile/omni.jar)
private data   size  92M total-ws  49M

trunk (20101208) with d2d disabled:
total          size 318M total-ws 113M
mapped files   size  20M total-ws   5M
private data   size 122M total-ws  75M

ff3.6.12:
total          size 219M total-ws  61M
mapped files   size  19M total-ws   5M (staticcache = 1.2M)
private data   size  64M total-ws  34M

try-c67837cd8351 (use gdi table loading)
total          size 359M total-ws  94M
mapped files   size  39M total-ws   4M (mostly not font related)
private data   size 106M total-ws  58M

        trunk  ff36
total   513M   199M
mapped  190M    19M
share   103M    25M
private  99M    62M
image    70M    60M
stack    36M    27M
heap     13M     5M

Just to put this in context, I think the fact that the average WS size is 50% larger on trunk vs. 3.6 is a much larger problem then mapped files using chunks of VM space.  The WS size directly affects performance while using lots of VM might cause OOM conditions in worse-case scenarios.  The increase in WS size doesn't appear to be related to dwrite.

Once cmaps are cached (bug 600713) there will be a lot less file mapping going on.  When fonts are explicitly used on a page via dwrite, mapping will still occur but the problem will be confined to fonts on the page.

Comment 17

[John Daggett (:jtd)]

(In reply to comment #15)
> tryserver build is a big improvement, but doesn't totally eliminate it. Fonts
> are still mapped, but most of them are only mapped once. It takes the
> regression down to ~80MB, with about 25MB of that being duplicate mappings.

With Windows 7 RTM + normal updates I can reproduce the numerous duplicate font file mappings.  It appears to be related to Direct2D, on a machine without Direct2D enabled but with DirectWrite enabled, the problem doesn't occur.

Steps:

1. Using latest trunk, open several tabs:

http://people.mozilla.org/~jdaggett/tests/opacitywaterfalls.html
http://twitter.com/muratamakoto
http://www.facebook.com/
http://www.wikipedia.org/

2. Run VMMap

Result: multiple mappings for several fonts.  On a Japanese Windows 7 x64, several mappings of MSGothic and MSMincho show up (I'm guessing on an English install several instances of Arial show up).  Total address space used for mapped files: just over 300M.

However, downloading the IE9 beta seems to cure the problem.  IE9 has several required updates that get installed, including one related to "graphics".  Looking over the updates on the page below, I imagine the fixes are related to KB971512 and KB2028560 (for x64).  After the updates, the usage for the steps above drops to 141M and there are no duplicate font file mappings showing up in the VMMap display.

  http://ie.microsoft.com/testdrive/info/installinstructions/

Could those seeing this try updating to IE9 beta, or at least downloading the required updates for IE9 beta and verify whether that solves the duplicate mappings or not?

Comment 18

[zhentar]

Installing KB2028560 resolved the duplicate mappings for me.

Comment 19

[timeless]

Can we detect the absence of KB2028560 in some way and disable d2d? the code would of course have to be limited to nt6.1

Comment 20

[John Daggett (:jtd)]

It would still be helpful if others test this.

Comment 21

[John Daggett (:jtd)]

(In reply to comment #19)
> Can we detect the absence of KB2028560 in some way and disable d2d? the code
> would of course have to be limited to nt6.1

It might be possible to detect this but I don't think that's the right path to go down.  Better to reduce our font table loading in the dwrite case which appears to cause the excess file mapping when the update is not installed.

Comment 22

[[Baboo]]

(In reply to comment #19)
> Can we detect the absence of KB2028560 in some way and disable d2d? the code
> would of course have to be limited to nt6.1

Better check the version numbers of the updated file(s).

Comment 23

[John Daggett (:jtd)]

Microsoft posted an optional update today, KB2454826, which replaces the KB2028560 mentioned above:

http://support.microsoft.com/?kbid=2454826

Comment 24

[John Daggett (:jtd)]

Since this is a system-level bug that's resolved in recent updates and we've landed patches as part of bug 602792 that reduce the usage of DirectWrite font API's for reading cmaps, I think we should mark this as WONTFIX.