1076349 - High peak GPU memory usage and slow scrolling speed due to landing Bug 902952

Status: RESOLVED INCOMPLETE

(Core :: Graphics, defect)

Description

[Alice0775 White]

Attachment: peak memory after landing Bug 902952

[Tracking Requested - why for this release]:

After landing Bug 902952, 
* Peak memory usage has become extremely large.
* The scrolling speed has become slow.


Steps To Reproduce:
1. Open testcase index.html
   https://dl.dropboxusercontent.com/u/33246373/20140922031026.7z   
2. Run the following code in ScratchPad with chrome privilege

   (function() {
    function scroll(event) {
      var top = content.document.documentElement.scrollTop ||
               content.document.body.scrollTop;
      
      if (top < 11000) {
        var evt = content.document.createEvent("KeyboardEvent");
        evt.initKeyEvent ("keypress", true, true, window, 
                              null, null, null, null, 
                              40, 0) 
        content.document.dispatchEvent(evt)
      } else {
        content.removeEventListener("scroll", scroll, false);
        content.scrollTo(0,0);
        alert(new Date().getTime() - start);
      }
    }

    content.focus();
    content.scrollTo(0,0);
    var start = new Date().getTime();
    content.addEventListener("scroll", scroll, false);
    scroll();
})();

Actual Results:
Peak memory usage has become extremely large, increased +6GB.
* The scrolling speed has become slow, 50% slow.


Regression window(m-i)

Good:
Peak memory usage less than +1.0GB
Scroll speed 13653, 13837 msec
https://hg.mozilla.org/integration/mozilla-inbound/rev/165c3fd176ec
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141001112720

Bad:
Peak memory usage +6GB
Scroll speed 21342, 21738 msec
https://hg.mozilla.org/integration/mozilla-inbound/rev/d954ed24e795
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141001113823

Pushlog:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=165c3fd176ec&tochange=d954ed24e795

Comment 1

[Alice0775 White]

Graphics
--------

Adapter Description: ATI Radeon HD 4300/4500 Series
Adapter Drivers: aticfx64 aticfx64 aticfx32 aticfx32 atiumd64 atidxx64 atiumdag atidxx32 atiumdva atiumd6a atitmm64
Adapter RAM: 512
ClearType Parameters: Gamma: 2200 Pixel Structure: R ClearType Level: 50 Enhanced Contrast: 200
Device ID: 0x954f
Direct2D Enabled: true
DirectWrite Enabled: true (6.2.9200.16492)
Driver Date: 4-29-2013
Driver Version: 8.970.100.1100
GPU #2 Active: false
GPU Accelerated Windows: 1/1 Direct3D 11 (OMTC)
Subsys ID: 00000000
Vendor ID: 0x1002
WebGL Renderer: Google Inc. -- ANGLE (ATI Radeon HD 4300/4500 Series Direct3D9Ex vs_3_0 ps_3_0)
windowLayerManagerRemote: true
AzureCanvasBackend: direct2d
AzureContentBackend: direct2d 1.1
AzureFallbackCanvasBackend: cairo
AzureSkiaAccelerated: 0

Comment 2

[Alice0775 White]

Attachment: peak memory before landing Bug 902952

Comment 3

[Elbart]

Similar regression in this test
https://developer.mozilla.org/media/uploads/demos/p/a/paulrouget/8bfba7f0b6c62d877a2b82dd5e10931e/hacksmozillaorg-achi_1334270447_demo_package/HWACCEL/

on this machine:

=====================================================================
Intel i5-430U @ 1.2GHz-1.7GHz

Application Basics
------------------

Name: Firefox
Version: 35.0a1
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
Multiprocess Windows: 0/1

Graphics
--------

Adapter Description: Intel(R) HD Graphics
Adapter Description (GPU #2): NVIDIA GeForce 310M
Adapter Drivers: igdumd64 igd10umd64 igdumdx32 igd10umd32
Adapter Drivers (GPU #2): nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Adapter RAM: Unknown
Adapter RAM (GPU #2): 1024
ClearType Parameters: Gamma: 2200 Pixel Structure: R ClearType Level: 100 Enhanced Contrast: 50
Device ID: 0x0046
Device ID (GPU #2): 0x0a70
Direct2D Enabled: true
DirectWrite Enabled: true (6.2.9200.16571)
Driver Date: 1-30-2013
Driver Date (GPU #2): 7-2-2014
Driver Version: 8.15.10.2993
Driver Version (GPU #2): 9.18.13.4052
GPU #2 Active: false
GPU Accelerated Windows: 1/1 Direct3D 11 (OMTC)
Subsys ID: 12d21043
Subsys ID (GPU #2): 12d21043
Vendor ID: 0x8086
Vendor ID (GPU #2): 0x10de
WebGL Renderer: Google Inc. -- ANGLE (Intel(R) HD Graphics Direct3D9Ex vs_3_0 ps_3_0)
windowLayerManagerRemote: true
AzureCanvasBackend: direct2d
AzureContentBackend: direct2d 1.1
AzureFallbackCanvasBackend: cairo
AzureSkiaAccelerated: 0

Important Modified Preferences
------------------------------

gfx.blacklist.direct2d: 4
gfx.direct2d.force-enabled: true
gfx.direct3d.last_used_feature_level_idx: 1
=====================================================================

Before: 14FPS
After:   8FPS

IE11:   20FPS
Chrome: 40FPS

Comment 4

[Elbart]

Doing a bisection with force-setting the prefs of https://hg.mozilla.org/mozilla-central/rev/d954ed24e795 I came to this regression-range:

Last good revision: bf5fcc0c4b27 (2014-09-15)
First bad revision: 55b46de164d8 (2014-09-16)
Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bf5fcc0c4b27&tochange=55b46de164d8

Last good revision: 2badc0f1f829
First bad revision: 8e28464849fa
Pushlog:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=2badc0f1f829&tochange=8e28464849fa

Is it the same for you, Alice?

Comment 5

[Robert Kaiser]

(In reply to Elbart from comment #4)
> Last good revision: 2badc0f1f829
> First bad revision: 8e28464849fa
> Pushlog:
> https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=2badc0f1f829&tochange=8e28464849fa

That makes sense, from what I can tell that's when D2D 1.1 support was first added at all. Not sure if it helps Bas and other gfx people a whole lot, though. What it can close out is any later and smaller change causing this, so in that sense it's a step forward - but in the end, I guess that push being the regression just means that D2D 1.1 being the issue, which doesn't point to a fix unfortunately. :(

Comment 6

[Jeff Muizelaar [:jrmuizel]]

So the page at the beginning of this bug has a bunch of large (3900x3900) images. I had a look at what's happening with API trace and we seem to be making copies of these into new textures everytime we draw. This is obviously quite bad. I suspect this is either a silly bug (we're getting the wrong kind of surfaces and end up going down a slow path, or has something to do with D2D1.1's use of ID2D1Images.

Comment 7

[Alice0775 White]

Regression window with force enable Direct2D1.1
user_pref("gfx.canvas.azure.backends", "direct2d1.1,direct2d,skia,cairo");
user_pref("gfx.content.azure.backends", "direct2d1.1,direct2d,cairo");
user_pref("gfx.direct2d.use1_1", true);

Inlocal build
Last Good: 20e97c8496e4
First Bad: 7b9201731195

Triggered by
7b9201731195	Matt Woodrow — Bug 1046550 - Part 3: Use Direct2D 1.1 when preffed on. r=bas

Comment 8

[Bas Schouten (:bas.schouten)]

On my desktop machine I cannot reproduce a difference between Direct2D 1.0 and 1.1 on this page. I get about 8000ms on the first run and 4000ms on subsequent runs with both Direct2D 1.0 and Direct2D 1.1. I did do some profiling though and it does look like there's some significant win to be had here, although none of it is D2D 1.1 specific. First of all it looks like we never optimize our high quality downscaled images, which means that we upload them again on every draw call. There's also a fair amount of time spent uploading the 3900x3900 images but that's not completely unexpected. Image decoding is the other main contributor of the profile, but as far as I can tell that is all spent off the main thread so shouldn't be the biggest issue. I'll get this running on hardware more similar to the hardware it was reported on and see if I can reproduce something there.

Comment 9

[Alice0775 White]

profiler log
http://people.mozilla.org/~bgirard/cleopatra/#report=93b4cd85c37981df04e107007d92186649add4bd

Comment 10

[Bas Schouten (:bas.schouten)]

Hrm, I don't see anything that should be D2D 1.1 specific in that profile. I've been testing more extensively with a laptop with NVidia optimus. Both using the discrete GPU or switching off the discrete GPU and using the integrated GPU, but I cannot reproduce any differences there either. The profile that you mosted suggests we spend a significant amount of time deallocating textures, particularly clearing out the optimized one we have in LockImageData it appears.

Jeff, did you say you had a machine which reproduced the D2D 1.0/1.1 difference?

Comment 11

[Alice0775 White]

Attachment: about_memory.txt

The huge memory usage is gpu-shared....

3,551,723,520 B ── gpu-shared

Comment 12

[Jeff Muizelaar [:jrmuizel]]

I haven't tried to reproduce. I will try though.

Comment 13

[Jeff Muizelaar [:jrmuizel]]

(In reply to Alice0775 White from comment #11)
> Created attachment 8500173 [details]
> about_memory.txt
> 
> 
> The huge memory usage is gpu-shared....
> 
> 3,551,723,520 B ── gpu-shared

When are you measuring this? After the run or during?

Comment 14

[Alice0775 White]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #13)
> (In reply to Alice0775 White from comment #11)
> > Created attachment 8500173 [details]
> > about_memory.txt
> > 
> > 
> > The huge memory usage is gpu-shared....
> > 
> > 3,551,723,520 B ── gpu-shared
> 
> When are you measuring this? After the run or during?

During test. 
Measuring it from another window(Ctrl+N).

STR
1. Start Nightly
2. Open the test html
3. Open ScratchPad and pase the Javascript Code 

4. Open new window(Ctrl+N) 
5. Open about:memory

6. Run the ScratchPad code with chrome privilege
7. Click [Measure] in the tab of about:memory when memory usage becomes Giga bytes.
   (it is slightly difficult timing)

Comment 15

[Jeff Muizelaar [:jrmuizel]]

I wasn't able to reproduce the timing differences on my machine.

Alice0775 White can you get profiles of both the fast and slow cases. That should give us a better idea of where the slowdown is coming from.

Comment 16

[Alice0775 White]

Nightly(2014-10-01) before landing Bug 902952
https://hg.mozilla.org/mozilla-central/rev/14665b1de5ee
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141001030205
http://people.mozilla.org/~bgirard/cleopatra/#report=51dcca5c8396567d6fedfb745b6fd608229a2185


Nightly(2014-10-02) after landing Bug 902952
https://hg.mozilla.org/mozilla-central/rev/2399d1ae89e9
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141002030202
http://people.mozilla.org/~bgirard/cleopatra/#report=3480a43da9a3aa0c99d69a5336d68a792bd0e9f9

Comment 17

[Alice0775 White]

Test case of Comment 3
https://developer.mozilla.org/media/uploads/demos/p/a/paulrouget/8bfba7f0b6c62d877a2b82dd5e10931e/hacksmozillaorg-achi_1334270447_demo_package/HWACCEL/

Nightly(2014-10-01) before landing Bug 902952
https://hg.mozilla.org/mozilla-central/rev/14665b1de5ee
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141001030205
http://people.mozilla.org/~bgirard/cleopatra/#report=3a36aada795504043aa61b1c7340524351cac257


Nightly(2014-10-02) after landing Bug 902952
https://hg.mozilla.org/mozilla-central/rev/2399d1ae89e9
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 ID:20141002030202
http://people.mozilla.org/~bgirard/cleopatra/#report=0e49ddedb5545343dc8947f3c314ab163423b65c

Comment 18

[Jeff Muizelaar [:jrmuizel]]

(In reply to Alice0775 White from comment #16)
> Nightly(2014-10-01) before landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/14665b1de5ee
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141001030205
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=51dcca5c8396567d6fedfb745b6fd608229a2185
> 
> 
> Nightly(2014-10-02) after landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/2399d1ae89e9
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141002030202
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=3480a43da9a3aa0c99d69a5336d68a792bd0e9f9

This looks like it might have to do with how we Optimize source surfaces. It would be interesting to make the D2D1.1 code more similar to the D2D1 code and see if it improves. That being said it looks like we're just blocked on a lock during texture creation so it might be unrelated.

(In reply to Alice0775 White from comment #17)
> Test case of Comment 3
> https://developer.mozilla.org/media/uploads/demos/p/a/paulrouget/
> 8bfba7f0b6c62d877a2b82dd5e10931e/hacksmozillaorg-
> achi_1334270447_demo_package/HWACCEL/
> 
> Nightly(2014-10-01) before landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/14665b1de5ee
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141001030205
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=3a36aada795504043aa61b1c7340524351cac257
> 
> 
> Nightly(2014-10-02) after landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/2399d1ae89e9
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141002030202
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=0e49ddedb5545343dc8947f3c314ab163423b65c


This looks like it might be caused by the D2D1 D2D1.1 mismatch between canvas and content?

We end up with a stack that looks like this:

CTexture2D::Map(unsigned int,D3D10_MAP,unsigned int,D3D10_MAPPED_TEXTURE2D *)
mozilla::gfx::DataSourceSurfaceD2DTarget::EnsureMapped()
mozilla::gfx::DataSourceSurfaceD2DTarget::Stride()
mozilla::gfx::DrawTargetD2D1::GetImageForSurface(mozilla::gfx::SourceSurface *,mozilla::gfx::Matrix &,mozilla::gfx::ExtendMode)
mozilla::gfx::DrawTargetD2D1::CopySurface(mozilla::gfx::SourceSurface *,mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gfx::IntPointTyped<mozilla::gfx::UnknownUnits> const &)
mozilla::layers::CopyableCanvasLayer::UpdateTarget(mozilla::gfx::DrawTarget *)
mozilla::layers::CanvasClient2D::Update(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>,mozilla::layers::ClientCanvasLayer *)
mozilla::layers::ClientCanvasLayer::RenderLayer()

Which is no good.

Comment 19

[Alice0775 White]

Attachment: memory/CPU usage

1. Open testcase index.html
   https://dl.dropboxusercontent.com/u/33246373/20140922031026.7z   
2. Repeat scroll to bottom and top by dragging thumb

Memory/CPU usage grow and grow

Comment 20

[Jeff Muizelaar [:jrmuizel]]

Here's a build that might help:
https://tbpl.mozilla.org/?tree=Try&rev=de32e9b7ea2a

Comment 21

[Jim Jeffery not reading bug-mail 1/2/11]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> Here's a build that might help:
> https://tbpl.mozilla.org/?tree=Try&rev=de32e9b7ea2a

This try-build seems to fix https://bugzilla.mozilla.org/show_bug.cgi?id=1077157 
where there is missing controls on HTML5 youtube vids with <96DPI settings....

Comment 22

[Alice0775 White]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> Here's a build that might help:
> https://tbpl.mozilla.org/?tree=Try&rev=de32e9b7ea2a

No thing to changed...
23476 msec and gigabyte memory usage...

(fortunately, it seems to fix the problem of Test case of Comment 3)

Comment 23

[Matt Woodrow (:mattwoodrow)]

(In reply to Alice0775 White from comment #16)
> Nightly(2014-10-01) before landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/14665b1de5ee
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141001030205
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=51dcca5c8396567d6fedfb745b6fd608229a2185
> 
> 
> Nightly(2014-10-02) after landing Bug 902952
> https://hg.mozilla.org/mozilla-central/rev/2399d1ae89e9
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
> ID:20141002030202
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=3480a43da9a3aa0c99d69a5336d68a792bd0e9f9

Looks like the majority of the time is spent reading back optimized d2d(1) surfaces into memory so that we can start scale requests. What's particularly exciting is that once we've done this we drop our reference to the d2d surface, and the paint that immediately follows has to upload it all again.

The two profiles actually look fairly similar to me, apart from the d2d 1.1 has two incredibly slow frames right at the end.

These frames spend time waiting on a monitor in RequestDecodeCore (20%), uploading a finished scale to d2d (10%), zero'ing memory in InternalAddFrame (18%), and waiting on d2d 1.1 to delete textures (48%).

Is that last piece possible because d2d delays destruction of textures until sometime in the future, and d2d 1.1 is delaying considerably longer than d2d 1.0? That would explain the memory usage rise, and the massively slow frame when it finally decides to do some cleanup. Not sure if this is d2d 1.1's fault, or ours for thrashing textures so much.

Comment 24

[Alice0775 White]

[Tracking Requested - why for this release]: PC hang 

Repeat scroll up and down by dragging thumb. Memory hogging and thrashing occurs. then PC become unresponsive.

Comment 25

[Alice0775 White]

I can no longer reproduce the problem, because I have changed Graphic Card.