1041112 - Optimize larger image and canvas memory usage

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Homescreen, defect)

Description

[Kevin Grandon :kgrandon]

We would like to use larger images in the gaia homescreen on memory constrained devices. Due to bug 1029902 we had to reduce the size for 256mb devices. On a memory constrained flame for example we had to reduce image size from 126px to 64px due to memory usage. The original 126px accounted for three columns, and display density. This is likely a meta bug.

Comment 1

[Kevin Grandon :kgrandon]

Benoit - Is there anything we can do here or are there any open bugs that might help this?

Comment 2

[Chris Lord [:cwiiis]]

I really think we shouldn't trigger accelerated canvas unless the canvas is attached to a document. There are many more cases where it's slower and causes memory issues than there are advantages for using gpu acceleration for off-screen canvases.

Comment 3

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

That would be quite easy to hack up ...

Comment 4

[Benoit Girard (:BenWa)]

(In reply to Chris Lord [:cwiiis] from comment #2)
> I really think we shouldn't trigger accelerated canvas unless the canvas is
> attached to a document. There are many more cases where it's slower and
> causes memory issues than there are advantages for using gpu acceleration
> for off-screen canvases.

The problem is you might do a drawing command to it and later attach it to the screen and drive it.

Ideally what we need is the ability to transfer from software to hardware (and back would be nice to). Then if the canvas is being driven by rAF and/or updating at 60 FPS then we switch to the right backend for it.

For gaia we should just be adding the moz use software flag (I don't recall the name but it's there).

Comment 5

[Chris Lord [:cwiiis]]

(In reply to Benoit Girard (:BenWa) from comment #4)
> The problem is you might do a drawing command to it and later attach it to
> the screen and drive it.
> 
> Ideally what we need is the ability to transfer from software to hardware
> (and back would be nice to). Then if the canvas is being driven by rAF
> and/or updating at 60 FPS then we switch to the right backend for it.

While yes, we need the ability to switch, I don't think people commonly create canvases, draw to them, then attach them to a document and continue to draw to them - at least, not in situations where not having hardware acceleration would be bad (I can imagine doing that for generated images, but in those cases, the updates are infrequent and likely faster in software).

That said, it's conjecture, numbers would be nice.

> For gaia we should just be adding the moz use software flag (I don't recall
> the name but it's there).

Ah, well, we should definitely be using this then...

Comment 6

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

willReadFrequently is the software flag

Comment 7

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I'll test using that on the homescreen later and see what it does.

Comment 8

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Benoit Girard (:BenWa) from comment #4)
> (In reply to Chris Lord [:cwiiis] from comment #2)
> > I really think we shouldn't trigger accelerated canvas unless the canvas is
> > attached to a document. There are many more cases where it's slower and
> > causes memory issues than there are advantages for using gpu acceleration
> > for off-screen canvases.
> 
> The problem is you might do a drawing command to it and later attach it to
> the screen and drive it.
> 
> Ideally what we need is the ability to transfer from software to hardware

We don't have this ability now, but it would be easy to do. There isn't very much state held in the actual DrawTarget.

> (and back would be nice to).

We do have this ability already (going from hardware -> software)

> Then if the canvas is being driven by rAF
> and/or updating at 60 FPS then we switch to the right backend for it.

I don't see how the update frequency or rAF usage could determine what backend we use. It really just depends on the type of workload. If you are doing a bunch of getImageData/putImageData, you want software. If it's a bunch of drawImage(), you want hardware. If it's in between, well, we'd have to figure something out.

> 
> For gaia we should just be adding the moz use software flag (I don't recall
> the name but it's there).

The long term plan was to use this flag for the initial state only, and then switch to hardware/software later as defined by the heuristic. We could also consider a canvas not attached to a document as implicitly having this flag. If you attach and start doing drawImage() frequently we could then upgrade it.

Comment 9

[James Willcox (:snorp) ([e-mail]) (he/him)]

Doing this heuristic-based thing would be a great project for an intern, I think. Others simply haven't had time to look into it.

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: DMD with hw acceleration

Comment 11

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: DMD without hw acceleration

Comment 12

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Unfortunately setting willReadFrequently: true seems to just move the unaccounted memory from one spot (the GL stuff) to another (something in libpng).  heap-unclassified does drop about 2 MB, but the amount of memory reported under imagelib goes up 1 MB.

Comment 13

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Actually, that appears to be an artifact of not having discarded images.  If I force that things look a lot better.

Comment 14

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: PR to test willReadFrequently

Comment 15

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

A bit of testing shows that setting willReadFrequently: true has a pretty similar effect to Vivien's patch from bug 1029902 once images are discarded.

Comment 16

[Kevin Grandon :kgrandon]

Comment on attachment 8459242 [details] [review]
PR to test willReadFrequently

I'll leave an R+ in case you want to land this. It sounds like it should fix the first-time load of the homescreen, but would not impact subsequent loads where we do not leverage canvas? I wonder if this shouldn't be it's own bug.

Comment 17

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I didn't take detailed measurements but it looked like it was helping both first run and second run pretty substantially.

Comment 18

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I took 16 sets of measurements: http://people.mozilla.org/~khuey/new-measurements.tgz

The 4 configuration tested were:

'baseline': tip of gaia/gecko
'big-icons' tip of gaia/gecko with kgrandon's changes to icon sizes backed out
'no-hw-accel': big-icons + willReadFrequently
'both': baseline + willReadFrequently

With two variables:

first run vs. second run
--minimize vs. not

Take aways:

The amount of memory used by images seems to be highly variable, probably based on what has been discarded/redecoded whenever the about:memory report runs.

In some cases we appear to have decoded substantially more images than the 47 icons the homescreen has in an engineering build.

If you compare baseline (which is just the smaller icons) vs. no-hw-accel (which is just the willReadFrequently), heap-unclassified drops by a couple MB in every comparison.  In the unminimized cases the memory used by images is higher.

The heap-unclassified in the baseline second-run case is in the PNG decoder, while in the first-run case it's in the SkiaGL stuff.  With willReadFrequently neither of these sets of stacks appear in the DMD log.

In the 1st run no-minimize case the smaller icons vs. willReadFrequently is effectively a wash:
-0.17 MB (100.0%) -- explicit
├──-4.13 MB (2402.90%) ── heap-unclassified
├───4.11 MB (-2391.22%) -- images/content/raster

In the 2nd run no-minimze case the heap-unclassified is close to a wash, and the image memory has increased by 11 MB.  This makes little sense, because the theoretical maximum image size is 182 px * 182 px * 4 bytes per px^2 * 47.  And we see that there are 80-something images floating around.  I don't have an explanation for this.

Comment 19

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So I took another measurement of the 2nd run no-minimize case and this time the image memory has increased by close to 6 MB which is pretty close to (182^2 - 64^2) * 4 * 47.

Comment 20

[Milan Sreckovic [:milan] (needinfo for best results)]

Great information, so I'm starting to get lost a bit :)  So far, do we have a combination of settings and approaches that would let us use large icons with comparable memory use of the current baseline (small icons), or at least, get us to the "acceptable" amount of memory used?

Comment 21

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Milan Sreckovic [:milan] from comment #20)
> Great information, so I'm starting to get lost a bit :)  So far, do we have
> a combination of settings and approaches that would let us use large icons
> with comparable memory use of the current baseline (small icons), or at
> least, get us to the "acceptable" amount of memory used?

Not comparable.  But we can use the large icons for 5-6 MB more when we're painting the homescreen.  Whether or not that's acceptable I don't know.

Comment 22

[Carsten Book [:Tomcat]]

master: https://github.com/mozilla-b2g/gaia/commit/09b9aff6f288584e6be5eaaae504e395451da447

Comment 23

[Ryan VanderMeulen [:RyanVM]]

v2.0: https://github.com/mozilla-b2g/gaia/commit/83fc38ade5f3a1303340c315ef26acb2026f2de6

Comment 24

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Comment on attachment 8459242 [details] [review]
PR to test willReadFrequently

I guess this does not need my feedback anymore.