Optimize larger image and canvas memory usage

RESOLVED FIXED in Firefox OS v2.0

Status

Firefox OS
Gaia::Homescreen
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: kgrandon, Assigned: khuey)

Tracking

({memory-footprint, perf})

unspecified
2.1 S1 (1aug)
ARM
Gonk (Firefox OS)
memory-footprint, perf
Dependency tree / graph

Firefox Tracking Flags

(blocking-b2g:2.0+, b2g-v2.0 fixed, b2g-v2.1 fixed)

Details

(Whiteboard: [MemShrink][systemsfe])

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

4 years ago
We would like to use larger images in the gaia homescreen on memory constrained devices. Due to bug 1029902 we had to reduce the size for 256mb devices. On a memory constrained flame for example we had to reduce image size from 126px to 64px due to memory usage. The original 126px accounted for three columns, and display density. This is likely a meta bug.
(Reporter)

Comment 1

4 years ago
Benoit - Is there anything we can do here or are there any open bugs that might help this?
Flags: needinfo?(bgirard)

Comment 2

4 years ago
I really think we shouldn't trigger accelerated canvas unless the canvas is attached to a document. There are many more cases where it's slower and causes memory issues than there are advantages for using gpu acceleration for off-screen canvases.
That would be quite easy to hack up ...
(In reply to Chris Lord [:cwiiis] from comment #2)
> I really think we shouldn't trigger accelerated canvas unless the canvas is
> attached to a document. There are many more cases where it's slower and
> causes memory issues than there are advantages for using gpu acceleration
> for off-screen canvases.

The problem is you might do a drawing command to it and later attach it to the screen and drive it.

Ideally what we need is the ability to transfer from software to hardware (and back would be nice to). Then if the canvas is being driven by rAF and/or updating at 60 FPS then we switch to the right backend for it.

For gaia we should just be adding the moz use software flag (I don't recall the name but it's there).
Flags: needinfo?(bgirard)

Comment 5

4 years ago
(In reply to Benoit Girard (:BenWa) from comment #4)
> The problem is you might do a drawing command to it and later attach it to
> the screen and drive it.
> 
> Ideally what we need is the ability to transfer from software to hardware
> (and back would be nice to). Then if the canvas is being driven by rAF
> and/or updating at 60 FPS then we switch to the right backend for it.

While yes, we need the ability to switch, I don't think people commonly create canvases, draw to them, then attach them to a document and continue to draw to them - at least, not in situations where not having hardware acceleration would be bad (I can imagine doing that for generated images, but in those cases, the updates are infrequent and likely faster in software).

That said, it's conjecture, numbers would be nice.

> For gaia we should just be adding the moz use software flag (I don't recall
> the name but it's there).

Ah, well, we should definitely be using this then...
willReadFrequently is the software flag
I'll test using that on the homescreen later and see what it does.
(In reply to Benoit Girard (:BenWa) from comment #4)
> (In reply to Chris Lord [:cwiiis] from comment #2)
> > I really think we shouldn't trigger accelerated canvas unless the canvas is
> > attached to a document. There are many more cases where it's slower and
> > causes memory issues than there are advantages for using gpu acceleration
> > for off-screen canvases.
> 
> The problem is you might do a drawing command to it and later attach it to
> the screen and drive it.
> 
> Ideally what we need is the ability to transfer from software to hardware

We don't have this ability now, but it would be easy to do. There isn't very much state held in the actual DrawTarget.

> (and back would be nice to).

We do have this ability already (going from hardware -> software)

> Then if the canvas is being driven by rAF
> and/or updating at 60 FPS then we switch to the right backend for it.

I don't see how the update frequency or rAF usage could determine what backend we use. It really just depends on the type of workload. If you are doing a bunch of getImageData/putImageData, you want software. If it's a bunch of drawImage(), you want hardware. If it's in between, well, we'd have to figure something out.

> 
> For gaia we should just be adding the moz use software flag (I don't recall
> the name but it's there).

The long term plan was to use this flag for the initial state only, and then switch to hardware/software later as defined by the heuristic. We could also consider a canvas not attached to a document as implicitly having this flag. If you attach and start doing drawImage() frequently we could then upgrade it.
Doing this heuristic-based thing would be a great project for an intern, I think. Others simply haven't had time to look into it.
Unfortunately setting willReadFrequently: true seems to just move the unaccounted memory from one spot (the GL stuff) to another (something in libpng).  heap-unclassified does drop about 2 MB, but the amount of memory reported under imagelib goes up 1 MB.
Actually, that appears to be an artifact of not having discarded images.  If I force that things look a lot better.
Created attachment 8459242 [details] [review]
PR to test willReadFrequently
Attachment #8459236 - Attachment is obsolete: true
Attachment #8459237 - Attachment is obsolete: true
A bit of testing shows that setting willReadFrequently: true has a pretty similar effect to Vivien's patch from bug 1029902 once images are discarded.
Attachment #8459242 - Flags: feedback?(kgrandon)
Attachment #8459242 - Flags: feedback?(21)
(Reporter)

Comment 16

4 years ago
Comment on attachment 8459242 [details] [review]
PR to test willReadFrequently

I'll leave an R+ in case you want to land this. It sounds like it should fix the first-time load of the homescreen, but would not impact subsequent loads where we do not leverage canvas? I wonder if this shouldn't be it's own bug.
Attachment #8459242 - Flags: feedback?(kgrandon) → review+
I didn't take detailed measurements but it looked like it was helping both first run and second run pretty substantially.
Blocks: 1011657
blocking-b2g: --- → 2.0?

Updated

4 years ago
Component: Graphics → Gaia
Keywords: footprint, perf
Product: Core → Firefox OS
Whiteboard: [MemShrink]

Updated

4 years ago
Whiteboard: [MemShrink] → [MemShrink][systemsfe]
I took 16 sets of measurements: http://people.mozilla.org/~khuey/new-measurements.tgz

The 4 configuration tested were:

'baseline': tip of gaia/gecko
'big-icons' tip of gaia/gecko with kgrandon's changes to icon sizes backed out
'no-hw-accel': big-icons + willReadFrequently
'both': baseline + willReadFrequently

With two variables:

first run vs. second run
--minimize vs. not

Take aways:

The amount of memory used by images seems to be highly variable, probably based on what has been discarded/redecoded whenever the about:memory report runs.

In some cases we appear to have decoded substantially more images than the 47 icons the homescreen has in an engineering build.

If you compare baseline (which is just the smaller icons) vs. no-hw-accel (which is just the willReadFrequently), heap-unclassified drops by a couple MB in every comparison.  In the unminimized cases the memory used by images is higher.

The heap-unclassified in the baseline second-run case is in the PNG decoder, while in the first-run case it's in the SkiaGL stuff.  With willReadFrequently neither of these sets of stacks appear in the DMD log.

In the 1st run no-minimize case the smaller icons vs. willReadFrequently is effectively a wash:
-0.17 MB (100.0%) -- explicit
├──-4.13 MB (2402.90%) ── heap-unclassified
├───4.11 MB (-2391.22%) -- images/content/raster

In the 2nd run no-minimze case the heap-unclassified is close to a wash, and the image memory has increased by 11 MB.  This makes little sense, because the theoretical maximum image size is 182 px * 182 px * 4 bytes per px^2 * 47.  And we see that there are 80-something images floating around.  I don't have an explanation for this.
So I took another measurement of the 2nd run no-minimize case and this time the image memory has increased by close to 6 MB which is pretty close to (182^2 - 64^2) * 4 * 47.
Great information, so I'm starting to get lost a bit :)  So far, do we have a combination of settings and approaches that would let us use large icons with comparable memory use of the current baseline (small icons), or at least, get us to the "acceptable" amount of memory used?
Assignee: nobody → khuey
Component: Gaia → Gaia::Homescreen
Keywords: checkin-needed
(In reply to Milan Sreckovic [:milan] from comment #20)
> Great information, so I'm starting to get lost a bit :)  So far, do we have
> a combination of settings and approaches that would let us use large icons
> with comparable memory use of the current baseline (small icons), or at
> least, get us to the "acceptable" amount of memory used?

Not comparable.  But we can use the large icons for 5-6 MB more when we're painting the homescreen.  Whether or not that's acceptable I don't know.
master: https://github.com/mozilla-b2g/gaia/commit/09b9aff6f288584e6be5eaaae504e395451da447
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → 2.1 S1 (1aug)
blocking-b2g: 2.0? → 2.0+
v2.0: https://github.com/mozilla-b2g/gaia/commit/83fc38ade5f3a1303340c315ef26acb2026f2de6
status-b2g-v2.0: --- → fixed
status-b2g-v2.1: --- → fixed
Comment on attachment 8459242 [details] [review]
PR to test willReadFrequently

I guess this does not need my feedback anymore.
Attachment #8459242 - Flags: feedback?(21)
You need to log in before you can comment on or make changes to this bug.