Closed Bug 1069534 Opened 10 years ago Closed 10 years ago

b2g process memleak in 2.1

Categories

(Firefox OS Graveyard :: Stability, defect, P1)

Product:
Firefox OS Graveyard
Component:
Stability
Platform:
ARM
Gonk (Firefox OS)
Type:
defect

Tracking

(blocking-b2g:2.1+)

Status:
RESOLVED WORKSFORME
Project Flags:
blocking-b2g 2.1+

People

(Reporter: tkundu, Unassigned)

References

Details

(Whiteboard: [caf priority: p3][CR 726725])

Attachments

(3 files)

unified-memory-report-1411035881-201.json.gz
104.33 KB, application/x-gzip
Details
memory-reports from a normal device which does not have memleak
2.62 MB, text/plain
Details
log patch - Add logout
13.71 KB, patch
Details | Diff | Splinter Review 
Hi,

We are seeing that b2g process is leaking memory. We are seeing growth in USS, VSS and KGSL memory usage linearly over a period of time.

STR:
Random stability testing for 24 hours

I will update DMD logs soon on this.
[Blocking Requested - why for this release]:
blocking-b2g: --- → 2.1?
Please note that we have an issue with DMD in bug 1069542
Flags: needinfo?(khuey)
Whiteboard: [CR 726725]
The 27MB at the very top of the report is bug 1044125.

heap-unclassified is a little high but not ridiculously so.  Once we have DMD working we'll have more insight there.

Everything else looks like gfx stuff (some of that may also be related to bug 1044125.
Flags: needinfo?(khuey)
how about this kgsl memory growth:

107.15 MB (100.0%) -- kgsl-memory
├───94.41 MB (88.10%) -- b2g (pid=201)
│   ├──49.84 MB (46.52%) ── texture [35]
│   ├──19.22 MB (17.94%) ── egl_image [42]
│   ├──15.94 MB (14.87%) ── egl_surface [2]
│   ├───4.04 MB (03.77%) ── gl [22]
│   ├───2.44 MB (02.27%) ── command [39]
│   ├───2.21 MB (02.07%) ── any(0) [41]
│   └───0.71 MB (00.66%) ++ (4 tiny)
└───12.75 MB (11.90%) -- Camera (pid=4387)
    ├───7.93 MB (07.40%) ── texture [6]
    ├───2.02 MB (01.88%) ── gl [11]
    ├───1.99 MB (01.86%) ── renderbuffer
    └───0.80 MB (00.75%) ++ (5 tiny)


I guess that DMD will help here.. Any blind guess by seeing growth
Flags: needinfo?(khuey)
@Sotaro: any suggestion, what can cause such high kgsl memory usage by b2g process ? I guess that DMD will tell only 'heap unclassified' memory usage which is only 12.89MB for b2g process.
Flags: needinfo?(sotaro.ikeda.g)
Whiteboard: [CR 726725] → [caf priority: p1][CR 726725]
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #4)
> how about this kgsl memory growth:
> 
> 107.15 MB (100.0%) -- kgsl-memory
> ├───94.41 MB (88.10%) -- b2g (pid=201)
> │   ├──49.84 MB (46.52%) ── texture [35]
> │   ├──19.22 MB (17.94%) ── egl_image [42]
> │   ├──15.94 MB (14.87%) ── egl_surface [2]
> │   ├───4.04 MB (03.77%) ── gl [22]
> │   ├───2.44 MB (02.27%) ── command [39]
> │   ├───2.21 MB (02.07%) ── any(0) [41]
> │   └───0.71 MB (00.66%) ++ (4 tiny)
> └───12.75 MB (11.90%) -- Camera (pid=4387)
>     ├───7.93 MB (07.40%) ── texture [6]
>     ├───2.02 MB (01.88%) ── gl [11]
>     ├───1.99 MB (01.86%) ── renderbuffer
>     └───0.80 MB (00.75%) ++ (5 tiny)
> 
> 
> I guess that DMD will help here.. Any blind guess by seeing growth

Camera kgsl-memory seems to come from canvas 2d's SkiaGL. It seems better to apply same change of Bug 1064126.
Flags: needinfo?(sotaro.ikeda.g)
Tapas, can you provide a memory report in normal condition? It would be very helpful if we could have that information. For example, I am not sure if "15.94 MB (14.87%) ── egl_surface [2]" is adequate value. On flame it is normally "3.16 MB (12.02%) ── egl_surface [2]".

> 107.15 MB (100.0%) -- kgsl-memory
> ├───94.41 MB (88.10%) -- b2g (pid=201)
> │   ├──49.84 MB (46.52%) ── texture [35]
> │   ├──19.22 MB (17.94%) ── egl_image [42]
> │   ├──15.94 MB (14.87%) ── egl_surface [2]
> │   ├───4.04 MB (03.77%) ── gl [22]
> │   ├───2.44 MB (02.27%) ── command [39]
> │   ├───2.21 MB (02.07%) ── any(0) [41]
> │   └───0.71 MB (00.66%) ++ (4 tiny)
Flags: needinfo?(tkundu)
Tapas, which device are you use for tests? What is the device's display size?
Bug 1045778 might help to reduce gralloc memory usage a bit.
(In reply to Sotaro Ikeda [:sotaro] from comment #8)
> Tapas, which device are you use for tests? What is the device's display size?

We are using msm8926, 1080p device.


I am seeing following memory allocation in a normal device (just after reboot and running same set of applications which memleak affected device was running [See previoud mem-reprot] ).

62.22 MB (100.0%) -- kgsl-memory
├──61.66 MB (99.09%) -- b2g (pid=215)
│  ├──41.05 MB (65.97%) ── egl_image [9]
│  ├──15.94 MB (25.61%) ── egl_surface [2]
│  ├───2.02 MB (03.25%) ── gl [11]
│  ├───1.88 MB (03.01%) ── any(0) [31]
│  ├───0.75 MB (01.21%) ── command [12]
│  └───0.02 MB (00.04%) ++ (2 tiny)
└───0.57 MB (00.91%) ── mm-qcamera-daem (pid=286)/2d [2]

If we compare this with Comment 6 then we can see total kgsl growth is almost 40MB over a period of time.

So, we are seeing a linear growth in kgsl memory usage over a period of time. I never saw b2g process using such huge memory (>100MB) under normal conditions.  I guess that is a leak in somewhere gfx layer which is not clearing graphics objects.

We asked our test team to reproduce again with fix from bug 1064126 as it has some potential fix for it (See comment 3) . 

As discussed in IRC, Could you please give us a patch to kgsl growth (94-61=33MB) for b2g process .
Flags: needinfo?(tkundu) → needinfo?(sotaro)
(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from
> comment #4)
> > how about this kgsl memory growth:
> > 
> > 107.15 MB (100.0%) -- kgsl-memory
> > ├───94.41 MB (88.10%) -- b2g (pid=201)
> > │   ├──49.84 MB (46.52%) ── texture [35]
> > │   ├──19.22 MB (17.94%) ── egl_image [42]
> > │   ├──15.94 MB (14.87%) ── egl_surface [2]
> > │   ├───4.04 MB (03.77%) ── gl [22]
> > │   ├───2.44 MB (02.27%) ── command [39]
> > │   ├───2.21 MB (02.07%) ── any(0) [41]
> > │   └───0.71 MB (00.66%) ++ (4 tiny)
> > └───12.75 MB (11.90%) -- Camera (pid=4387)
> >     ├───7.93 MB (07.40%) ── texture [6]
> >     ├───2.02 MB (01.88%) ── gl [11]
> >     ├───1.99 MB (01.86%) ── renderbuffer
> >     └───0.80 MB (00.75%) ++ (5 tiny)
> > 
> > 
> > I guess that DMD will help here.. Any blind guess by seeing growth
> 
> Camera kgsl-memory seems to come from canvas 2d's SkiaGL. It seems better to
> apply same change of Bug 1064126.

Fix from bug 1064126 is applicable for video app but here kgsl growth is for camera app. Do we need new fix for this ? Please let me know if I need to raise new bug for it or not.
(In reply to Sotaro Ikeda [:sotaro] from comment #9)
> Bug 1045778 might help to reduce gralloc memory usage a bit.

bug 1045778 could be a memory optimization for low memory device . I don't see any patches there . Do you still think that it is related to kgsl memleak ?
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #12)
> (In reply to Sotaro Ikeda [:sotaro] from comment #9)
> > Bug 1045778 might help to reduce gralloc memory usage a bit.
> 
> bug 1045778 could be a memory optimization for low memory device . I don't
> see any patches there . Do you still think that it is related to kgsl
> memleak ?

camera app also have same code to generate video thumbnails.
Sorry, please ignore comment 13. It is for comment 11.
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #12)
> (In reply to Sotaro Ikeda [:sotaro] from comment #9)
> > Bug 1045778 might help to reduce gralloc memory usage a bit.
> 
> bug 1045778 could be a memory optimization for low memory device . I don't
> see any patches there . Do you still think that it is related to kgsl
> memleak ?

It is not related to memory leak. It is about redundant memory allocation.
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #11)
> > 
> > Camera kgsl-memory seems to come from canvas 2d's SkiaGL. It seems better to
> > apply same change of Bug 1064126.
> 
> Fix from bug 1064126 is applicable for video app but here kgsl growth is for
> camera app. Do we need new fix for this ? Please let me know if I need to
> raise new bug for it or not.

camera app also have same code to generate video thumbnails. By the same way, camera app could reduce memory usage. camera app's kgsl seems to come from SkiaGL.
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #10)
> Created attachment 8492370 [details]
> memory-reports from a normal device which does not have memleak
> 
> (In reply to Sotaro Ikeda [:sotaro] from comment #8)
> > Tapas, which device are you use for tests? What is the device's display size?
> 
> We are using msm8926, 1080p device.
> 
> 
> I am seeing following memory allocation in a normal device (just after
> reboot and running same set of applications which memleak affected device
> was running [See previoud mem-reprot] ).
> 
> 62.22 MB (100.0%) -- kgsl-memory
> ├──61.66 MB (99.09%) -- b2g (pid=215)
> │  ├──41.05 MB (65.97%) ── egl_image [9]
> │  ├──15.94 MB (25.61%) ── egl_surface [2]
> │  ├───2.02 MB (03.25%) ── gl [11]
> │  ├───1.88 MB (03.01%) ── any(0) [31]
> │  ├───0.75 MB (01.21%) ── command [12]
> │  └───0.02 MB (00.04%) ++ (2 tiny)
> └───0.57 MB (00.91%) ── mm-qcamera-daem (pid=286)/2d [2]
> 
> If we compare this with Comment 6 then we can see total kgsl growth is
> almost 40MB over a period of time.
> 
> So, we are seeing a linear growth in kgsl memory usage over a period of
> time. I never saw b2g process using such huge memory (>100MB) under normal
> conditions.  I guess that is a leak in somewhere gfx layer which is not
> clearing graphics objects.

I do not think gfx layers cause the this bug. This seems to be caused by more higher level like gaia apps.
blocking-b2g: 2.1? → 2.1+
Adding NI kevin for Comment 17. Please also note that I am re-running same test again with fix from bug 1044125 (See comment 3)
Flags: needinfo?(kgrandon)
Sotaro or Kyle - do we have any tools that would give us visibility into what images are sticking around  inside of egl_image in comment 10?
Flags: needinfo?(kgrandon) → needinfo?(sotaro.ikeda.g)
Not that I know of.
Flags: needinfo?(khuey)
(In reply to Kevin Grandon :kgrandon from comment #19)
> Sotaro or Kyle - do we have any tools that would give us visibility into
> what images are sticking around  inside of egl_image in comment 10?

EGLImages are OpenGL binding of gralloc buffers. So, they are just layer's buffers. About layer's buffer. LayerScope seems be used. But I do not know how to use it.

https://wiki.mozilla.org/Platform/GFX/LayerScope

And about layers, layer dumps already provides a lot of information about layers.
Flags: needinfo?(sotaro.ikeda.g)
(In reply to Sotaro Ikeda [:sotaro] from comment #21)
> (In reply to Kevin Grandon :kgrandon from comment #19)
> > Sotaro or Kyle - do we have any tools that would give us visibility into
> > what images are sticking around  inside of egl_image in comment 10?
> 
> EGLImages are OpenGL binding of gralloc buffers. So, they are just layer's
> buffers. About layer's buffer. LayerScope seems be used. But I do not know
> how to use it.

It is about b2g process's EGLImage that is related to layers. Even in b2g, WebGL and SkiaGL uses EGLImage in SurfaceStream. If threa are EGLImage that is realated to SurfaceStream, kgsl memory typically includes "renderbuffer".
Between two memory report, foreground app seems different.
- attachment 8491717 [details] : homescreen is foreground app
- attachment 8492370 [details] : camera is foreground app
See Also: → 1045778
(In reply to Sotaro Ikeda [:sotaro] from comment #23)
> Between two memory report, foreground app seems different.
> - attachment 8491717 [details] : homescreen is foreground app
> - attachment 8492370 [details] : camera is foreground app

Tapas, can you take a normal memory consumption when homescreen is foreground?
Flags: needinfo?(tkundu)
The patch might help to analyze kgsl memory grow.
(In reply to Sotaro Ikeda [:sotaro] from comment #26)
> Created attachment 8493302 [details] [diff] [review]
> log patch - Add logout
> 
> The patch might help to analyze kgsl memory grow.

Thanks sotaro. I will update you soon here after running with this patch.
I am not seeing memleak issue in our build recently. We may need more time for testing. I will update asap.
(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #28)
> I am not seeing memleak issue in our build recently. We may need more time
> for testing. I will update asap.

Have you seen this bug again?
We are not seeing this issue anymore. But I will reopen it if I see it again.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(tkundu)
Resolution: --- → WORKSFORME
Whiteboard: [caf priority: p1][CR 726725] → [caf priority: p3][CR 726725]
Flags: needinfo?(sotaro)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: