Closed Bug 941789 Opened 6 years ago Closed 6 years ago

Crash in OMXCodecProxy

Categories

(Core :: Audio/Video, defect, major)

ARM
Gonk (Firefox OS)
defect
Not set
major

Tracking

()

RESOLVED FIXED
mozilla28
blocking-b2g koi+
Tracking Status
firefox26 --- wontfix
firefox27 --- wontfix
firefox28 --- fixed
b2g-v1.2 --- fixed

People

(Reporter: bhargavg1, Assigned: sotaro)

References

Details

(Keywords: crash, Whiteboard: [caf priority: p3][CR 580024][PRISM:580024][MemShrink:P2])

Crash Data

Attachments

(7 files, 2 obsolete files)

Upon running camera tests over a period of time see that there is a crash in OMXCodecProxy with the following signature

[@ android::MediaResourceManagerService::cancelClientLocked(android::spandroid::IBinder const&) | android::MediaResourceManagerService::cancelClient(android::spandroid::IMediaResourceManagerClient const&) | android::OMXCodecProxy::~OMXCodecProxy | android::OMXCodecProxy::~OMXCodecProxy ]

+++ This bug was initially created as a clone of Bug #937849 +++

When using camera app in overnight tests runs see that there is a crash in mediaserver.

The root cause seems that the camera app is passing bad data to the mediaserver somehow.

These seems to be part of series of camera app bugs

bug 937758, bug 924039
blocking-b2g: --- → koi?
Crash Signature: [@ android::MediaResourceManagerService::cancelClientLocked(android::spandroid::IBinder const&) | android::MediaResourceManagerService::cancelClient(android::spandroid::IMediaResourceManagerClient const&) | android::OMXCodecProxy::~OMXCodecProxy | android…
Summary: Crash in OMXDecoderProxy → Crash in OMXCodecProxy
Whiteboard: [PRISM:572266] → [PRISM:579547]
Looking through the (huge) logcat, it looks like the camera driver can't get memory for its snapshot buffers (is this pmem or just regular memory?):

11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*, int): main image: rotation = 90, yoff = 0, cbcroff = 0, size = 4718592, width = 2048, height = 1536
11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d106c. stream_buf =0x0, pmem_type =20, num_of_buf=1. buf_len=4718592, cbcr_off=0
11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =-1, camera_memory=0x6c1d10
11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d07fc. stream_buf =0x6b9f40, pmem_type =4, num_of_buf=1. buf_len=4718592, cbcr_off=0
11-20 07:04:26.789   124   124 E MemoryHeapBase: mmap(fd=118, size=4718592) failed (Out of memory)
11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =114, camera_memory=0x6c2df0
11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*, int): thumbnail: rotation = 90, yoff = 0, cbcroff = 0, size = 460800, width = 640, height = 480
11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d0acc. stream_buf =0x6ba5c8, pmem_type =3, num_of_buf=1. buf_len=460800, cbcr_off=0
11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =118, camera_memory=0x6c1c48
11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_stream_qbuf: VIDIOC_QBUF error = -1, stream type=3
11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_stream_util_reg_buf: VIDIOC_QBUF rc = -1
11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*, int):reg snapshot buf err=-1
11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d0acc
11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d07fc
11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d106c
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::handleError(): E
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::deinitSnapshotChannel(mm_camera_channel_type_t): E
11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_ch_util_reg_buf_cb: Trying to register
11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_ch_util_reg_buf_cb: Done register
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::deinitSnapshotChannel(mm_camera_channel_type_t): Release snapshot channel
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::deinitSnapshotChannel(mm_camera_channel_type_t): X
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot state to: 0
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::handleError(): X
11-20 07:04:26.789   124   124 D QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*, int): X
11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::initJPEGSnapshot(int): Failure allocating memory for Snapshot buffers
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::handleError(): E
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot state to: 0
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::handleError(): X
11-20 07:04:26.789   124   124 V QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::initJPEGSnapshot(int): X
11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::start() : Error while Initializing snapshot
11-20 07:04:26.789   124   124 I QCameraHWI_Still: void android::QCameraStream_Snapshot::deInitBuffer(): E
11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::deinitSnapshotBuffers(): E
11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::deinitSnapshotBuffers(): Unpreparing Snapshot Buffer
11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_stream_fsm_notused: Invalid evt=8, stream_state=0
11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::deinitSnapshotBuffers():unreg snapshot buf err=-1
11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t android::QCameraStream_Snapshot::deinitSnapshotBuffers(): X
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot state to: 1
11-20 07:04:26.789   124   124 D QCameraHWI_Still: void android::QCameraStream_Snapshot::deInitBuffer(): X
11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t android::QCameraStream_Snapshot::start(): X
11-20 07:04:26.789   124   124 E QCameraHWI: android::status_t android::QCameraHardwareInterface::takePicture(): error - can't start Snapshot stream!
Bhargav, can you run get_about_memory.py so that we can see where the memory is going?
Flags: needinfo?(bhargavg1)
(In reply to Mike Habicher [:mikeh] from comment #3)
> Bhargav, can you run get_about_memory.py so that we can see where the memory
> is going?

Unfortunately the test setup is gone. Attaching the only logs that we have. 

This is similar to what we see in other cases as well where we run out of memory and b2g process keeps getting fatter, bug 937849 comment 11
Flags: needinfo?(bhargavg1)
Attached file b2g_procrank.txt
Attached file b2g_ps_oom.zip
(In reply to Mike Habicher [:mikeh] from comment #2)
> Looking through the (huge) logcat, it looks like the camera driver can't get
> memory for its snapshot buffers (is this pmem or just regular memory?):
> 
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): main image: rotation = 90, yoff = 0, cbcroff = 0, size = 4718592,
> width = 2048, height = 1536
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d106c. stream_buf
> =0x0, pmem_type =20, num_of_buf=1. buf_len=4718592, cbcr_off=0
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =-1,
> camera_memory=0x6c1d10
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d07fc. stream_buf
> =0x6b9f40, pmem_type =4, num_of_buf=1. buf_len=4718592, cbcr_off=0
> 11-20 07:04:26.789   124   124 E MemoryHeapBase: mmap(fd=118, size=4718592)
> failed (Out of memory)
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =114,
> camera_memory=0x6c2df0
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): thumbnail: rotation = 90, yoff = 0, cbcroff = 0, size = 460800, width
> = 640, height = 480
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d0acc. stream_buf
> =0x6ba5c8, pmem_type =3, num_of_buf=1. buf_len=460800, cbcr_off=0
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =118,
> camera_memory=0x6c1c48
> 11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_stream_qbuf:
> VIDIOC_QBUF error = -1, stream type=3
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_stream_util_reg_buf: VIDIOC_QBUF rc = -1
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int):reg snapshot buf err=-1
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d0acc
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d07fc
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d106c
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): E
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_ch_util_reg_buf_cb: Trying to register
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_ch_util_reg_buf_cb: Done register
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): Release snapshot channel
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 0
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): X
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::initJPEGSnapshot(int): Failure allocating
> memory for Snapshot buffers
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 0
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): X
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::initJPEGSnapshot(int): X
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::start() : Error while Initializing snapshot
> 11-20 07:04:26.789   124   124 I QCameraHWI_Still: void
> android::QCameraStream_Snapshot::deInitBuffer(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): Unpreparing
> Snapshot Buffer
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_stream_fsm_notused: Invalid evt=8, stream_state=0
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers():unreg snapshot buf
> err=-1
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 1
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::deInitBuffer(): X
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::start(): X
> 11-20 07:04:26.789   124   124 E QCameraHWI: android::status_t
> android::QCameraHardwareInterface::takePicture(): error - can't start
> Snapshot stream!

I am hoping that if its a pmem issue we should see something like,
"/dev/pmem_adsp: failed to map pmem fd: Out of memory"
and finallly see something, 
OMXNodeInstance: !!! Observer died. Quickly, do something, ... anything...
Is this a gaia/marionette/mochitest? If so, can we get a copy of your script, or STR?
(In reply to Mike Habicher [:mikeh] from comment #2)
> Looking through the (huge) logcat, it looks like the camera driver can't get
> memory for its snapshot buffers (is this pmem or just regular memory?):
> 
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): main image: rotation = 90, yoff = 0, cbcroff = 0, size = 4718592,
> width = 2048, height = 1536
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d106c. stream_buf
> =0x0, pmem_type =20, num_of_buf=1. buf_len=4718592, cbcr_off=0
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =-1,
> camera_memory=0x6c1d10
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d07fc. stream_buf
> =0x6b9f40, pmem_type =4, num_of_buf=1. buf_len=4718592, cbcr_off=0
> 11-20 07:04:26.789   124   124 E MemoryHeapBase: mmap(fd=118, size=4718592)
> failed (Out of memory)
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =114,
> camera_memory=0x6c2df0
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): thumbnail: rotation = 90, yoff = 0, cbcroff = 0, size = 460800, width
> = 640, height = 480
> 11-20 07:04:26.789   124   124 E QCameraHWI: Init Heap =0x6d0acc. stream_buf
> =0x6ba5c8, pmem_type =3, num_of_buf=1. buf_len=460800, cbcr_off=0
> 11-20 07:04:26.789   124   124 E QCameraHWI: heap->fd[0] =118,
> camera_memory=0x6c1c48
> 11-20 07:04:26.789   124   124 E mm-libcamera2: mm_camera_stream_qbuf:
> VIDIOC_QBUF error = -1, stream type=3
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_stream_util_reg_buf: VIDIOC_QBUF rc = -1
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int):reg snapshot buf err=-1
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d0acc
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d07fc
> 11-20 07:04:26.789   124   124 E QCameraHWI: Release 0x6d106c
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): E
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_ch_util_reg_buf_cb: Trying to register
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_ch_util_reg_buf_cb: Done register
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): Release snapshot channel
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::
> deinitSnapshotChannel(mm_camera_channel_type_t): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 0
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::initSnapshotBuffers(cam_ctrl_dimension_t*,
> int): X
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::initJPEGSnapshot(int): Failure allocating
> memory for Snapshot buffers
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 0
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::handleError(): X
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::initJPEGSnapshot(int): X
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::start() : Error while Initializing snapshot
> 11-20 07:04:26.789   124   124 I QCameraHWI_Still: void
> android::QCameraStream_Snapshot::deInitBuffer(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): E
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): Unpreparing
> Snapshot Buffer
> 11-20 07:04:26.789   124   124 E mm-libcamera2:
> mm_camera_stream_fsm_notused: Invalid evt=8, stream_state=0
> 11-20 07:04:26.789   124   124 E QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers():unreg snapshot buf
> err=-1
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: android::status_t
> android::QCameraStream_Snapshot::deinitSnapshotBuffers(): X
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::setSnapshotState(int): Setting snapshot
> state to: 1
> 11-20 07:04:26.789   124   124 D QCameraHWI_Still: void
> android::QCameraStream_Snapshot::deInitBuffer(): X
> 11-20 07:04:26.789   124   124 V QCameraHWI_Still: virtual android::status_t
> android::QCameraStream_Snapshot::start(): X
> 11-20 07:04:26.789   124   124 E QCameraHWI: android::status_t
> android::QCameraHardwareInterface::takePicture(): error - can't start
> Snapshot stream!

I am hoping that if its a pmem issue we should see something like,
"/dev/pmem_adsp: failed to map pmem fd: Out of memory"
and finallly see something, 
OMXNodeInstance: !!! Observer died. Quickly, do something, ... anything...
(In reply to bhargavg1 from comment #4)
> (In reply to Mike Habicher [:mikeh] from comment #3)
> > Bhargav, can you run get_about_memory.py so that we can see where the memory
> > is going?
> 
> Unfortunately the test setup is gone. Attaching the only logs that we have. 
> 
> This is similar to what we see in other cases as well where we run out of
> memory and b2g process keeps getting fatter, bug 937849 comment 11

Will check if I can salvage something from earlier runs
(In reply to Mike Habicher [:mikeh] from comment #8)
> Is this a gaia/marionette/mochitest? If so, can we get a copy of your
> script, or STR?

These are marionette based tests. We should endup in this situation pretty much after using the camera/camcorder for 4-5 hrs
If you can provide us with your test script(s), we can try to reproduce here.
(In reply to bhargavg1 from comment #9)
> (In reply to Mike Habicher [:mikeh] from comment #2)
> > Looking through the (huge) logcat, it looks like the camera driver can't get
> > memory for its snapshot buffers (is this pmem or just regular memory?):
> > 
> Will check if I can salvage something from earlier runs

Attached logs from earlier runs 

See that for the B2G process (pid:127) we have duplication of strings
>> 7.90 MB (06.21%) -- string(length=9646, copies=404, "data:image/png;base64,iV....
Not sure if this is a leak or something, I havent had a chance to go through this yet, will leave it to the experts
Attached file memory_reports.zip
For memory-report-<X>-<PID>.gz, does larger X mean later report? Or just a different instance?
X is just the device timestamp of the report.
Whiteboard: [PRISM:579547] → [PRISM:580024]
Recent builds of Firefox can't open the memory-report in comment 13. Firefox 26.0b6 works: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/26.0b6/linux-x86_64/en-US/firefox-26.0b6.tar.bz2

It looks like we're leaking a LOT of strings:

180.15 MB (100.0%) -- explicit
├──124.83 MB (69.29%) -- js-non-window
│  ├──115.06 MB (63.87%) -- zones
│  │  ├──113.17 MB (62.82%) -- zone(0x40458800)
│  │  │  ├───66.31 MB (36.81%) -- compartment([System Principal])
│  │  │  │   ├──64.55 MB (35.83%) -- objects
│  │  │  │   │  ├──55.01 MB (30.54%) -- malloc-heap
│  │  │  │   │  │  ├──30.78 MB (17.09%) ── elements/non-asm.js
│  │  │  │   │  │  ├──24.20 MB (13.44%) ── slots
│  │  │  │   │  │  └───0.03 MB (00.02%) ── ctypes-data
│  │  │  │   │  ├───9.54 MB (05.29%) -- gc-heap
│  │  │  │   │  │   ├──7.92 MB (04.39%) ── ordinary
│  │  │  │   │  │   └──1.62 MB (00.90%) ++ (3 tiny)
│  │  │  │   │  └───0.00 MB (00.00%) ── non-heap/code/asm.js
│  │  │  │   └───1.76 MB (00.98%) ++ (4 tiny)
│  │  │  ├───41.97 MB (23.30%) -- strings
│  │  │  │   ├──39.54 MB (21.95%) -- notable
│  │  │  │   │  ├──31.66 MB (17.57%) ++ (577 tiny)
│  │  │  │   │  └───7.88 MB (04.37%) -- string(length=9646, copies=403, "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiY

This particular data:image URL is the same as the one mentioned here: https://bugzilla.mozilla.org/show_bug.cgi?id=897684#c6

Also, the memory report shows hundreds to tens-of-thousands of copies of smaller strings:

│  │  │  │   │  ├──31.66 MB (17.57%) -- (577 tiny)
│  │  │  │   │  │  ├───1.37 MB (00.76%) ++ string(length=35, copies=14911, "https://github.com/mozilla-b2g/gaia")
│  │  │  │   │  │  ├───1.33 MB (00.74%) ++ string(length=11, copies=29016, "/index.html")
│  │  │  │   │  │  ├───1.16 MB (00.64%) ++ string(length=9, copies=25389, "installed")
│  │  │  │   │  │  ├───0.85 MB (00.47%) ++ string(length=9, copies=18538, "certified")
│  │  │  │   │  │  ├───0.85 MB (00.47%) ++ string(length=9, copies=18538, "readwrite")
│  │  │  │   │  │  ├───0.61 MB (00.34%) ++ string(length=13, copies=13299, "The Gaia Team")
│  │  │  │   │  │  ├───0.58 MB (00.32%) ++ string(length=5, copies=18942, "en-US")
│  │  │  │   │  │  ├───0.50 MB (00.28%) ++ string(length=8, copies=10881, "readonly")
│  │  │  │   │  │  ├───0.34 MB (00.19%) ++ string(length=16, copies=5642, "portrait-primary")
│  │  │  │   │  │  ├───0.30 MB (00.16%) ++ string(length=16, copies=4836, "Web app template")
│  │  │  │   │  │  ├───0.30 MB (00.16%) ++ string(length=17, copies=4836, "Busts your memory")
│  │  │  │   │  │  ├───0.30 MB (00.16%) ++ string(length=6, copies=9672, "inline")
│  │  │  │   │  │  ├───0.26 MB (00.14%) ++ string(length=32, copies=2821, "/dialer/index.html#keyboard-view")
│  │  │  │   │  │  ├───0.22 MB (00.12%) ++ string(length=6, copies=7254, "window")
│  │  │  │   │  │  ├───0.22 MB (00.12%) ++ string(length=9, copies=4837, "Membuster")
│  │  │  │   │  │  ├───0.22 MB (00.12%) ++ string(length=8, copies=4836, "Template")
Component: Gaia::Camera → Gaia::System
Whiteboard: [PRISM:580024] → [PRISM:580024][MemShrink]
Component: Gaia::System → General
blocking-b2g: koi? → koi+
Please reproduce this result passing --minimize to get_about_memory.py so that we can confirm the results in comment 16 are valid; or make a copy of your test script available/provide STR so that we can try to reproduce this here.
Flags: needinfo?(bhargavg1)
Keywords: crash
Expected the logs to be available today from test-runs, waiting for those still will try to get logs tomorrow
Flags: needinfo?(bhargavg1)
Attached file compact_memory_reports (obsolete) —
Updated with compact logs
Attachment #8338350 - Attachment is obsolete: true
Were those attachments meant to be flagged as obsolete?

I just looked through the logs (*24576* is the memory report for the b2g parent process) and things look a lot more sane:

Main Process (pid 24576)
Explicit Allocations

37.28 MB (100.0%) -- explicit
├──13.37 MB (35.86%) ++ js-non-window
├───5.54 MB (14.87%) ++ workers/workers()
├───4.99 MB (13.40%) ── heap-unclassified
├───4.30 MB (11.52%) ++ window-objects
├───3.07 MB (08.23%) ++ heap-overhead
├───2.59 MB (06.95%) ++ images
├───1.92 MB (05.14%) ++ (13 tiny)
├───0.88 MB (02.37%) ── xpti-working-set
└───0.62 MB (01.66%) ++ storage/sqlite

What has changed between the reports in attachment 8338350 [details] and attachment 8336339 [details]? Just the addition of the --minimize argument to get_about_memory.py?
Flags: needinfo?(bhargavg1)
these are the latest reports. The only difference is that now we are saying,
"minimize memory report"
Flags: needinfo?(bhargavg1)
mlee, can anyone on your team make (more) sense of the above reports? In comment 16 it looks like we're leaking huge amounts of small-string memory, but apparently minimizing the memory report (which, as I understand it, runs GC and CCs first) is showing more sane reports in comment 20 and comment 21.
Flags: needinfo?(mlee)
Kyle's team will be more helpful here. Kyle can you have someone on the MemShrink team look into these reports?

(In reply to Mike Habicher [:mikeh] from comment #22)
> mlee, can anyone on your team make (more) sense of the above reports? In
> comment 16 it looks like we're leaking huge amounts of small-string memory,
> but apparently minimizing the memory report (which, as I understand it, runs
> GC and CCs first) is showing more sane reports in comment 20 and comment 21.
Flags: needinfo?(mlee) → needinfo?(khuey)
Whiteboard: [PRISM:580024][MemShrink] → [PRISM:580024][MemShrink:P2]
(In reply to Mike Habicher [:mikeh] from comment #16)
> Recent builds of Firefox can't open the memory-report in comment 13.

Sorry about that.  Firefox builds fromn a window of about 2 weeks were producing dumps
that no longer be read.  Fixing Firefox to handle them is a pain and not worth
it for a handful of extant dumps, but you can work around it by deleting all
the records that contain a path starting with "redundant".  For example, delete
this whole line:

    {"process": "Main Process (pid 127)", "path": "redundant/js-main-runtime-com
partments/system", "kind": 2, "units": 1, "amount": 33, "description": ""},

There will only be handful of such records in each dump file, and as their path
suggest, they are redundant w.r.t. other measurements anyway.


(In reply to Mike Lee [:mlee] from comment #23)
> Kyle's team will be more helpful here. Kyle can you have someone on the
> MemShrink team look into these reports?

FWIW, Kyle doesn't have a team, and the MemShrink "team" isn't a team in any official sense, but simply a small collection of people from various "real" teams who happen to be interested in memory consumption and so pay attention to MemShrink-tagged bugs :)
Maybe it's qcom camera issue, we can't reproduce it on fugu monkey test. We almost run over one week.
And I can't find the keyword 'MediaResourceManagerService::cancelClientLocked' in our android buzilla.
Component: General → Video/Audio
Product: Firefox OS → Core
From attachment 8336245 [details], the following call stack says OMXCodec was instantiated in b2g process. 

------------------------------------------
0  libxul.so!android::MediaResourceManagerService::cancelClientLocked() 
1  libxul.so!android::MediaResourceManagerService::cancelClient() 
2  libxul.so!android::OMXCodecProxy::~OMXCodecProxy
3  libxul.so!android::OMXCodecProxy::~OMXCodecProxy
5  libutils.so!android::RefBase::decStrong()
6  libxul.so!android::OmxDecoder::ReleaseMediaResources()
Can we get the script to reproduce it?
In attachment 8336245 [details], I saw about 25 of following functions. Does the test instantiate more than 25 number of audio/video files?
- MediaDecoderStateMachine::DecodeThreadRun()
- android::OMX::CallbackDispatcher::loop()
Does the crash happens more than once?
Flags: needinfo?(bhargavg1)
Assignee: nobody → sotaro.ikeda.g
Access to member is not protected in MediaResourceManagerService::binderDied().
Attachment #8342122 - Flags: review?(chris.double)
bhargavg1, is it possible to re-Test by attachment 8342122 [details] [diff] [review]?
Attachment #8342122 - Flags: review?(chris.double) → review+
(In reply to Sotaro Ikeda [:sotaro] from comment #31)
> bhargavg1, is it possible to re-Test by attachment 8342122 [details] [diff] [review]
> [review]?

Sure, might take 1/2 days
Flags: needinfo?(bhargavg1)
Committable patch. Carry "doublec review+".
Attachment #8342122 - Attachment is obsolete: true
Attachment #8342375 - Flags: review+
check-in of attachment 8342375 [details] [diff] [review] is necessary, anyways. If the problem is still reproducible. Please reopen the bug.
Keywords: checkin-needed
(In reply to Sotaro Ikeda [:sotaro] from comment #35)
> check-in of attachment 8342375 [details] [diff] [review] is necessary,
> anyways. If the problem is still reproducible. Please reopen the bug.

Does this solve the memory low issues that we are seeing in comment 2. I guess we have another problem as well that we run out of memory and the memreports indicate lots of duplicate strings.
I feel memory problem is not related to the fix.
(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> I feel memory problem is not related to the fix.

So should we clone another bug, we have marked as memshrink not sure if there is any analysis from that point yet. We might trip on the same issue in v1.3 as well
https://hg.mozilla.org/mozilla-central/rev/41fce02031ab
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
Well passing --minimize is going to throw away a lot of stuff.  The original report looks like we're leaking manifest objects.  Is this reproducible?
Flags: needinfo?(khuey) → needinfo?(mhabicher)
To make sure that we dont loose track of the bug, will clone a new one
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #42)
>
> Well passing --minimize is going to throw away a lot of stuff.  The original
> report looks like we're leaking manifest objects.  Is this reproducible?

Bhargav, is the leak still reproducible?
Flags: needinfo?(bhargavg1)
(In reply to Mike Habicher [:mikeh] from comment #44)
> (In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #42)
> >
> > Well passing --minimize is going to throw away a lot of stuff.  The original
> > report looks like we're leaking manifest objects.  Is this reproducible?
> 
> Bhargav, is the leak still reproducible?

Don't think we had any specific fixes for the memleak issue, so this should be still seen
Flags: needinfo?(bhargavg1)
(In reply to bhargavg1 from comment #45)
> 
> Don't think we had any specific fixes for the memleak issue, so this should
> be still seen

If you're still seeing it, please open a new bug and please attach steps to reproduce. It doesn't look like any STR were ever attached to this bug.
Flags: needinfo?(mhabicher)
(In reply to Sotaro Ikeda [:sotaro] from comment #31)
> bhargavg1, is it possible to re-Test by attachment 8342122 [details] [diff] [review]
> [review]?

Seems to be crashing again. Attached are the logs
Flags: needinfo?(sotaro.ikeda.g)
From the following log, attachment 8345014 [details] seems different crash. The crash happens within camra hal in mediaserver process. It seems better to handle as a different bug.

-----------------------
01-06 02:03:03.809 E/QCameraHWI(  133):  setPreviewWindow : X, mPreviewState = 0
01-06 02:03:03.819 E/libgenlock(  133): perform_lock_unlock_operation: handle is invalid
01-06 02:03:03.819 E/QCameraHWI_Preview(  133): processPreviewFrameWithDisplay: genlock_unlock_buffer failed
01-06 02:03:03.819 F/libc    (  133): Fatal signal 11 (SIGSEGV) at 0x00000004 (code=1)
01-06 02:03:03.819 E/QualcommCamera(  133): Qint android::close_camera_device(hw_device_t*): device =0x94a808 E
Flags: needinfo?(sotaro.ikeda.g)
Blocks: 950577
(In reply to Mike Habicher [:mikeh] from comment #46)
> (In reply to bhargavg1 from comment #45)
> > 
> > Don't think we had any specific fixes for the memleak issue, so this should
> > be still seen
> 
> If you're still seeing it, please open a new bug and please attach steps to
> reproduce. It doesn't look like any STR were ever attached to this bug.

https://bugzilla.mozilla.org/show_bug.cgi?id=950577
Whiteboard: [PRISM:580024][MemShrink:P2] → [CR 580024][PRISM:580024][MemShrink:P2]
Whiteboard: [CR 580024][PRISM:580024][MemShrink:P2] → [caf priority: p3][CR 580024][PRISM:580024][MemShrink:P2]
You need to log in before you can comment on or make changes to this bug.