1042356 - [Open 2 only] Camera crashes during video recording

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Vendcom, defect)

Description

[Marcia Knous [:marcia]]

Attachment: open2.txt

Open 2, while running:

Gaia   b9d19011123487009c80d1200937652d58c434a0
SourceStamp 00f4b3a7046f
BuildID 20140722000200
Version 32.0

STR:
1. Launch the camera and with to video mode
2. Begin recording, and move camera around to change focus to a different area
3. Camera crashes (sometimes the app crashes and sometimes the device reboots)

Logcat attached

Comment 1

[Marcia Knous [:marcia]]

Display specs for the device:

Display 	Type 	TFT capacitive touchscreen
Size 	320 x 480 pixels, 3.5 inches (~165 ppi pixel density)

Sorry I pasted this comment in the wrong bug.

Comment 2

[Marcia Knous [:marcia]]

[Blocking Requested - why for this release]: Video must work on this device for the 2.0 release.

Comment 3

[Hema Koka [:hema]]

Blocking because this device is targeted for 2.0

Andrew, we only have one open2 device (it is with marcia) so it will be hard to debug this. But, can you take a look at the logs and see what could be going on?


Thanks a lot!
Hema

Comment 4

[Mike Habicher [:mikeh] (high bugzilla latency)]

I see a SIGSEGV in the Gonk camera library:

D/QCamera2HWI(  297): static void qcamera::QCamera2HardwareInterface::metadata_stream_cb_routine(mm_camera_super_buf_t*, qcamera::QCameraStream*, void*): hdr_scene_data: 1 0 0.000000
E/QCamera2HWI(  297): int32_t qcamera::QCamera2HardwareInterface::processHDRData(cam_asd_hdr_scene_data_t) : hdr_scene_data: processHDRData: 0 0.000000
E/OMXNodeInstance(  297): !!! Observer died. Quickly, do something, ... anything...
D/ALSADevice(  297): setHardwareParams: buffer_size 32768, period_size 4096, period_cnt 8
W/AudioFlinger(  297): write blocked for 2378 msecs, 1 delayed writes, thread 0xb5de7008
D/ALSADevice(  297): standby: handle 0xb743cd70 h 0x0
D/ALSADevice(  297): standby handle h 0xb747c3a0
F/libc    (  297): Fatal signal 11 (SIGSEGV) at 0x00000008 (code=1), thread 14007 (VideoEncMsgThre)

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

I am putting together a debug image we should be able to flash to the device, in order to capture a core file from the crashing process. That will (hopefully) give as a better idea of what exactly went wrong. Might take a while to sync and build however, so I'll provide it tomorrow.

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

From http://wwwen.zte.com.cn/en/press_center/news/201402/t20140224_418353.html, I can see the Open 2 only has 256 MB of RAM. It may be suffering from the same issue in bug 1027592. Mike suggested it is likely a null pointer dereference, and perhaps that is from a memory allocation failure - it would make sense if it is a random process which crashes.

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

So it does not appear we are able to produce full images for Open 2 at this time. I built the gecko libraries with some extra debug logs in camera to see if we can isolate the problem a bit closer.

https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz

To load the libraries, execute the following:
adb remount && adb push ./b2g /system/b2g && adb reboot

Please begin two logs collections before starting:
adb logcat -v threadtime >debug.log
adb shell top -d 1 -s rss -m 20 >top.log

However given that the STR indicates different crash types (app crashes, device reboot, log shows a third case of mediaserver crashing), I don't believe that this is a camera specific issue, but rather something more fundamental. Either that or we have a bunch of root causes :).

Comment 8

[Marcia Knous [:marcia]]

(In reply to Andrew Osmond [:aosmond] from comment #7)
> So it does not appear we are able to produce full images for Open 2 at this
> time. I built the gecko libraries with some extra debug logs in camera to
> see if we can isolate the problem a bit closer.
> 
> https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz
> 
> To load the libraries, execute the following:
> adb remount && adb push ./b2g /system/b2g && adb reboot
> 
> Please begin two logs collections before starting:
> adb logcat -v threadtime >debug.log
> adb shell top -d 1 -s rss -m 20 >top.log
> 
> However given that the STR indicates different crash types (app crashes,
> device reboot, log shows a third case of mediaserver crashing), I don't
> believe that this is a camera specific issue, but rather something more
> fundamental. Either that or we have a bunch of root causes :).

Andrew: I tried pushing that file to the Open 2, but unfortunately it causes the device to get stuck at the Firefox OS loading screen (tried rebooting, battery pull, etc but no luck).

This device has the latest ZTE Open 2 base image + shallow flash of Gecko and Gaia on it.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Updated build to actually be based on 2.0 instead of b2g-inbound. Same link as before:

https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz

Comment 10

[Joshua Mitchell (Inactive)]

Qa-wanted tag added without any actual qa-wanted request: removing

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

Please try loading this in combination with a base build from the 25th:
https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz

It won't solve the problem but there should be more hopefully useful info in the logcat.

Comment 12

[Marcia Knous [:marcia]]

Attachment: debug.txt

Debug log. I actually used a 2.1 build to test this, not actually a 2.0 build.

Will attach top log next.

Comment 13

[Marcia Knous [:marcia]]

Attachment: top.txt

Steps I followed when testing this:

1. Launch camera app.
2. Immediately press the video button and start recording.
3. Wait for app to crash.

Comment 14

[Joshua Mitchell (Inactive)]

(In reply to Andrew Osmond [:aosmond] (Vacation Aug 1-10) from comment #11)
> Please try loading this in combination with a base build from the 25th:
> https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz
> 
> It won't solve the problem but there should be more hopefully useful info in
> the logcat.

sorry Andrew - Qa-wanted does not have any Open-2 devices. Marcia seems to have supplied the needed logcats though.

Comment 15

[Mike Habicher [:mikeh] (high bugzilla latency)]

Marcia, does your device have any files in /data/core ? (i.e. what does |adb shell ls -l /data/core| output?)

Comment 16

[Marcia Knous [:marcia]]

(In reply to Mike Habicher [:mikeh] from comment #15)
> Marcia, does your device have any files in /data/core ? (i.e. what does |adb
> shell ls -l /data/core| output?)

In adb shell I get:

data/core: No such file or directory

Comment 17

[Marcia Knous [:marcia]]

(In reply to Andrew Osmond [:aosmond] (Vacation Aug 1-10) from comment #11)
> Please try loading this in combination with a base build from the 25th:
> https://dl.dropboxusercontent.com/u/85398890/bug1042356.b2g.tar.gz
> 
> It won't solve the problem but there should be more hopefully useful info in
> the logcat.

nhirata helped me get the logging build up and running, but I will say that even if I follow the instructions with a base build from the 25th and push the dropbox files, the device won't start. So the latest logging I created as I noted earlier is actually based off a 2.1 build, not a 2.0 build.

Comment 18

[Mike Habicher [:mikeh] (high bugzilla latency)]

Got the Open 2 from No-Jun and can reproduce this problem; STR:
1. open Camera app
2. switch to video mode
3. tap on the sart-recording button

Expected: record video

Observed: viewfinder is extremly laggy and evntually the Camera app is killed.

Kernel log confirms that the Camera (and the Homescreen too) app is LMKed:

# adb shell dmesg | grep sigkill
<6>[07-31 08:58:47.334] send sigkill to 970 (Homescreen), adj 534, size 233
<6>[07-31 08:58:47.739] send sigkill to 1083 (Camera), adj 134, size 503

Comment 19

[Mike Habicher [:mikeh] (high bugzilla latency)]

And sometimes the entire phone just reboots. :(

Comment 20

[Mike Habicher [:mikeh] (high bugzilla latency)]

Further to comment 19, actually, just _everything_ gets killed, including the b2g parent:

adb shell dmesg | grep sigkill
<6>[07-31 09:06:53.986] send sigkill to 985 (Homescreen), adj 534, size 1452
<6>[07-31 09:06:58.164] send sigkill to 1093 (Camera), adj 134, size 393
<6>[07-31 09:06:58.523] send sigkill to 1232 ((Preallocated a), adj 67, size 219
<6>[07-31 09:06:58.983] send sigkill to 299 (b2g), adj 0, size 4098

Comment 21

[Mike Habicher [:mikeh] (high bugzilla latency)]

Attachment: watch -n 1 "adb shell b2g-info 2>&1 | tee -a watch_open2.txt"

The b2g-info item "Non-B2G procs" shows that some non-B2G process begins consuming significant amounts of memory when recording starts. The reported values are:

- before starting camera:  ~47 MB
- after starting camera:   ~70 MB
- switching to video mode: ~90 MB
- starting recording:      ~100 to 130 MB!
- post-crash:              ~53 MB

Comment 22

[Mike Habicher [:mikeh] (high bugzilla latency)]

Attachment: watch -n 1 "adb shell ps | grep \"mm-qcamera-daemon\|mediaserver\" | tee -a watch_open2_ps.txt"

Fields in attachment are:
USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME

VSIZE numbers for each process:
- mediaserver:
  - before starting camera:  ~28 MB
  - after starting camera:   ~58 MB
  - switching to video mode: ~84 MB
  - starting recording:      ~98 to 111MB
  - after crash:             ~30 MB
- mm-qcamera-daemon:
  - before starting camera:  ~14 MB
  - after starting camera:   ~36 MB
  - switching to video mode: ~57 MB
  - starting recording:      ~57 to 69 MB
  - after crash:             ~14 MB

Based on the RSS numbers, it looks like the mediaserver is thrashing, while the camera daemon approximately doubles in size when recording.

Comment 23

[Mike Habicher [:mikeh] (high bugzilla latency)]

We don't have source to these components, so this is POVB.

Comment 24

[Hema Koka [:hema]]

POVB. NI Vance (TAM) to move this forward.

Comment 25

[Jason Smith [:jsmith]]

Dropping the blocking flag since this is a vendor bug.

Comment 26

[BMO Automation]

Firefox OS is not being worked on