950479 - getUserMedia camera auto-focus needs center weighting

Status: NEW

(Core :: WebRTC: Audio/Video, defect)

Description

[A Capesius]

User Agent: Mozilla/5.0 (Windows NT 5.2; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 (Beta/Release)
Build ID: 20131209204824

Steps to reproduce:

I'm testing using a v28 local test build provided in bug 932295 with auto-focus capability on Galaxy S4 and attempting to do close-up video capture.



Actual results:

The Galaxy S4 rear camera is auto-focus, and is properly auto-focusing for general video capture when using the test build.

When doing close-ups, the S4 appears to use full screen focus area rather than center-weighted focus as a camera default. The camera loses focus on closer objects in the center of the field and focuses on the background instead. With light changes or image shift, the camera initiates re-focus to the foreground object, but then quickly loses it as the background is a larger focus area. 

Attempting to override the weighting by filling the field of view with the closer object brings the object too close to get focus at all (camera minimum distance limitation).



Expected results:

Camera should auto-focus using a center weighted area. 

A selectable focus weighting constraint is one solution. But IMO applying a small fixed center-weighted focus area seems to make the most sense as a default for embedding video. This allows for close up and distant work and is more intuitive for the user.

For reference, Nikon SLR cameras use center weighting based on 6-13% of the screen. So a ~10% focus area seems appropriate.

Comment 1

[Kevin Brosnan [Ex-Mozilla]]

what is the expected behavior of autofocus?

Comment 2

[(not currently active) Ted Mielczarek]

I don't think we defined it in that level of detail, I just said "it would be nice if the camera autofocused" and so that's what was implemented. I agree that center weighting makes a lot of sense. It looks like this ought to be fairly straightforward:
http://developer.android.com/guide/topics/media/camera.html#metering-focus-areas

Comment 3

[Gian-Carlo Pascutto [:gcp]]

We currently just set autofocus and assume the defaults are good. We're still a bit away from implementing the WebRTC constraints that allow a webapp to specify autofocus etc., so trying to set a better default (when possible, i.e. when the device supports it) sounds good to me.

Comment 4

[A Capesius]

This page has some specs on newer devices that put them all around the same general f-stop and focal length, so a single setting should be pretty effective across the board.

http://connect.dpreview.com/post/9219904986/shootout-samsunggalaxys4-vs-htcone-vs-iphone5-vs-lumia920

For setFocusAreas()  10% still seems good.
http://developer.android.com/reference/android/hardware/Camera.Parameters.html#getFocusAreas%28%29

For setMeteringAreas() full screen would probably be better otherwise the image may wash out or go dark in varying light conditions.

Comment 5

[swaroop.rao]

If nobody's working on this, can I take it up?

Comment 6

[Gian-Carlo Pascutto [:gcp]]

Sure, go right ahead! See if you can make a Firefox for Android build, get a WebRTC call going, then look at bug 932295 for some idea where to find the relevant code.

Comment 7

[swaroop.rao]

I already have a Firefox for Android build. I had submitted a fix for another defect a while back (https://bugzilla.mozilla.org/show_bug.cgi?id=936727). I just updated using hg but I'm not sure if I've done it right. What's the command to use to update my local repository to the latest label? And, can you help me with understanding what this WebRTC call is about. In the meantime, I will review the other bug that you referred me to and try to figure out how it was fixed.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

>What's the command to use to update my local repository to the latest label? 

hg pull -u
https://developer.mozilla.org/en/docs/Mercurial_FAQ

>And, can you help me with understanding what this WebRTC call is about.

WebRTC is a JavaScript API that allows Firefox (and Chrome) instances to make video calls with each other. Some sites using that API that you can use:

http://apprtc.appspot.com/
http://www.talky.io/

We already implement WebRTC, but our handling of the camera's focus could be improved. I think for testing this we don't need to make actual calls, but can use a test page that just captures the camera video image:

http://mozilla.github.io/webrtc-landing/gum_test.html

You'd need to go into the Java code that initializes the Android Camera and see what needs to be modified to use a better auto-focus algorithm.

Comment 9

[swaroop.rao]

I'm a bit confused. If it has already been fixed in bug #932295 (https://bugzilla.mozilla.org/show_bug.cgi?id=932295), how is this bug different? I found a file called GeckoAppShell.java in which there is a method called initCamera(). I may be way off base here, but is this the place where I need to think about making changes?

Comment 10

[Gian-Carlo Pascutto [:gcp]]

>how is this bug different?

Bug 932295 enables auto-focus. This bug observed that the default auto-focus algorithm isn't very good because it's trying to focus on things at the edge of the screen. See the original comment in this bug.

>I found a file called GeckoAppShell.java in which there is a method called initCamera(). I may be way off base here, but is this the place where I need to think about making changes?

No, I'm not even sure that code is used anywhere. Look at bug 932295 to see where auto-focus was enabled in the source. That's in the WebRTC/Android video capture driver. This change almost certainly needs to go in the same place.

Comment 11

[swaroop.rao]

OK, I finished updating my local repository and building the APK file. When I installed it on my Nexus 5 (4.4.2) and ran it, it started up and a second later, it just terminated. No errors, just quietly closed. I had Android SDK v4.3 installed at that time. I'm now downloading v4.4. Does that matter, though? How do I debug why the browser is closing?

Comment 12

[swaroop.rao]

So, I tried running 'adb logcat' and running Fennec again. It closed again after a second and I captured the following log message(s). Hope I captured the relevant part. If so, can somebody please help me figure out why it's crashing?

********************
E/Profiler( 8972): BEGIN mozilla_sampler_init
E/Profiler( 8972): BPUnw: [1 total] thread_register_for_profiling(me=0x77452798, stacktop=0x77f73972)
E/Profiler( 8972): SPS:
E/Profiler( 8972): SPS: Unwind mode       = pseudo
E/Profiler( 8972): SPS: Sampling interval = 0 ms (zero means "platform default")
E/Profiler( 8972): SPS: Entry store size  = 0 (zero means "platform default")
E/Profiler( 8972): SPS: UnwindStackScan   = 0 (max dubious frames per unwind).
E/Profiler( 8972): SPS: Use env var MOZ_PROFILER_MODE=help for further information.
E/Profiler( 8972): SPS:
E/Profiler( 8972): Registering start signal
I/GeckoHealth( 8972): fennec_swarooprao :: HealthReportBroadcastService :: Registering HealthReportUploadService.
I/GeckoHealth( 8972): fennec_swarooprao :: BackgroundService :: Setting inexact repeating alarm for interval 3600000
E/Profiler( 8972): BPUnw: [2 total] thread_register_for_profiling(me=0x7747f170, stacktop=0x801b2cbf)
I/Gecko:MemoryInfoDumper( 8972): Fifo watcher disabled via pref.
E/GeckoConsole( 8972): Could not read chrome manifest 'file:///data/data/org.mozilla.fennec_swarooprao/chrome.manifest'.
D/FaviconCache( 8972): Favicon cache fullness: 36864/524288
D/FaviconCache( 8972): Favicon cache fullness: 73728/524288
D/dalvikvm( 8972): GC_CONCURRENT freed 458K, 3% free 19315K/19808K, paused 1ms+2ms, total 13ms
E/GeckoConsole( 8972): While creating services from category 'profile-after-change', could not create service for entry 'Disk Space Watcher Service', contract ID '@mozilla.org/toolkit/disk-space-watcher;1'
W/ResourceType( 8972): Too many attribute references, stopped at: 0x01010099
E/Profiler( 8972): BPUnw: [3 total] thread_register_for_profiling(me=0x821bc0d0, stacktop=0x856fdcbf)
E/Profiler( 8972): BPUnw: [4 total] thread_register_for_profiling(me=0x821bc318, stacktop=0x857fbcbf)
E/GeckoConsole( 8972): zerdatime 1389712491026 - browser chrome startup finished.
F/libc    ( 8972): Fatal signal 11 (SIGSEGV) at 0x00000059 (code=1), thread 9012 (Gecko)
I/WindowState(  750): WIN DEATH: Window{430e26f0 u0 org.mozilla.fennec_swarooprao/org.mozilla.fennec_swarooprao.App}
I/ActivityManager(  750): Process org.mozilla.fennec_swarooprao (pid 8972) has died.
W/ActivityManager(  750): Force removing ActivityRecord{42602e88 u0 org.mozilla.fennec_swarooprao/.App t45}: app died, no saved state
W/WindowManager(  750): Force-removing child win Window{432458c8 u0 SurfaceView} from container Window{430e26f0 u0 org.mozilla.fennec_swarooprao/org.mozilla.fennec_swarooprao.App}
W/WindowManager(  750): Failed looking up window
W/WindowManager(  750): java.lang.IllegalArgumentException: Requested window android.os.BinderProxy@426b3d00 does not exist
W/WindowManager(  750): 	at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:7923)
W/WindowManager(  750): 	at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:7914)
W/WindowManager(  750): 	at com.android.server.wm.WindowState$DeathRecipient.binderDied(WindowState.java:1047)
W/WindowManager(  750): 	at android.os.BinderProxy.sendDeathNotice(Binder.java:493)
W/WindowManager(  750): 	at dalvik.system.NativeStart.run(Native Method)
I/WindowState(  750): WIN DEATH: null
I/DEBUG   ( 4579): unexpected waitpid response: n=9012, status=00000b00
I/DEBUG   ( 4579): ptrace detach from 9012 failed: No such process
I/DEBUG   ( 4579): debuggerd committing suicide to free the zombie!
D/Zygote  (  182): Process 8972 exited cleanly (11)
I/DEBUG   ( 9048): debuggerd: Dec  4 2013 23:14:10
W/Binder  ( 1021): Caught a RuntimeException from the binder stub implementation.
W/Binder  ( 1021): java.lang.NullPointerException
W/Binder  ( 1021): 	at android.inputmethodservice.IInputMethodWrapper.setSessionEnabled(IInputMethodWrapper.java:280)
W/Binder  ( 1021): 	at com.android.internal.view.IInputMethod$Stub.onTransact(IInputMethod.java:129)
W/Binder  ( 1021): 	at android.os.Binder.execTransact(Binder.java:404)
W/Binder  ( 1021): 	at dalvik.system.NativeStart.run(Native Method)
W/InputMethodManagerService(  750): Got RemoteException sending setActive(false) notification to pid 8972 uid 10059
D/dalvikvm( 1084): GC_CONCURRENT freed 8033K, 20% free 48171K/59808K, paused 2ms+5ms, total 45ms
D/dalvikvm( 8733): GC_CONCURRENT freed 344K, 3% free 17379K/17756K, paused 3ms+3ms, total 37ms
D/dalvikvm( 9065): Late-enabling CheckJNI
I/ActivityManager(  750): Start proc org.mozilla.fennec_swarooprao for service org.mozilla.fennec_swarooprao/org.mozilla.gecko.background.announcements.AnnouncementsService: pid=9065 uid=10059 gids={50059, 3003, 1028, 1015}
D/dalvikvm(  750): GC_EXPLICIT freed 1394K, 17% free 44937K/53876K, paused 2ms+6ms, total 77ms
D/dalvikvm( 9065): Trying to load lib /data/app-lib/org.mozilla.fennec_swarooprao-2/libmozglue.so 0x425a1ea8
D/dalvikvm( 9065): Added shared lib /data/app-lib/org.mozilla.fennec_swarooprao-2/libmozglue.so 0x425a1ea8
D/dalvikvm( 9065): No JNI_OnLoad found in /data/app-lib/org.mozilla.fennec_swarooprao-2/libmozglue.so 0x425a1ea8, skipping init
D/dalvikvm( 1547): GC_CONCURRENT freed 1082K, 6% free 19847K/21068K, paused 2ms+4ms, total 42ms
D/dalvikvm( 1547): WAIT_FOR_CONCURRENT_GC blocked 12ms
D/dalvikvm( 1547): GC_FOR_ALLOC freed 212K, 6% free 19847K/21068K, paused 21ms, total 21ms
D/dalvikvm( 1547): GC_FOR_ALLOC freed 449K, 6% free 19879K/21068K, paused 19ms, total 19ms
D/dalvikvm( 1547): GC_FOR_ALLOC freed 138K, 6% free 19901K/21068K, paused 18ms, total 18ms
E/GoogleConversionPing( 8844): Error sending ping

********************

Comment 13

[Gian-Carlo Pascutto [:gcp]]

Figuring out why your local build doesn't work isn't really on-topic for this bug. Are you able to come to #mobile on IRC or to the mobile-firefox-dev (https://mail.mozilla.org/listinfo/mobile-firefox-dev) mailinglist to debug this?

Comment 14

[swaroop.rao]

Apologies. I didn't realize that. I'm on IRC right now (id is srao).

Comment 15

[swaroop.rao]

OK, sorry for the long delay in my response. Work pressure. Anyway, I've now managed to do a clean build and install Fennec on my Nexus 5 and got it to run without crashing. Before I make the necessary code changes, I wanted to try it out for myself and see the manifestation of the problem. I went to http://mozilla.github.io/webrtc-landing/gum_test.html and checked out the video and still camera and did not notice any problem. If I'm not able to reproduce the problem, how will I know if my code fix actually addressed the problem?

Comment 17

[Chris Peterson [:cpeterson]]

This camera bug still exists and affects GeckoView and Fenix.

Comment 18

[pranshukumar1999]

I would like to work on this bug. But a little heads up will be required. Thanks.

Comment 19

[sharathbattu123987]

different? I found a file called GeckoAppShell.java in which there is a

where can I able to see GeckoAppShell.java