1049086 - Camera App is not power friendly.

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Camera, defect, P1)

Description

[Sushil]

The layout of Camera App is not power friendly. Concern has been raised on the Power numbers in Camcorder recording use case on v2.0 Flame device.

In Camcorder recording frame, there are too many layers in layer tree:
1. ColorLayerComposite: dst=[0 36 480 800] src=[0 0 480 764]
2. ColorLayerComposite: dst=[0 0 480 800] src=[0 0 480 800]
3. ThebesLayerComposite: dst=[0 0 480 800] src=[0 0 480 800] Tr=0 Blend=0x100
4. ImageLayerComposite: dst=[0 1 480 800] src=[0 24 719 456] Tr=4 Blend=0x100
5. ThebesLayerComposite: dst=[21 26 317 785] src=[0 0 296 759] Tr=0 Blend=0x105

Three Layers: [3], [4] and [5] are composed on screen, since they are visible.
Where as, on Android there is only 1 single full screen RGBA layer which has Video content + Camcorder UI content (timer + button). There is no blending, no scaling and rotation. Here is SF dumpsys:

    type    |  handle  |   hints  |   flags  | tr | blend |  format  |          source crop            |           frame           name
------------+----------+----------+----------+----+-------+----------+---------------------------------+--------------------------------
    FB_BLIT | b8ee1048 | 00000000 | 00000000 | 00 | 00100 | 00000001 | [    0.0,    0.0,  480.0,  800.0] | [    0,    0,  480,  800] com.android.camera2/com.android.camera.CameraLauncher


So on Firefox OS Camera App, there are overheads which contribute to Power usage, due to difference in layer tree as compared to Android.

1. Layer[5] is a big translucent RGBA layer [296 x 759] which needs to be blended on top of Video layer, just to show 2 small UI controls (timer and button). Data fetch + Blending operation contributes to power numbers. We checked, there is 13 mA of power savings by skipping Layer[5] in Composition.

2. There is rotation on Layer[4], we are checking how much we can save if there were no rotation on Video content layer.

3. Only 1 pixel line is visible of Layer[3] so it should not contribute much to power numbers since we fetch only the visible 1 pixel line of data. We will also check by dropping Layer[3] in Composition.

4. Layers [2] and [1] are entirely hidden under full screen opaque Thebes layer [3], that's why they are never composed. But they are still being painted by CPU, which should be processing overhead. I will raise another bugzilla on this, it is seen on other Camera use cases also. Sometimes I see layer[1] as Thebes layer. Is GPU used for painting background Thebes layer ?

Summary: Can we have a simple layer tree in Camera App to avoid all these overheads which contribute to power numbers (similar to Android Camera App) ?

Comment 1

[Sushil]

Benwa, who can look into this bug ?

Comment 2

[bhargavg1]

[Blocking Requested - why for this release]:

Comment 3

[Benoit Girard (:BenWa)]

We should probably start by solving bug 1040952 first?

Comment 4

[Jason Smith [:jsmith]]

Jon - Is this a blocker from a power perspective?

Comment 5

[Jon Hylands [:jhylands]]

Jason - without specific target numbers to reach, or a before/after patch scenario, or something like a comparison between FxOS and Android, I can't really answer that question. I can give you a number that says "The video camera uses X mA while running", but that really isn't helpful unless we have something to compare it to.

Are you looking for a comparison between Android and FxOS, like we're trying to resolve in bug 1010638 for Video playback?

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

Sushil, is this OK now that bug 1040952 has been fixed?

Comment 7

[Sushil]

(In reply to Milan Sreckovic [:milan] from comment #6)
> Sushil, is this OK now that bug 1040952 has been fixed?

This bug is for power concerns raised due to Camera App layout (higher numbers of layers & data fetch).
I do not know how much improvement Bug 1040952 can bring. Bhargav, can you please have someone to check the power numbers with patch from Bug 1040952 to verify ?

Comment 8

[bhargavg1]

(In reply to Sushil from comment #7)
> (In reply to Milan Sreckovic [:milan] from comment #6)
> > Sushil, is this OK now that bug 1040952 has been fixed?
> 
> This bug is for power concerns raised due to Camera App layout (higher
> numbers of layers & data fetch).

The major issue with the camera app is the number of layers. Untill we fix that we wont have much savings. What we are specifically after is,

Video content + Camcorder UI content (timer + button) to be one layer with no blending or scaling

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

Thanks for the details.  I'd like the camera app team to take a look at this and make sure the CSS is setup for it first, or give us a reduced test case that points out the bug on the platform side.

Comment 10

[Benoit Girard (:BenWa)]

(In reply to bhargavg1 from comment #8)
> no blending

The camera controls are non rectangular buttons. This will require blending the UI layer into the video layer.

Comment 11

[Hema Koka [:hema]]

Sushil, 

Can we get comparison numbers between fxos 2.0 and android?

Justin and David are investigating in the meantime on the camera app side. 

Thanks
Hema

Comment 12

[Sushil]

(In reply to Benoit Girard (:BenWa) from comment #10)
> (In reply to bhargavg1 from comment #8)
> > no blending
> 
> The camera controls are non rectangular buttons. This will require blending
> the UI layer into the video layer.

Recording timer [~ 100 x 75] and button [~ 64 x 64] rect sizes are very small. But the translucent RGBA layer used for Recording timer and button is very big [296 x 759]. So most of the pixels on this layer have nothing to blend with underlying Video layer.

So it is possible to have:

1. Single full screen opaque RGBA layer with Video + UI contols (timer & button) painted on it. It will surely improve power numbers as frame will have only 1 layer and 1 full screen RGBA data to be fetched.

OR

2. Have 2 small separate layers for Recording timer and button. I am not sure, if it will improve power numbers much since # of layers will increase from 3 to 4. But data fetched will be less. And if we can make these 2 layers as Opaque, it will avoid blending operation. I am not sure if and how much this option can improve the power numbers, we will need to check if you can provide such an App.

Comment 13

[Sushil]

(In reply to Hema Koka [:hema] from comment #11)
> Sushil, 
> 
> Can we get comparison numbers between fxos 2.0 and android?
> 
> Justin and David are investigating in the meantime on the camera app side. 
> 

Bhargav should have this info.

Comment 14

[Jon Hylands [:jhylands]]

Just as an additional data point - I have my phone sitting on a small ledge right now, with the camera app running. Switching back and forth between camcorder and camera mode (with the same image displayed on the preview screen), the camcorder mode takes 50mA more power (~565mA versus ~515mA) than the camera mode. This is averaged in each case over 30 seconds. I'm running current 2.0 (built yesterday evening).

Comment 15

[bhargavg1]

(In reply to Sushil from comment #13)
> (In reply to Hema Koka [:hema] from comment #11)
> > Sushil, 
> > 
> > Can we get comparison numbers between fxos 2.0 and android?
> > 
> > Justin and David are investigating in the meantime on the camera app side. 
> > 
> 
> Bhargav should have this info.

We are seeing a difference of around 15%

Comment 16

[David Flanagan [:djf]]

(In reply to bhargavg1 from comment #15)
> > 
> > Bhargav should have this info.
> 
> We are seeing a difference of around 15%

If I understand this bug report correctly, your concern for this bug is the power consumption while recording video, is that right?

Are the Android and FirefoxOS camera apps recording videos of the same resolution?  If we're recording a higher-resolution video than Android, then power consumption may be related to that instead of the graphics system.

Comment 17

[Benoit Girard (:BenWa)]

(In reply to Sushil from comment #12)
> (In reply to Benoit Girard (:BenWa) from comment #10)
> > (In reply to bhargavg1 from comment #8)
> > > no blending
> > 
> > The camera controls are non rectangular buttons. This will require blending
> > the UI layer into the video layer.
> 
> Recording timer [~ 100 x 75] and button [~ 64 x 64] rect sizes are very
> small. But the translucent RGBA layer used for Recording timer and button is
> very big [296 x 759]. So most of the pixels on this layer have nothing to
> blend with underlying Video layer.
> 
> So it is possible to have:
> 
> 1. Single full screen opaque RGBA layer with Video + UI contols (timer &
> button) painted on it. It will surely improve power numbers as frame will
> have only 1 layer and 1 full screen RGBA data to be fetched.

That wont be useful because we're going to have to composite the UI on each new decoded video frame which is what the compositor is doing anyways.

> 
> OR
> 
> 2. Have 2 small separate layers for Recording timer and button. I am not
> sure, if it will improve power numbers much since # of layers will increase
> from 3 to 4. But data fetched will be less. And if we can make these 2
> layers as Opaque, it will avoid blending operation. I am not sure if and how
> much this option can improve the power numbers, we will need to check if you
> can provide such an App.

We can tweak the UI layer to have a smaller surface area. This will reduce the useless blend area between UI elements. This will result in more draw calls however (HWC layers).

Comment 18

[bhargavg1]

(In reply to David Flanagan [:djf] from comment #16)
> If I understand this bug report correctly, your concern for this bug is the
> power consumption while recording video, is that right?

True

 
> Are the Android and FirefoxOS camera apps recording videos of the same
> resolution?  If we're recording a higher-resolution video than Android, then
> power consumption may be related to that instead of the graphics system.

Yes they are measured with similar resolution and in similar environment as well, as the environment would have large impact on the sensors functionality

Comment 19

[Sushil]

(In reply to Benoit Girard (:BenWa) from comment #17)
> > So it is possible to have:
> > 
> > 1. Single full screen opaque RGBA layer with Video + UI contols (timer &
> > button) painted on it. It will surely improve power numbers as frame will
> > have only 1 layer and 1 full screen RGBA data to be fetched.
> 
> That wont be useful because we're going to have to composite the UI on each
> new decoded video frame which is what the compositor is doing anyways.
> 
Android Camera App is doing this way and it results in less power numbers.

> 
> We can tweak the UI layer to have a smaller surface area. This will reduce
> the useless blend area between UI elements. This will result in more draw
> calls however (HWC layers).

If you can provide a test App, Bhargav can help to get power numbers. I am not sure how much improvement it can bring in power numbers, due to less data fetch.

Comment 20

[David Flanagan [:djf]]

We have a couple of work-in-progress patches in the child bug 1049198, and Jon Hylands is going to see what sort of difference they make for power consumption.

Meanwhile, I'd like to spend some time trying to fix layer 5 in comment 0 to see if we can tweak the HTML or CSS to turn that into two much smaller Thebes layers instead of one big one.

Also, now that I've read Mason's excellent description of how to understand layer trees (in http://www.masonchang.com/blog/2014/3/2/wow-such-checkerboard) I'd like to go back and re-read the layer trees for the camera in four cases:

1) camera preview
2) video preview
3) video recording
4) video playback in the camera preview gallery

Comment 21

[David Flanagan [:djf]]

Probably worth checking what the layer tree looks like with the HDR and self-timer icons displayed on the screen, and also with the grid lines enabled.

Also, all the work so far has been done against master, but I should probably be looking at the layer trees for 2.0 since that is the relevant release for this bug.

Comment 22

[David Flanagan [:djf]]

In the child bug, Justin suggest capturing layer trees using the front-facing camera because it does not display a focus ring to confuse things.  That seems like a valuable suggestion.  In trying that out, I notice that the front-facing camera preview on my Flame has a frame rate of 17 instead of the 29-30 that we get with the rear-facing camera. Maybe that is a hardware limitation, or maybe something really bad is happening in the layer tree, so I'll want to look at the trees for front vs. rear.

Comment 23

[David Flanagan [:djf]]

Hmm. Actually the frame rate is also pretty bad with the rear-facing camera in still photo mode. When it auto-focuses we have a frame rate near 30, but then it goes down to 23 or so. This does not happen in video mode which makes me think that the extra cpu reqiurements of face tracking or CAF are making the still photo preview frame rate low.  Probably not something to chase down on this particular bug.

Comment 24

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to David Flanagan [:djf] from comment #23)

> Hmm. Actually the frame rate is also pretty bad with the rear-facing camera
> in still photo mode. When it auto-focuses we have a frame rate near 30, but
> then it goes down to 23 or so. This does not happen in video mode which
> makes me think that the extra cpu reqiurements of face tracking or CAF are
> making the still photo preview frame rate low.  Probably not something to
> chase down on this particular bug.

Note that the camera viewfinder frame rate is variable. The camera driver will change it based on the brightness of the scene (or really for any other internal reasons).

Comment 25

[David Flanagan [:djf]]

Since I don't have a way to measure power consumption directly, I'm going to use the overdraw percentage as a proxy. This is the 3rd number drawn on the screen when you enable 

  Settings->Developer->Developer HUD-> Frames per second

I get numbers that seem stable and repeatable on multiple tests. See https://bugzilla.mozilla.org/show_bug.cgi?id=1049198#c38 for an example.

Comment 26

[David Flanagan [:djf]]

Unfortunately, experience in bug 1049198 shows that the overdraw percentage is not a good proxy for power usage. Justin's 2.0 patch increased the power usage while recording video but decreased the overdraw number.  It left the power usage unchanged while just displaying the video stream, but increased the overdraw percentage there.  Counting the number of layers is not a good proxy for power usage because some layers are easy and some are hard. What I've learned is that counting pixels is not a good proxy either.

Benoit suggests that the overdraw number is still very useful and that we should strive to get it much lower than where it is now.

Comment 27

[David Flanagan [:djf]]

Given that we were not very successful in reducing power consumption in bug 1049198, I'm beginning to thing that the premise of this bug--that we can reduce power consumption by improving the layer tree--may just not be true, and I'm not really sure how to move forward on this.

Tomorrow I'll investigate the size of the previews streams we're displaying to see if we can reduce power by reducing stream resolution. Though that may lead to less crisp video.

Milan and/or Benoit: when you look at the layer trees attached to bug 1049198 do you see any red flags that we should be investigating?  I fee like Justin and I are at our wits end here.

Comment 28

[David Flanagan [:djf]]

Hmm.... I notice that inthe 2.0 layer trees in bug 1049198 the layer immediately below the ref layer has a scrollableRect and scrollId=3 on it.  There is nothing in the camera that should be scrollable. Could this be a performance issue?

Comment 29

[Milan Sreckovic [:milan] (needinfo for best results)]

Kats, can you take a look at why we get the scrollable layer?

Comment 30

[Milan Sreckovic [:milan] (needinfo for best results)]

Listing here the patches from bug 1049198; we didn't see a lot of improvement, but maybe it's different on the other devices:
https://bugzilla.mozilla.org/attachment.cgi?id=8469519
https://bugzilla.mozilla.org/attachment.cgi?id=8469614

Comment 31

[Kartikaya Gupta (email:[e-mail])]

(In reply to David Flanagan [:djf] from comment #28)
> Hmm.... I notice that inthe 2.0 layer trees in bug 1049198 the layer
> immediately below the ref layer has a scrollableRect and scrollId=3 on it. 
> There is nothing in the camera that should be scrollable. Could this be a
> performance issue?

Not sure I looked at the correct attachments on that bug, but I looked at a bunch of the 2.0 layer dumps. In all of them the scrolling ContainerLayer was the root layer of the child process, so it is expected to be "scrollable" (the viewport frame, which corresponds to the root container layer, is always scrollable). None of them had a scrollableRect or displayport larger than the composition bounds, which means that they can't actually be scrolled, because the content fits exactly in the visible area.

I don't think that the existence of that scrolling ContainerLayer alone should make much different with respect to power consumption, but I don't know enough to say that for sure. Let me know if you have other questions about that layer tree, or if I looked at the wrong ones or something.

Comment 32

[Benoit Girard (:BenWa)]

(In reply to David Flanagan [:djf] from comment #27)
> Given that we were not very successful in reducing power consumption in bug
> 1049198, I'm beginning to thing that the premise of this bug--that we can
> reduce power consumption by improving the layer tree--may just not be true,
> and I'm not really sure how to move forward on this.

We can't draw this conclusion from the lack of success. It's not a linear function. Even if it does turn out to be wrong (I doubt it) getting the right layer tree will end up being a memory and performance improvement.


> 
> Tomorrow I'll investigate the size of the previews streams we're displaying
> to see if we can reduce power by reducing stream resolution. Though that may
> lead to less crisp video.
> 
> Milan and/or Benoit: when you look at the layer trees attached to bug
> 1049198 do you see any red flags that we should be investigating?  I fee
> like Justin and I are at our wits end here.

I'll have a look at this in depth early next week. Keeping ni? as a reminder.

Comment 33

[David Flanagan [:djf]]

Bug 1051101 is for whatever work needs to be done to get the camera app layer tree to go through the HWC path instead of the GPU.  The existing bug 1049198 will remain open for whatever CSS and HTML work is needed.

Comment 34

[Hema Koka [:hema]]

Inder,

Based on a irc conversation with Benwa, he is working on https://bugzilla.mozilla.org/show_bug.cgi?id=1052751 to optimize further for reducing overdraw numbers (he is looking into culling layers). However that work will not improve the camera app and will most likely end up with same results.

Now that with Justin's patch in bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1049198 we are falling back to HWC and also reducing layers, we should see improvements on the power consumption for camera. Please test and evaluate if any further optimizations are required for 2.0 release. If not, lets get a closure on this one.

Thanks so much for helping throughout the test/fix cycle!
Hema

Comment 35

[Inder]

Hema - looks like feedback on Justin's patch will come early next week because of long weekend in India. We need to wait for confirmation that we have solved the power issue before calling it done here. Bhargav will update here with the latest results as soon as it's available.

Comment 36

[Yeojin Chung [:YeojinC]]

This issue is not applicable to create a test case.

Comment 37

[KTucker [:KTucker][Inactive 3/4/2016]]

Not enough information provided to write a valid test case here to monitor power usage.

Comment 38

[bhargavg1]

Justin, we don't see any regression because if the change please land it. Sorry for the delay had issues in test set up.

Comment 39

[bhargavg1]

Justin, we don't see any regression because if the change please land it. Sorry for the delay had issues in test set up.

Comment 40

[Justin D'Arcangelo [:justindarc]]

All dependencies are resolved. Closing.