957288 - [B2G][Tech Eva][Desert Rescue] Extremely poor frame rate when playing Desert Rescue on Buri device

Status: RESOLVED WONTFIX

(Tech Evangelism Graveyard :: Preinstalled B2G Apps, defect, P1)

Description

[Cody Roesch [:croesch]]

Attachment: Poor Framerate Video

When the user tries to play the Desert Rescue app on the Buri device, the user experiences very poor framerate.

Repro Steps:
1) Updated Buri to Build ID: 20140106004001
2) Navigate to marketplace and download "Desert Rescue."
3) Launch "Desert Rescue" after installation and proceed to play the game.
4) Notice that the framerate is very poor and the game is mostly unplayable.

Actual:
Game is unplayable due to low framerate.

Expected:
Framerate should be high enough to be playable and fun.

Environmental Variables
Device: Buri v 1.3.0 Moz RIL
Build ID: 20140106004001
Gecko: http://hg.mozilla.org/releases/mozilla-aurora/rev/a43cb4b322d3
Gaia: 35a60b82f8cf2d759939a350e2dadbb9d8b2f5dc
Platform Version: 28.0a2


Notes:
Repro frequency: 5/5 100%

See Attached: Video

Comment 1

[Cody Roesch [:croesch]]

This issue also occurs on the latest Buri v 1.2.0 Mozilla RIL

Environmental Variables
Device: Buri v 1.2.0 Mozilla RIL
Build ID: 20140106004001
Gecko: http://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/d552c08a72d0
Gaia: 8441587c3b352e052fee07665c21fd192540f19f
Platform Version: 26.0

Comment 2

[Jason Smith [:jsmith]]

What about 1.1?

Comment 3

[Cody Roesch [:croesch]]

The bug does not occur in v1.1 on the Buri Device. The game has great framerate in 1.1

Environmental Variables
Device: Buri v 1.1.0 COM RIL
Build ID: 20140102041202
Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/bdac595a4e46
Gaia: 6ff3a607f873320d00cb036fa76117f6fadd010f
Platform Version: 18.1
RIL Version: 01.01.00.019.281

Comment 4

[Jason Smith [:jsmith]]

The attachment here is a picture, not a video. Can you include two videos here - one showcasing the behavior on 1.1, the other showcasing the behavior on 1.3?

Comment 5

[Jason Smith [:jsmith]]

Okay - I saw this for myself on a 1.3 buri build - this looks like very choppy. This is a partner app, so this needs to block.

Comment 6

[Cody Roesch [:croesch]]

Attachment: The actual video

Added the choppy video. Waiting for the 1.1 device to add the video for that device.

Comment 7

[Jason Smith [:jsmith]]

Milan - Poor framerate would be a graphics bug, right? Or could this be caused by something else?

Comment 8

[Cody Roesch [:croesch]]

Attachment: 1.1 video smooth framerate

Comment 9

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Not necessarily graphics, could be anything.  Can we get QA to grab a profile? (for this and any "X is slow" type bugs)

Comment 10

[Jason Smith [:jsmith]]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #9)
> Not necessarily graphics, could be anything.  Can we get QA to grab a
> profile? (for this and any "X is slow" type bugs)

Can you point at directions we can follow to get this profile?

Comment 11

[Jason Smith [:jsmith]]

Moving back to general until we get more details on the root cause.

Comment 12

[Jason Smith [:jsmith]]

Kevin mentioned in IRC he would be willing to look into getting a cleopatra profile for this bug.

Comment 13

[Kevin Grandon :kgrandon]

(In reply to Jason Smith [:jsmith] from comment #12)
> Kevin mentioned in IRC he would be willing to look into getting a cleopatra
> profile for this bug.

Argh, I really wanted to help out, but I appear to have bricked my hamachi =/

I'm going to NI? on bkelly or mchang to see if either would have cycles to grab a profile.

Comment 14

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Jason Smith [:jsmith] from comment #10)
> (In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #9)
> > Not necessarily graphics, could be anything.  Can we get QA to grab a
> > profile? (for this and any "X is slow" type bugs)
> 
> Can you point at directions we can follow to get this profile?

The main instructions are at https://developer.mozilla.org/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler -- we can also set up a time for bgirard to walk you and any other team members through doing the profiling, to get that knowledge out there more.

Comment 15

[Mason Chang [Inactive] [:mchang]]

James, can you try to get a profile for this with the instructions? If you have some issues, feel free to ping me. Thanks!

Comment 16

[Ben Kelly [:bkelly, not reviewing]]

Here is a profile of the game running without user input on my buri running a fresh v1.3:

  http://people.mozilla.org/~bgirard/cleopatra/#report=47d4a04e3be124c37a838234c26bfa70bad3c6ac

About 60% of the time is in skCanvas::flush().  There seems to be significant amounts of time spent in javascript drawing to the canvas.

Comment 17

[Ben Kelly [:bkelly, not reviewing]]

Vladimir, does the profile in comment 16 show anything out of the ordinary or unexpected?

Comment 18

[Ben Kelly [:bkelly, not reviewing]]

It seems skCanvas::flush() is calling GLContext::CreateProgram() and GLContext::LinkProgram() repeatedly.  Is this normal for an opengl application like this?

Comment 19

[Ben Kelly [:bkelly, not reviewing]]

Milan, do have anyone in gfx who can look at this more closely?  The profile suggests most of the time is in skia and opengl which I'm not too familiar with.

Comment 20

[Milan Sreckovic [:milan] (needinfo for best results)]

bjacob or snorp, can you comment on comments 16+19+20?  Just a sanity check.

Comment 21

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Nope, definitely not normal to be spending all the time creating programs.  Looks like cache thrashing, or cache fail.

Thanks for doing the profile -- this helps us immensely as we can route things directly to the right people.

Comment 22

[sarsenyev]

Attachment: Image when tried to launch the game

Noticed on the latest 1.3, if during the game press on the power button twice, the game is returned with smooth frame

Cannot provide the accurate regression window for the low frame issue, because after the last working build, the game crashes with later builds after is launched. 
On some builds the game isn't available from Marketplace, see the attached image

The last working build:
Device: Buri 1.2 MOZ
BuildID: 20130820040204
Gaia: 7249b5d61b955c23efe9e18f80f0da2e78d827eb
Gecko: bb025b6949e8
Version: 26.0a1

The first broken build when the crash appears:
report bp-24c8f165-553c-414d-ac41-be9152140108.

Device: buri 1.2 Moz
BuildID: 20130821050136
Gaia: 70033ba9cbac65551f68fcb3a28f7daf3105e6ff
Gecko: b2486721572e
Version: 26.0a1
Firmware Version: v1.2_20131115

Comment 23

[James Willcox (:snorp) ([e-mail]) (he/him)]

I agree, cache thrashing. I think this stuff gets flushed when we are low on resources, is that the reason here?

Comment 24

[Ben Kelly [:bkelly, not reviewing]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #23)
> I agree, cache thrashing. I think this stuff gets flushed when we are low on
> resources, is that the reason here?

I can check overall memory usage tomorrow.  If this is the case, though, would it be possible throttle cache flushes based on time-since-last-used?  So, avoid flushing shader programs that have been used within the last X time period.

Comment 25

[James Willcox (:snorp) ([e-mail]) (he/him)]

According to gdb, this app is resetting dimensions on every frame, causing us to reset the context (which with SkiaGL means GLContext, SurfaceStream, etc). Real bad.

Comment 26

[James Willcox (:snorp) ([e-mail]) (he/him)]

The reason this would runs "ok" on software is because a software canvas reset is a lot less work. With GL, we have a lot more stuff to recreate.

Comment 27

[James Willcox (:snorp) ([e-mail]) (he/him)]

I think the our options here are:

1) Get the developer to fix the app so it stop setting width/height on every frame.
2) Blacklist the app on 1.2+
3) Add something that allows us to disable SkiaGL just for this app

Comment 28

[James Willcox (:snorp) ([e-mail]) (he/him)]

To explain comment 22, I would guess the canvas is getting demoted to software at some point there do to memory pressure.

Comment 29

[James Willcox (:snorp) ([e-mail]) (he/him)]

um, due to memory pressure...

Comment 30

[Ben Kelly [:bkelly, not reviewing]]

I wonder if there is any way to speed up canvas resets by preserving the GLContext and the ProgramCache.  From talking to James on IRC, it sounds non-trivial.

Comment 31

[Milan Sreckovic [:milan] (needinfo for best results)]

And it's probably dangerous - doing a canvas reset, people would justly expect things to be, well, reset :)

#1 from Comment 27 makes the most sense.  I don't want to just put this to TechEvangelism so that it doesn't get lost, but it shouldn't be treated as a graphics bug for 1.3 (or ever).  Jason, what do you think?

Comment 32

[Jason Smith [:jsmith]]

(In reply to Milan Sreckovic [:milan] from comment #31)
> And it's probably dangerous - doing a canvas reset, people would justly
> expect things to be, well, reset :)
> 
> #1 from Comment 27 makes the most sense.  I don't want to just put this to
> TechEvangelism so that it doesn't get lost, but it shouldn't be treated as a
> graphics bug for 1.3 (or ever).  Jason, what do you think?

That works for me. I'll send this over to Elisa to get the partner to look into this.

Comment 33

[Ben Kelly [:bkelly, not reviewing]]

After researching a bit it does sound like setting dimensions is generally discouraged these days for precisely this reason.  Instead it appears they should be using:

  canvas.clearRect(0, 0, canvas.width, canvas.height);

Comment 34

[Elisa Helton]

Hello Space Monster Games ([e-mail]): Can you please review this bug and get back to us with feedback and a fix date? Thank you.

Comment 35

[Nicole Fong]

Hello again Space Monster Games ([e-mail]): Can you please provide us with a status regarding this bug and a fix date?  Thank you.

Comment 36

[Nicole Fong]

@[e-mail] : Are there any updates regarding the status of this bug and a potential fix date?  Please let us know.  Thank you!

Comment 37

[wilcom1970]

Do the target devices support the requestAnimationFrame API?
If so I can implement this. Also I have a fix for scaling which you should be able to see at my site: http://playstar.mobi/play.php?game=desertrescue on your mobile device.

Comment 38

[wilcom1970]

I also fixed the sloppy canvas sizing/wiping on every frame.

Comment 39

[Ben Kelly [:bkelly, not reviewing]]

(In reply to wilcom1970 from comment #37)
> Do the target devices support the requestAnimationFrame API?

I am using requestAnimationFrame() for some things in 1.3/1.4.  As far as I can tell it was in gecko long before gecko18, so I imagine it should work well on 1.0.1 and 1.1 too.

Comment 40

[Nicole Fong]

What's the status?

Comment 41

[Nicole Fong]

Issue still occurs on Flame v1.3.  @[e-mail] - can you provide a fix by September 15th, please?  Thanks!

Comment 42

[Michael Ellis]

@[e-mail] : Are there any updates regarding the status of this bug and a potential fix date?  Please let us know.  Thank you!

Comment 43

[Sylvestre Ledru [:Sylvestre]]

Mass closing on Tech Evangelism::Preinstalled B2G App as Firefox OS is no longer a thing.

Comment 44

[Sylvestre Ledru [:Sylvestre]]

Mass closing on Tech Evangelism::Preinstalled B2G App as Firefox OS is no longer a thing.

Comment 45

[Sylvestre Ledru [:Sylvestre]]

Mass closing on Tech Evangelism::Preinstalled B2G App as Firefox OS is no longer a thing.