827407 - [adbe 3563826] java.lang.NullPointerException: at com.adobe.flashplayer.FlashPaintSurface$2$1.run(FlashPaintSurface.java) on ICS and above

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Plugins, defect)

Description

[Scoobidiver (away)]

It's #63 top crasher in 17.0, #22 in 18.0b7, and #83 in 19.0a2.

Here is a crash report: bp-38a0ebaa-8c3b-470d-82b6-ff0c32130107.

java.lang.NullPointerException
	at com.adobe.flashplayer.FlashPaintSurface$2$2.run(FlashPaintSurface.java:253)
	at android.os.Handler.handleCallback(Handler.java:725)
	at android.os.Handler.dispatchMessage(Handler.java:92)
	at android.os.Looper.loop(Looper.java:137)
	at android.app.ActivityThread.main(ActivityThread.java:5039)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:511)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)
	at dalvik.system.NativeStart.main(Native Method)

More reports at:
https://crash-stats.mozilla.com/report/list?signature=java.lang.NullPointerException%3A+at+com.adobe.flashplayer.FlashPaintSurface%242%242.run%28FlashPaintSurface.java%29

Comment 1

[Scoobidiver (away)]

More reports also at:
https://crash-stats.mozilla.com/report/list?signature=java.lang.NullPointerException%3A+at+com.adobe.flashplayer.FlashPaintSurface%242%241.run%28FlashPaintSurface.java%29

Comment 2

[Scoobidiver (away)]

With combined signatures, it's #48 top crasher in 19.0.2, #4 in 20.0b5, #6 in 21.0a2, and #22 in 22.0a1.

Comment 3

[Scoobidiver (away)]

If it's higher in Beta, it's likely because there are more JB users in this channel because of bug 780831.

Comment 4

[Scoobidiver (away)]

It's now #7 in 20.0, #4 in 21.0b1, and #4 in 22.0a2. If the volume becomes higher, it's likely because many users have upgraded to ICS and above where Flash is not officially supported.

Is there something actionable on the Firefox side?

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

Is volume up? or is it climbing the charts because overall crash volume is falling?

Comment 6

[Scoobidiver (away)]

(In reply to Brad Lassey [:blassey] from comment #5)
> Is volume up? or is it climbing the charts because overall crash volume is
> falling?
The volume (in a steady state) is up and this seems caused only by a regression in 20.0:
19.0.2 (last week of March): 0.05 crashes/100 ADU
20.0 (last six days):        0.21 crashes/100 ADU
21.0b1 (last four days):     0.33 crashes/100 ADU

Comment 7

[bhavana bajaj [:bajaj]]

(In reply to Brad Lassey [:blassey] from comment #5)
> Is volume up? or is it climbing the charts because overall crash volume is
> falling?

Brad, based on comment# 6 any suspected bugs that could have regressed this ? Also can you please help find an assignee here ?

CCing QA as well to see they can help find the regressing bug here.

Comment 8

[Chris Peterson [:cpeterson]]

I believe Flash is crashing in a TimerTask callback. Presumably, this is a race condition related to unloading Flash content. I will review the pushlog between Fx 19 and 20 for any suspicious changes.

Comment 9

[Chris Peterson [:cpeterson]]

Like Scoobidiver said, this spike looks like a regression in some build of Nightly 20.0a1.

Comment 10

[Chris Peterson [:cpeterson]]

Some interesting data points from the crash reports' logcat dumps:

1. Flash is hitting a CalledFromWrongThreadException before crashing:

System.err: android.view.ViewRootImpl$CalledFromWrongThreadException: Only the original thread that created a view hierarchy can touch its views.
    at android.view.ViewRootImpl.checkThread(ViewRootImpl.java:4746)
    at android.view.ViewRootImpl.recomputeViewAttributes(ViewRootImpl.java:2610)
    at android.view.ViewGroup.recomputeViewAttributes(ViewGroup.java:1062)
    at android.view.ViewGroup.recomputeViewAttributes(ViewGroup.java:1062)
    at android.view.View.setSystemUiVisibility(View.java:16021)
    at com.adobe.flashplayer.FlashPaintSurface$2.surfaceCreated(FlashPaintSurface.java:234)
    at android.view.SurfaceView.updateWindow(SurfaceView.java:569)
    at android.view.SurfaceView.access$000(SurfaceView.java:86)
    at android.view.SurfaceView$4.setFormat(SurfaceView.java:740)
    at com.adobe.flashplayer.FlashPaintSurface$7.run(FlashPaintSurface.java:717)
    at android.os.Handler.handleCallback(Handler.java:725)
    at android.os.Handler.dispatchMessage(Handler.java:92)
    at org.mozilla.gecko.GeckoAppShell.pumpMessageLoop(GeckoAppShell.java:2110)
    at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
    at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
    at org.mozilla.gecko.GeckoAppShell.runGecko(GeckoAppShell.java:290)
    at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:104)


2. The logs have many JNI error messages about "Something's grabbing the JNIEnv from the wrong thread!" I think we should make these JNI errors fatal crashes, so we can debug the problem when it happens instead of waiting for a random crash to happen later. Or at least make them fatal on the Nightly channel.

https://hg.mozilla.org/mozilla-central/file/2949e808ed33/widget/android/AndroidBridge.h#l125


3. Some of the logs have error messages about "Error: Bad NPObject as private data!" from:

https://mxr.mozilla.org/mozilla-central/search?string=Bad+NPObject+as+private+data


4. Also, most of these crashes involve Flash video. This might not be useful, though because video is probably the most popular use of Flash.

Comment 11

[James Willcox (:snorp) ([e-mail]) (he/him)]

We have had this crash forever. I don't believe we did anything to cause it to be worse, it could just be that people are using Flash more often now. One of the reasons Firefox has had some more uptake recently is that Chrome and stock browser don't support Flash, so it seems plausible at least.

Still, interesting stuff in comment #10.

1) I think this is just a bug in Flash. It's scheduling a callback on the gecko thread and doing bad stuff there.

2) I have seen these before but never tracked them down. I like your suggestion. Looks like you filed bug 800661 about this a while back.

3) I have never seen that before, but it's consistent with some of the crashes we've had in other plugin stuff. I fixed at least two bugs where Flash was calling into the browser after the plugin had been destroyed. This seems to be another manifestation of that.

Comment 12

[Scoobidiver (away)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #11)
> We have had this crash forever. I don't believe we did anything to cause it
> to be worse, it could just be that people are using Flash more often now.
I disagree. The volume per ADU is four times higher in 20.0 than in 19.0.2, in less than one week. See comment 6.

Comment 13

[Kevin Brosnan [Ex-Mozilla]]

A list of affected devices would be good.

Comment 14

[Scoobidiver (away)]

(In reply to Kevin Brosnan [:kbrosnan] from comment #13)
> A list of affected devices would be good.
Mainly Galaxy Tab 2.

From https://crash-analysis.mozilla.com/rkaiser/2013-04-10/2013-04-10.fennecandroid.release.20.0.devices.html (4 devices min):
* java.lang.NullPointerException: at com.adobe.flashplayer.FlashPaintSurface$2$1.run(FlashPaintSurface.java) 	386
Samsung GT-P5110 	31
Samsung GT-P5100 	27
HTC Sensation XL with Beats Audio X315e 	23
HTC Sensation XE with Beats Audio Z715e 	18
Samsung GT-P3100 	18
Samsung GT-P3110 	12
Sony Tablet S 	12
Asus Nexus 7 	11
LGE LG-P705 	9
HTC EVO 3D X515m 	7
Sony SGPT12 	7
HTC Sensation Z710e 	7
Samsung GT-I9300 	6
Samsung GT-N8000 	6
HTC One V 	5
Samsung GT-I9100 	5
Motorola MZ601 	5
Acer A500 	5
HTC Desire C 	5
Sony Ericsson LT26i 	4
Sony SGPT13 	4
Unknown cm_tenderloin 	4
Rockchip N101 DUAL CORE2 V11 	4
* java.lang.NullPointerException: at com.adobe.flashplayer.FlashPaintSurface$2$2.run(FlashPaintSurface.java) 	82
Asus Nexus 7 	32
Samsung GT-P5100 	5
LGE Nexus 4 	4

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #11)
> 1) I think this is just a bug in Flash. It's scheduling a callback on the
> gecko thread and doing bad stuff there.

snorp, Flash is probably confused that it is running on a non-UI thread (i.e. our Gecko thread). Can we do anything to call Flash only on the UI thread? I am assuming that running Flash out-of-process is not viable..

Comment 16

[Robert Kaiser]

The URLs are more or less just random video sites, all with only one hit:

1 	http://www.bbc.co.uk/news/technology-22031894
1 	https://www.humblebundle.com/weekly
1 	http://tvnz.co.nz/rookie-blue/s3-ep7-video-5386060
1 	http://www.qiwireless.com/nokia-md-51w-jbl-playup-unboxing-and-demo-video/
1 	https://play.google.com/store/apps/details?id=com.rovio.angrybirdsspace.premium&
1 	http://www.freeandroidtv.com/
1 	http://www.1channel.ch/external.php?title=Right+at+Your+Door&url=aHR0cDovL2dvcml
1 	http://www.youtube.com/watch?v=CX9LOktn7xk
1 	http://www.youtube.com/watch?v=QenDbHO0oxw
1 	http://rt.com/on-air/rt-america-air/
1 	http://www.tabletowo.pl/2013/04/07/recenzja-tabletu-krugermatz-km1010-wideo/
1 	http://www.narutoget.com/watch/22-naruto-shippuden-episode-26-english-subbed/
1 	http://www.amazon.com/gp/product/B004K71YGI/ref=atv_dp_season_select?ie=UTF8&red
1 	http://www.filmesonlinegratis.net/vh/?id=fubt5ten594f
1 	http://www.battlefield.com/battlefield-4/videos/bf4-announce-trailer-with-rihann
1 	http://theoffice.so/0812.html
1 	http://www.youtube.com/watch?v=mEVzeZLhbSc
1 	http://www.1channel.ch/external.php?title=30+Rock&url=aHR0cDovL3NoYXJlcmVwby5jb2
1 	http://www.filmesonlinegratis.net/quebrando-regras.html
1 	about:blank
1 	http://www.jeuxvideo.com/news/2013/00064874-rediffusion-du-direct-concernant-l-a
1 	http://www.anime44.com/oreimo-episode-7
1 	http://played.to/q43pu402zs8u
1 	http://clipiki.ru/video/225160/Otbrosyi-4-sezon-8-seriya-Kubik-v-Kube
1 	http://www.kinomaniak.tv/film/7-psychopatow/55975
1 	http://www.1channel.ch/external.php?title=30+Rock&url=aHR0cDovL3NoYXJlc2l4LmNvbS

Comment 17

[bhavana bajaj [:bajaj]]

:kbrosnan, checked with :cpeterson on if he needs any qa help here, and we think it will be help us with some solid STR given the co-relation to devices and the urls in comment #16.Can you please give a stab at it ? Thanks !

Comment 18

[Chris Peterson [:cpeterson]]

:kbrosnan, it would also be helpful to know if there is a correlation with a Flash version. Do you have a list of Flash module IDs and version numbers?

Comment 19

[Chris Peterson [:cpeterson]]

I haven't been able to reproduce the NullPointerException from comment 0, but I have found some interesting problems when playing the following embedded YouTube video on my Nexus 4:

http://www.qiwireless.com/nokia-md-51w-jbl-playup-unboxing-and-demo-video/


1. When playing the video, I hit an GL assertion failure because the SurfaceCaps don't match the PixelBufferFormat:

F/MOZ_Assert( 5455): Assertion failure: caps.color == !!format.red, at gfx/gl/GLContext.cpp:877
F/libc    ( 5455): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1), thread 5473 (FP_DoPlay)

(gdb) bt
#0  0x7a652408 in mozilla::gl::GLContext::UpdatePixelFormat (this=0x78e9e400) at gfx/gl/GLContext.cpp:878
#1  0x7a65f614 in mozilla::gl::GLContext::InitOffscreen (this=0x78e9e400, size=..., caps=...) at gfx/gl/GLContext.h:1201
#2  0x7a65ffb2 in mozilla::gl::GLContextProviderEGL::CreateOffscreen (size=..., caps=..., flags=<optimized out>) at gfx/gl/GLContextProviderEGL.cpp:2361
#3  0x7a1594f2 in EnsureGLContext () at dom/plugins/base/nsNPAPIPluginInstance.cpp:87
#4  0x7a15b1ac in nsNPAPIPluginInstance::CreateSurfaceTexture (this=0x78257980) at dom/plugins/base/nsNPAPIPluginInstance.cpp:976
#5  0x7a15b246 in nsNPAPIPluginInstance::AcquireContentWindow (this=0x78257980) at dom/plugins/base/nsNPAPIPluginInstance.cpp:1001
#6  0x81ef9122 in ?? ()
#7  0x81ef9122 in ?? ()

(gdb) print this->mCaps
$4 = {any = false, color = false, alpha = false, bpp16 = false, depth = false, stencil = false, antialias = false, preserve = false}

(gdb) print format
$6 = {red = 8, green = 8, blue = 8, alpha = 0, depth = 0, stencil = 0, samples = 1}


2. If I ignore the SurfaceCaps assertions, the video starts playing but the image is upside down! Panning the page fixes the video orientation.


3. While the video is playing, Android dumps tons of GL warnings:

W/Adreno200-ES20( 7767): <glBindTexture:721>: GL_INVALID_OPERATION
W/SurfaceTexture( 7767): [unnamed-7767-0] updateTexImage: clearing GL error: 0x502


4. When I unload the page, I hit some "grabbing the JNIEnv from the wrong thread" warnings. Setting a breakpoint, I see that Gecko is trying to release a Java Surface on a worker thread, not the Java UI thread. I wonder if these JNIEnv warnings is related to the CalledFromWrongThreadException from comment 10.

We currently "recover" from these JNIEnv warnings by returning early without doing any work. I wonder whether the NullPointerException from comment 0 could be related to our failure to UnregisterSurfaceTextureFrameListener(). As I noted in comment 8, Flash is hitting a NullPointerException when trying to execute a TimerTask that has not been cancelled.


I/AndroidBridge( 9891): ###!!!!!!! Something's grabbing the JNIEnv from the wrong thread! (thr 0x7026fc88 should be 0x40e71a90)

#0  0x7a0548f0 in mozilla::AndroidBridge::GetJNIEnv () at widget/android/AndroidBridge.h:127
#1  0x7a0a0006 in mozilla::AndroidBridge::UnregisterSurfaceTextureFrameListener (this=0x74a48170, surfaceTexture=0x1d500276) at widget/android/AndroidBridge.cpp:2314
#2  0x7a50f76a in nsSurfaceTexture::~nsSurfaceTexture (this=0x80152540, __in_chrg=<optimized out>) at gfx/thebes/nsSurfaceTexture.cpp:212
#3  0x7a0585ec in nsSurfaceTexture::Release (this=0x80152540) at ../../../dist/include/nsSurfaceTexture.h:25
#4  0x7a55cb56 in assign_assuming_AddRef (newPtr=0x0, this=0x7f99c838) at ../../dist/include/nsAutoPtr.h:868
#5  assign_with_AddRef (rawPtr=0x0, this=0x7f99c838) at ../../dist/include/nsAutoPtr.h:852
#6  operator= (rhs=0x0, this=0x7f99c838) at ../../dist/include/nsAutoPtr.h:936
#7  mozilla::gl::SurfaceTextureWrapper::~SurfaceTextureWrapper (this=0x7f99c830, __in_chrg=<optimized out>) at gfx/gl/GLContextProviderEGL.cpp:785
#8  0x7a55cb7e in mozilla::gl::SurfaceTextureWrapper::~SurfaceTextureWrapper (this=0x7f99c830, __in_chrg=<optimized out>) at gfx/gl/GLContextProviderEGL.cpp:786
#9  0x7a55c780 in mozilla::gl::GLContextEGL::ReleaseSharedHandle (this=<optimized out>, shareType=<optimized out>, sharedHandle=2140784688) at gfx/gl/GLContextProviderEGL.c
pp:998
#10 0x7a54c9a8 in mozilla::layers::SharedTextureHostOGL::DeleteTextures (this=0x7ff78700) at gfx/layers/opengl/TextureHostOGL.cpp:222
#11 0x7a54d28e in mozilla::layers::SharedTextureHostOGL::SetCompositor (this=0x7ff78700, aCompositor=0x0) at gfx/layers/opengl/TextureHostOGL.cpp:211
#12 0x7a54a584 in mozilla::layers::ImageHostSingle::SetCompositor (this=<optimized out>, aCompositor=<optimized out>) at gfx/layers/composite/ImageHost.cpp:23
#13 0x7a51de96 in Detach (this=<optimized out>) at gfx/layers/composite/CompositableHost.h:156
#14 mozilla::layers::CompositableParent::ActorDestroy (this=<optimized out>, why=<optimized out>) at gfx/layers/composite/CompositableHost.cpp:75
#15 0x7a174ff6 in mozilla::plugins::PPluginBackgroundDestroyerParent::DestroySubtree (this=0x8028fb40, why=why@entry=mozilla::ipc::IProtocolManager<mozilla::ipc::RPCChannel::RPCListener>::Deletion) at objdir-android/ipc/ipdl/PPluginBackgroundDestroyerParent.cpp:332
#16 0x7a1b06fc in OnMessageReceived (__msg=..., this=0x8028fb40) at objdir-android/ipc/ipdl/PCompositableParent.cpp:171
#17 mozilla::layers::PCompositableParent::OnMessageReceived (this=0x8028fb40, __msg=...) at objdir-android/ipc/ipdl/PCompositableParent.cpp:144
#18 0x7a1b2a32 in mozilla::layers::PCompositorParent::OnMessageReceived (this=0x780b7800, __msg=...) at objdir-android/ipc/ipdl/PCompositorParent.cpp:311
#19 0x7a156e2e in mozilla::ipc::AsyncChannel::OnDispatchMessage (this=0x780b7808, msg=...) at ipc/glue/AsyncChannel.cpp:494
#20 0x7a15dca6 in mozilla::ipc::RPCChannel::OnMaybeDequeueOne (this=0x780b7808) at ipc/glue/RPCChannel.cpp:402
#21 0x7a097f66 in DispatchToMethod<ThumbnailRunnable, tag_nsresult (ThumbnailRunnable::*)()> (method=(tag_nsresult (ThumbnailRunnable::*)(ThumbnailRunnable * const)) 0x7a15dbd5 <mozilla::ipc::RPCChannel::OnMayb
eDequeueOne()>, obj=<optimized out>, arg=...) at ipc/chromium/src/base/tuple.h:383
#22 RunnableMethod<ThumbnailRunnable, tag_nsresult (ThumbnailRunnable::*)(), Tuple0>::Run (this=<optimized out>) at ipc/chromium/src/base/task.h:307
#23 0x7a15c856 in Run (this=<optimized out>) at ../../dist/include/mozilla/ipc/RPCChannel.h:425
#24 mozilla::ipc::RPCChannel::DequeueTask::Run (this=<optimized out>) at ../../dist/include/mozilla/ipc/RPCChannel.h:448
#25 0x7a4c8b2a in MessageLoop::RunTask (this=0x785ffdec, task=0x7792c3a0) at ipc/chromium/src/base/message_loop.cc:334
#26 0x7a4c9506 in MessageLoop::DeferOrRunPendingTask (this=<optimized out>, pending_task=...) at ipc/chromium/src/base/message_loop.cc:342
#27 0x7a4ca8bc in DoWork (this=<optimized out>) at ipc/chromium/src/base/message_loop.cc:442
#28 MessageLoop::DoWork (this=0x785ffdec) at ipc/chromium/src/base/message_loop.cc:421
#29 0x7a4caaac in base::MessagePumpDefault::Run (this=0x78275440, delegate=0x785ffdec) at ipc/chromium/src/base/message_pump_default.cc:23
#30 0x7a4c8ca2 in MessageLoop::RunInternal (this=this@entry=0x785ffdec) at ipc/chromium/src/base/message_loop.cc:216
#31 0x7a4c8d00 in RunHandler (this=0x785ffdec) at ipc/chromium/src/base/message_loop.cc:209
#32 MessageLoop::Run (this=0x785ffdec) at ipc/chromium/src/base/message_loop.cc:183
#33 0x7a4d23cc in base::Thread::ThreadMain (this=0x7825b3d0) at ipc/chromium/src/base/thread.cc:156
#34 0x7a4de10e in ThreadFunc (closure=<optimized out>) at ipc/chromium/src/base/platform_thread_posix.cc:39
#35 0x402a341c in __thread_entry () from /Users/cpeterson/Code/mozilla/jimdb/lib/0172ddd29f0b445c/system/lib/libc.so
#36 0x402a2b10 in pthread_create () from /Users/cpeterson/Code/mozilla/jimdb/lib/0172ddd29f0b445c/system/lib/libc.so
#37 0x00000000 in ?? ()

Comment 20

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Chris Peterson (:cpeterson) from comment #19)
> I haven't been able to reproduce the NullPointerException from comment 0,
> but I have found some interesting problems when playing the following
> embedded YouTube video on my Nexus 4:
> 
> http://www.qiwireless.com/nokia-md-51w-jbl-playup-unboxing-and-demo-video/
> 
> 
> 1. When playing the video, I hit an GL assertion failure because the
> SurfaceCaps don't match the PixelBufferFormat:

This looks like a regression from bug 716859. I think changing 'dummyCaps' in nsNPAPIPluginInstance::EnsureGLContext() to use SurfaceCaps::FromRGBA() should fix things.
 
> 
> 2. If I ignore the SurfaceCaps assertions, the video starts playing but the
> image is upside down! Panning the page fixes the video orientation.
> 

Hmm. It sounds like we're missing a layer invalidation. Maybe try nsNPAPIPluginInstance::RedrawPlugin() after calling SetInverted in anp_native_window_invertPluginContent() (ANPNativeWindow.cpp)

> 
> 3. While the video is playing, Android dumps tons of GL warnings:
> 
> W/Adreno200-ES20( 7767): <glBindTexture:721>: GL_INVALID_OPERATION
> W/SurfaceTexture( 7767): [unnamed-7767-0] updateTexImage: clearing GL error:
> 0x502
> 

Yeah I've seen this before. Android's SurfaceTexture checks for any existing GL errors before it does its own stuff. I think something in our compositor sometimes generates errors on Adreno, so you see this. Probably not the cause of any plugin issues.

> 
> 4. When I unload the page, I hit some "grabbing the JNIEnv from the wrong
> thread" warnings. Setting a breakpoint, I see that Gecko is trying to
> release a Java Surface on a worker thread, not the Java UI thread. I wonder
> if these JNIEnv warnings is related to the CalledFromWrongThreadException
> from comment 10.

It looks like AndroidBridge::UnregisterSurfaceTextureListener() should use GetJNIForThread() instead of GetJNIEnv()

> 
> We currently "recover" from these JNIEnv warnings by returning early without
> doing any work. I wonder whether the NullPointerException from comment 0
> could be related to our failure to UnregisterSurfaceTextureFrameListener().
> As I noted in comment 8, Flash is hitting a NullPointerException when trying
> to execute a TimerTask that has not been cancelled.
> 

Maybe? I remember seeing reports of this bug before I even implemented support for ICS or Honeycomb, though, so I'm skeptical. I think this part could just be a Flash bug.

Comment 21

[bhavana bajaj [:bajaj]]

Hey Chris, what are the next steps here ? Do we still need information from :kbrosnan for comment# 18 or was that resolved offline ?

Comment 22

[Chris Peterson [:cpeterson]]

bajaj, I have no news. I'm still investigating. I have fixed a couple related Flash bugs I found during my investigation, but not the root cause.

Comment 23

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to bhavana bajaj [:bajaj] from comment #21)
> Hey Chris, what are the next steps here ? Do we still need information from
> :kbrosnan for comment# 18 or was that resolved offline ?

bajaj, I think getting a break down of the crashes by flash version might be useful.

Comment 24

[bhavana bajaj [:bajaj]]

(In reply to Brad Lassey [:blassey] from comment #23)
> (In reply to bhavana bajaj [:bajaj] from comment #21)
> > Hey Chris, what are the next steps here ? Do we still need information from
> > :kbrosnan for comment# 18 or was that resolved offline ?
> 
> bajaj, I think getting a break down of the crashes by flash version might be
> useful.

Filed Bug 865492 to pull out this information .

Comment 25

[Chris Peterson [:cpeterson]]

Based on the data from bug 865492, this crash affects Flash 10.3, 11.0, and 11.1. However, 95% of the crashes are from Flash 11.1, but that's probably because our users are good about updating to the latest version of Flash.

Comment 26

[Brad Lassey [:blassey] (use needinfo?)]

bug 865492 only says that the majority of crashes are happening in 11.1.115, how ever there are 13 released versions of 11.1.115 (the latest being 11.1.115.54). I'd like to know the breakdown of those minor versions.

Comment 27

[bhavana bajaj [:bajaj]]

(In reply to Brad Lassey [:blassey] from comment #26)
> bug 865492 only says that the majority of crashes are happening in 11.1.115,
> how ever there are 13 released versions of 11.1.115 (the latest being
> 11.1.115.54). I'd like to know the breakdown of those minor versions.

I was under the impression that is not needed here, based on :cpeterson's comment https://bugzilla.mozilla.org/show_bug.cgi?id=865492#c13 .Also prior comments on 865492 indicate that we do not have the capability to get that at the minute ,but if that is blocking the investigation here and that's attainable we can definitely file a bug and try to get it prioritized .

Comment 28

[Kevin Brosnan [Ex-Mozilla]]

I asked that question in bug 865492 comment 6 #8, 9, and 12 answer. Crash stats does not collect minor version info.

Comment 29

[Chris Peterson [:cpeterson]]

In comment 6, Scoobidiver says this may have been a regression in Firefox 20, perhaps exacerbated by more users upgrading to ICS.

Comment 30

[Robert Kaiser]

(In reply to Kevin Brosnan [:kbrosnan] from comment #28)
> I asked that question in bug 865492 comment 6 #8, 9, and 12 answer. Crash
> stats does not collect minor version info.

To be precise, the annotation that Firefox for Android adds to the crash report doesn't collect more than e.g. "11.5 r115" on the client, so the crash-stats server doesn't have that info.

Comment 31

[Chris Peterson [:cpeterson]]

I don't know if something changed recently in Nightly 23 or if the sample size is too small to be meaningful, but the "FlashPaintSurface" crashes in Nightly 23 have dropped to about the same level as Firefox 19 (before the apparently regression in 20):

* Firefox 19.0.2: (0+247)/139177=0.2%
* Firefox 20.0.1: (6987+2075)/556061=1.6%
* Beta 21.0b4: (222+138)/15090=2.4%
* Aurora 22.0a2: (28+30)/2741=2.1%
* Nightly 23.0a1: (9+11)/7281=0.3%

Also interesting is that *none* of Firefox 19's crashes (in my sample range) were from "FlashPaintSurface$2$1", whereas Firefox 20 had 3x as many "FlashPaintSurface$2$1" crashes as "FlashPaintSurface$2$2".

Comment 32

[Mark Finkle (:mfinkle) (use needinfo?)]

Re-noming for firefox21. This should probably not track.

Comment 33

[Chris Peterson [:cpeterson]]

I have asked Adobe's Flash team to investigate this bug from the Flash side. Adobe's internal bug number is 3563826.

Comment 34

[Scoobidiver (away)]

Here are recent daily crash ratios and ranks:
* 21.0: (2,215+600)/(7,379,160/7)=0.27%    #4
* 22.0b1: (76+63)/(209,533/7)=0.46%        #5
* 23.0a2: 1/(2,296/7)=0.3%                #23
* 24.0a1: 6/(5,628/7)=0.75%                #8
It's still higher than 0.05% in 19.0.2.

Comment 35

[Scoobidiver (away)]

Renominating for 22 (see comment 34) as untracking was based on a false assumption (see comment 31).

Comment 36

[Alex Keybl [:akeybl]]

Is there reason to believe there's anything we can do on our side? This is being investigated on Adobe's dide. This is a very old stability issue, although it is ongoing.

Comment 37

[Scoobidiver (away)]

(In reply to Alex Keybl [:akeybl] from comment #36)
> Is there reason to believe there's anything we can do on our side?
Because of a Firefox regression in 20.0 (see comment 6).

Comment 38

[Chris Peterson [:cpeterson]]

(In reply to Alex Keybl [:akeybl] from comment #36)
> Is there reason to believe there's anything we can do on our side? This is
> being investigated on Adobe's dide. This is a very old stability issue,
> although it is ongoing.

akeybl, I don't have any new leads on this bug. I spoke with an Adobe QA manager about this bug about two weeks ago, but I have not received any updates.

Summary:

1. This might be a Firefox regression because of this crash spiked in Firefox 20.

2. We are calling into the Flash plugin on our Gecko thread, not the main UI thread. The exception stack trace in comment 10 points to Flash calling Android View methods on a different thread than the View was created (which does NOT necessarily have to be the UI thread). Did we start calling Flash on multiple threads in Firefox 20?

Comment 39

[Chris Peterson [:cpeterson]]

snorp, could this Flash crash be related to your Flash fullscreen bug 809055? That fix landed in Firefox 20 and is related to changing Views as Flash goes fullscreen.

Comment 40

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Chris Peterson (:cpeterson) from comment #39)
> snorp, could this Flash crash be related to your Flash fullscreen bug
> 809055? That fix landed in Firefox 20 and is related to changing Views as
> Flash goes fullscreen.

I guess it's plausible, but I kinda doubt it? I remember seeing this crash very early when I was working on Flash support. I guess it could've been a different NPE, though.

Comment 41

[Brad Lassey [:blassey] (use needinfo?)]

Tracking plus, long standing crash. The recent spike moves it from a - to a +.

Comment 42

[Alex Keybl [:akeybl]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #40)
> (In reply to Chris Peterson (:cpeterson) from comment #39)
> > snorp, could this Flash crash be related to your Flash fullscreen bug
> > 809055? That fix landed in Firefox 20 and is related to changing Views as
> > Flash goes fullscreen.
> 
> I guess it's plausible, but I kinda doubt it? I remember seeing this crash
> very early when I was working on Flash support. I guess it could've been a
> different NPE, though.

Could we backout or disable for a single release to verify?

Comment 43

[Chris Peterson [:cpeterson]]

(In reply to Alex Keybl [:akeybl] from comment #42)
> Could we backout or disable for a single release to verify?

Backing out bug 809055 would be very easy; it's just two lines of code. The downside is that Firefox will hang if the user tries to switch apps while playing fullscreen Flash video.

I think that experiment would be worthwhile. This Flash crash is consistently the #4 topcrash for Firefox 21 and above, which is probably more frequent than users switching apps while playing fullscreen Flash video.

Comment 44

[Alex Keybl [:akeybl]]

(In reply to Chris Peterson (:cpeterson) from comment #43)
> Backing out bug 809055 would be very easy; it's just two lines of code. The
> downside is that Firefox will hang if the user tries to switch apps while
> playing fullscreen Flash video.
> 
> I think that experiment would be worthwhile. This Flash crash is
> consistently the #4 topcrash for Firefox 21 and above, which is probably
> more frequent than users switching apps while playing fullscreen Flash video.

Let's perform this experiment early in FF23 on beta (or while it's still on Aurora).

Comment 45

[Chris Peterson [:cpeterson]]

Attachment: Back out Flash fullscreen video bug 809055 to test whether FlashPaintSurface crashes are correlated. r=

Comment 46

[Chris Peterson [:cpeterson]]

Comment on attachment 761013 [details] [diff] [review]
Back out Flash fullscreen video bug 809055 to test whether FlashPaintSurface crashes are correlated. r=

snorp: This patch temporarily backs out your fix for Flash fullscreen video bug 809055 to test whether it is correlated with these FlashPaintSurface crashes.

Comment 47

[bugsy]

This Flash crash is happening when i go from background to full screen on my Android Tablet Acer A500 ICS Firefox 21.0. Firefox hangs up After Crash .. This was in Gamespot.com

Comment 48

[Chris Peterson [:cpeterson]]

(In reply to bugsy from comment #47)
> This Flash crash is happening when i go from background to full screen on my
> Android Tablet Acer A500 ICS Firefox 21.0. Firefox hangs up After Crash ..
> This was in Gamespot.com

bugsy: that's promising news! Is the gamespot.com crash listed in your browser's "about:config" list of crash reports? Can you share the crash report URL or ID? That would be a big help.

Is there a particular gamespot URL that demonstrates the problem? Can you retest with Firefox Beta (version 22), which includes some Flash crash fixes? Thanks!

Comment 49

[bugsy]

Hi i will get back to you with this Test ... the url is http://uk.gamespot.com/e3/stage-1-day-2/

Comment 50

[bugsy]

(In reply to Chris Peterson (:cpeterson) from comment #48)
> (In reply to bugsy from comment #47)
> > This Flash crash is happening when i go from background to full screen on my
> > Android Tablet Acer A500 ICS Firefox 21.0. Firefox hangs up After Crash ..
> > This was in Gamespot.com
> 
> bugsy: that's promising news! Is the gamespot.com crash listed in your
> browser's "about:config" list of crash reports? Can you share the crash
> report URL or ID? That would be a big help.
> 
> Is there a particular gamespot URL that demonstrates the problem? Can you
> retest with Firefox Beta (version 22), which includes some Flash crash
> fixes? Thanks!

Flash does not load for me in Firefox Beta (version 22) at http://uk.gamespot.com/e3/stage-2-day-2/

Comment 51

[Chris Peterson [:cpeterson]]

When I try to load the gamespot videos on my Nexus 4 (using Firefox 21 or Nightly 24), the videos fail to load (showing just a black rectangle) and logcat shows dozens of the following warnings:

W/SurfaceTexture( 2510): [unnamed-2510-1] updateTexImage: clearing GL error: 0x502
W/Adreno200-ES20( 2510): <glBindTexture:721>: GL_INVALID_OPERATION
W/SurfaceTexture( 2510): [unnamed-2510-1] updateTexImage: clearing GL error: 0x502
W/Adreno200-ES20( 2510): <glBindTexture:721>: GL_INVALID_OPERATION

Comment 52

[bugsy]

Gamespot wont change to fullscreen in HD mode and i was running there site for 6 hrs a day over 4 days while at E3 Expo and advertisements ran over the window at times i  was using Firefox 21.0 and Firefox was crashing at least 5 times a day through this period of time im using up to date flash om Acer A500 Tablet,ICS

Comment 53

[Chris Peterson [:cpeterson]]

Speculatively back out Flash fullscreen video bug 809055 to test whether FlashPaintSurface crashes are correlated:
https://hg.mozilla.org/integration/mozilla-inbound/rev/97e842bc8760

Comment 54

[Ed Morley [:emorley]]

(In reply to Chris Peterson (:cpeterson) from comment #53)
> Speculatively back out Flash fullscreen video bug 809055 to test whether
> FlashPaintSurface crashes are correlated:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/97e842bc8760

Merge of backout:
https://hg.mozilla.org/mozilla-central/rev/97e842bc8760

Comment 55

[Chris Peterson [:cpeterson]]

My speculative backout has been in Nightly 24 for about one week (since July 18). Nightly builds from July 18-24 had 1 crash. In comparison, Nightly builds from July 11-17 had 6 crashes (during July 11-17). So the crash volume *seems* to be down, but it is not zero.

I'll wait another week to see if we see any different numbers for Aurora 24 before deciding whether to back out the backout.

Comment 56

[Scoobidiver (away)]

(In reply to Chris Peterson (:cpeterson) from comment #55)
> In comparison, Nightly builds from July 11-17 had 6 crashes (during July 11-17).
I counted 9 crashes. See https://crash-stats.mozilla.com/report/list?version=FennecAndroid%3A24.0a1&range_value=4&range_unit=weeks&signature=java.lang.NullPointerException%3A%20at%20com.adobe.flashplayer.FlashPaintSurface%242%241.run%28FlashPaintSurface.java%29

Comment 57

[Chris Peterson [:cpeterson]]

(In reply to Scoobidiver from comment #56)
> (In reply to Chris Peterson (:cpeterson) from comment #55)
> > In comparison, Nightly builds from July 11-17 had 6 crashes (during July 11-17).
> I counted 9 crashes. See
> https://crash-stats.mozilla.com/report/list?version=FennecAndroid%3A24.
> 0a1&range_value=4&range_unit=weeks&signature=java.lang.
> NullPointerException%3A%20at%20com.adobe.flashplayer.
> FlashPaintSurface%242%241.run%28FlashPaintSurface.java%29

But all those crash reports have Build timestamps that are less than 20130618000000.

Comment 58

[Scoobidiver (away)]

(In reply to Chris Peterson (:cpeterson) from comment #57)
> But all those crash reports have Build timestamps that are less than
> 20130618000000.
It was for the comparison. It's nine (before the fix) versus one (after the fix). They are very good results. You now need to find another way to fix bug 809055.

Comment 59

[Chris Peterson [:cpeterson]]

Scoobidiver, do you think we can call this bug fixed? The crash stats for the last 7 days since the last channel uplift look very good:

22.0   = 2490 crashes = 2.3% of 106639 crashes
23.0b1 = 109 crashes = 3.0% of 3660 crashes
24.0a2 = 0 crashes! = 0% of 461 crashes (expected ~12)
25.0a1 = 0 crashes! = 0% of 109 crashes (expected ~3)

I will nom for uplifting this fix to 23.0b1.

Comment 60

[Chris Peterson [:cpeterson]]

Comment on attachment 761013 [details] [diff] [review]
Back out Flash fullscreen video bug 809055 to test whether FlashPaintSurface crashes are correlated. r=

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 809055
User impact if declined: This Flash crash is responsible for about 3% of Fennec crashes.
Testing completed (on m-c, etc.): m-c and m-a for the past two weeks with 0 crashes over the past 7 days.
Risk to taking this patch (and alternatives if risky): This speculative fix just backed out the fix for Flash fullscreen crash bug 809055, which would be reopened.
String or IDL/UUID changes made by this patch: N/A

Comment 61

[Scoobidiver (away)]

(In reply to Chris Peterson (:cpeterson) from comment #59)
> Scoobidiver, do you think we can call this bug fixed?
The high volume part is fixed but I think the pre-bug 809055's fix volume (before Fx 20.0) will remain. But let's consider it as fixed for now.

Comment 62

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 761013 [details] [diff] [review]
Back out Flash fullscreen video bug 809055 to test whether FlashPaintSurface crashes are correlated. r=

we'll take the backout and track bug 809055 hoping for a potential fix that doesn't cause crashes like this.

Comment 63

[Chris Peterson [:cpeterson]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f863495cdd41

Comment 64

[Scoobidiver (away)]

Only the "java.lang.NullPointerException: at com.adobe.flashplayer.FlashPaintSurface$2$1.run(FlashPaintSurface.java)" signature is fixed.

Comment 65

[Lukas Blakk [:lsblakk] use ?needinfo]

In that case we can leave it open for the other signature, but this is no longer a topcrasher so removing the keyword and the tracking status.

Comment 66

[Scoobidiver (away)]

This signature is fixed.