Closed Bug 772097 Opened 12 years ago Closed 4 years ago

[adbe 3318859] crash in F_835252764

Categories

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect, P3)

x86
Windows NT
defect

Tracking

(firefox47 affected, firefox48 affected, firefox49 affected, firefox-esr45 affected, firefox50 affected, firefox51 affected, firefox52 wontfix, firefox53 affected)

RESOLVED WORKSFORME
Tracking Status
firefox47 --- affected
firefox48 --- affected
firefox49 --- affected
firefox-esr45 --- affected
firefox50 --- affected
firefox51 --- affected
firefox52 --- wontfix
firefox53 --- affected

People

(Reporter: benjamin, Unassigned)

References

Details

(Keywords: crash, Whiteboard: [Flash 11.3])

Crash Data

Attachments

(1 file)

27.37 KB, application/octet-stream
Details
Summary: crash in F_835252764_________________________________________________________ → crash in F_835252764
Whiteboard: [Flash-11.3] → [Flash-11.3][startupcrash]
Version: unspecified → 15 Branch
This signature also appears to be a low-volume crash signature which has been around in prior versions of Flash, but has spiked significantly with protected mode.
When I look at the breakdown by Firefox version, it seems almost all of the Aurora crashes are startup crashes, but those on 13.0.1 and trunk have uptime.
Here are the steps to reproduce, reported by Adobe:

Method 
1:Clear  firefox cache and Go to http://www.vudu.com/movies/#
2:Choose New Trailers
3:Player tailers one by one  -- Once the video starts playing, you can seek to 5-10 seconds before the end of the video, then let it complete.

Result:
Crash after playback  3 or more of the trailers (eg. play The Loved Ones->Madagascar3->Magic Mike->The Loved Ones)

Apparently this isn't an issue in Firefox 12.

I tried this but can't get Vudu to load at all... I'll try again on a different box. In the meantime, Marcia is it possible to get other people to try this and get a nightly regression window? Specifically checking the nightlies before and after bug 90268 landed, since it's the obvious candidate.
Not sure whether this is revelant. I ran these STR in nightly under a debugger and got a different stack trace:

>	NPSWF32_11_3_300_265.dll!F1046602669___________________________()  Line 2423	C++
 	NPSWF32_11_3_300_265.dll!F1092009539__________()  Line 3811	C++
 	NPSWF32_11_3_300_265.dll!F661466345____________________()  Line 3756	C++
 	NPSWF32_11_3_300_265.dll!NPP_SetWindow()  Line 976	C++
 	FlashPlayerPlugin_11_3_300_265.exe!F_952353214____________________()  Line 200	C++

The disassembly is:

--- F_1885880_________________________________________________________ ---------
675623C0  mov         edx,dword ptr [ecx+4Ch]  
675623C3  mov         eax,dword ptr [esp+4]  
675623C7  mov         dword ptr [eax+4],edx  
675623CA  mov         edx,dword ptr [ecx+4Ch]  
675623CD  test        edx,edx  
675623CF  je          F1046602669___________________________+14h (675623D4h)  
* 675623D1  mov         dword ptr [edx+8],eax   <--crash here, EDX = 00D66660
675623D4  mov         dword ptr [ecx+4Ch],eax  
675623D7  ret         4

0xD66660 is accessible but readonly memory, this is as access-violation-write
Oddly, following these steps I can consistently reproduce the F1046602699 crash when in a debugger, and the F_835252764 signature when running Firefox normally with the crash reporter.
hrm! I crashed with crash report https://crash-stats.mozilla.com/report/index/bp-9386a084-5e5d-4c65-84c1-48adc2120717 which says that the stack is:

0: F_835252764_________________________________________________________ (F_1885880_________________________________________________________:2423)
1: F1018235994______________________________________ (F_1963533604__________________________________________________:3810)
2: F_1417225323______________________________________ (F_1963533604__________________________________________________:3756)
3: NPP_SetWindow (F_1417895389__________________________________________________________________:976)

MSVC says the stack for the same crash is:

>	NPSWF32_11_3_300_265.dll!F1046602669___________________________()  Line 2423	C++
 	NPSWF32_11_3_300_265.dll!F1092009539__________()  Line 3811	C++
 	NPSWF32_11_3_300_265.dll!F661466345____________________()  Line 3756	C++
 	NPSWF32_11_3_300_265.dll!NPP_SetWindow()  Line 976	C++

So the source/line information matches, but the function names don't. This is confusing; it must be something about the obfuscation technique?
Adding qawanted, but I will take a look at the STR in Comment 3 and see if I can help find a regression range.
Keywords: qawanted
QA Contact: mozillamarcia.knous
Using the latest nightly in the lab (Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0), I keep freezing while the video is buffering - http://www.vudu.com/movies/# and playing the three stooges video.

I will try another machine next.
So far on several different Windows 7 machines I have not been able to reproduce the crash using the STR in Comment 3.
Hrm, I can almost always reproduce it, though it sometimes requires repeating the trailer load 10 or more times. It was easier to reproduce when I tied all the processes to a single CPU to emulate a single-CPU system. Perhaps it's easier to see with slower hardware?
I can reproduce the bug in today's Nightly and Windows 7. I followed the steps in #3. 
The crashes are: 

https://crash-stats.mozilla.com/report/index/bp-0d4bba26-adab-4ab5-858a-bdcfd2120720

https://crash-stats.mozilla.com/report/index/bp-345cfca6-84c0-4d45-b9ba-6d48a2120720

https://crash-stats.mozilla.com/report/index/bea313b5-cd9d-4abd-9d55-0b4682120720

https://crash-stats.mozilla.com/report/index/bp-0d4bba26-adab-4ab5-858a-bdcfd2120720

https://crash-stats.mozilla.com/report/index/bp-0d9d5a07-b379-4091-982f-877e42120720

https://crash-stats.mozilla.com/report/index/bp-67fe1dc8-5a7a-4ee8-82a8-2a3822120720

https://crash-stats.mozilla.com/report/index/bp-0d4bba26-adab-4ab5-858a-bdcfd2120720

https://crash-stats.mozilla.com/report/index/bp-4f2bc468-c1f9-4453-8760-875802120720

Graphics
       
Adapter DescriptionSiS Mirage 3 GraphicsVendor ID0x1039Device ID0x6351Adapter RAMUnknownAdapter DriversSISGRUMD SiSClone SiSFunc SiSKrl SiSGlvDriver Version7.14.10.5290Driver Date12-15-2010Direct2D EnabledBlocked for your graphics card because of unresolved driver issues.DirectWrite Enabledfalse (6.1.7600.16972)ClearType ParametersGamma: 2200 Pixel Structure: RGB ClearType Level: 0 Enhanced Contrast: 400 WebGL RendererBlocked for your graphics card because of unresolved driver issues.GPU Accelerated Windows0. Blocked for your graphics card because of unresolved driver issues.

I have a 6 years old computer, it's quite fast but....
It's #6 top plugin crasher in 15.0b1.
Keywords: topcrash
This crash should be fixed in Flash 11.3.300.268.
Gabriela did some preliminary testing last evening and her testing indicates that .268 did address this crash. She indicated she would also do some additional testing this evening.
Crashes were reduced in 11.3.300.268, but are still around with .270 in smaller numbers.
our internal bugs (#3286062/3219934) have been marked resolved fixed in Dolores 11.4.  we shipped 11.4.402.265 on 8/21.  closing.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
There are still crashes in 11.4.402.265. Here is the breakdown per Flash version:
11.3.300.262 	32.172 % 	157
11.3.300.265 	26.844 % 	131
11.3.300.271 	12.295 % 	60
11.3.300.257 	9.836 % 	48
11.4.402.265 	7.582 % 	37
11.4.400.252 	3.074 % 	15
11.1.102.55 	1.844 % 	9
11.4.400.231 	1.639 % 	8
11.3.300.268 	1.434 % 	7
11.3.300.231 	0.82 % 	        4
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
BTW, you can find this breakdown at the bottom of the Signature Summary tab in https://crash-stats.mozilla.com/report/list?signature=F_835252764_________________________________________________________
thanks.  bug#3318859 has been opened>for review in ellis...
Summary: crash in F_835252764 → [adbe 3318859] crash in F_835252764
Any additional reproducible scenarios would be extremely helpful in getting this resolved.
It's #64 top crasher in 15.0.1.
Keywords: topcrash
This is still present in latest builds for last 4 weeks at it shows in Soccoro.

Version:  Percentage      Nr.Crashes

18.0.1 	   35.625 %	      1404
18.0.2 	   31.337 %	      1235
19.0b4 	    5.303 %	       209
19.0b3 	    3.349 %	       132
19.0b6 	    3.324 %	       131
19.0b5 	    3.096 %	       122
19.0b2 	    2.614 %            103
MarioMi: As this is a crash in Flash, the versions of Firefox it happens in are not very useful. What is more useful is which versions of Flash it happens in.
There continues to be traffic on this signature for current Flash Player versions.  It happens under extreme low memory conditions.  Our audit of related code did not reveal opportunities for a blind fix.

I've pulled some 11.6 crash reports and added them to our bug in an attempt to get some new traction on this, but reproducible steps would be incredibly helpful as we're hitting the end of what we can glean from the stacks.
Keywords: steps-wanted
Priority: -- → P2
Whiteboard: [Flash-11.3][startupcrash] → [Flash-11.3][Flash-11.6][startupcrash]
Let's keep the Flash 11.6 whiteboard for regressions in Flash 11.6.
Whiteboard: [Flash-11.3][Flash-11.6][startupcrash] → [Flash-11.3][startupcrash]
There are still thousands crashes in last 4 weeks via soccoro, most of them on 11.6:

Flash Version	Percentage	Number Of Crashes
11.6.602.180	 61.838 %	      2369
11.3.300.262	   9.58 %	       367
11.3.300.265	   7.93 %	       304
11.3.300.257	   7.70 %	       295
Matt Talistu noticed that I had put these comments in the wrong bug.  They belong here. Sorry for the confusion!

From Magnus:
It would be useful to have the call stack for the PIC process.  The current attached dumps are in the sandbox process.  Difficult to know what's going on as Jimson said earlier in the notes.

From Jimson: 
Seems to be due to a corrupt AbortUnwindList. May be because we don't remove the last AbortUnwindObject from the list. This would indicate that the player didn't invoke RemoveAbortUnwindObject() for one of the added objects. However, no missing invocations stand out.
Priority: P2 → P3
https://www.youtube.com/watch?v=o9GLl6kI4hQ
switching from fullscreen to normal mode with flash player hardware acceleration (hwa) enabled (with hwa disabled I don't have those crashes)
https://bugzilla.mozilla.org/show_bug.cgi?id=774281#c17

fyi: Both reports were created at the same time and belong together (correlation).
2 reports for 1 action
Thanks MrX1980. How reproducible are these steps for you?

Georg, is there anything actionable for you in comment 30-32? If not, what would help here? A Firefox regression window? A Flash regression window?
Keywords: qawanted
While there's a common signature in play, my guess is that the huge uptick we're seeing with Flash Player 11.8 has a unique root-cause.  

We reopened this last week as 3550712 and are currently investigating.  There has been persistent low-volume activity on this signature for years that I don't believe will be resolved by this fix.  I expect we'll have a fix for the immediate issue in a future Flash Player 11.8 beta build, and will be looking for parity with Flash Player 11.7 as our short-term success case.
Summary: [adbe 3318859] crash in F_835252764 → [adbe 3318859][adbe 3550712 for Flash 11.8 spike] crash in F_835252764
a) switching from fullscreen to normal mode = always
b) scrolling after playing = once (until yet)
I can repro this, thanks!
(In reply to MrX1980 from comment #35)
> a) switching from fullscreen to normal mode = always
I confirm this with Flash 11.8.800.50 on FF 21, Win 7 x64.
Keywords: steps-wanted
Keywords: reproducible
Jeromie, do you need us to find a Firefox regression window or do you have sufficient information to go on?
We believe this is fixed on our side, but we had a nasty installer bug that prevented us from shipping the Beta last week.  That issue is also resolved, and we should be on track for shipping a beta this week with this fix.
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #38)
> Jeromie, do you need us to find a Firefox regression window or do you have
> sufficient information to go on?
The regression window is for adbe 3318859 (Firefox's fault), not adbe 3550712 (Flash-11.8 fault).
(In reply to Jeromie Clark from comment #39)
> We believe this is fixed on our side
It's back to Flash 11.7 volume in Flash 11.8.800.64: around #30 Flash crasher in current Firefox channels.
Keywords: reproducible
Summary: [adbe 3318859][adbe 3550712 for Flash 11.8 spike] crash in F_835252764 → [adbe 3318859] crash in F_835252764
Whiteboard: [Flash 11.3][Flash 11.8] → [Flash 11.3][Flash 11.8][fixed in Flash 11.8.800.64]
We shipped a blind fix for this in 11.8.800.64, and I see a sharp reduction in frequency as compared to 11.8.800.50.  The low volume in 11.8.800.75 is early, but also supports a positive resolution. 

I've closed 3550712 as Fixed.  

In terms of the baseline issue (3318859), I think we're stuck on the issues in Comment 28
We really should have had a separate bug for the 11.8 spike to have it cleanly tracked separately. But what's done is done, so we need to live with having both issues in this bug.
Removing the whiteboard that biases queries with Flash 11.8.
Whiteboard: [Flash 11.3][Flash 11.8][fixed in Flash 11.8.800.64] → [Flash 11.3]
We're still hitting this with Flash 19, but I don't think finding a regression window from 2012 is going to help us much in resolving it at this point.
Crash volume for signature 'F_835252764_________________________________________________________':
 - nightly(version 50):5 crashes from 2016-06-06.
 - aurora (version 49):28 crashes from 2016-06-07.
 - beta   (version 48):647 crashes from 2016-06-06.
 - release(version 47):1304 crashes from 2016-05-31.
 - esr    (version 45):31 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       1       1       1       0       0       1       0
 - aurora        3       9       3       2       5       4       1
 - beta         91     109     111      84      81      61      77
 - release     204     175     195     194     159     155     155
 - esr           5       5       3       4       3       1       1

Affected platform: Windows
Crash volume for signature 'F_835252764_________________________________________________________':
 - nightly (version 51): 2 crashes from 2016-08-01.
 - aurora  (version 50): 10 crashes from 2016-08-01.
 - beta    (version 49): 321 crashes from 2016-08-02.
 - release (version 48): 195 crashes from 2016-07-25.
 - esr     (version 45): 38 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       0       0       0
 - aurora        2       4       2
 - beta         83     121      82
 - release      57      52      42
 - esr           1       3       3

Affected platform: Windows

Crash rank on the last 7 days:
             Browser   Content Plugin
 - nightly                     #63
 - aurora                      #70
 - beta                        #20
 - release                     #23
 - esr                         #83
Crash volume for signature 'F_835252764_________________________________________________________':
 - nightly (version 54): 0 crashes from 2017-01-23.
 - aurora  (version 53): 1 crash from 2017-01-23.
 - beta    (version 52): 117 crashes from 2017-01-23.
 - release (version 51): 259 crashes from 2017-01-16.
 - esr     (version 45): 99 crashes from 2016-08-10.

Crash volume on the last weeks (Week N is from 02-06 to 02-12):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       0       0
 - aurora        0       0
 - beta         75      20
 - release     156      54       0
 - esr           3       6       5       3       2       0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly
 - aurora                      #170
 - beta    #5824               #62
 - release                     #23
 - esr                         #109
Component: Plug-ins → Flash (Adobe)
Product: Core → External Software Affecting Firefox
Version: 15 Branch → unspecified
Mass wontfix for bugs affecting firefox 52.

Closing because no crashes reported for 12 weeks.

Status: REOPENED → RESOLVED
Closed: 12 years ago4 years ago
Resolution: --- → WORKSFORME
Product: External Software Affecting Firefox → External Software Affecting Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: