Closed Bug 995182 Opened 10 years ago Closed 8 years ago

[adbe 3744253] crash in npswf32_13_0_0_182.dll@0x23b1bd

Categories

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect, P5)

Product:
External Software Affecting Firefox Graveyard
Component:
Flash (Adobe)
Platform:
x86
Windows 8.1
Type:
defect

Tracking

(firefox29 ?, firefox30 affected, firefox31 ?)

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox29 --- ?
firefox30 --- affected
firefox31 --- ?

People

(Reporter: andrei, Unassigned)

References

(
URL
)

Details

(Keywords: crash)

Crash Data

Attachments

(2 files)

crash_npswf.txt
21.38 KB, text/plain
Details
minidump
67.54 KB, application/octet-stream
Details
Attached file crash_npswf.txt — 
This bug was filed from the Socorro interface and is 
report bp-a665622a-d560-4504-8100-4ff192140411.
=============================================================

Crashed during a mozmill test. To note that we just recently updated flash to 13.0.0.182
And we have the debug version installed on these machines.

Attached is the full testrun log.
status-firefox29: --- → ?
status-firefox30: --- → affected
status-firefox31: --- → ?
Component: General → Flash (Adobe)
OS: Windows NT → Windows 8.1
Product: Toolkit → Plugins
Version: unspecified → 13.x
Wow, this crashes like hell in our security related Mozmill tests. Andrei, can you please find the log file of Flash and attach it here? Given that we have a debug version Adobe might be interested to get this additional information.

Crash Reason 	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address 	0x4

0 	NPSWF32_13_0_0_182.dll 	NPSWF32_13_0_0_182.dll@0x23b1bd 	
1 	NPSWF32_13_0_0_182.dll 	NPSWF32_13_0_0_182.dll@0x23b327 	
2 	NPSWF32_13_0_0_182.dll 	NPSWF32_13_0_0_182.dll@0x23b3e8 	
3 	ntdll.dll 	__RtlUserThreadStart 	
4 	ntdll.dll 	_RtlUserThreadStart
Group: core-security
Flags: needinfo?(andrei.eftimie)
(In reply to Andrei Eftimie from comment #0)
> To note that we just recently updated flash to 13.0.0.182

Is 13 released now or is it still a beta?
It didn't crashed again when I ran aurora remote on mm-win-81-32-4, after enabling the logs for the debug version. I'll keep trying.
Do you guys need symbols for this build?  I'm only seeing the one crash in crash-stats.

Also, we'd like to reproduce this if possible.  We have mozmill running.  A little more specificity in the STRs would be greatly appreciated.
Flags: needinfo?(kairo)
The crash as reported here happened on a Windows 8.1 32bit version when using an Aurora build of Firefox. If you have the mozmill environment (https://mozqa.com/mozmill-env/) running you can run the following:

testrun_remote %path_to_Aurora_installer%

But as Andreea said above, she was not able to reproduce it anymore on that machine. I would propose that we watch those tests over the weekend, and on Monday we will continue in figuring out what could be wrong here.
Flags: needinfo?(kairo)
Cool, sounds good.
(In reply to Jeromie Clark from comment #5)
> Do you guys need symbols for this build?

Looking at stats like https://crash-analysis.mozilla.com/rkaiser/2014-04-11/2014-04-11.firefox.flash.13-0-0-168.html I think we do have the symbols, not sure why they weren't used in this case.

Henrik, do we have a symbolized signature for later crashes?
Flags: needinfo?(hskupin)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #8)
> Looking at stats like
> https://crash-analysis.mozilla.com/rkaiser/2014-04-11/2014-04-11.firefox.
> flash.13-0-0-168.html I think we do have the symbols, not sure why they
> weren't used in this case.
> 
> Henrik, do we have a symbolized signature for later crashes?

What do you mean with we have symbols? For Flash, or what? Not sure I understand the question yet.
Flags: needinfo?(hskupin)
We could try to enable the Flash Error Reporting[1] on all windows machines where we have the Flash Debugger version installed.

This way we might have more info if/when it crashes again.

[1] http://helpx.adobe.com/flash-player/kb/configure-debugger-version-flash-player.html
Flags: needinfo?(andrei.eftimie)
Henrik, 

Adobe provides a set of obfuscated symbols for Flash Player to Mozilla.  The crash reporting infrastructure will automatically apply those symbols, but the crashes that you've reported just show offsets.  This typically occurs in instances where symbols have not been provided or applied for a particular build.

Ideally, the crash dump would have symbols applied, and I would be able to look at the body of crashes that match the signature to understand crash volume, affected configurations, pull multiple stack traces, etc.  

Running the ActionScript debugger version of Flash Player is probably not helpful.  It doesn't provide additional C++ debugging information.  We fuzz the ActionScript APIs extensively, so the odds are pretty slim that we're crashing because of something that would be logged at the ActionScript level. 

Thanks,
Jeromie
(In reply to Jeromie Clark from comment #11)
> Running the ActionScript debugger version of Flash Player is probably not
> helpful.  It doesn't provide additional C++ debugging information.  We fuzz
> the ActionScript APIs extensively, so the odds are pretty slim that we're
> crashing because of something that would be logged at the ActionScript
> level. 

Do you have a link to the download we need here? Would be good to get some instructions then, so we can dive into more tomorrow.
What download are you talking about?  It appears that you're processing them locally without the Flash symbols. If you submit the crashes to crash-stats, we will have useful symbols. Otherwise they won't.
Andrei reported such a crash to crash-stats. So not sure what the problem is here. Maybe he can give a clear answer in how this has been reported, and where it has been processed.
Flags: needinfo?(andrei.eftimie)
Andrei, which Mozmill-test was affected here? Also has there been a bug filed for? I don't see a mozmill-test related bug on the dependency list.
Priority: -- → P5
I have not logged a mozmill-test failure. 

From the log (attachment 8405350 [details]) it appears to have failed in `testSecurity\testDVCertificate.js` (which doesn't have anything to do with flash).
Right before that test we ran `testPrivateBrowsing\testFlashCookie.js`
Flags: needinfo?(andrei.eftimie)
Andrei, you missed to give feedback on comment 14.
Flags: needinfo?(andrei.eftimie)
(In reply to Andrei Eftimie from comment #16)
> Right before that test we ran `testPrivateBrowsing\testFlashCookie.js`

Given that it might be a dupe of bug 980938. Now we only need the correct symbols for the Flash crash.
(In reply to Henrik Skupin (:whimboo) from comment #14)
> Andrei reported such a crash to crash-stats. So not sure what the problem is
> here. Maybe he can give a clear answer in how this has been reported, and
> where it has been processed.

Oh, I'm not sure what other information I can give here.

- we had a testrun fail
- I checked the Jenkins console log
- noticed we had a crash
- logged into the machine
- opened Firefox (a Nightly from the shared network location)
- opened 'about:crashes'
- the recent crash was logged there, clicked on the link
- from crashstats I clicked on the link from Bugzilla - Report this bug in [...]
- filed this bug (some information was prefilled from the link I clicked) and attached the Jenkins console log
Flags: needinfo?(andrei.eftimie)
Thanks Andrei, so this sounds like something is not working correctly on crash-stats for the new 13.x release?

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #8)
> Looking at stats like
> https://crash-analysis.mozilla.com/rkaiser/2014-04-11/2014-04-11.firefox.
> flash.13-0-0-168.html I think we do have the symbols, not sure why they
> weren't used in this case.

Do you know when crash-stats got this symbols? Maybe the submission of this crash (2014-04-11 12:48:58) was before the update on crash-stats? We have updated our boxes right after the release went out.

Given that we have debug versions running now to prevent the crash on bug 980938, we might not see this crash again for the foreseeable future. :/
(In reply to Henrik Skupin (:whimboo) from comment #20)
> Do you know when crash-stats got this symbols? Maybe the submission of this
> crash (2014-04-11 12:48:58) was before the update on crash-stats? We have
> updated our boxes right after the release went out.

That's what I expect and that's why I asked yesterday if you can have this run again and a new crash being generated that would get all the symbolication correctly.
(In reply to Jeromie Clark from comment #11)
> Ideally, the crash dump would have symbols applied, and I would be able to
> look at the body of crashes that match the signature to understand crash
> volume, affected configurations, pull multiple stack traces, etc.  

Jeromie, this apparently is from a debug version of Flash, and I guess we do not have symbols for those. Is there something we can do here?
In the end, it might just be a case of bug 980938, but on the debug build of Flash.
Flags: needinfo?(jeclark)
Given that it seems to be a null pointer crash, I don't a reason right now to keep this as security sensitive.
Group: core-security
I can share symbols, or if you want to attach the minidump, I could take a look in visual studio.
Flags: needinfo?(jeclark)
I think we should be able to share the minidump, there shouldn't be any PII issue from testing machines, I guess. And on IRC, Andrei said, the dump should still be around. Andrei?
Unfortunately it seems the dumps are now saved in each testruns data/ folder, which gets deleted when a new testrun of the same type is executed.

The dump is no longer available.

I've checked the path mentioned in the log: `c:\jenkins\workspace\mozilla-aurora_remote\data\profile\minidumps\` and I've checked the default paths in %user%\AppData\Mozilla\[...]
Andrei, please check the logs carefully. mozcrash is backing up the minidumps for us in the general crash report folder:

05:26:07 mozcrash INFO | Saved minidump as C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash Reports\pending\7aa26678-f316-4448-9dd7-11466e426ce1.dmp
05:26:07 mozcrash INFO | Saved app info as C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash Reports\pending\7aa26678-f316-4448-9dd7-11466e426ce1.extra
(In reply to Jeromie Clark from comment #24)
> I can share symbols, or if you want to attach the minidump, I could take a
> look in visual studio.

Jeromie, are only the minidumps from debug builds helpful or do those from opt builds also work for you? I ask because of the other bug where we also have a minidump.
Flags: needinfo?(jeclark)
(In reply to Henrik Skupin (:whimboo) from comment #27)
> Andrei, please check the logs carefully. mozcrash is backing up the
> minidumps for us in the general crash report folder:
> 
> 05:26:07 mozcrash INFO | Saved minidump as
> C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash
> Reports\pending\7aa26678-f316-4448-9dd7-11466e426ce1.dmp
> 05:26:07 mozcrash INFO | Saved app info as
> C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash
> Reports\pending\7aa26678-f316-4448-9dd7-11466e426ce1.extra

As said in comment 26 I've also checked the AppData/Roaming location.
> C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash Reports\pending
is empty, and
> C:\Users\mozauto\AppData\Roaming\Mozilla\Firefox\Crash Reports\submitted
has 2 files (with the timestamp of our crash here) both txt files which only contain the crashid
> bp-a665622a-d560-4504-8100-4ff192140411.txt
> Crash ID: bp-a665622a-d560-4504-8100-4ff192140411
and
> bp-hr-20140411-dbd41cd1-6685-46cc-b555-d8e23089a129.txt
> Crash ID: bp-hr-20140411-dbd41cd1-6685-46cc-b555-d8e23089a129

No dump I am afraid :(
Oh wait. I can get it from crash-stats directly. Totally missed that feature! I will attach it shortly.
The minidump was perfect, thanks.  This is Adobe 3744253.  It's assigned for investigation.  STRs would be awesome if you discover them, but we'll try and do what we can.  If you happen to capture other similar minidumps, they would probably be useful.
Flags: needinfo?(jeclark)
Summary: crash in npswf32_13_0_0_182.dll@0x23b1bd → [adbe 3744253] crash in npswf32_13_0_0_182.dll@0x23b1bd
After analysis, we believe that this is resolved by a recent fix to another but and should be fixed in current versions.  I'm not able to verify the developer's assertion directly by looking at crash stats for this signature because it lacked symbols.  Do you guys have a straightforward way to confirm that this is resolved in with the current Flash Player?
Robert, can you do some analysis on crash stats? There is not a single crash for it in the last month. But maybe it had a low appearance anyway.
URL: https://crash-stats.mozilla.com/repor...
Flags: needinfo?(kairo)
(In reply to Henrik Skupin (:whimboo) from comment #34)
> Robert, can you do some analysis on crash stats? There is not a single crash
> for it in the last month. But maybe it had a low appearance anyway.

I don't know more than crash-stats does, it's also the primary tool I use myself. Of course one can search for a longer rage with SuperSearch, e.g. https://crash-stats.mozilla.com/search/?signature=~npswf32_13_0_0_182.dll%400x23b1bd&date=>%3D2014-08-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform but it looks like there just are no crashes at this address. It might have moved to a different address with a different version (13.x is old nowadays) and it might actually have symbols and show up with an actually useful signature nowadays, though.
Flags: needinfo?(kairo)
So closing as WFM?
I'm closing a lot of bugs which are filed as Adobe Flash bugs which are either irrelevant, not actionable, or not serious enough to track in the Mozilla bug tracker. For the most part, Flash bugs should be filed in Adobe bugbase, and we'll only track a few highly-critical issues in the Mozilla tracker.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INCOMPLETE
Version and milestone values are being reset to defaults as part of product refactoring.
Version: 13.x → unspecified
Product: External Software Affecting Firefox → External Software Affecting Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: