Closed Bug 980938 Opened 10 years ago Closed 9 years ago

Crash of Adobe Flash with protected mode enabled and clearing cookies [@ F2102588022]

Categories

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Product:
External Software Affecting Firefox Graveyard
Component:
Flash (Adobe)
Platform:
All
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(firefox27 affected, firefox28 affected, firefox29 affected, firefox30 affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox27 --- affected
firefox28 --- affected
firefox29 --- affected
firefox30 --- affected

People

(Reporter: whimboo, Unassigned)

References

(
URL
)

Details

(Keywords: crash, Whiteboard: [mozmill][qa-automation-blocked][fixed in flash player 15])

Crash Data

Attachments

(2 files, 3 obsolete files)

minimized testcase
501 bytes, text/plain
Details
dmp file
19.42 KB, application/vnd.tcpdump.pcap
Details 
In our recent Mozmill testruns with crash reporting enabled have caught crashes of the most recent version of Flash 12.0.0.70 on Windows. 

Crash report: bp-48e7ea6c-61dc-40af-8d05-6139b2140307.

Crash Reason 	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address 	0x88

0 	FlashPlayerPlugin_12_0_0_70.exe 	F2102588022______________________________________________________________________________________________________ 	F_1997051416__________________________________________________________________
Whiteboard: [mozmill]
Version: 11.x → 12.x
Crash graph:
Windows 7 	68.63 %	105
Windows Vista 	18.30 %	28
Windows 8.1 	11.11 %	17
Windows 8 	1.96 %	3 

Also all branches of Firefox seem to be affected.
URL: https://crash-stats.mozilla.com/repor...
status-firefox27: --- → affected
status-firefox28: ? → affected
status-firefox29: ? → affected
Is there a test-case to reproduce this with?
Flags: needinfo?(hskupin)
It looks like a constant crash on shutdown and only occurs since the upgrade to that version 2 days ago. I will try to investigate that later today or on Monday.
Flags: needinfo?(hskupin)
Attached file testClearFormHistory.js (obsolete) — 
The test which is causing the crash is:
http://hg.mozilla.org/qa/mozmill-tests/file/e1347f0f5e3d/firefox/tests/functional/testFormManager/testClearFormHistory.js

Attached you can find a minimized version of it for testing.

Steps:
1. Setup Mozmill and the test repository: https://developer.mozilla.org/en-US/docs/Mozmill_Tests
2. Overwrite the mentioned test by the attached minimized testcase
3. Run: mozmill -b %path_to_fx% -t %path_to_test%

This might be a crash when we try to clear the Flash cookies? I will try with a debug version of Flash to get more information.
So the debug version of Flash 12.0.0.70 doesn't crash. It only happens for the end-user version.
Jeromie, this looks like it could be a good test-case to investigate?
Flags: needinfo?(jeclark)
No problem.  This is Adobe 3720665
Flags: needinfo?(jeclark)
Jeromie, if you need any other information or someone for testing a build, please let me know. It's 100% reproducible for us.
If it's easy to boil that test down into an HTML document that I can load and reproduce, that would be very helpful.
Attached file windows7_64_firefox_32.txt (obsolete) — 
Run some tests today with the minimized test provided and found this:

On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash.

On windows 7 64, in about 100 runs, it didn't crashed at all using the 64Bits version of firefox.

On the same machine, with the 32Bits version of firefox, we always get a firefox crash logged.
I have attached the console log.

When checking in about:crashes, I've seen a big list of crashed, but the links go to https://crash-stats.mozilla.com/about/throttling .

I'll run tests again on windows 8.1 with the 32Bits version and on a windows 7 32Bits with same firefox, see what happens and come back with more details.
(In reply to daniel.gherasim from comment #11)
> On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash.

Just to add to this comment.  The machine used here is not part of our Mozmill CI in the data center, but was a local one by Daniel. So this doesn't help. Please really use the machines on CI staging for your investigation.

> On windows 7 64, in about 100 runs, it didn't crashed at all using the
> 64Bits version of firefox.

The 64bit version of Firefox is not supported by ourselves. But good to know that this doesn't happen. Or wasn't it the machine from CI staging too?

> When checking in about:crashes, I've seen a big list of crashed, but the
> links go to https://crash-stats.mozilla.com/about/throttling .

Due to bug 981641 we sometimes have broken crash reports which cannot be submitted.

> I'll run tests again on windows 8.1 with the 32Bits version and on a windows
> 7 32Bits with same firefox, see what happens and come back with more details.

When you do this please really use the CI staging machines. Thanks.
(In reply to Henrik Skupin (:whimboo) from comment #12)
> (In reply to daniel.gherasim from comment #11)
> 
> > On windows 7 64, in about 100 runs, it didn't crashed at all using the
> > 64Bits version of firefox.
> 
> The 64bit version of Firefox is not supported by ourselves. But good to know
> that this doesn't happen. Or wasn't it the machine from CI staging too?
> 

It was the machine from CI staging here.

Continued testing on our staging machines.

On Windows 7 32 - Fails almost all the time with the latest Aurora and Nightly.

Interesting fact is that the first time we run the test with any clean install of firefox (or unzipped one) it doesn't fail, it starts failing with the second run.

On Windows 8 32 Bits, in 200 runs it didn't reproduce at all.
(In reply to daniel.gherasim from comment #11)
> On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash.
> 
> On windows 7 64, in about 100 runs, it didn't crashed at all using the
> 64Bits version of firefox.

As Henrik has stated, and also to make it crystal-clear: We do not ship 64bit Firefox and will not do so for quite some time, the 64bit builds of Nightly that do exist are experimental only.

That said, it might still be an interesting data point for Jeromie that the official 32bit Firefox versions crash while the experimental 64bit builds do not cause the crash apparently.
It failed with Shockwave Flash Version:12.0.0.44 too, I dind't find the bug in the first place as it has a different signature, but I can see now that is the same. So I mark it as dependant on this one as is an older Flash version.
Bug 946723
Blocks: 946723
Can you please explain why this blocks bug 946723?
Flags: needinfo?(cosmin.malutan)
Is bug 982669
Blocks: 982669
No longer blocks: 946723
Flags: needinfo?(cosmin.malutan)
Crash Signature: [@ F2102588022______________________________________________________________________________________________________] → [@ F2102588022______________________________________________________________________________________________________] [@ F_1324485019______________ ]
Summary: Crash of Adobe Flash 12.0.0.70 [@ F2102588022______________________________________________________________________________________________________] → Crash of Adobe Flash 12.0.0.44/70 [@ F2102588022______________________________________________________________________________________________________]
No longer blocks: 982669
Tested today several times, and it's interesting that first time we run our tests on a clean build of firefox it doesn't log any crash. Then with the second tests we have every time.

Other thing is that if we clear the history using the FormHistory.jsm library, we don't get any crash.

Manually, if we click on the clearButton, we still don't have crashes,
Let's make the summary a bit nicer to read, out tools catch the full signature from the dedicated Crash Signature field anyhow. :)
Summary: Crash of Adobe Flash 12.0.0.44/70 [@ F2102588022______________________________________________________________________________________________________] → Crash of Adobe Flash 12.0.0.44/70 in F2102588022
I installed 12.0.0.77 on the Windows 7 64bit machine and crashes with the same signature as 12.0.0.70:
bp-081336e0-cb6a-4964-af8a-200cf2140312
Attachment #8389056 - Attachment is obsolete: true
Jeromie, does the attached minidump file help you to get this investigated? Or don't you see it given that it is private? If that is the case I could send it via email.
Flags: needinfo?(jeclark)
The interesting thing is that the latest Firefox 24ESR doesn't cause Firefox to crash. Daniel, please do a regression test on our side so we can find out when this crash starts in Firefox.
Flags: needinfo?(daniel.gherasim)
The signature summary over the last month indicates a large uptick at Firefox 26 with a long tail of old versions.  Flash Player shows similar results, with affected versions back a couple years.

The stack traces are all one line.  We're attempting to release an observer.  I'm guessing we get a bad handle somehow, but there's not much there to work with. 

I'm not able to see the minidumps, but it's okay.  I've asked someone to set up a Mozmill instance on our side to try and reproduce the problem.  It's going to be much easier to figure this out with an interactive debugging session and a C++ debug version of Flash Player.
Flags: needinfo?(jeclark)
I run tests with different versions of firefox (older Nightly & older ESR24 versions) and even with older builds then this ones, Firefox still crashes. ( on windows 7 64 bits).
Flags: needinfo?(daniel.gherasim)
(In reply to daniel.gherasim from comment #25)
> I run tests with different versions of firefox (older Nightly & older ESR24
> versions) and even with older builds then this ones, Firefox still crashes.
> ( on windows 7 64 bits).

This information totally lacks which versions of Firefox you have tested! If you are still not sure how to do a regression test, please talk to me on IRC.
I tried to find a regression range but I couldnt' find a last good version of firefox on which we don't see crashes.

List of firefox versions on which I tried runing the test:

* Firefox 19.0
Status: _crashes_
https://crash-stats.mozilla.com/report/index/1314b629-4174-4e6b-8ea7-b2ab62140313

* Firefox 20.0
Status: _crashes_
https://crash-stats.mozilla.com/report/index/7e57a3a0-784a-43d3-bcf8-e0b512140313

* Firefox 24.0.3ESR 
Status: _crashes_
https://crash-stats.mozilla.com/report/index/76588b06-d586-45b3-b93e-e894a2140313

* Firefox 27.0.1
Status: _crashes_
https://crash-stats.mozilla.com/report/index/7e57a3a0-784a-43d3-bcf8-e0b512140313

* Firefox Autora 29.0a2 build
Status: _crashes_
https://crash-stats.mozilla.com/report/index/081336e0-cb6a-4964-af8a-200cf2140312


I also tested older versions of flash, with the latest version of Nightly:

* 12.0.0.44
https://crash-stats.mozilla.com/report/index/90d3efc6-972a-499e-b241-c24372140313

* 12.0.0.70
https://crash-stats.mozilla.com/report/index/b18bc1a0-f9ff-42d5-ade4-8c6112140313

* 11,9,900,170
https://crash-stats.mozilla.com/report/index/e3db18f0-e529-4224-b991-54f202140313
An update here.
Using the latest Firefox Release & older flash versions and found out this:

* Flash 9.0.283.0 
Status: NO CRASH in 100 RUNS

* Flash 10.1.85.3
Status: NO CRASH in 50 RUNS

* Flash 10.1.102.64
Status: NO CRASH in 50 RUNS

* Flash 11.0.1.152
Status: NO CRASH in 50 RUNS

* Flash 11.2.202.235 (Released 5/04/2012)
Status: NO CRASH IN 100 RUNS

---
* Flash: 11.3.300.257 (Released 6/08/2012)
Status: CRASHES a lot, I guess about 70-90% crash rate.
---
Thanks Daniel! Looks like that this has been introduced with version 11.3.300.257 of Flash. The release notes (http://forums.adobe.com/message/4476911) list the protected mode for Firefox as a new feature! I would say that exactly this might be the reason for the crash. This would also correlate to the platform version information and that Windows XP is not affected. So a problem with the Flash cookies and clear recent history?
Please test later versions as well to see if the rate of crashes changes. We know that 11.3, where protected mode and hw acceleration were introduced (for Win7 [or Vista?] and later but NOT for WinXP), was way more crash-prone then any version before. Some fixes in Firefox and Flash were made in 2013 to improve the situation though and general crash rates improved much. There might still be issues around protected mode for sure, but let's not automatically blame everything on it.

(In reply to Jeromie Clark from comment #24)
> The signature summary over the last month indicates a large uptick at
> Firefox 26 with a long tail of old versions. 

That might just be a by-product of that matching the distribution of Firefox versions "in the wild" in that month. If only few people are running an older version, it surely also has fewer Flash crashes. ;-)
Well, something I will try now it to disable the protected mode. If the crashes will not stop we can be sure it is something else.

http://forums.adobe.com/thread/1018071#TemporaryWorkaround
Ok, so with the protected mode turned off the crashes are gone. So my assumption from above may be right. Now we only miss more details how the steps of the cleanup history test are involved here.
Summary: Crash of Adobe Flash 12.0.0.44/70 in F2102588022 → Crash of Adobe Flash with protected mode enabled [@ F2102588022]
Attached file minimized testcase — 
Minimized version of the testcase, which shows that this is related to the removal of all cookies by the sanitizer.
Attachment #8388515 - Attachment is obsolete: true
Attachment #8388534 - Attachment is obsolete: true
(In reply to Henrik Skupin (:whimboo) from comment #34)
> Created attachment 8391494 [details]
> minimized testcase
> 
> Minimized version of the testcase, which shows that this is related to the
> removal of all cookies by the sanitizer.

Ah, removing all cookies also makes a call to delete all Flash cookies, I guess that call runs into the crash. then.
Summary: Crash of Adobe Flash with protected mode enabled [@ F2102588022] → Crash of Adobe Flash with protected mode enabled and clearing cookies [@ F2102588022]
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #35)
> Ah, removing all cookies also makes a call to delete all Flash cookies, I
> guess that call runs into the crash. then.

And that might make the testcase even smaller. But lets see if Adobe can reproduce. For reference the code which triggers the removal of plugin cookies:

http://mxr.mozilla.org/mozilla-central/source/browser/base/content/sanitize.js#158
Blocks: 984308
We were unable to reproduce this crash in Firefox 27.0.1 and Firefox 28.0 after running the following Mozmill tests with Flash Player 12.0.0.77 and 11.3.300.257 for many times.

$ hg branch
mozilla-release

$ hg branches
default                     3755:8fa4cef860a5
mozilla-aurora              3754:666180a8068d
mozilla-release             3752:d4480bedbc93
mozilla-esr24               3743:96e9aa990cad
mozilla-beta                3750:9237323ed082 (inactive)

$ mozmill -b '/c/Program Files (x86)/Mozilla Firefox/firefox.exe' -t /c/Users/labuser/Desktop/mozmill-tests/firefox/tests/functional/

$ mozmill -b '/c/Program Files (x86)/Mozilla Firefox/firefox.exe' -t /c/Users/labuser/Desktop/mozmill-tests/firefox/tests/functional/testFormManager/

Before running the tests, we have already replaced the mozmill-tests\firefox\tests\functional\testFormManager\testClearFormHistory.js file with either one of the two reduced versions attached in this bug report.

We have also tried clearing the Flash cookies manually via Firefox's History menu > Clear Recent History > Time range to clear: Everything > select "Cookies" then Clear Now, but still couldn't get it to crash.

Any other suggestions or manual steps that might help to reproduce this crash?
Blocks: 991736
Sorry that I haven't had time in the last days to deeper dive into this issue. As you can see for the newly added blocked bug we have another incarnation of this crash in our endurance tests. We will report here once we have more information.
Whiteboard: [mozmill] → [mozmill][qa-automation-blocked]
Hey Henrik, 

On a side note, we still can't repro 980938 and I'd really like to figure out why not.  It's bugging me.  We just re-ran everything against a 32-bit Win7 install and still no luck there.  We recently investigated a crash for Microsoft that only happens with their debugger attached.  :)

I'd like to get a handle on what's unique about that configuration.  Likely candidates are available RAM -- we do some last-minute clean-up when we get close to using all of the available memory.  It's possible that the issue is related to the out-of-memory (OOM) cleanup and we're not hitting the OOM condition that you are.   It could be hardware acceleration related, but if you're still crashing after disabling Hardware Acceleration in Flash, that would rule it out (Right Click a Flash movie > Settings > the left-most tab, uncheck Use Hardware Acceleration).   Maybe we need to be using a debug Firefox target?  

Anyway, suggestions would be appreciated.  We're happy to try and figure it out.  I'd like to see this get fixed, but the stack dumps are useless on this, so we really need to reproduce it.

Thanks,
Jeromie
At this moment I'm not reproducing the crashes anymore, on mm-win-vista-32-4, with Flash release version 13 or with 12.0.0.77. Tried running both endurance flash tests and the cookie one, with  latest nightly and esr24 from 15th March.

It's curious why and I'll look to check for another machine as well to be sure.
Andreea, please see comment 1. I think you should better try to reproduce this crash on a Windows 7 machine which has the highest crash rate. Sorry for mentioning Windows 8 on IRC late yesterday.
Yes, I'm reproducing on Win 7 on the ci, don't know what happened with vista as we also had high failures there.

Jeromie, I used hardware acceleration disabled and we still crash with the testcase attached here. I'll try a debug Firefox build. 

Unfortunately, I couldn't reproduce it on my local machine either, at the moment I'm trying to use proxy as on the remote machines.
The flash cookie test did not crash so far, only with the testcase I managed to reproduce at this time.

Adrian, is there any chance we can create a downloadable archive for this machine which I could then use locally? Thanks.
Flags: needinfo?(afernandez)
(In reply to Andreea Matei [:AndreeaMatei] from comment #43)
> The flash cookie test did not crash so far, only with the testcase I managed
> to reproduce at this time.

Have you modified the flash cookie test, so it doesn't store a cookie before calling clean-up? I wonder if we only crash when no Flash cookies are present.
Flags: needinfo?(andreea.matei)
I modified testRemoveAllCookies.js to not store cookies but it still didn't failed(it looks like it uses a different clean-up method).
In minimized testcase I navigated to:
>http://domain1.mozqa.com/data/firefox/cookies/cookie_single.html
in order to have cookies but I still saw the crash afterwards.
The only way I couldn't reproduce the crash with this testcase was to navigate to a page where we use flash:
>  controller = mozmill.getBrowserController();
>  controller.open("http://mozqa.com/data/firefox/plugins/flash/sample-swf-video-10s.swf");
>  controller.sleep(1000);
If prior of cleaning all history I open the page above I can't reproduce the crash.
I will continue to investigate this today, hope I can bring more useful info.
Flags: needinfo?(andreea.matei)
Would you mind posting the minidump(s) from the test machines as attachments?  We're not able to repro in house, but maybe we can make some progress on a blind fix.
Attachment #8388534 - Attachment is private: false
Jeromie, please check the attachment list. I previously added a minidump via the private flag, so you were most likely not able to see it. It's one for the latest 12.0 release. I hope that helps.
That worked, thanks. :)
No need for a VM for now. Lets see if Adobe can work this out with the given minidump file.
Flags: needinfo?(afernandez)
I was able to take a look at the minidump, but I just get the one line callstack there as well.  If you have a VM that reproduces the issue, that would be huge.
We have a couple VMs where we can reproduce this crash, yes. What shall we do?
If we can get a copy that would be ideal.  Email me a dropbox link or something -- whatever is most convenient for you.
Banakon, are you seeing this crash yourself? Is it reproducible? If yes, do you have steps how to get Flash to crash?
Flags: needinfo?(banakon)
it happens often, without precise steps, for examplke today by opening 6 tabs with flash content, then scrollin into one, then the hang.
Flags: needinfo?(banakon)
protected mode on.
Can you try if you can reproduce with those flash sites? It would be helpful if you could give us the list of them so that we can also try ourselves. Also Adobe might be able to figure the affected code.
I guess that all is in the crash report.
I am wondering that a crash report in the year 2014 is unable to help further.

the page are:
https://www.youtube.com/user/Utopianer1
https://www.youtube.com/user/Fledermaus1990
https://www.youtube.com/user/waldteufel78
https://www.youtube.com/user/rogga93
https://www.youtube.com/user/Yellowman2org/videos
https://www.youtube.com/user/Charly01H/videos
https://www.youtube.com/user/Quaso258/videos

I guess that this is very very unusefull. the only soution is to tell the dev if they might to develop a better crash report.
otherwise we remain here with this bug for years ;) as we did in the past years.
No, the crash lacks this information. It might be because the problem is inside the Flash plugin and not Firefox. So thank you for those URLs. Andreea, could you quickly check those on our machines?

Banakon, what kind of setup do you have? Are you behind a firewall? Do you have to use a proxy?
hi, noproxy.
Norton internet security, with *no* addons for firefox.
Hm, I see that we haven't attached a minidump on this bug yet. Andreea, can you please re-trigger such a crash on one of our systems, and attach the minidump file?
Flags: needinfo?(andreea.matei)
I have tried yesterday on a staging win 7 machine, with latest Flash (non debug version) to reproduce the crash for the minidump file. Usually before, it did best with the flash endurance tests and ESR24. Ran several times and didn't had a crash. Will check again today, but from what I see, it's still happening on the stats, for the last 3 days we had 24 crashes on Win 7 with the latest version:
https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&range_value=3&signature=F2102588022______________________________________________________________________________________________________
Flags: needinfo?(andreea.matei)
Best you would try it with the minimized testcase.
Attached file dmp file — 
Added the file.
Jerome, does this minidump file help you?
Flags: needinfo?(jeclark)
I've added the minidump to our bug.  This issue is still in the queue, but it's open to an engineer for further investigation.  I'll let you know if we have additional questions.  Thanks!
I can't seem to clear the needinfo flag on this, just FYI.
Flags: needinfo?(jeclark)
It's magic!
I'd like to indicate I've had the crash problem for many months now, though I imagine that flash being as important to the web as it is currently, it'll cause a significant pressure to move to other browsers on Windows platforms if not fixed (plugin side, or browser side).

I currently turn protectedmode/IPC mode off on one of my Windows computers (flashplayer crashes immediately on YouTube video playback), but this isn't a general solution for most users. Ironically, the slower laptop doesn't see the crashes, but it does see general slothiness on all web pages that have flash if not also disabled.
(In reply to Jeromie Clark from comment #66)
> I've added the minidump to our bug.  This issue is still in the queue, but
> it's open to an engineer for further investigation.  I'll let you know if we
> have additional questions.  Thanks!

Jermomie, it would be great if we could get a bit of action here. The crashes are a permanent pain for our Mozmill tests given that those are always happening. We are forced to use the debug builds of Flash, but would happily go back to the release builds. Are there any news two months later?
There is no update here in a long time.

I come to bring some good news. We've been running the Flash Debugger version on our Windows (>XP) CI machines from version 12 upwards.

Time has come to upgrade to Flash 15, and I am unable to reproduce the crash anymore.
(I can't vouch for it being fixed in 15 as I have also tested a few older versions - couple 14 and 13 - with which I was also unable to reproduce the crash, it might have been fixed a while ago, I am not sure of this).

I have updated our staging Win CI machines to Flash 15 on Friday, we have 4 days without a crash.
It really does look like the problem has been fixed.
Jerome, could you check your internal bug tracker if that problem has indeed been fixed? Thanks.
Flags: needinfo?(jeclark)
We believe that this is resolved as of Flash Player 15.0.0.117.
Flags: needinfo?(jeclark)
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Whiteboard: [mozmill][qa-automation-blocked] → [mozmill][qa-automation-blocked][fixed in flash player 15]
Version and milestone values are being reset to defaults as part of product refactoring.
Version: 12.x → unspecified
Product: External Software Affecting Firefox → External Software Affecting Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.