Closed Bug 980938 Opened 7 years ago Closed 6 years ago
Crash of Adobe Flash with protected mode enabled and clearing cookies [@ F2102588022]
In our recent Mozmill testruns with crash reporting enabled have caught crashes of the most recent version of Flash 22.214.171.124 on Windows. Crash report: bp-48e7ea6c-61dc-40af-8d05-6139b2140307. Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0x88 0 FlashPlayerPlugin_12_0_0_70.exe F2102588022______________________________________________________________________________________________________ F_1997051416__________________________________________________________________
Version: 11.x → 12.x
Crash graph: Windows 7 68.63 % 105 Windows Vista 18.30 % 28 Windows 8.1 11.11 % 17 Windows 8 1.96 % 3 Also all branches of Firefox seem to be affected.
Is there a test-case to reproduce this with?
It looks like a constant crash on shutdown and only occurs since the upgrade to that version 2 days ago. I will try to investigate that later today or on Monday.
The test which is causing the crash is: http://hg.mozilla.org/qa/mozmill-tests/file/e1347f0f5e3d/firefox/tests/functional/testFormManager/testClearFormHistory.js Attached you can find a minimized version of it for testing. Steps: 1. Setup Mozmill and the test repository: https://developer.mozilla.org/en-US/docs/Mozmill_Tests 2. Overwrite the mentioned test by the attached minimized testcase 3. Run: mozmill -b %path_to_fx% -t %path_to_test% This might be a crash when we try to clear the Flash cookies? I will try with a debug version of Flash to get more information.
So the debug version of Flash 126.96.36.199 doesn't crash. It only happens for the end-user version.
Jeromie, this looks like it could be a good test-case to investigate?
No problem. This is Adobe 3720665
Jeromie, if you need any other information or someone for testing a build, please let me know. It's 100% reproducible for us.
If it's easy to boil that test down into an HTML document that I can load and reproduce, that would be very helpful.
Run some tests today with the minimized test provided and found this: On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash. On windows 7 64, in about 100 runs, it didn't crashed at all using the 64Bits version of firefox. On the same machine, with the 32Bits version of firefox, we always get a firefox crash logged. I have attached the console log. When checking in about:crashes, I've seen a big list of crashed, but the links go to https://crash-stats.mozilla.com/about/throttling . I'll run tests again on windows 8.1 with the 32Bits version and on a windows 7 32Bits with same firefox, see what happens and come back with more details.
(In reply to daniel.gherasim from comment #11) > On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash. Just to add to this comment. The machine used here is not part of our Mozmill CI in the data center, but was a local one by Daniel. So this doesn't help. Please really use the machines on CI staging for your investigation. > On windows 7 64, in about 100 runs, it didn't crashed at all using the > 64Bits version of firefox. The 64bit version of Firefox is not supported by ourselves. But good to know that this doesn't happen. Or wasn't it the machine from CI staging too? > When checking in about:crashes, I've seen a big list of crashed, but the > links go to https://crash-stats.mozilla.com/about/throttling . Due to bug 981641 we sometimes have broken crash reports which cannot be submitted. > I'll run tests again on windows 8.1 with the 32Bits version and on a windows > 7 32Bits with same firefox, see what happens and come back with more details. When you do this please really use the CI staging machines. Thanks.
(In reply to Henrik Skupin (:whimboo) from comment #12) > (In reply to daniel.gherasim from comment #11) > > > On windows 7 64, in about 100 runs, it didn't crashed at all using the > > 64Bits version of firefox. > > The 64bit version of Firefox is not supported by ourselves. But good to know > that this doesn't happen. Or wasn't it the machine from CI staging too? > It was the machine from CI staging here. Continued testing on our staging machines. On Windows 7 32 - Fails almost all the time with the latest Aurora and Nightly. Interesting fact is that the first time we run the test with any clean install of firefox (or unzipped one) it doesn't fail, it starts failing with the second run. On Windows 8 32 Bits, in 200 runs it didn't reproduce at all.
(In reply to daniel.gherasim from comment #11) > On windows 8.1 64 with the latest Nightly 64, I haven't seen any crash. > > On windows 7 64, in about 100 runs, it didn't crashed at all using the > 64Bits version of firefox. As Henrik has stated, and also to make it crystal-clear: We do not ship 64bit Firefox and will not do so for quite some time, the 64bit builds of Nightly that do exist are experimental only. That said, it might still be an interesting data point for Jeromie that the official 32bit Firefox versions crash while the experimental 64bit builds do not cause the crash apparently.
It failed with Shockwave Flash Version:188.8.131.52 too, I dind't find the bug in the first place as it has a different signature, but I can see now that is the same. So I mark it as dependant on this one as is an older Flash version. Bug 946723
Can you please explain why this blocks bug 946723?
Crash Signature: [@ F2102588022______________________________________________________________________________________________________] → [@ F2102588022______________________________________________________________________________________________________] [@ F_1324485019______________ ]
Summary: Crash of Adobe Flash 184.108.40.206 [@ F2102588022______________________________________________________________________________________________________] → Crash of Adobe Flash 220.127.116.11/70 [@ F2102588022______________________________________________________________________________________________________]
No longer blocks: 982669
Duplicate of this bug: 982669
Tested today several times, and it's interesting that first time we run our tests on a clean build of firefox it doesn't log any crash. Then with the second tests we have every time. Other thing is that if we clear the history using the FormHistory.jsm library, we don't get any crash. Manually, if we click on the clearButton, we still don't have crashes,
Let's make the summary a bit nicer to read, out tools catch the full signature from the dedicated Crash Signature field anyhow. :)
Summary: Crash of Adobe Flash 18.104.22.168/70 [@ F2102588022______________________________________________________________________________________________________] → Crash of Adobe Flash 22.214.171.124/70 in F2102588022
I installed 126.96.36.199 on the Windows 7 64bit machine and crashes with the same signature as 188.8.131.52: bp-081336e0-cb6a-4964-af8a-200cf2140312
Attachment #8389056 - Attachment is obsolete: true
Jeromie, does the attached minidump file help you to get this investigated? Or don't you see it given that it is private? If that is the case I could send it via email.
The interesting thing is that the latest Firefox 24ESR doesn't cause Firefox to crash. Daniel, please do a regression test on our side so we can find out when this crash starts in Firefox.
The signature summary over the last month indicates a large uptick at Firefox 26 with a long tail of old versions. Flash Player shows similar results, with affected versions back a couple years. The stack traces are all one line. We're attempting to release an observer. I'm guessing we get a bad handle somehow, but there's not much there to work with. I'm not able to see the minidumps, but it's okay. I've asked someone to set up a Mozmill instance on our side to try and reproduce the problem. It's going to be much easier to figure this out with an interactive debugging session and a C++ debug version of Flash Player.
I run tests with different versions of firefox (older Nightly & older ESR24 versions) and even with older builds then this ones, Firefox still crashes. ( on windows 7 64 bits).
(In reply to daniel.gherasim from comment #25) > I run tests with different versions of firefox (older Nightly & older ESR24 > versions) and even with older builds then this ones, Firefox still crashes. > ( on windows 7 64 bits). This information totally lacks which versions of Firefox you have tested! If you are still not sure how to do a regression test, please talk to me on IRC.
I tried to find a regression range but I couldnt' find a last good version of firefox on which we don't see crashes. List of firefox versions on which I tried runing the test: * Firefox 19.0 Status: _crashes_ https://crash-stats.mozilla.com/report/index/1314b629-4174-4e6b-8ea7-b2ab62140313 * Firefox 20.0 Status: _crashes_ https://crash-stats.mozilla.com/report/index/7e57a3a0-784a-43d3-bcf8-e0b512140313 * Firefox 24.0.3ESR Status: _crashes_ https://crash-stats.mozilla.com/report/index/76588b06-d586-45b3-b93e-e894a2140313 * Firefox 27.0.1 Status: _crashes_ https://crash-stats.mozilla.com/report/index/7e57a3a0-784a-43d3-bcf8-e0b512140313 * Firefox Autora 29.0a2 build Status: _crashes_ https://crash-stats.mozilla.com/report/index/081336e0-cb6a-4964-af8a-200cf2140312 I also tested older versions of flash, with the latest version of Nightly: * 184.108.40.206 https://crash-stats.mozilla.com/report/index/90d3efc6-972a-499e-b241-c24372140313 * 220.127.116.11 https://crash-stats.mozilla.com/report/index/b18bc1a0-f9ff-42d5-ade4-8c6112140313 * 11,9,900,170 https://crash-stats.mozilla.com/report/index/e3db18f0-e529-4224-b991-54f202140313
It's reproducible even with Firefox 15.0: https://crash-stats.mozilla.com/report/index/e682e6df-0b30-4d46-8010-f71fd2140313
An update here. Using the latest Firefox Release & older flash versions and found out this: * Flash 9.0.283.0 Status: NO CRASH in 100 RUNS * Flash 10.1.85.3 Status: NO CRASH in 50 RUNS * Flash 10.1.102.64 Status: NO CRASH in 50 RUNS * Flash 18.104.22.168 Status: NO CRASH in 50 RUNS * Flash 22.214.171.124 (Released 5/04/2012) Status: NO CRASH IN 100 RUNS --- * Flash: 11.3.300.257 (Released 6/08/2012) Status: CRASHES a lot, I guess about 70-90% crash rate. ---
Thanks Daniel! Looks like that this has been introduced with version 11.3.300.257 of Flash. The release notes (http://forums.adobe.com/message/4476911) list the protected mode for Firefox as a new feature! I would say that exactly this might be the reason for the crash. This would also correlate to the platform version information and that Windows XP is not affected. So a problem with the Flash cookies and clear recent history?
Please test later versions as well to see if the rate of crashes changes. We know that 11.3, where protected mode and hw acceleration were introduced (for Win7 [or Vista?] and later but NOT for WinXP), was way more crash-prone then any version before. Some fixes in Firefox and Flash were made in 2013 to improve the situation though and general crash rates improved much. There might still be issues around protected mode for sure, but let's not automatically blame everything on it. (In reply to Jeromie Clark from comment #24) > The signature summary over the last month indicates a large uptick at > Firefox 26 with a long tail of old versions. That might just be a by-product of that matching the distribution of Firefox versions "in the wild" in that month. If only few people are running an older version, it surely also has fewer Flash crashes. ;-)
Well, something I will try now it to disable the protected mode. If the crashes will not stop we can be sure it is something else. http://forums.adobe.com/thread/1018071#TemporaryWorkaround
Ok, so with the protected mode turned off the crashes are gone. So my assumption from above may be right. Now we only miss more details how the steps of the cleanup history test are involved here.
Summary: Crash of Adobe Flash 126.96.36.199/70 in F2102588022 → Crash of Adobe Flash with protected mode enabled [@ F2102588022]
Minimized version of the testcase, which shows that this is related to the removal of all cookies by the sanitizer.
(In reply to Henrik Skupin (:whimboo) from comment #34) > Created attachment 8391494 [details] > minimized testcase > > Minimized version of the testcase, which shows that this is related to the > removal of all cookies by the sanitizer. Ah, removing all cookies also makes a call to delete all Flash cookies, I guess that call runs into the crash. then.
Summary: Crash of Adobe Flash with protected mode enabled [@ F2102588022] → Crash of Adobe Flash with protected mode enabled and clearing cookies [@ F2102588022]
(In reply to Robert Kaiser (:firstname.lastname@example.org) from comment #35) > Ah, removing all cookies also makes a call to delete all Flash cookies, I > guess that call runs into the crash. then. And that might make the testcase even smaller. But lets see if Adobe can reproduce. For reference the code which triggers the removal of plugin cookies: http://mxr.mozilla.org/mozilla-central/source/browser/base/content/sanitize.js#158
We were unable to reproduce this crash in Firefox 27.0.1 and Firefox 28.0 after running the following Mozmill tests with Flash Player 188.8.131.52 and 11.3.300.257 for many times. $ hg branch mozilla-release $ hg branches default 3755:8fa4cef860a5 mozilla-aurora 3754:666180a8068d mozilla-release 3752:d4480bedbc93 mozilla-esr24 3743:96e9aa990cad mozilla-beta 3750:9237323ed082 (inactive) $ mozmill -b '/c/Program Files (x86)/Mozilla Firefox/firefox.exe' -t /c/Users/labuser/Desktop/mozmill-tests/firefox/tests/functional/ $ mozmill -b '/c/Program Files (x86)/Mozilla Firefox/firefox.exe' -t /c/Users/labuser/Desktop/mozmill-tests/firefox/tests/functional/testFormManager/ Before running the tests, we have already replaced the mozmill-tests\firefox\tests\functional\testFormManager\testClearFormHistory.js file with either one of the two reduced versions attached in this bug report. We have also tried clearing the Flash cookies manually via Firefox's History menu > Clear Recent History > Time range to clear: Everything > select "Cookies" then Clear Now, but still couldn't get it to crash. Any other suggestions or manual steps that might help to reproduce this crash?
Sorry that I haven't had time in the last days to deeper dive into this issue. As you can see for the newly added blocked bug we have another incarnation of this crash in our endurance tests. We will report here once we have more information.
Whiteboard: [mozmill] → [mozmill][qa-automation-blocked]
Hey Henrik, On a side note, we still can't repro 980938 and I'd really like to figure out why not. It's bugging me. We just re-ran everything against a 32-bit Win7 install and still no luck there. We recently investigated a crash for Microsoft that only happens with their debugger attached. :) I'd like to get a handle on what's unique about that configuration. Likely candidates are available RAM -- we do some last-minute clean-up when we get close to using all of the available memory. It's possible that the issue is related to the out-of-memory (OOM) cleanup and we're not hitting the OOM condition that you are. It could be hardware acceleration related, but if you're still crashing after disabling Hardware Acceleration in Flash, that would rule it out (Right Click a Flash movie > Settings > the left-most tab, uncheck Use Hardware Acceleration). Maybe we need to be using a debug Firefox target? Anyway, suggestions would be appreciated. We're happy to try and figure it out. I'd like to see this get fixed, but the stack dumps are useless on this, so we really need to reproduce it. Thanks, Jeromie
At this moment I'm not reproducing the crashes anymore, on mm-win-vista-32-4, with Flash release version 13 or with 184.108.40.206. Tried running both endurance flash tests and the cookie one, with latest nightly and esr24 from 15th March. It's curious why and I'll look to check for another machine as well to be sure.
Andreea, please see comment 1. I think you should better try to reproduce this crash on a Windows 7 machine which has the highest crash rate. Sorry for mentioning Windows 8 on IRC late yesterday.
Yes, I'm reproducing on Win 7 on the ci, don't know what happened with vista as we also had high failures there. Jeromie, I used hardware acceleration disabled and we still crash with the testcase attached here. I'll try a debug Firefox build. Unfortunately, I couldn't reproduce it on my local machine either, at the moment I'm trying to use proxy as on the remote machines.
The flash cookie test did not crash so far, only with the testcase I managed to reproduce at this time. Adrian, is there any chance we can create a downloadable archive for this machine which I could then use locally? Thanks.
(In reply to Andreea Matei [:AndreeaMatei] from comment #43) > The flash cookie test did not crash so far, only with the testcase I managed > to reproduce at this time. Have you modified the flash cookie test, so it doesn't store a cookie before calling clean-up? I wonder if we only crash when no Flash cookies are present.
I modified testRemoveAllCookies.js to not store cookies but it still didn't failed(it looks like it uses a different clean-up method). In minimized testcase I navigated to: >http://domain1.mozqa.com/data/firefox/cookies/cookie_single.html in order to have cookies but I still saw the crash afterwards. The only way I couldn't reproduce the crash with this testcase was to navigate to a page where we use flash: > controller = mozmill.getBrowserController(); > controller.open("http://mozqa.com/data/firefox/plugins/flash/sample-swf-video-10s.swf"); > controller.sleep(1000); If prior of cleaning all history I open the page above I can't reproduce the crash. I will continue to investigate this today, hope I can bring more useful info.
Would you mind posting the minidump(s) from the test machines as attachments? We're not able to repro in house, but maybe we can make some progress on a blind fix.
Attachment #8388534 - Attachment is private: false
Jeromie, please check the attachment list. I previously added a minidump via the private flag, so you were most likely not able to see it. It's one for the latest 12.0 release. I hope that helps.
That worked, thanks. :)
No need for a VM for now. Lets see if Adobe can work this out with the given minidump file.
I was able to take a look at the minidump, but I just get the one line callstack there as well. If you have a VM that reproduces the issue, that would be huge.
We have a couple VMs where we can reproduce this crash, yes. What shall we do?
If we can get a copy that would be ideal. Email me a dropbox link or something -- whatever is most convenient for you.
Banakon, are you seeing this crash yourself? Is it reproducible? If yes, do you have steps how to get Flash to crash?
it happens often, without precise steps, for examplke today by opening 6 tabs with flash content, then scrollin into one, then the hang.
protected mode on.
Can you try if you can reproduce with those flash sites? It would be helpful if you could give us the list of them so that we can also try ourselves. Also Adobe might be able to figure the affected code.
I guess that all is in the crash report. I am wondering that a crash report in the year 2014 is unable to help further. the page are: https://www.youtube.com/user/Utopianer1 https://www.youtube.com/user/Fledermaus1990 https://www.youtube.com/user/waldteufel78 https://www.youtube.com/user/rogga93 https://www.youtube.com/user/Yellowman2org/videos https://www.youtube.com/user/Charly01H/videos https://www.youtube.com/user/Quaso258/videos I guess that this is very very unusefull. the only soution is to tell the dev if they might to develop a better crash report. otherwise we remain here with this bug for years ;) as we did in the past years.
No, the crash lacks this information. It might be because the problem is inside the Flash plugin and not Firefox. So thank you for those URLs. Andreea, could you quickly check those on our machines? Banakon, what kind of setup do you have? Are you behind a firewall? Do you have to use a proxy?
hi, noproxy. Norton internet security, with *no* addons for firefox.
Hm, I see that we haven't attached a minidump on this bug yet. Andreea, can you please re-trigger such a crash on one of our systems, and attach the minidump file?
I have tried yesterday on a staging win 7 machine, with latest Flash (non debug version) to reproduce the crash for the minidump file. Usually before, it did best with the flash endurance tests and ESR24. Ran several times and didn't had a crash. Will check again today, but from what I see, it's still happening on the stats, for the last 3 days we had 24 crashes on Win 7 with the latest version: https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&range_value=3&signature=F2102588022______________________________________________________________________________________________________
Best you would try it with the minimized testcase.
Added the file.
Jerome, does this minidump file help you?
I've added the minidump to our bug. This issue is still in the queue, but it's open to an engineer for further investigation. I'll let you know if we have additional questions. Thanks!
I can't seem to clear the needinfo flag on this, just FYI.
I'd like to indicate I've had the crash problem for many months now, though I imagine that flash being as important to the web as it is currently, it'll cause a significant pressure to move to other browsers on Windows platforms if not fixed (plugin side, or browser side). I currently turn protectedmode/IPC mode off on one of my Windows computers (flashplayer crashes immediately on YouTube video playback), but this isn't a general solution for most users. Ironically, the slower laptop doesn't see the crashes, but it does see general slothiness on all web pages that have flash if not also disabled.
(In reply to Jeromie Clark from comment #66) > I've added the minidump to our bug. This issue is still in the queue, but > it's open to an engineer for further investigation. I'll let you know if we > have additional questions. Thanks! Jermomie, it would be great if we could get a bit of action here. The crashes are a permanent pain for our Mozmill tests given that those are always happening. We are forced to use the debug builds of Flash, but would happily go back to the release builds. Are there any news two months later?
There is no update here in a long time. I come to bring some good news. We've been running the Flash Debugger version on our Windows (>XP) CI machines from version 12 upwards. Time has come to upgrade to Flash 15, and I am unable to reproduce the crash anymore. (I can't vouch for it being fixed in 15 as I have also tested a few older versions - couple 14 and 13 - with which I was also unable to reproduce the crash, it might have been fixed a while ago, I am not sure of this). I have updated our staging Win CI machines to Flash 15 on Friday, we have 4 days without a crash. It really does look like the problem has been fixed.
Jerome, could you check your internal bug tracker if that problem has indeed been fixed? Thanks.
We believe that this is resolved as of Flash Player 220.127.116.11.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Whiteboard: [mozmill][qa-automation-blocked] → [mozmill][qa-automation-blocked][fixed in flash player 15]
Version and milestone values are being reset to defaults as part of product refactoring.
Version: 12.x → unspecified
You need to log in before you can comment on or make changes to this bug.