Closed Bug 711564 Opened 13 years ago Closed 12 years ago

[10.5] Firefox Crash [@ GLEngine@0x620cf ]

Categories

(Core :: Graphics, defect)

10 Branch
All
macOS
defect
Not set
critical

Tracking

()

RESOLVED FIXED
mozilla12
Tracking Status
firefox10 + affected
firefox11 + verified
firefox12 + verified
firefox-esr10 13+ verified

People

(Reporter: marcia, Assigned: BenWa)

References

()

Details

(Keywords: crash, regression, topcrash, Whiteboard: [qa+])

Crash Data

Attachments

(2 files, 2 obsolete files)

Seen while looking at crash stats. https://crash-stats.mozilla.com/report/list?signature=GLEngine@0x620c to the crashes which are all Mac 10.5 and all on Aurora except for two that show on trunk. Some appears to be dupes and may be the same user. Will take a look and see if there is a site this is happening on consistently. 

https://crash-stats.mozilla.com/report/index/ab192914-f8b1-4bb4-bab6-726202111215

Frame 	Module 	Signature [Expand] 	Source
0 	GLEngine 	GLEngine@0x620cf 	
1 	QuartzCore 	CARendererDestroy 	
2 	QuartzCore 	-[CARenderer dealloc] 	
3 	Foundation 	NSPopAutoreleasePool
CARenderer indicates that we're using a Core Animation plugin. I don't think Core Animation is well supported on those platforms. We could disable the plugin support for the Core Animation drawing model in 10.5.
This is our top crash on 10b2 for Mac right now. Adding the top crash keyword.
Keywords: topcrash
I can't reproduce this bug but I have a few ideas of things we can try. I'd like to get a hold of someone that can and put up a try build.

I tried reproducing the crash with my 10.5 server on 10b2 without success by accessing Youtube and some links from the description.
I tried on two 10.5 machines with various URLs and so far no luck reproducing this crash.
Also I assume that Bug 715238 is a dupe of this bug?
#13 top crash overall in early B3 data. Adding the tracking flag so we don't lose the fact we have to do what Benoit indicates in Comment 1. All the comments in the bug indicate users are crashing on youtube.com although as I indicated I have not yet been able to reproduce.
Attached patch Disable Core Animation on 10.5 (obsolete) — Splinter Review
Here's a patch that disables Core Animation on 10.5. I would like to understand why this issues is araising but in the mean time here is a try build so we can start doing some testing. CA is said to be buggy on 10.5 while QuickDraw+CoreGraphics drawing models are supported.
(In reply to Benoit Girard (:BenWa) from comment #3)
> I can't reproduce this bug but I have a few ideas of things we can try. I'd
> like to get a hold of someone that can and put up a try build.
> 
> I tried reproducing the crash with my 10.5 server on 10b2 without success by
> accessing Youtube and some links from the description.

Is there any risk to taking this in FF10beta4 and backing out for beta 5 if there are any regressions? If you agree that we should land this early, please nominate for landing on beta 10 today.
This patch needs more verification locally before I am ready to nominate it for beta landing. I will need some users/QA to help me test the try build that will be ready in a few hours:

https://hg.mozilla.org/try/rev/984fcbafe963

Without step to reproduce and/or a regression window this patch is largely an educated shot in the dark unfortunately :(
I have tried my build from http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-984fcbafe963 on my 10.5 machine with Quicktime/Flash. It would be worth while having others try the build with flash+quicktime and other plugins they use.

It's not clear what has caused this issue to suddenly appear. Did some plug-in start using Core Animation suddenly on 10.5?
Forgot to include both CA/ICA
Attachment #587041 - Attachment is obsolete: true
Attachment #587188 - Attachment is obsolete: true
Adding qawanted and CC'ing Anthony/Juan to help line up exploratory testing of this try build on OS X 10.5. What specifically should they be looking for?

This would preferably be completed tomorrow so that we can land the fix prior to cutting beta 4.
Keywords: qawanted
(In reply to Alex Keybl [:akeybl] from comment #14)
> Adding qawanted and CC'ing Anthony/Juan to help line up exploratory testing
> of this try build on OS X 10.5. What specifically should they be looking for?
> 
> This would preferably be completed tomorrow so that we can land the fix
> prior to cutting beta 4.

Sure, given specific instruction, we can hand this off to our Romanian test team to bang on overnight.
I went back to the manual report that rhelmer ran for me the other day for another bug and attached the correlations.
Benoit, let us know if you have any suggestions as to the testing we should be using for this. So far I have not been able to reproduce the problem on the machines in the lab to begin with, so it's hard to see the before and after.
Build with v2 is out:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-de6feef0e3d0/

Testing should consist of trying out any plugin you can get your hands on under 10.5 with this build. Testing only needs to verify that the plugin is able to refresh smoothly. Testing should be done with Quicktime, Flash (Youtube/Game), Googletalk (Google plus hang out), Java Applet and any other plugins we can get our hands on.

I'll ping Josh Aas when I see him tomorrow since blacklist Core Animation on 10.5 is a significant change to plug-ins. It would make approving this change easier if we can prove that this patch helps.
Ok, so I digged into this issue for a while and now I can see those crashes on my old white MBP 13" with 10.5.8 installed. I haven't had installed Flip4Mac and Firefox crashed after some seconds when I have installed the plugin manually:

1. Open http://www.pny.com/movies
2. Click on e.g. German
3. You will see the missing plugin notificiation
4. Try to auto-install Flip4Mac
5. Click the install manually button
6. Install Flip4Mac player
7. Wait until Firefox crashes

Crash report: bp-53314118-b4df-4382-9647-489212120110

I will now try to be able to reproduce the crash and find better STR. Once I have those in hands I can search for the regression range.
Hardware: x86 → All
Updated steps to reproduce:

1. Ensure Flip4Mac is installed, like 2.4.0.11
2. Open http://www1.movie-promo.com/PNY/RedirectToWhiteLabel
3. Click on Germany
Here my installed plugins:
* Silverlight.plugin Version: 4.0.60531.0
* Flip4Mac WMV Plugin.plugin Version: 2.4.0.11
* Flip4Mac WMV Plugin.plugin Version: 2.4.0.11

Once more some updated steps:
1. Ensure the above plugins are installed
2. Create a fresh profile
3. Open http://www1.movie-promo.com/pny-de/Movie/468/3-Engel-f%FCr-Charlie

We do not crash with Firefox 3.6 but all later versions. In 3.6 I can see an error of the Flip4Mac plugin, which states: "461 Unsupported Transport". Also the Quicktime logo is displayed shifted by about 300x200 pixels.

In Firefox 4 a lot has been changed, so not sure if you need further investigation on it or not.

I will now try the tryserver build and check if the proposed patch fixes the issue.
Assignee: nobody → bgirard
Status: NEW → ASSIGNED
Keywords: qawanted
(In reply to Benoit Girard (:BenWa) from comment #11)
> I have tried my build from
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.
> com-984fcbafe963 on my 10.5 machine with Quicktime/Flash. It would be worth
> while having others try the build with flash+quicktime and other plugins
> they use.

This build doesn't fix the crash for me with Flip4Mac.


(In reply to Benoit Girard (:BenWa) from comment #18)
> Build with v2 is out:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.
> com-de6feef0e3d0/

This build now disables the plugin completely, which fixes the crash but as pointed out would mean a massive change for our 10.5 user base.

Benoit, I hope you can reproduce the crash now on your own system, which would make it easier to get a proper fix in place. If you need something else from me, please ask. I'm always around for proofing patches (tryserver builds).

Also if you want to have the real regression range please come back to me.
I can confirm the crash on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:10.0) Gecko/20100101 Firefox/10.0 beta 3

I've followed the steps from comment19 and when installing Flip4Mac v2.4.0.11, at plugin installation process, Firefox crashed. The installation was finished as expected and when I restarted Firefox and repeated the steps, Firefox crashed again. 
On http://www.pny.com/movies and on http://mozqa.com/data/firefox/plugins/mediaplayer.html with Flip4Mac installed, Firefox crashes every time.

I have tried also on the try-build provided in comment18 (try-macosx64):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:12.0a1) Gecko/20120109 Firefox/12.0a1

Firefox did not crash using http://www.pny.com/movies. On http://mozqa.com/data/firefox/plugins/mediaplayer.html Firefox didn't crash either but displayed a grey page with no content.
(In reply to Henrik Skupin (:whimboo) from comment #21)

Thanks for the steps Henrik! I'll be in the office soon to try it out on my Mac 10.5.

> We do not crash with Firefox 3.6 but all later versions. 

Does this happen specifically on 4.0 (or 9.0 or below)? I don't see any crash reports on 9.0 or below. Also some users with this signature report that they are seeing the problem with Youtube but I don't believe Flip4Mac is used there.

I think once we have the above two questions answered we should have a good idea of what exactly is causing the regression. I'll be in soon to look into this.
(In reply to Benoit Girard (:BenWa) from comment #24)

> > We do not crash with Firefox 3.6 but all later versions. 
> 
> Does this happen specifically on 4.0 (or 9.0 or below)? I don't see any
> crash reports on 9.0 or below. Also some users with this signature report
> that they are seeing the problem with Youtube but I don't believe Flip4Mac
> is used there.

Any version of Firefox starting with 4.0 is crashing. I haven't checked Youtube yet and will not be able to in the next couple of hours due to other topcrasher investigations on 10.6/10.7 - which are installed on the same box. So I really hope you can reproduce the crash.

> I think once we have the above two questions answered we should have a good
> idea of what exactly is causing the regression. I'll be in soon to look into
> this.

Shall I check for the regression range during our Firefox 4 cycle?
(In reply to Henrik Skupin (:whimboo) from comment #25)
> Shall I check for the regression range during our Firefox 4 cycle?

No thanks, I can reproduce the crash myself. Thanks for the help so far Henrik and Vlad.
(In reply to Marcia Knous [:marcia] from comment #7)
> All the
> comments in the bug indicate users are crashing on youtube.com although as I
> indicated I have not yet been able to reproduce.

What comments are you referring to Marcia? I took another and can't find any mention of youtube.com? The correlation is at about 16% for Flip4Mac.
Chromium disabled Core Animation on 10.5. I've never disabled it on 10.5 because I though a plug-in could try to use it if it didn't crash, but seeing it cause a topcrasher is making me more willing to disable it.

http://www.google.com/codesearch#OAMlx_jo-ck/o3d/plugin/mac/main_mac.mm&exact_package=chromium&q=CoreAnimation&type=cs&l=563

I've contacted Flip4Mac.
Comment on attachment 587189 [details] [diff] [review]
Disable Core Animation on 10.5 v2

Let's get this landed on trunk now so that once we've confirmed with Flip4Mac we can land this change.
Attachment #587189 - Flags: review?(smichaud)
Attachment #587189 - Flags: review?(smichaud) → review+
(In reply to Henrik Skupin (:whimboo) from comment #21)
> 3. Open http://www1.movie-promo.com/pny-de/Movie/468/3-Engel-f%FCr-Charlie

Sorry if I missed this elsewhere, but does this occur any time a Flip4Mac video is loaded in the browser?
On 10.5 I think so, yes.
The user crash comments have mention crashing on youtube.com. If you click on the comments link - https://crash-stats.mozilla.com/report/list?signature=GLEngine@0x620c.

(In reply to Benoit Girard (:BenWa) from comment #27)
> (In reply to Marcia Knous [:marcia] from comment #7)
> > All the
> > comments in the bug indicate users are crashing on youtube.com although as I
> > indicated I have not yet been able to reproduce.
> 
> What comments are you referring to Marcia? I took another and can't find any
> mention of youtube.com? The correlation is at about 16% for Flip4Mac.
Per the triage meeting, marcia to confirm the answer to the question posed in Comment 30.
I tried reproducing this on 10.5 using many of the sites that had URLs - I wasn't able to reproduce on every site, http://asx.clevver.com/wm/asx.php?vtype=movies&vfile=mo_TheIdesOfMarch_MakeItMandatory was one other site besides the one Henrik listed in Comment 21 that crashes reliably. Some of the sites in the crash data appeared to be sites that were using flash and not WMP.

I tested with

* Silverlight.plugin Version: 4.0.60531.0
* Flip4Mac WMV Plugin.plugin Version: 2.4.0.11

http://asx.clevver.com/wm/asx.php?vtype=movies&vfile=mo_TheIdesOfMarch_MakeItMandatory plays fine in Safari and Chrome on the same machine.
So, if this existed in Fx4, why is it suddenly a topcrash now?
Given Marcia's latest findings, I don't think we should land this for beta 4 (or even m-c really)

1) It's not happening for all Flip4Mac video (Comment 35), and the patch will disable the plugin entirely (Comment 22)
2) Chromium disabled Core Animation on 10.5 (Comment 28), but video seems to play fine for Marcia in Chrome on the same machine (Comment 35)
3) This affects even Firefox 4 (Comment 21)

As Christian suggests, we should understand why this is a top crash now, and subsequently try to find a root cause.
It appears some of it was caused by Flip4Mac release an update.

Have we been able to reproduce the crash with Flash? It would be important to test with Flash if we can confirm that it's also causing crashes. The volume is higher than expected if it's only caused by Flip4Mac.
https://hg.mozilla.org/mozilla-central/rev/8eb5fd522461
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla12
(In reply to Alex Keybl [:akeybl] from comment #37)
> Given Marcia's latest findings, I don't think we should land this for beta 4
> (or even m-c really)

So it has been landed now on m-c. Shall we backout the patch?

> 1) It's not happening for all Flip4Mac video (Comment 35), and the patch
> will disable the plugin entirely (Comment 22)

I'm not sure which sites Marcia is referencing. It would be great to get those URLs for verification from my side. Given the one URL Firefox crashes by only loading http://asx.clevver.com/wm/asx.php.

> 2) Chromium disabled Core Animation on 10.5 (Comment 28), but video seems to
> play fine for Marcia in Chrome on the same machine (Comment 35)

The given link from comment 35 doesn't work for me. It only wants to download the PHP file, and nothing more.

> As Christian suggests, we should understand why this is a top crash now, and
> subsequently try to find a root cause.

As the release notes for the 2.4.0.11 version state I would assume that the new WMV plugin is causing this problem:

Includes a new Flip4Mac WMV Plugin that supports Safari 4 and 5, Firefox 3 and 4, and Google Chrome 6 or later. This also replaces the old 'webplugin', which has been removed.

I will try to find a former release to download, which is actually quiet difficult.
Nothing where Flash is involved crashes for me. Only Flip4Mac content is triggering crashes. You can take any wmv file you can find like:

http://www.bartlett-equip.com.au/media/safety_chain_installation.wmv

Can we be sure that users have been crashing on youtube? Means can we always trust the included URL for crashes? What happens i.e. if the crash happens in a background tab. Do we get this? Ted, I think you are the best person to ask for.
Re-open until we're sure if there is a Flash issue and it is solved.

I've contacted the authors for Flip4Mac and the plugin should fall back to CoreGraphics when CA/ICA is not available (10.5). The fact that it does not is a separate issue.

(In reply to Alex Keybl [:akeybl] from comment #37)
> Given Marcia's latest findings, I don't think we should land this for beta 4
> (or even m-c really)
> 
> 1) It's not happening for all Flip4Mac video (Comment 35), and the patch
> will disable the plugin entirely (Comment 22)
We are contacting them asking them why the plugin doesn't fall back to using CoreGraphics like it's intended. Using CA/ICA in 10.5 was never a solution and should of never been allowed.
> 2) Chromium disabled Core Animation on 10.5 (Comment 28), but video seems to
> play fine for Marcia in Chrome on the same machine (Comment 35)
I tested with http://www.pny.com/movies and the video did not play correctly. If the video doesn't play correctly with CoreGraphics I think that is a separate issue. Where it does play correctly is presumably because CA/ICA isn't available and it falls back correctly to CG.
> 3) This affects even Firefox 4 (Comment 21)
It does because CA/ICA should of been blocked there on 10.5 starting in FF4.0 but was never a problem because no plug-in was using it to my knowledge.
> 
> As Christian suggests, we should understand why this is a top crash now, and
> subsequently try to find a root cause.

I'm fine with waiting to understand the issue more before landing this on beta but I feel that the fix will at least in part include this fix and most probably some outreach with plugin vendors. Therefore I think we should keep this fix on trunk.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Henrik Skupin (:whimboo) from comment #41)
> Can we be sure that users have been crashing on youtube? Means can we always
> trust the included URL for crashes? What happens i.e. if the crash happens
> in a background tab. Do we get this? Ted, I think you are the best person to
> ask for.

The URL data is not 100% reliable. We try to set the URL when a navigation takes place, or when the user switches tabs, but if a background tab crashes it's likely that the data will be wrong. It's also possible that this interacts badly with in-process plugins, I don't actually know how that works.

FWIW, if you look at the "Modules" tab of the crash in comment 0, Flip4Mac is not present. There are a number of other plugins there, like Flash Player, QuickTime, and some Google stuff.
Adding need URLs keyword to get some URLs. The youtube site references were comments from users, not actual URLs.
Keywords: needURLs
Here are some recent URLs:

 10 about:blank
      6 http://biggovernment.com/kavrech/2012/01/01/lets-move-a-government-initiative-to-lie-to-kids/
      6 http://apps.facebook.com/livescrabble/?ref=bookmarks&count=0&fb_source=bookmarks_apps&fb_bmpos=1_0
      5 \N
      5 http://www.manager.co.th/Entertainment/ViewNews.aspx?NewsID=9550000000742
      3 http://blogs.lanacion.com.ar/maldito-tango/maestros/luto-por-la-muerte-de-andrea-missa
      2 http://www.manager.co.th/Entertainment/ViewNews.aspx?NewsID=9540000162567
      2 http://www.lecturespositives.schoupinou.com/
      2 http://www.honolulu.gov/multimed/waikiki.asp
      2 http://twistyshardfree.com/watch.php?tag=p9tWZ5F6B2
      2 https://www.birchbox.com/video/perfume-dos-and-donts/
      2 https://apps.facebook.com/tetris_battle/?ref=bookmarks&count=0&fb_source=bookmarks_apps&fb_bmpos=1_0
      2 http://kitchenplanner.ikea.com/fr/UI/Pages/VPUI.htm
      2 http://ch131.com/
      2 http://audio.rambler.ru/player?id=vesti
      1 http://www.youtube.com/watch?v=Lw86NUNxgRg
      1 http://www.youtube.com/watch?v=K9CDfFGfCu4
      1 http://www.youtube.com/watch?v=BbLCOIqXCtw&feature=fvwrel
      1 http://www.youtube.com/watch?v=0UNKApfAt_o
      1 http://www.youtube.com/signin?action_handle_signin=true&nomobiletemp=1&hl=en_US&next=%2Fwatch%3Fv%3D9ULPym86nS0%26oref%3Dhttp%253A%252F%252Fwww.youtube.com%252Fresults%253Fsearch_query%253Dedge%252Bof%252Bdesire%252Bjohn%252Bmayer%2526oq%253Dedge%252Bof%2
      1 http://www.yogwf.com/viewtopic.php?f=17&t=31338
      1 http://www.yahoo.co.jp/_ylh=X3oDMTB0NWxnaGxsBF9TAzIwNzcyOTYyNjUEdGlkAzEyBHRtcGwDZ2Ex/f/topics/top/6/*-http://dailynews.yahoo.co.jp/fc/sports/marathon/?1325598956
      1 http://www.weightwatchers.co.uk/plan/mgr/PlanManager.aspx?deepLink=deepLinkToFoodTracker
      1 http://www.vidxden.com/404.html
      1 http://www.photoways.com/studio?creation_id=21880289
      1 http://www.photogriffon.com/photographes-du-monde/50-jill-greenberg/jill-greenberg-1.html
      1 http://www.palottery.state.pa.us/
      1 http://www.odatv.com/
      1 http://www.lyricsfreak.com/d/doors/love+her+madly_20042701.html
      1 http://www.jambajuice.ph/
      1 http://www.itsallgay.eu/
      1 http://www.google.ca/search?q=get+fit+guy&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
      1 http://www.belongilfields.com.au/
      1 https://www.birchbox.com/shop/best-sellers/ybf-beauty-automatic-universal-taupe-eyebrow-pencil
      1 http://ohana.me/
      1 http://mylifewithedanamia.blogspot.com/?zx=cee4ff7306b0f265
      1 http://jambajuice.ph/
      1 http://galleries.adult-empire.com/55/5528/adult-3d-porn-03/42/3d-porn-movie-02.wmv
      1 http://fhg.pixandvideo.com/v2284/images/movie_1.wmv
      1 http://fhg.dacash.com/gals/newul/video/19/8/gallery/movie1.wmv
      1 http://dailyburn.com/
Keywords: needURLs
(In reply to Henrik Skupin (:whimboo) from comment #40)
> (In reply to Alex Keybl [:akeybl] from comment #37)
> > Given Marcia's latest findings, I don't think we should land this for beta 4
> > (or even m-c really)
> 
> So it has been landed now on m-c. Shall we backout the patch?

We can leave this in for now if we're still considering as an uplift candidate.

I have 2 questions that I hope QA can clarify my understanding of

1) What is the user experience for tested plugins before taking this patch?
my  understanding is that:
* Flip4Mac crashes on some videos on 10.5
* external users appear to be crashing without Flip4Mac, we suspect Flash /may/ also be affected (we haven't been able to repro)
2) What is the user experience for tested plugins after taking this patch?
my understanding is that:
* Flip4Mac is completely disabled
* we have not been able to repro a non-Flip4Mac crash, but we also haven't found any regressions with the try build
(In reply to Alex Keybl [:akeybl] from comment #46)
> * Flip4Mac crashes on some videos on 10.5

For me it crashes for any video which gets played with Flip4Mac. I'm still waiting for an answer from Marcia (see comment 40) which would prove it otherwise.

> * Flip4Mac is completely disabled

So far as I can see yes.

> * we have not been able to repro a non-Flip4Mac crash, but we also haven't
> found any regressions with the try build

Benoit, do you know other plugins which make use of Core Animation? I think it would be great to compile a list those.
It happens mainly with Flash for Mac OS X 10.6 while this crash is on Mac OS X 10.5 (see http://people.mozilla.org/~rhelmer/temp/Firefox-10.0b3-correlation/20120111_Firefox_10.0-interesting-modules.txt):
  GLEngine@0x620cf|EXC_BAD_ACCESS / KERN_INVALID_ADDRESS (64 crashes)
     83% (53/64) vs.  20% (73/371) FlashPlayer-10.6
         83% (53/64) vs.  17% (63/371) 61128DE8A338EBBFEF3FBF66C456D66A0

The latest Flash version for Mac OS X 10.5 is 10.3 (see http://kb2.adobe.com/cps/908/cpsid_90893.html).

The Flip4Mac correlation is low:
     11% (7/64) vs.   3% (11/371) Flip4Mac WMV Plugin
That's a high correlation for a version that's does not appear to be compatible with OSX 10.5. Perhaps it would be worth while trying to reproduce the crash with OSX 10.5 and Flash-10.6. Can we get Adobe in the loop and ask them if they have any idea how users on OSX 10.5 might be ending up on Flash-10.6?

While the Flip4Mac correlation is low, I do believe it explains some crash, although not all. This issue appears to be have multiple causes so we should look for a single culprit. My guess could be that both Flash-10.6 and the latest Flip4Mac ask for Core Animation on 10.5 unintentionally.

If no one has a chance to test it this today I'll try it Monday on my 10.5 machine at work.
(In reply to Benoit Girard (:BenWa) from comment #49)
> This issue appears to be have multiple causes so we should
> look for a single culprit.
shouldn't*
I couldn't get my hands on Flash-10.6, the best resource I found was:
http://kb2.adobe.com/cps/142/tn_14266.html
(In reply to Benoit Girard (:BenWa) from comment #51)
> I couldn't get my hands on Flash-10.6, the best resource I found was:
> http://kb2.adobe.com/cps/142/tn_14266.html

Smadayag, could you please help out? Thanks.
i believe the last supported plugin for OS X 10.5 is 10,3,183,11.  you can download the archive in the technote URL above.

the way it worked prior to Flash Player 11 is the both plugins are actually installed but based on the OS, we determined and serve the correct plugin(10.4-10.5 carbon or 10.6 CA).

to find the 10.6 plugin open and view the package contents of the installed Flash Player.plugin.

Path: Flash Player.plugin > Contents > Plugins

hope that helps...
During the channel meeting today it was requested that qa also try to find out a regression range for this bug. It seems they want to understand why this started happening in the 10 timeframe.
See https://crash-stats.mozilla.com/report/list?query_search=signature&query_type=contains&reason_type=contains&range_value=4&range_unit=weeks&date=2011-12-10&hang_type=any&process_type=any&signature=GLEngine%400x620cf
According to the Table and Graph tabs, it first appeared in 10.0a2/20111206 and, according to the Reports tab, it first appeared in 10.0a2/20111207.
There's a bug in Socorro.

Guessing the Reports tab is right, the regression range is:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=12c96ed8154d&tochange=ca986aa710c7
(In reply to Scoobidiver from comment #55)
> There's a bug in Socorro.
I filed bug 718958 for that.

There's also one crash in 10.0a2/20111130 so the regression range might also be:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=58b665c43963&tochange=53b1db0a0d4b
(In reply to Scoobidiver from comment #56)
> There's also one crash in 10.0a2/20111130 so the regression range might also
> be:

Please see my comment 21. This crash has been started with Firefox 4. Nothing we would be able to get from any 10.0a2 crash reports.
This crash with the latest version of Flip4Mac starts between the builds 2010032203 and 2010032303.

WORKS: http://hg.mozilla.org/mozilla-central/rev/e9b7e0b5821d
FAILS: http://hg.mozilla.org/mozilla-central/rev/e9312d05488f

Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e9b7e0b5821d&tochange=e9312d05488f

Add support for the NPAPI Core Animation drawing model. b=497225 r=josh sr=roc
author	Benoit Girard <bgirard@mozilla.com>
	Tue Mar 23 20:51:11 2010 -0400 (at Tue Mar 23 20:51:11 2010 -0400)

With Flip4Mac there is no crash with version 2.3.8, so I still think its related to their reimplementation of the plugin for Firefox.

Regarding Flash I will contact some of the crash reporters and ask which version they have installed exactly.
Socorro does not show the flash version in the module section on Mac so as Henrik suggests in Comment 58 we would have to get it from the reporters. You cannot install Flash 11 on 10.5 from what I can see, so it would have to be an earlier version for sure.
I have contacted 5 reporters who have frequently sent crash reports to us. Waiting for feedback now.
The fix for Flip4Mac involves the patch I landed on trunk to stop the crashing, and an update that Flip4Mac will have to make.
(In reply to Benoit Girard (:BenWa) from comment #61)
> The fix for Flip4Mac involves the patch I landed on trunk to stop the
> crashing, and an update that Flip4Mac will have to make.

Benoit, have we been in touch with the devs of Flip4Mac yet?
Yes, they're working on an update.
(In reply to Henrik Skupin (:whimboo) from comment #58)
> This crash with the latest version of Flip4Mac starts between the builds
> 2010032203 and 2010032303.

Thanks for pulling this together Henrik.

(In reply to Benoit Girard (:BenWa) from comment #61)
> The fix for Flip4Mac involves the patch I landed on trunk to stop the
> crashing, and an update that Flip4Mac will have to make.

In the interest of still trying to fix this for FF10, can we identify what portion of

> Add support for the NPAPI Core Animation drawing model. b=497225 r=josh
> sr=roc
> author	Benoit Girard <bgirard@mozilla.com>
> 	Tue Mar 23 20:51:11 2010 -0400 (at Tue Mar 23 20:51:11 2010 -0400)

is tickling the crash and fix that for beta 6 (going to build 1/23)?
(In reply to Alex Keybl [:akeybl] from comment #64)
> In the interest of still trying to fix this for FF10, can we identify what
> portion of
> 
> > Add support for the NPAPI Core Animation drawing model. b=497225 r=josh
> > sr=roc
> > author	Benoit Girard <bgirard@mozilla.com>
> > 	Tue Mar 23 20:51:11 2010 -0400 (at Tue Mar 23 20:51:11 2010 -0400)
> 
> is tickling the crash and fix that for beta 6 (going to build 1/23)?

This change adds a fundamental new drawing model, it's not tickling the crash. NPAPI advertises a list of feature supported, one of which is Core Animation drawing model. The crash is that the plugin incorrectly asks for a feature on 10.5 which leads to a crash. The proper fix is not allowing them to use it on 10.5 which is https://bugzilla.mozilla.org/show_bug.cgi?id=711564#attach_587189 that already landed on trunk.
(In reply to Benoit Girard (:BenWa) from comment #65)
> This change adds a fundamental new drawing model, it's not tickling the
> crash. NPAPI advertises a list of feature supported, one of which is Core
> Animation drawing model. The crash is that the plugin incorrectly asks for a
> feature on 10.5 which leads to a crash. The proper fix is not allowing them

Just to note. With the second latest version of Flip4Mac users don't see anything. So disabling it for the latest version on 10.5 doesn't change anything for them but stops those nasty crashes.
(In reply to Henrik Skupin (:whimboo) from comment #66)
> Just to note. With the second latest version of Flip4Mac users don't see
> anything. So disabling it for the latest version on 10.5 doesn't change
> anything for them but stops those nasty crashes.

Henrik, can you clarify? Did you use the try build in https://bugzilla.mozilla.org/show_bug.cgi?id=711564#c10 with the latest version of Flip4Mac and verify that video is working across the board without crashes?
(In reply to Alex Keybl [:akeybl] from comment #67)
> Henrik, can you clarify? Did you use the try build in
> https://bugzilla.mozilla.org/show_bug.cgi?id=711564#c10 with the latest
> version of Flip4Mac and verify that video is working across the board
> without crashes?

No, with the latest version of Flip4Mac on 10.5 you will not be able to watch video, because we disabled it. Without Benoit's patch people would crash. Older versions of Flip4Mac do not crash because they are not using CA (right Benoit?). Those users will probably only see an empty plugin container. That's what I can see on my box.

Regarding Flash I got feedback from one reporter. His crash report is:
https://crash-stats.mozilla.com/report/index/73299870-b527-449c-ab3e-29c112120112

Here are the installed plugins:

    File: QuickTime Plugin.plugin
    Version: 7.6.1

    File: Flash Player.plugin
    Version: 11.1.102.55
    Shockwave Flash 11.1 r102

    File: FacebookVideoCalling.webplugin
    Version: 1.1.0.12
    Facebook Video Calling by Skype

    File: OVSHelper.plugin
    Version: 1.1
    DivX VOD Helper Plug-in

    File: Silverlight.plugin
    Version: 5.0.61118.0
    5.0.61118.0

    File: JavaPlugin2_NPAPI.plugin
    Version: 12.9.0
    Java Plug-In 2 for NPAPI Browsers

As you can see he is on 10.5.8 and he got Flash 11.1 installed on his system. I will tell him how to fix it but also want to ask how Flash 11 has been installed. Seems like the version check during its installation failed. Anything else you want to know?
(In reply to Henrik Skupin (:whimboo) from comment #68)
> Older versions of Flip4Mac do not crash because they are not using CA
> (right Benoit?). 

That's correct.

I'd like to know if they crash every time with Flash 11.1.102.55 and if reverting to the correct flash version fixes the crash.
Comment on attachment 587189 [details] [diff] [review]
Disable Core Animation on 10.5 v2

[Approval Request Comment]
Regression caused by (bug #): Bug in Flip4Mac update, users running flash version not supported for 10.5
User impact if declined: Crash with the current Flip4Mac plugin and some unsupported version for Flash.
Testing completed (on m-c, etc.): This has landed on m-c for 12 days and no crashes have been reported since.
Risk to taking this patch (and alternatives if risky): Flip4Mac will not render on 10.5 until the plug-in is fixed.
Attachment #587189 - Flags: approval-mozilla-beta?
Attachment #587189 - Flags: approval-mozilla-aurora?
The one user who replied to my questions regarding the crash with Flash doesn't know how and when Flash got upgraded to 11.0 on his box. As he stated it doesn't crash on all sites but mainly on universe.chacha.com and channelfireball.com.
There are some PPC users who are using a modified Flash 10.1 plugin that bumps the reported version number to 11. The hack is very simple, so I imagine there is an Intel version that does the same thing for 10.5.
(In reply to Cameron Kaiser from comment #72)
> There are some PPC users who are using a modified Flash 10.1 plugin that
> bumps the reported version number to 11. The hack is very simple, so I

Do you have a link where the hack is described?
In early B5 data, this ranks as the top Mac crash - I also added another signature that I saw.

I believe Alex can help with the question in Comment 73 and it will be something we try in the lab.
Crash Signature: [@ GLEngine@0x620cf ] → [@ GLEngine@0x620cf ] [@ GLEngine@0x606ef ]
Both Marcia and I have attempted to install Flash v11 on 10.5 without success. We also found no mention of the hack that Cameron mentioned in comment#72 when searching on the web.

Marcia's going to do some exploratory testing with 10.5 to make sure there are no adverse affects on Flash or other plugins, and then 10.6 to make sure that CA is still enabled without regressions.

Barring any major issues we find, we intend to approve for Aurora/Beta in the next 1-2 hours.
(In reply to Alex Keybl [:akeybl] from comment #75)
> Both Marcia and I have attempted to install Flash v11 on 10.5 without
> success. We also found no mention of the hack that Cameron mentioned in
> comment#72 when searching on the web.

I have found the following when searching for a hack at least for the PPC case:
https://discussions.apple.com/thread/3558589?start=0&tstart=0

I haven't digged into because I don't have time for, but hopefully Marcia could do.
One thing I noticed while loading some sites is that the Shockwave plugin Version: 11.6.3r633 does not get recognized while running the build in Comment 18. It works in other versions of Firefox so I assuming that Core Animation comes into play here?
I was able to install the hack in Version 76 and I can get that version of Flash to show (Version: 11.1.102.55). But I haven't yet generated a crash.
(In reply to Marcia Knous [:marcia] from comment #77)
> One thing I noticed while loading some sites is that the Shockwave plugin
> Version: 11.6.3r633 does not get recognized while running the build in
> Comment 18. It works in other versions of Firefox so I assuming that Core
> Animation comes into play here?

This must be on 10.5? As a general rule Core Graphics is used for ads and Core Animation is used for video (youtube) and games.
Yes, testing on 10.5.8 with miniclip.com site playing the Saloon game. It doesn;t work in the try build (cannot recognize the plugin) but works on other versions of Firefox such as B5.

(In reply to Benoit Girard (:BenWa) from comment #79)
> (In reply to Marcia Knous [:marcia] from comment #77)
> > One thing I noticed while loading some sites is that the Shockwave plugin
> > Version: 11.6.3r633 does not get recognized while running the build in
> > Comment 18. It works in other versions of Firefox so I assuming that Core
> > Animation comes into play here?
> 
> This must be on 10.5? As a general rule Core Graphics is used for ads and
> Core Animation is used for video (youtube) and games.
We've requested a report in bug 720534 to help us understand what percentage of crashes are caused by

* Flash Player 11 (unsupported) on 10.5
* Flip4Mac (broken) on 10.5

If a majority of the crashes are correlated with these software versions, we will not take action that disables unrelated plugins for the sake of this subset of crashing users.
We just heard back about 720534 but the results were not conclusive. Even still, I don't think we should take this patch for our final beta. Here are our unknowns:

* Why aren't we seeing this crash on FF9.0.1 with the latest version of F4M?
* What is the full list of plugins that will be disabled on 10.5 if we take this patch (we know of F4M and Shockwave currently)?
* Why can't we reproduce the crash with Flash?

This seems too risky for our final beta.
Attachment #587189 - Flags: approval-mozilla-beta?
Attachment #587189 - Flags: approval-mozilla-beta-
Attachment #587189 - Flags: approval-mozilla-aurora?
Attachment #587189 - Flags: approval-mozilla-aurora-
Regarding disabled plugins, I did test the latest nightly Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:12.0a1) Gecko/20120123 Firefox/12.0a1 on 10.5.8 and the miniclip site does work. It did not work in the build in Comment 18 since the plugin was not recognized. http://www.miniclip.com/games/saloon-brawl/en/ was the site I was testing - so I did not get consistent results there.

As far as the flash reproducibility, could it be hardware related?

I agree with the assessment in Comment 82 that we still have some unknowns to consider so close to final.
The crash can be easily reproduced with a modified Flash version as given in comment 76 by opening the following URL and waiting until the commercial has been finished playing and the real video starts. It will crash immediately at this time. Here my crash report:

https://crash-stats.mozilla.com/report/index/bp-6ed1e99c-4e62-47ac-98c6-c3b402120124

Steps:
1. Modify your Flash Player
2. Open http://www.playtv.fr/television/#skynews
3. Wait until the commercial has been finished playing

I have asked some more users for their feedback.
Wow, now I even crash with the latest Firefox 10 Beta with the original Flash player (FlashPlayer-10.4-10.5) on that given site. Here the crash report:

https://crash-stats.mozilla.com/report/index/bp-0287d3a0-7432-4a68-aae9-0f9582120124

With a Nightly build we only disable Flash for content which makes use of CA. Videos on YouTube or other platforms still work for me.
Status: REOPENED → ASSIGNED
(In reply to Marcia Knous [:marcia] from comment #83)
> It did not work in the build in
> Comment 18 since the plugin was not recognized.

Do you have any idea why the plugin was not recognized? Is it recognized with the latest Beta but not my try build?
In Comment 83 I was testing with the hacked version of flash (forcing the 11 version on Mac 10.5). If I retest this with the latest shipping version of Flash, both your try build and FF 10.b.6 play http://www.miniclip.com/games/saloon-brawl/en/ fine.

(In reply to Benoit Girard (:BenWa) from comment #86)
> (In reply to Marcia Knous [:marcia] from comment #83)
> > It did not work in the build in
> > Comment 18 since the plugin was not recognized.
> 
> Do you have any idea why the plugin was not recognized? Is it recognized
> with the latest Beta but not my try build?
As requested, here are my results testing with 9.0.1 on a 10.5 machine in the lab:

STR:
1. Install the hack described in Comment 76
2. Load http://www.playtv.fr/television/#skynews
3. The browser freezes, but I don't crash. In the player content area I see the Flip for Mac logo
(In reply to Marcia Knous [:marcia] from comment #88)
> 3. The browser freezes, but I don't crash.

Does it jank or does it permanently freeze?
From IRC Marcia confirms that it hangs permanently which would explain why we see less crashes on 9.0.1 since a hang wont be reported.
I'm working with a white MBP 13" from early 2007. Again, could be hardware related if Firefox crashes or freezes?
Yes, your video card could make a big difference here. 2007 would be an intel correct?
It's a Intel GMA 950 (Device ID: 0x27a2, Rev. ID: 0x0003).
Benoit and I had a phone conference and he indicated he would create a Google doc so we could sort out the hardware differences here. My two lab machines are both iMacs.
Crash Signature: [@ GLEngine@0x620cf ] [@ GLEngine@0x606ef ] → [@ GLEngine@0x620cf ] [@ GLEngine@0x606ef ] [@ -[CARenderer dealloc] ]
Try run for 368d2791227d is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=368d2791227d
Results (out of 5 total builds):
    failure: 5
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-368d2791227d
Try run for d9f98ab3c299 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=d9f98ab3c299
Results (out of 50 total builds):
    success: 45
    warnings: 2
    failure: 3
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-d9f98ab3c299
 Timed out after 06 hours without completing.
Try run for 90c1e659a517 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=90c1e659a517
Results (out of 26 total builds):
    success: 20
    warnings: 6
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-90c1e659a517
 Timed out after 06 hours without completing.
(In reply to Mozilla RelEng Bot from comment #98)
> Builds (or logs if builds failed) available at:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.
> com-90c1e659a517

This build has preference:
'plugins.tenfive.coreanimation.enabled' that can be toggled. Browser restart is recommended in case the plugin caches the value.
I have filled out my results for one of the 10.5 machines in the QA lab.

(In reply to Benoit Girard (:BenWa) from comment #95)
> https://docs.google.com/spreadsheet/
> ccc?key=0AvYoHvO43KG_dGMxc3F2UHpadHo3elpkWTU2THV4cGc#gid=0
Comment on attachment 587189 [details] [diff] [review]
Disable Core Animation on 10.5 v2

[Triage Comment]
Upon further review, we've decided to approve for Aurora 11 to get testing coverage with our Beta population - we'll watch for fallout. Our decision on beta 10 stands due to  inconsistencies in testing, the fact that FF9 and prior were affected, and where we are in FF10 (between beta 6 and RC).
Attachment #587189 - Flags: approval-mozilla-aurora- → approval-mozilla-aurora+
Please land no later than tomorrow, as our next merge date is Tuesday.
Benoit, can you please give a wrap-up for the status of the bug? It's now marked as fixed for 12/11 but not for 13. Have we reached the goal here or do you expect more work to do?
Core Animation is disabled on 10.5 on trunk already and Flip4Mac is working on an update.

(In reply to Ed Morley [:edmorley] from comment #39)
> https://hg.mozilla.org/mozilla-central/rev/8eb5fd522461

I don't expect to see this crash post FF10, if we see crashes that prove otherwise please reopen.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Whiteboard: [qa+]
Crash Signature: [@ GLEngine@0x620cf ] [@ GLEngine@0x606ef ] [@ -[CARenderer dealloc] ] → [@ GLEngine@0x620cf ] [@ GLEngine@0x606ef ] [@ -[CARenderer dealloc] ] [@ @0x0 | -[CARenderer dealloc] ]
I've verified this on 
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:11.0) Gecko/20100101 Firefox/11.0 beta 4

I've followed the steps from comment19, comment20 and comment21 and Firefox 11 beta 4 isn't crashing anymore.

This bug is verified fixed on Firefox 11 Beta.
Whiteboard: [qa+] → [qa+][qa!:11]
No crashes on 
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:12.0) Gecko/20100101 Firefox/12.0 beta 2 using the steps from comment19, comment20 and comment21.

Setting resolution to Verified Fixed.
Status: RESOLVED → VERIFIED
Whiteboard: [qa+][qa!:11] → [qa!]
It's currently #1 top crasher in 10.0.4esr on Mac OS X.
(In reply to Scoobidiver from comment #108)
> It's currently #1 top crasher in 10.0.4esr on Mac OS X.

According to MXR 10.5 doesn't have the fix here. Please file a new bug and CC' me. I will ask for approval to uplift this fix to 10-ESR.
(In reply to Benoit Girard (:BenWa) from comment #109)
> According to MXR 10.5 doesn't have the fix here. Please file a new bug and
> CC' me. I will ask for approval to uplift this fix to 10-ESR.
The process is to ask the ESR uplift in the original bug, not in a dedicated one.
Ok sure, 100+ comment bugs are noisy. I'm pulling mozilla-esr10 to get a patch that applies cleanly, I'll post a mozilla-esr10 version soon.
Comment on attachment 587189 [details] [diff] [review]
Disable Core Animation on 10.5 v2

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
Top crasher for ESR, fix has been successfully landed and verified the fix on central, aurora for 3 months without a regression. The fix has since shipped in Firefox 12.

User impact if declined: Crashes on 10.5 when a Core Animation plugin is requested.

Risk to taking this patch (and alternatives if risky): Low, this fix has already shipped.

String changes made by this patch: None
Attachment #587189 - Flags: approval-mozilla-esr10?
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Comment on attachment 587189 [details] [diff] [review]
Disable Core Animation on 10.5 v2

low risk and fixes an ESR topcrash, looks good to me.
Attachment #587189 - Flags: approval-mozilla-esr10? → approval-mozilla-esr10+
Vlad, can you please verify this fixed with the latest-mozilla-esr10 nightly?
Whiteboard: [qa!] → [qa+]
I've verified this on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:10.0.5esrpre) Gecko/20120522 Firefox/10.0.5esrpre

I've used the steps from comment19, comment20 and comment21 and Firefox didn't crash.
Setting the flag to Verified on ESR.

What I noticed instead is the fact that the site: http://www.pny.com/movies is loading very slowly and sometimes takes a lot of time to actually select a movie. Randomly the trailers are displayed black even though the clips started playing. 
Perhaps this has something to do with my Mac configuration and it will be great if someone else can also check the functionality of the site.

I have the following plugins installed:
Flip4Mac Media Plug-in 2.4.2.4
Silverlight Plug-in 5.1.10411.0

Disabled Core Animation on 10.5 v2 but same hanging issue is existing.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: