Closed Bug 816976 Opened 7 years ago Closed 7 years ago

[adbe 3519647][10.8] crash in glcGetIOAccelService in Flash plugin-container process

Categories

(External Software Affecting Firefox :: Flash (Adobe), defect, P3, critical)

x86_64
macOS
defect

Tracking

(firefox17 unaffected, firefox18 affected, firefox19- affected, firefox20- affected, firefox21- affected)

RESOLVED FIXED
Tracking Status
firefox17 --- unaffected
firefox18 --- affected
firefox19 - affected
firefox20 - affected
firefox21 - affected

People

(Reporter: scoobidiver, Unassigned)

Details

(4 keywords, Whiteboard: [Flash 11.5, 11.6] [STR in comment #58][fixed in Flash 11.7.700.146])

Crash Data

Attachments

(3 files)

It first showed up in 18.0 and above on November 27 at 20:40 GMT.
It's currently #14 top crasher in 18.0b1 on Mac OS X.

Signature 	glcGetIOAccelService More Reports Search
UUID	60946f65-6cd4-4133-868b-3b23d2121129
Date Processed	2012-11-29 06:45:33
Process Type	plugin Version: Filename: Flash Player.plugin
Uptime	2593
Install Age	2.0 days since version was first installed.
Install Time	2012-11-27 07:13:52
Product	Firefox
Version	18.0
Build ID	20121121075611
Release Channel	beta
OS	Mac OS X
OS Version	10.8.2 12C60
Build Architecture	amd64
Build Architecture Info	family 6 model 58 stepping 9
Crash Reason	EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash Address	0x30163f00
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x 166GL Context? GL Context+ GL Layers? GL Layers+ 
Processor Notes 	/data/socorro/stackwalk/bin/exploitable: ERROR: unable to analyze dump
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x 166

Frame 	Module 	Signature 	Source
0 	OpenGL 	glcGetIOAccelService 	
1 	OpenGL 	CGLUpdateContext 	
2 	CoreGraphics 	CGSReconfigNotifierCalloutListInvokeAll 	
3 	CoreGraphics 	displayConfigFinalizedProc 	
4 	CoreGraphics 	displayAcceleratorChangedProc 	
5 	CoreGraphics 	CGSPostLocalNotification 	
6 	CoreGraphics 	notify_datagram_handler 	
7 	CoreGraphics 	CGSDispatchDatagramsFromStream 	
8 	CoreGraphics 	snarf_events 	
9 	CoreGraphics 	CGSGetNextEventRecordInternal 	
10 	CoreGraphics 	CGEventCreateNextEvent 	
11 	HIToolbox 	PullEventsFromWindowServerOnConnection 	
12 	CoreFoundation 	__CFMachPortPerform 	
13 	CoreFoundation 	__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ 	
14 	CoreFoundation 	__CFRunLoopDoSource1 	
15 	CoreFoundation 	__CFRunLoopRun 	
16 	CoreFoundation 	CFRunLoopRunSpecific 	
17 	HIToolbox 	RunCurrentEventLoopInMode 	
18 	HIToolbox 	ReceiveNextEventCommon 	
19 	HIToolbox 	BlockUntilNextEventMatchingListInMode 	
20 	AppKit 	_DPSNextEvent 	
21 	AppKit 	-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] 	
22 	AppKit 	-[NSApplication run] 	
23 	XUL 	base::MessagePumpNSApplication::DoRun 	ipc/chromium/src/base/message_pump_mac.mm:677
24 	XUL 	base::MessagePumpCFRunLoopBase::Run 	ipc/chromium/src/base/message_pump_mac.mm:213
25 	XUL 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:215
26 	XUL 	XRE_InitChildProcess 	toolkit/xre/nsEmbedFunctions.cpp:485
27 	plugin-container 	main 	ipc/app/MozillaRuntimeMain.cpp:48
28 	plugin-container 	plugin-container@0xee3

More reports at:
https://crash-stats.mozilla.com/report/list?signature=glcGetIOAccelService
(In reply to Scoobidiver from comment #0)
> It first showed up in 18.0 and above on November 27 at 20:40 GMT.
Notice that it's the release date of the latest Beta Flash 11.5: http://labsdownload.adobe.com/pub/labs/flashplatformruntimes/shared/air3-5_flashplayer11-5_releasenotes.pdf
Whiteboard: [Flash 11.5]
I'm not so sure Flash 11.5 is involved here.

I looked at the "debug identifiers" for the "Flash Player" and "FlashPlayer-10.6" modules for a couple of these crash records, and found that they match those for the current Flash release (11.5.502.110).

It's really too bad you need to look at the "debug identifier" to identify the version.  I think Socorro should add actual version numbers for the most popular plugins, especially Flash.

I'll post the debug IDs for the 11.5.502.110 version of the two Flash modules in my next comment.
> I'm not so sure Flash 11.5 is involved here.

Flash 11.5 -> the latest Flash 11.5 beta
Debug IDs for Flash 11.5.502.110:

Flash Player
i386   BC6B4FD5C5D5733904DFD4B250ABEBCC0
x86_64 3A7B2454F9E04AD8C7E8D9400CCCB57C0

FlashPlayer-10.6
i386   2774DAA3A8D428BD2D104A604D6A4C9A0
x86_64 33B5B76EF1128CA29F0EB85607763E3E0

These were generated using dump_syms, which can be found in $objdir/dist/host/bin of a custom Firefox build.
(In reply to Steven Michaud from comment #2)
> It's really too bad you need to look at the "debug identifier" to identify
> the version.  I think Socorro should add actual version numbers for the most
> popular plugins, especially Flash.

Hm, the linked data for the correlation script hasn't been updated in a while either:
http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/
I wonder if this is still being used.
All the reports I looked at have AppleIntelHD4000GraphicsGLDriver - perhaps that is a factor.
For the Flash version <-> debugID connections, there is a table for those in Socorro, if we update that one, Socorro would be able to display those Flash versions correctly in their crash report displays and summary reports.
It's #5 top crasher in 18.0.2, #2 in 19.0b6 and 20.0a2 and #4 in 21.0a1 on Mac OS X.

Stack traces are various.

One comment talk about waking up from the sleep mode.
Hm, i haven't looked into the IOSurface based implementation yet, but would it be possible that the associated context doesn't survive sleep/wakeup and needs some "before sleep" handling?

Steps for this would be great if some variation of sleep/wakeup with Flash active is the issue here.
Keywords: steps-wanted
Unlikely that this has regressed since 18.0.2. We'll need to investigate for first resolution in FF20.

I'm hoping QA can take a stab at reproducing this after FF19 is released on Tuesday.
Keywords: qawanted
QA Contact: jbecerra
These are 100% associated with the Flash plugin, as best I can tell.
Summary: [10.8] crash in glcGetIOAccelService → [10.8] crash in glcGetIOAccelService in Flash plugin-container process
With FF 20 and up, you can see the actual Flash plugin version under "Process Type" in the Details tab.

Checking the 10 most recent crashes, I see the following versions:

11.5.502.149 (7 crashes)
11.6.602.167 (3 crashes)

I suspect the fact that most crashes are associated with 11.5.502.149 isn't particularly meaningful.  11.6 hasn't been out very long, and I suspect most people haven't yet upgraded.
Whiteboard: [Flash 11.5] → [Flash 11.5, 11.6]
Priority: -- → P2
(Following up comment #7)

> All the reports I looked at have AppleIntelHD4000GraphicsGLDriver

Of the 10 reports I looked at for comment #13, 7 had this driver.  But 3 had the following drivers:

AMDRadeonX3000GLDriver
AppleIntelHD3000GraphicsGLDriver
It may be worthwhile trying to find out more about the code path on which these crashes happen, as I tried to do for bug 804606.  Specifically, it'd be very interesting to know if the "event" being processed as the crash happens comes from the Flash plugin.

I can get to that eventually.  But it'll probably be several weeks before I have the time.
Tracking for FF20, if a low risk fix was available we could take uplift until Beta 4 goes to build on March 12. Steven, I'm assigning to you but if you don't have time before March 12 let's come up with other ideas of who might be able to take this on.
Assignee: nobody → smichaud
Juan, have you had a chance to look at this yet? If not I'll assign to someone at Softvision to look at this overnight.
Frankly, I think this is most likely to be a Flash bug.  We may still be able to work around it (as I did for another Flash bug at bug 804606).  But the chances of us (that is me) having that kind of fix ready for FF 20 are no better than 50/50.

It'd help a lot if someone could figure out how to reproduce these crashes.  So I think that should be the priority for QA.  A clue is provided by the single comment about waking from sleep mode mentioned in comment #9:

"i think the crash occurred when i woke my machine from sleep mode. not sure though."

I know it's very difficult to figure out how to reproduce a bug with the little information available here.  But it's worth a try.
Keywords: needURLs
This is the only URL I saw in the list:

http://www.i-have-a-dreambox.com/wbb2/hmportal.php
Keywords: needURLs
Ioana, can you have your team attempt to reproduce this overnight?
QA Contact: jbecerra → ioana.budnar
Tested on Mac OS X 10.8.2, FF 18.0.2 and 20b1, Flash 11.5.502.149 and 11.6.602.167.
I tried several times sleep/wake with the link in comment 19, several youtube videos, flash games sites, facebook, but couldn't crash Firefox.
I noticed Firefox moved very slowly with ~30 flash tabs opened, sometimes small hangs occurred and for couple of times "quit" option from dock bar didn't work, I had to force quit to be able to really quit.
FWIW, I managed to get a plugin crash once: https://crash-stats.mozilla.com/report/index/bp-31c423e4-b357-4e8a-9701-0f9f62130222
Looking in about:crashes I see there are other crashes, but none of them are valid. Each crash links to http://crash-stats.mozilla.com/about/throttling
I tried to crash FF beta 20 and FF 18.0 using the info from comment 18 and comment 19, I tested the sleep test on different intervals (no crashes), loaded some heavy loaded flash pages, the browser did not crash only frozen for a few seconds and moved very slow until I closed some of the pages.
I have been trying some of the ideas in comment #21 and #22 on a couple of machines, one desktop and one laptop, running 10.8 and Flash 11.6.latest, but so far I haven't been able to reproduce the crashes.
When I have time to start really working on this (sometime in the next few weeks), if I'm able to gain a better understanding of the code path on which the crashes happen, this may provide additional clues about how to reproduce them.
OK, I've started doing an analysis of the code patch on which these crashes happen.

I've already found that that glcGetIOAccelService isn't normally called at all in the Flash plugin-container process -- whether or not Flash is set to use hardware acceleration.  This greatly increases the likelihood that this is a Flash bug, which (like bug 804606) only happens when Flash detects certain hardware.  (See comment #14.)
(Following up comment #25)

I tested on a MacPro, running OS X 10.8.2, with ATI Radeon HD 5770 graphics hardware.

I've got a Retina MBP with Intel HD Graphics 4000 graphics hardware, and another MBP with Intel HD Graphics 3000 hardware.  I'll test on them next.
(Following up comment 26)

Actually I don't see calls to glcGetIOAccelService (in the plugin-container process or the main process) running on that hardware, either.

So I'm beginning to think this might be an Apple bug.

The following might be relevant:
http://code.google.com/p/chromium/issues/detail?id=155938
This might also be relevant (note the similarity of the stack below the top):

http://forums.plexapp.com/index.php/topic/49462-crash-upon-start-on-retina-mbp/

Do we have any way of knowing whether these crashes only happen on Retina MBPs (if they do)?
I've found that displayAcceleratorChangedProc gets called if you switch to using a different kind of graphics harware (say on a laptop, which usually has two kinds).

I used gfxCardStatus to force a switch, but I suppose it can also happen spontaneously.

I'll keep digging.
Steven, is there anything QA can do to assist you? So far we've been unable to reproduce this.
No, I can't yet reproduce it.  And I've had to put it off to work on bug 837539, which seems more urgent (and which I'd managed to forget about).

But I've got more to say on this bug, and more leads to follow ... once I get back to it.  When that happens I may be able to provide clues to help narrow the search for STR, or even (if I'm very lucky) to find them myself.
(In reply to Steven Michaud from comment #33)
> No, I can't yet reproduce it.  And I've had to put it off to work on bug
> 837539, which seems more urgent (and which I'd managed to forget about).
> 
> But I've got more to say on this bug, and more leads to follow ... once I
> get back to it.  When that happens I may be able to provide clues to help
> narrow the search for STR, or even (if I'm very lucky) to find them myself.

Okay, thanks Steven. I'll leave qawanted on this bug until you come back. Feel free to drop the keyword if you think there's nothing left for us to do.
Hey Guys,

just encountered this crash https://crash-stats.mozilla.com/report/index/bp-25646c8f-5d0a-4d11-94ff-133a72130306

There was no user interaction or something, the only steps i did was listening to the bayern3 webradio via their player http://mediathek-audio.br.de/index.html?playeronly=true&channelId=b3

then after awhile i noticed that the radio stream has stopped and checked the player window and there was the crash notice.

Maybe related to the crash that they update webcam pics there from time to time
Hi Carsten,
Can you reproduce the crash again ?
Flags: needinfo?(cbook)
Tomcat, was this on a laptop with multiple kinds of graphics hardware?  Could the crash have been triggered by your machine starting to sleep?  Did you have an external monitor connected, and if so which monitor was FF on?

Was the Flash plugin playing the audio invisible, or offscreen?
(In reply to Steven Michaud from comment #37)
> Tomcat, was this on a laptop with multiple kinds of graphics hardware? 

its a   Model Identifier:	MacBookPro9,1

and seem the maschine has 2 graphics hardware
  Chipset Model:	Intel HD Graphics 4000

  Chipset Model:	NVIDIA GeForce GT 650M
  Type:	GPU
  Bus:	PCIe
  PCIe Lane Width:	x8
  VRAM (Total):	1024 MB
  Vendor:	NVIDIA (0x10de)
  Device ID:	0x0fd5
  Revision ID:	0x00a2
  ROM Revision:	3682
  gMux Version:	1.9.23


> Could the crash have been triggered by your machine starting to sleep?  Did
> you have an external monitor connected, and if so which monitor was FF on?
>
i have a external monitor, but was not connected at that time and also not powered on.

 
> Was the Flash plugin playing the audio invisible, or offscreen?

was invisible in a different window (kind of background window). Was only listening to the audio stream
Flags: needinfo?(cbook)
>> Was the Flash plugin playing the audio invisible, or offscreen?
>
> was invisible in a different window (kind of background window). Was
> only listening to the audio stream

This may be significant.

Please try the following steps.  With luck they'll reproduce the
crash.

1) Download and run gfxCardStatus from http://gfx.io/.

2) Make your audio stream run from its invisible plugin.

3) Use gfxCardStatus to switch your video hardware from what's
   currently in use.

   Usually programs on a MacBook Pro will default to using the
   "integrated video hardware" (which in your case is, I believe, the
   Intel HD Graphics 4000 hardware).  Try switching to "Discrete only"
   and "Integrated only".

   With luck you'll crash!
QA Contact: ioana.budnar → paul.silaghi
Flags: needinfo?(cbook)
(Following up comment #39)

After further investigation, I no longer have any reason to believe these crashes are more likely to happen with invisible plugins.

> http://mediathek-audio.br.de/index.html?playeronly=true&channelId=b3

And in any case I see only a single (Flash) plugin on this page, which isn't invisible.

I have a couple of MBPs, one with Intel HD 3000 graphics hardware built in, and the other with Intel HD 4000 graphics hardware built in (both of which are associated with this bug -- see comment #14).  I tested with this page on both of these machines, with OS X 10.8.2 running, and I didn't see crashes on either.  I used gfxCardStatus to switch video hardware, I used a hot corner to turn on the screensaver, I closed and opened the laptop lid, and I messed around with Firefox's fullscreen mode.

Tomcat, do you have any plugins, extensions and so forth that might make a difference?  Or any utilities that, like gfxCardStatus, run "in the background"?

My machines all have pretty much the standard plugins -- Flash, QuickTime, Silverlight, Java.  And none of the FF configurations I tested with have any extensions.
I'm going to have to give up on this bug, at least for the time being.  It's unlikely I'll be able to understand it any better until and unless we find a way to reproduce it.

Nonetheless these crashes happen in Flash code, and this particular Flash code is only ever exercised on OS X 10.8.  So, though I can't (yet) prove it, I think the chances are very good that this is a Flash bug.  I'll have more to say about this in subsequent comments (where, among other things, I'll explain why our crash stacks make it look like the crashes happen in system code).

I've figured out a way to stop the crashing Flash code from ever being called.  Doing this doesn't appear to have any bad consequences in my tests.  But the patch is very hackish ... even for me :-)  So I think we'd only want to use it under one or both of the following circumstances:

1) As a test, to see if my patch actually stops the crashes (or has bad side effects).

2) If the test from option 1 has passed, we're truly desperate for a fix, and we still haven't learned anything more about this bug.
Figuring out this bug is somewhat complicated by the fact that there are two different versions of OS X 10.8.2 -- build 12C3012 (which so far I've only seen on my Retina MBP) and build 12C60 (on my non-Retina MBP and MacPro).  For some reason Socorro only contains symbols for stacks generated on machines running build 12C60.

But the OpenGL framework seems to be the same in both builds, because all this bug's crashes take place at offset 0xacb from the address of its glcGetIOAccelService symbol.  This offset *isn't* inside the glcGetIOAccelService method (rather it's inside a non-exported method some distance below it, which for convenience I'll call sub_32f4).  But glcGetIOAccelService is the nearest public/exported symbol, so our crash stacks say these crashes happen in glcGetIOAccelService.

(I wasted a bunch of time trying to find out when glcGetIOAccelService gets called (the answer is only when the display subsystem is seriously broken), and how it's related to this bug (the answer is "not at all").  Oh well.)

What's more interesting is that all this bug's crashes happen when sub_32f4 is called from CGLUpdateContext, when the latter is in turn called (indirectly) from CGSReconfigNotifierCalloutListInvokeAll -- in other words when OS X calls each of the handlers that have been registered with it using CGDisplayRegisterReconfigurationCallback().  This happens whenever a machine's displays are "reconfigured" -- for example by switching from one kind of graphics hardware to another, or by plugging in or unplugging an external monitor.

Current versions of the Flash plugin (presumably starting with Flash 11.5) call CGDisplayRegisterReconfigurationCallback to register a callback.  But for some reason they only do this on OS X 10.8.X (not on earlier versions of OS X).  The callback is very short (only 48 bytes long), and is at offset 0x56b5b0 in the current version (11.6.602.171) of the FlashPlayer-10.6 binary (for convenience I'll call it sub_56b5b0).  The OS calls this method with a userInfo == a pointer to a CGLContextObj object.  When (flags & kCGDisplaySetModeFlag) is non-zero, sub_56b5b0 "calls" CGLUpdateContext on this CGLContextObj.  Or rather (and more precisely) it *jmp*s to the address of CGLUpdateContext that's been returned by a call to dlsym.  This use of "jmp" explains why sub_56b5b0 doesn't show up in our crash stacks, even under a false name.

Once I'd figured all this out, I tried to reproduce this bug by triggering "display reconfigurations" under all kinds of different circumstances.  All of them failed.

I'm sure these crashes happen when CGLUpdateContext tries to reference a bad pointer to a CGLContextObj -- presumably one that's been deleted.  But I haven't been able to figure out how this happens.  The one thing I'm reasonably confident of is that this isn't because of anything that Firefox has done -- Firefox code never calls CGLReleaseContext or CGLDestroyContext or CGLClearContext on that object.

In my next comment I'll post a debug logging patch that can be used to substantiate everything I say here.
Here's the debug logging patch I promised in my previous comment.  I've also started a tryserver build.

Note that it contains code (ifdefed out, in ReconfigurationCallback()) that can be used to "synthesize" this bug's crashes:  If you recompile with that code, the first "display reconfiguration" triggers a crash whose crash stack is exactly like this bug's crash stacks.
Here's the source for an interpose library to debug how Flash behaves in other browsers.  With it I found that Flash does *not* register a display reconfiguration callback when hosted by Chrome, but *does* when hosted by Opera (version 12.14, testing on OS X 10.8.2).  This probably means this bug's crashes are also happening with Opera.  I'll check tomorrow (if nobody's beaten me to it).

This interpose library doesn't work with Safari -- because Safari somehow blocks dyld interposing :-(  If need be I can probably do a binary hack of dyld to get around this ... but that would be time consuming.

To test with this interpose library, you first need to compile it (just run make).  Then set the DYLD_INSERT_LIBRARIES environment variable as follows at a Terminal prompt:

export DYLD_INSERT_LIBRARIES=[/full/path/to/]interpose.dylib

Then run Chrome or Opera (or Firefox) from the same Terminal prompt.
> This probably means this bug's crashes are also happening with
> Opera.  I'll check tomorrow (if nobody's beaten me to it).

Opera's bug tracking system is closed, or in other words not
publicly accessible :-(

https://bugs.opera.com/secure/Dashboard.jspa
(In reply to Steven Michaud from comment #45)
> Opera's bug tracking system is closed
likely definitively for the engine part: http://business.opera.com/press/releases/general/opera-gears-up-at-300-million-users
(In reply to Steven Michaud from comment #40)
> (Following up comment #39)
> 
> After further investigation, I no longer have any reason to believe these
> crashes are more likely to happen with invisible plugins.
> 
> > http://mediathek-audio.br.de/index.html?playeronly=true&channelId=b3
> 
> And in any case I see only a single (Flash) plugin on this page, which isn't
> invisible.
> 
> I have a couple of MBPs, one with Intel HD 3000 graphics hardware built in,
> and the other with Intel HD 4000 graphics hardware built in (both of which
> are associated with this bug -- see comment #14).  I tested with this page
> on both of these machines, with OS X 10.8.2 running, and I didn't see
> crashes on either.  I used gfxCardStatus to switch video hardware, I used a
> hot corner to turn on the screensaver, I closed and opened the laptop lid,
> and I messed around with Firefox's fullscreen mode.
> 
> Tomcat, do you have any plugins, extensions and so forth that might make a
> difference?  Or any utilities that, like gfxCardStatus, run "in the
> background"?
> 


Hey Steven,

plugins are only flash and the disabled java on 10.8. The only other background app i had was viscocity, so standard...also was not able to reproduce so far
Flags: needinfo?(cbook)
Thanks, Tomcat, for the info.  Too bad it provides so few clues :-(

> viscocity

I assume you mean this:  http://www.sparklabs.com/viscosity/

I can't think how it could be related.
Here's the hackish workaround I promised in comment #41.

Though it looks kind of scary, it should actually be quite safe.  Current Flash plugins (11.5 and 11.6) only contain one call to CGDisplayRegisterReconfigurationCallback() -- the one we want to prevent.  And Flash only ever calls it on OS X 10.8.  Furthermore this call only exists on OS X 10.7 and up (i.e. not on OS X 10.6).

If people think it's OK, I'd like to land this patch on the trunk as an experiment -- to see if it gets rid of the crashes, and to shake out any bad effects it may have.  Then in another couple of weeks we'll need to decide whether we want to keep it on the trunk, and even promote it to one or more of the branches.

For now I'm only asking for feedback.  If the general idea is approved, then I'll ask Benoit for a review.

I've started a tryserver build, which should be available in a few hours.
Attachment #724098 - Flags: feedback?(lsblakk)
Attachment #724098 - Flags: feedback?(bgirard)
Apparently there's a Flash 11.7 beta:
http://labs.adobe.com/downloads/flashplayer.html

I'll test with it and see what happens.
With respect to this bug, the Flash 11.7 beta I tested (version 11.7.700.128) seems to behave exactly the same as the release versions (11.5 and 11.6).  My hacky workaround also worked just fine on it.

(I tested on a Retina MBP running OS X 10.8.2.)
Comment on attachment 724098 [details] [diff] [review]
Hackish workaround

I've figured out how to reproduce these crashes, in both Firefox and Opera.

The STR are so obvious it's a bit embarrassing ... though there's a twist.  I also now have proof that this is a Flash bug.

Oddly, though, this patch may still be the best workaround.  But let me think about that.

More news in a bit :-)
Attachment #724098 - Flags: feedback?(lsblakk)
Attachment #724098 - Flags: feedback?(bgirard)
STR for Firefox or Opera:

1) Open a Terminal prompt and enter the following two commands:
   a) export NO_MAC_JEMALLOC=1 (only needed for Firefox)
   b) DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
2) Run Firefox or Opera from the Terminal prompt.
   Make sure the current page contains no Flash plugins.
3) Load a Flash plugin, for example http://mirrors.creativecommons.org/getcreative/.
3) Press the Back button to go back to the previous page.
   This should also unload the last Flash plugin.
4) Do something to cause your display(s) to be reconfigured.  Either:
   a) Plug in or unplug an external monitor.
   b) Use gfxCardStatus (http://gfx.io/) to switch from "discrete" to
      "integrated", or vice versa.

In step 4 you'll crash the Flash plugin.  In Opera this will be obvious -- you'll get their crash dump submission dialog.  In Firefox nothing will appear to happen.  But if you go to about:crashes you'll notice a new entry.

I tested on a Retina MBP running OS X 10.8.2.
Whiteboard: [Flash 11.5, 11.6] → [Flash 11.5, 11.6] [STR in comment #55]
The Flash bug is that Flash calls CGDisplayRegisterReconfigurationCallback() without ever calling CGDisplayRemoveReconfigurationCallback().  When the last Flash plugin is unloaded, this leaves a dangling pointer to a CGLContextObj, which will get passed to the Flash plugin's display reconfiguration callback whenever your displays get reconfigured.

Most of the time this doesn't cause a crash.  Maybe the memory formerly occupied by the CGLContextObj hasn't yet been reallocated.  Or maybe (and more likely) the memory has been reallocated to some other kind of object.  The CGL apis (including CGLUpdateContext()) check whether or not they're dealing with a valid CGLContextObj, and won't normally crash unless the CGLContextObj pointer points to inaccessible memory.  (One of the possible values for CGLError is kCGLBadContext, which means an "invalid context object".)

Using libgmalloc forces crashes to happen whenever the code tries to dereference an invalid pointer.  It and Firefox's jemalloc don't get along, so you need to disable jemalloc to use libgmalloc with Firefox.
(Following up comment #55)

> b) DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib

Oops, this should be:

  b) export DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
The following STR also works (with either Firefox or Opera):

1) Open a Terminal prompt and enter the following two commands:
   a) export NO_MAC_JEMALLOC=1 (only needed for Firefox)
   b) export DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
2) Run Firefox or Opera from the Terminal prompt.
3) Visit http://mirrors.creativecommons.org/ and load one of the Flash
   plugins.
4) Press the Back button, then either load the same Flash plugin
   again, or load the other one.
5) Do something to cause your display(s) to be reconfigured.  Either:
   a) Plug in or unplug an external monitor.
   b) Use gfxCardStatus (http://gfx.io/) to switch from "discrete" to
      "integrated", or vice versa.

With this STR, the crash will be obvious (visible) in both Opera and
Firefox.
Whiteboard: [Flash 11.5, 11.6] [STR in comment #55] → [Flash 11.5, 11.6] [STR in comment #58]
smichaud, I think you have gone well above the call of duty here ;-) We really should have cc'ed our Adobe contacts and gotten them to look at this long before you were manually disassembling their code! smadayag, can you make sure there's a adbe report on file about this?

As a plugin crash that doesn't take down Firefox, I really don't think we need to track this for Firefox releases.
Assignee: smichaud → nobody
Component: Plug-ins → Flash (Adobe)
Flags: needinfo?(smadayag)
Priority: P2 → P3
Product: Core → Plugins
Version: 18 Branch → unspecified
> smichaud, I think you have gone well above the call of duty here ;-)

You're most welcome.

> We really should have cc'ed our Adobe contacts and gotten them to
> look at this long before you were manually disassembling their code!

I disagree.

Adobe's most likely response would be the one they gave at bug 804606
-- "we can't reproduce this, so it isn't our problem".

I needed to reverse engineer the Flash plugin to find out exactly what
kind of bug this is.  And (as it turned out) to find proof that this
is an Adobe bug.
Also note that this is a Mac topcrasher:  Currently #4 on the 20, 21 and 22 branches together.
Thank you for the in-depth analysis.  This is Adobe Internal 3519647.
Flags: needinfo?(smadayag)
Summary: [10.8] crash in glcGetIOAccelService in Flash plugin-container process → [adbe 3519647][10.8] crash in glcGetIOAccelService in Flash plugin-container process
Given comment 62, untracking for upcoming releases (no changes possible on our side).
> no changes possible on our side

This isn't quite true -- see comment #50.

But it'd be much better for Adobe to fix this bug than for us to work around it.
Our candidate fix should be in this week's beta drop; however, I'm not sure how to tell if it worked by looking at crash-stats because of the version detection issue on Mac.
> Our candidate fix should be in this week's beta drop

Very glad to hear it.  Please tell me when it's available and I'll
test it.

> however, I'm not sure how to tell if it worked by looking at
> crash-stats because of the version detection issue on Mac

I'm afraid I don't understand this.
Cool, thanks for the interest.

When you look at the aggregate crash stats for Flash Player on Mac in crash-stats, the versions are all detected as 0.0.0.0.  

Ideally, we'd see the crashes bucketed by the various plug-in versions that reported the crash, such that we can tell whether or not the crash was being reported in the versions equal or higher to the one with the fix.  This is what happens with Windows bugs today.

I think there's a debug handle we can pull from the candidate binary that Benjamin's team can check for with custom reports, but it's not something I can do from Adobe.  I believe that there's also a fix for this in the pipeline already, and I have a bug open for our next version on our side to have an engineer investigate whether or not there's anything we can do from our side to help. 

We typically publish beta builds on Tuesday or Wednesday.  They're announced via twitter at https://twitter.com/FlashPlayerBeta, or if you'd like a direct email notification, by subscribing to notifications on the Flash Player beta forums: http://forums.adobe.com/thread/981567?tstart=0

If you don't want to sign up for something, I'd just hit the download page on Thursday at http://www.adobe.com/go/flashplayerbeta/.
> When you look at the aggregate crash stats for Flash Player on Mac
> in crash-stats, the versions are all detected as 0.0.0.0.

Yes :-(

For Firefox versions 20 and up, the actual version number is available
on the Details page under Process Type.  For example (from
bp-81a55f6e-009e-4bc3-83c7-b40992130318):

plugin Shockwave Flash Version:11.6.602.180 Filename: Flash Player.plugin

But this isn't included in the aggregate crash stats.  And the number
of users of Flash betas is presumably low enough that it won't be
practical to check "by hand" for the glcGetIOAccelService crash's
disappearance.

As far as I know, even this version information is only listed for the
Flash plugin.  It's my fond hope we'll eventually do it for *all*
plugins that use the "bundle" format, and that the version numbers
will appear in the "regular" location, so that they *will* get
included in the aggregate crash stats.

But my STR from comment #58 should be reliable enough to tell whether
or not your fix "works".
(In reply to Steven Michaud from comment #68)
> As far as I know, even this version information is only listed for the
> Flash plugin.

We actually have the plugin name and version annotated in the crash reports now via bug 818664, see e.g. 44a62a52-d061-49c3-bd85-923682130314.
(In reply to Georg Fritzsche [:gfritzsche] [away Mar 8 - Mar 17] from comment #69)
> see e.g. 44a62a52-d061-49c3-bd85-923682130314.

bp-44a62a52-d061-49c3-bd85-923682130314
(In reply to Jeromie Clark from comment #65)
> I'm not sure how to tell if it worked by looking at crash-stats because of the version
> detection issue on Mac.
Currently, there are crashes in Flash 11.7.700.128 but none in Flash 11.7.700.141 where it's not yet fixed: https://crash-stats.mozilla.com/query/query?product=Firefox&platform=mac&query=glcGetIOAccelService&process_type=plugin&plugin_field=name&plugin_query_type=contains&plugin_query=Flash&do_query=1
OK, I stand corrected on both counts.  Bug 818664 (on the 20 branch and up) records version information for all CFBundle plugins (not just Flash), and it's possible to get aggregate crash stats (for FF 20 and up) by plugin name and version number.

But the plugin information gets written to a catchall "miscellaneous information" field in Breakpad -- not (for example) to where Breakpad normally records version information.  And it's therefore difficult to find this information or search on it.

A better fix, in the long term, would be for Breakpad itself to be able to read versions numbers out of CFBundles, which it currently can't do.
@scoobidiver - Looked at our source control and confirmed that the candidate fix is introduced in Flash Player 11.7.700.144.  We shipped 11.7.700.146 earlier today.  I'm not sure why it disappeared, but I'll continue to keep an eye out for it. 

Thanks for the pointer on the search.  :)
(In reply to Jeromie Clark from comment #73)
> I'm not sure why it disappeared, but I'll continue to keep an eye out for it. 
There are about 490 active daily Mac users in 20.0 Beta and above (see https://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=22.0a1&v[]=21.0a2&v[]=20.0b5&v[]=&hang_type=any&os[]=Mac+OS+X&date_range_type=report&submit=Generate). Assuming only 5% of them have installed Flash Beta, the probability that those 25 users hit this crash is low.
I just downloaded and tested with Flash 11.7.700.146 (which I downloaded from http://www.adobe.com/go/flashplayerbeta/) in a recent mozilla-central nightly on OS X 10.8.2.

I didn't see any crashes using my STR from comment #58.  I also noticed that every call to CGDisplayRegisterReconfigurationCallback() from the Flash plugin (when a plugin is loaded) is now matched by a call to CGDisplayRemoveReconfigurationCallback() (when the plugin is unloaded).

So yes, this bug seems to be fixed.

Thanks, Adobe people, for the quick work!

But of course the next question is, when will this fix appear in a Flash plugin release? :-)
Thanks for the feedback.  We don't comment publicly on future release dates; however, in general, we're shooting for a monthly release cadence, with a dot-version ever 3 months.  Flash Player 11.6 has been in the market for a while, which should inform the target window.

That said, we release builds in the context of navigating the business and technical needs of all major browser vendors, the realities of the security landscape, supporting new and emerging platforms where possible, and meeting our own business goals.  Problem-solving in our scheduling meetings is just as challenging as the problem-solving in our technical ones.  :)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Whiteboard: [Flash 11.5, 11.6] [STR in comment #58] → [Flash 11.5, 11.6] [STR in comment #58][fixed in Flash 11.7.602.146]
Whiteboard: [Flash 11.5, 11.6] [STR in comment #58][fixed in Flash 11.7.602.146] → [Flash 11.5, 11.6] [STR in comment #58][fixed in Flash 11.7.700.146]
I just got prompted to download and install Flash 11.7.700.169.  And checking at http://get.adobe.com/flashplayer/?promoid=JZEFT, this appears to be the current release version.

So, Jeromie, can you confirm that Adobe has now fixed this bug in a Flash release?
Flags: needinfo?(rajesh_g)
You need to log in before you can comment on or make changes to this bug.