971811 - plugin-container crashes with "dyld: could not load inserted library ''" when website with Flash content gets opened

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0 ID:20140210030201 CSet: ecf20a2484b6

Recently I have upgraded to Flash 12.0.0.44 on this machine and I was facing crashes of the plugin container when I tried to open web pages which contain Flash objects. It crashes a couple times in a row for e.g. our vidyo site. Sadly the crash reporter doesn't catch the crash, but the Apple crash reporter:

Process:         plugin-container [1521]
Path:            /Applications/Firefox/Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
Identifier:      plugin-container
Version:         1.0
Code Type:       X86-64 (Native)
Parent Process:  firefox [92052]
Responsible:     plugin-container [1521]
User ID:         501

Date/Time:       2014-02-12 17:28:57.332 +0100
OS Version:      Mac OS X 10.9.1 (13B42)
Report Version:  11
Anonymous UUID:  2A720B63-8C01-90A4-F56E-212360C1933D

Sleep/Wake UUID: 3A7A2142-6552-4711-B3DA-F0DBDFA9EBB6

Crashed Thread:  0

Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000

Application Specific Information:
dyld: launch, loading dependent libraries

Dyld Error Message:
  could not load inserted library '' because image not found

Binary Images:
       0x100000000 -        0x100000ff7 +org.mozilla.plugincontainer (1.0) <D4451DA4-92CC-3063-88AD-1041011E438C> /Applications/Firefox/Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
    0x7fff6caac000 -     0x7fff6cadf817  dyld (239.3) <D1DFCF3F-0B0C-332A-BCC0-87A851B570FF> /usr/lib/dyld

After restarting Nightly I'm no longer able to reproduce the problem. I tried to replace Flash while keeping Nightly open, but no way to get it to crash.

Comment 1

[Steven Michaud [:smichaud] (Retired)]

> Dyld Error Message:
>  could not load inserted library '' because image not found

This is very mysterious.  I could only find one other reference to it, concerning the "Google Chrome Helper EH" (https://twitter.com/moeffju/status/413636243474235392).

I doubt you'll see this crash again.  But if you do, please attach whatever information you can find.  Don't forget to look in the console log for interesting messages.

> I tried to replace Flash while keeping Nightly open

This may have had something to do with it.  Possibly your Flash upgrade/install was incomplete?

Comment 2

[Steven Michaud [:smichaud] (Retired)]

> Sadly the crash reporter doesn't catch the crash

Probably because this was a SIGTRAP "crash".

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There is nothing in the syslog.log which is related to this crash except the entry for the above crash report, which I have already attached.

Comment 4

[Georg Fritzsche [:gfritzsche]]

This looks like we might be using some outdated plugin information and thus loading the plugin library fails.
Was Nightly running during the Flash update? Did the Flash updater ask you to close or restart Nightly and did you do it?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Nightly was running and as stated in comment 0 I haven't restarted it since then. That's why I tried to replicate it by installing the newest version over again. But it might be that due to the same file, I don't get the crash. It might be worse testing from the last -> current version. Also there is no request from the Flash installer to restart the browser.

Comment 6

[Georg Fritzsche [:gfritzsche]]

Yes, quite likely that it only happens for actual updates.
Was this a manual update or an automatic one?

Comment 7

[Steven Michaud [:smichaud] (Retired)]

> Dyld Error Message:
>  could not load inserted library '' because image not found

By looking at the source code for dyld on http://opensource.apple.com, I found that this message has to do with the value of the DYLD_INSERT_LIBRARIES environment variable, if any.

http://opensource.apple.com/source/dyld/dyld-239.3/src/dyld.cpp

Search on "loadInsertedDylib".

I also found I can reproduce this crash by entering the following at the command line, then running Firefox (a recent m-c nightly) from the command line, then loading a Flash "movie" (such as one of those at http://mirrors.creativecommons.org/.

export DYLD_INSERT_LIBRARIES=;

So Henrik ... do you have that environment variable set by any chance?

And if not (as I presume), I wonder if the Flash installer sets DYLD_INSERT_LIBRARIES at some point, and perhaps then (normally) unsets it.

Comment 8

[Steven Michaud [:smichaud] (Retired)]

> export DYLD_INSERT_LIBRARIES=;

This also "works":

export DYLD_INSERT_LIBRARIES=

Both set DYLD_INSERT_LIBRARIES to an empty value, which presumably gets interpreted as an empty string by dyld code.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No, it's not set. So the Flash installer might be doing that probably.

Comment 10

[Steven Michaud [:smichaud] (Retired)]

I have an OS X 10.9.1 partition on which Flash hasn't yet been upgraded to its latest version.

Henrik, lease let us know *exactly* how you did your upgrade.  For example (as Georg asked above), was it an automatic or a manual upgrade?  And if it was a manual one, how did you convince the Flash updater to proceed without quitting Firefox?

Once I know *in sufficient detail* what you did, I'll try doing the same on my OS X 10.9.1 partition.

Another thing:

Look at your Flash Player system pref panel and let us know what setting you have for "Updates" in the advanced panel.  I see the following three choices.  I've chosen the second, but the first is the default value.

Allow Adobe to install updates (recommended)
Notify me to install updates
Never check for updates (not recommended)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Surprisingly the setting is set to allow Adobe to install updates. Usually I don't let it do something like that. So I'm not fully sure if it had already been upgraded. But I did an upgrade manually by running the updater while Firefox was open. And I didn't restart Firefox since then.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

So just to get this straight:

1) You've set Flash to upgrade automatically (to allow Adobe to install updates)
2) You upgraded manually by running the updater while Firefox was open/running

I assume the updater you ran was downloaded from http://get.adobe.com/flashplayer/.  Is that correct?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Totally correct, yes.

Comment 14

[Steven Michaud [:smichaud] (Retired)]

A few more things before I do my test:

1) Do you have the Profile Manager set to start whenever you start Firefox?
2) Which is your default browser:  Firefox release, m-c nightly, or something else?

Testing on a partition (running OS X 10.7.5) that already has the latest Flash version, I find that the Flash updater *doesn't* force FF to quit if you've set Flash to upgrade automatically.  But it does try to launch the "default browser" after it's finished.  And if the version of FF you're running is the default browser, it displays an error that FF is already running.

Comment 15

[Steven Michaud [:smichaud] (Retired)]

The Flash installer comes in two pieces -- what you download from Adobe and what it in turn downloads and runs.  After mounting them I grepped both for the string DYLD_INSERT_LIBRARIES and found no matches.

So it looks like the Flash installer isn't at fault here, and that this bug is due to some weird initial condition in Henrik's setup when he saw the crash.

If that's so, we'll likely never fully know what's going on here.

Henrik, please do answer my questions from comment #14.  But unless you can figure out how to reproduce it, or lots of others see it, it's probably best to just drop this bug.

Comment 16

[Steven Michaud [:smichaud] (Retired)]

For what it's worth, I tried updating my Flash player on my OS X 10.9.1 partition, and didn't see this bug.

Before I did it, I

1) Chose "allow Adobe to install updates" in the Flash Player pref panel.
2) Ran today's m-c nightly and made it my default browser, then left it running.

When the installer finished, it opened a "thank you" page in Firefox -- in other words I didn't get an error that FF was already running.  In all other respects what happened is exactly what happened when I redid the update on OS X 10.7.5.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Steven Michaud from comment #14)
> 1) Do you have the Profile Manager set to start whenever you start Firefox?

I have multiple profiles on that machine. So it always comes up, yes. Also NO_REMOTE is set.

> 2) Which is your default browser:  Firefox release, m-c nightly, or
> something else?

It's Nightly, the version I'm running in the last couple of days.

I might have to wait for the next update of the flash player to test this. I never have seen this before. I don't think something is wrong with my environment, given that I don't change it and Firefox is also getting started via the dock and not through a Terminal window, when it could inherit custom environmental variables.

Comment 18

[Steven Michaud [:smichaud] (Retired)]

If this ever happens again, I suspect it's because the plugin is setuid or setgid.

Comment 19

[Steven Michaud [:smichaud] (Retired)]

Another (probably more likely) possibility is that the plugin was codesigned with "entitlements".

http://stackoverflow.com/questions/12522837/dyld-insert-libraries-doesnt-work-for-app-signed-with-entitlements-on-mountain

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I hit this issue again today. Not sure when exactly I did a Flash update the past days, but the Firefox version I'm using now, I haven't had up for a couple of weeks. And Flash is always complaining about this issue when I'm loading https://www.op.fi.

Steven, how can I provide further details for you? I will leave this Firefox instance open for now, so I could check that later.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Also cc'ing Jeromie from the Adobe team for information.

Comment 22

[Steven Michaud [:smichaud] (Retired)]

> Steven, how can I provide further details for you?

What I'd really like to see is the environment the "Firefox version I'm using now" is running in (presuming it's this instance where plugin-container always crashes with "dyld: could not load inserted library" when you try to load Flash).  (By "environment" I mean the complete set of environment variables.)  But I can't think of a way to get at it.

Please leave the instance open for now, and I'll look around for a way to print out the environment of a running program (from another program).

Comment 23

[Steven Michaud [:smichaud] (Retired)]

(Following up comment #22)

Figured it out.  It's not hard at all.

From a Terminal prompt do the following:

ps -E [firefoxpid] > ff-env.txt

Then upload the results here.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I'm very sorry, but the Firefox instance was frozen due to bug 1000235. So I had to kill it. For the next start of the application the plugin-container crash doesn't happen anymore. So we have to wait for the next time it happens.

Comment 25

[Steven Michaud [:smichaud] (Retired)]

Oh well :-(

At least the problem's not urgent.

Comment 26

[Steven Michaud [:smichaud] (Retired)]

By the way, bug 476541 doesn't really "freeze" Firefox (it doesn't shut down the UI, it cripples it).  So you could have used the steps from comment #23 even after bug 476541 happened.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I restarted Firefox already an hour ago. So way earlier as when you commented the steps. Sorry.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: ff-env.txt

Steven, I finally hit this problem again when upgrading Flash. So here is what I did:

1. Had Firefox open to download the new package. Here I got triggered to do so by Flash itself.

2. Download the Flash installer and start the installation

3. Flash will request to close all Firefox instances

4. Download the latest update of Firefox and restarted Firefox

5. Quit Firefox 

6. Continued with the Flash installation / upgrade

7. Started Firefox 

After step 7 I can see this problem all the time (even in a loop) with Rosetta Stone. This website doesn't seem to continue, and ends-up in an infinite loop. As attached you can find the environment variables as attached to the Firefox process.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Steven Michaud from comment #7)
> export DYLD_INSERT_LIBRARIES=;
> 
> So Henrik ... do you have that environment variable set by any chance?
> 
> And if not (as I presume), I wonder if the Flash installer sets
> DYLD_INSERT_LIBRARIES at some point, and perhaps then (normally) unsets it.

Looking at the attached env variable output, this variable is set to nothing. If the installer sets it, how would Firefox pick it up? Maybe the installer is doing that system-wide, and the manual restart of Firefox picks it up? Then when the installer quits it removes it?

Comment 30

[Björn Kahl]

As another datapoint, same happened today to me.  Details:

OS: Mac OSX 10.6.8

Firefox: 24.7.0 (esr update channel)

Observation:

The plugin-container crashes in a loop, approx. one crash every 45 seconds with Firefox most of the time unresponsive (showing the "Beachball") and very short intervals in between where Firefox is responsive.

I have three repeating messages in the system log:
01.09.14 15:49:41	[0x0-0xee0ee].org.mozilla.firefox[4296]	WARNING: parent WaitForMessage() failed: 0x10004003 (ipc/rcv) timed out: file /builds/slave/rel-m-esr24-osx64_bld-00000000/build/ipc/glue/GeckoChildProcessHost.cpp, line 668

01.09.14 15:50:16	[0x0-0xee0ee].org.mozilla.firefox[4296]	dyld: could not load inserted library: 

01.09.14 15:50:16	ReportCrash[4624]	Saved crash report for plugin-container[4625] version ??? (???) to .../Library/Logs/DiagnosticReports/plugin-container_2014-09-01-155016_bkmp.crash

01.09.14 15:50:26	[0x0-0xee0ee].org.mozilla.firefox[4296]	WARNING: parent WaitForMessage() failed: 0x10004003 (ipc/rcv) timed out: file /builds/slave/rel-m-esr24-osx64_bld-00000000/build/ipc/glue/GeckoChildProcessHost.cpp, line 668


Steps that caused the situation:

- I had Firefox open for browsing when he Flash Update Reminder appeared, asking me if I want the install e new Flash version now or if I want to be reminded later.
- I left the reminder open, i.e. did not clicked either option
- Firefox was stable usable, including watching Flash videos
- I closed Firefox to prepare for installing a new Flash version (it typical wants to have Firefox, Safari and the System Preferences not running)
- Hit "install new flash" on the Flash Update Reminder
- Firefox started and showed the session selection dialog, asking we which session I want to continue (I have it configured to ask always)
- I hit cancel, since I just wanted to get the new Flash downloaded
- Firefox went to http://get.adobe.com/de/flashplayer/
- Firefox froze and above mentioned 45 seconds loop of frozen UI, plugin-container crash and short periods of usable UI started.

The environment contains the empty DYLD_INSERT_LIBRARIES variable.

Quitting Firefox and manuell restarting from the Dock produces a working Firefox, which can successfully go to http://get.adobe.com/de/flashplayer/ and download the new Flash.

Comment 31

[Steven Michaud [:smichaud] (Retired)]

> Looking at the attached env variable output, this variable [DYLD_INSERT_LIBRARIES] is set to nothing.

This is exactly what triggers the bug.  See comment #8.

Now we need to figure out why it happens.

I frankly am not going to have time for this bug for a while.  Could you try to track it down?

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I'm very limited too at the moment. I will see what I will get with the next update of Flash.

Comment 33

[Samuel Bronson]

Attachment: plugin-container_2015-05-17-150410_Behemoth.crash

My mother just hit this, with the following in the Firefox environment:

(gdb) print *(char**)environ @ 29
$9 = {0x7fff5fbffb41 "PATH=/usr/bin:/bin:/usr/sbin:/sbin", 0x7fff5fbffb64 "TMPDIR=/var/folders/ff/npl19n0d3j7_zt9qb56_n3vc0000gp/T/", 0x7fff5fbffb9d "SHELL=/bin/bash", 0x7fff5fbffbad "HOME=/Users/pambronson", 0x7fff5fbffbc4 "USER=pambronson", 0x7fff5fbffbd4 "LOGNAME=pambronson", 0x7fff5fbffbe7 "SSH_AUTH_SOCK=/tmp/launch-AXoHhS/Listeners", 0x7fff5fbffc12 "Apple_PubSub_Socket_Render=/tmp/launch-nTF2i8/Render", 0x1003c59e0 "MOZ_LAUNCHED_CHILD=", 0x1249e8bc0 "MOZ_CRASHREPORTER_EVENTS_DIRECTORY=/Users/pambronson/Library/Application Support/Firefox/Profiles/4eelqckj.default/crashes/events", 0x7fff5fbffcdd "SECURITYSESSIONID=186a4", 0x10bf570e0 "XRE_PROFILE_LOCAL_PATH=", 0x100315380 "MOZ_CRASHREPORTER_STRINGS_OVERRIDE=/Applications/Firefox.app/Contents/Resources/browser/crashreporter-override.ini", 0x10bf57120 "XRE_START_OFFLINE=", 0x7fff5fbffd93 "DYLD_LIBRARY_PATH=/Applications/Firefox.app/Contents/MacOS", 0x7fff5fbffdce "__CF_USER_TEXT_ENCODING=0x1F6:0:0", 0x7fff5fbffdf0 "COMMAND_MODE=unix2003", 0x10865fb20 "__CHECKFIX1436934=1", 0x10bcfc310 "XUL_APP_FILE=", 0x10bf57140 "XRE_BINARY_PATH=", 0x1003ad6a0 "MOZ_CRASHREPORTER_DATA_DIRECTORY=/Users/pambronson/Library/Application Support/Firefox/Crash Reports", 0x10bf570c0 "XRE_PROFILE_PATH=", 0x10bcfc300 "NO_EM_RESTART=", 0x7fff5fbffebf "MOZ_CRASHREPORTER_RESTART_ARG_2=", 0x1003c6250 "MOZ_CRASHREPORTER_RESTART_ARG_1=", 0x7fff5fbfff0d "DYLD_INSERT_LIBRARIES=", 0x10034f520 "MOZ_CRASHREPORTER_RESTART_ARG_0=/Applications/Firefox.app/Contents/MacOS/firefox", 0x10bf57100 "XRE_PROFILE_NAME=", 0x0}

For some reason, Firefox couldn't take a hint and tried this ~10 times about 10 seconds apart, taking in total about 2 minutes, before eventually giving up on it.

Comment 34

[Samuel Bronson]

Attachment: Behemoth's about:support info

Comment 35

[Steven Michaud [:smichaud] (Retired)]

Thanks, Samuel, for your information.

But (as best I can tell) it doesn't add anything to what we currently know.  We've known since comment #8 that setting DYLD_INSERT_LIBRARIES to an empty value triggers this bug.  But we still don't know why or how this happens.

Can your mother reproduce this consistently, or only intermittently?  If she can reproduce it consistently, does disabling all her extensions make it go away?  If so, please try to find out exactly which extension triggers the problem.

Comment 36

[Markus Stange [:mstange]]

Attachment: patch

I still don't know what causes an empty DYLD_INSERT_LIBRARIES env variable to be set on the parent process, but working around it is really simple and I don't see a reason not to do it.
I hit this bug today after updating Flash and it really is an extremely bad experience.

Comment 37

[Steven Michaud [:smichaud] (Retired)]

Comment on attachment 8620408 [details] [diff] [review]
patch

This looks fine to me.

Comment 38

[Markus Stange [:mstange]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9718bc90b5c2

Comment 39

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/9718bc90b5c2