Closed Bug 655117 Opened 9 years ago Closed 8 years ago

Citrix ICA Client Version: 12.1.0.30 (npicaN.dll) crashes other plugins

Categories

(Core :: Plug-ins, defect)

x86_64
Windows 7
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla9
Tracking Status
firefox8 --- fixed

People

(Reporter: plee82, Assigned: bbondy)

Details

(Keywords: verified-beta, Whiteboard: [qa+])

Attachments

(1 file, 1 obsolete file)

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1

When the Citrix ICA plugin gets loaded as a plugin-container no other plugin is able to start. Once the Citrix ICA plugin starts, plugin-containers for flash, silverlight, etc will not be able to start and will hang the browser for minutes. I was able to fix the issue by modifying about:config and adding dom.ipc.plugins.enabled.npican.dll to false.

Reproducible: Always

Steps to Reproduce:
1. Install the Citrix ICA plugin (XenApp Online plugin 12.1.0.30)
2. Navigate to rsactx.jmfamily.com. The plugin container will get started.
3. Now go to any website using Flash or Silverlight.
4. Browser will hang.

Actual Results:  
Browser hangs.

Expected Results:  
Flash or silverlight plugin should start smoothly.

As mentioned above, if I disable the plugin-container for the Citrix plugin, all other plugins are able to start with no issues.
Component: General → Plug-ins
Product: Firefox → Core
QA Contact: general → plugins
Version: unspecified → Trunk
How does one obtain the plugin? Are you proficient at using visual studio? If so, you can use the Mozilla symbol server to get a stack trace of the hang: https://developer.mozilla.org/En/Using_the_Mozilla_symbol_server
I am sorry I am not very familiar with Visual Studio. The plugin can be downloaded from http://www.citrix.com/English/ss/downloads/details.asp?downloadId=2304987&productId=186#top

I noticed the Version is set to "Trunk". Shouldn't this be the 2.0 branch? This is happening in Firefox 4.0.1
I see the same behavior with the Citrix plugin. Stack trace for pluginContainer during hang follows:

>	ntdll.dll!_NtWaitForMultipleObjects@20()  + 0x15 bytes	
 	ntdll.dll!_NtWaitForMultipleObjects@20()  + 0x15 bytes	
 	user32.dll!_CallHookWithSEH@16()  + 0x33 bytes	
 	kernel32.dll!_WaitForMultipleObjectsExImplementation@20()  + 0x8e bytes	
 	user32.dll!_RealMsgWaitForMultipleObjectsEx@20()  + 0xe2 bytes	
 	xul.dll!base::MessagePumpForUI::WaitForWork()  Line 268	C++
 	xul.dll!base::MessagePumpForUI::DoRunLoop()  Line 238 + 0x6 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate, base::MessagePumpWin::Dispatcher * dispatcher)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate)  Line 78 + 0xc bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219 + 0x9 bytes	C++
 	xul.dll!MessageLoop::RunHandler()  + 0x1e5628 bytes	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!XRE_InitChildProcess(int aArgc, char * * aArgv, GeckoProcessType aProcess)  Line 519	C++
 	plugin-container.exe!wmain(int argc, wchar_t * * argv)  Line 128 + 0x33 bytes	C++
 	plugin-container.exe!__tmainCRTStartup()  Line 591 + 0x19 bytes	C
 	kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes	
 	ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes	
 	ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes
same issue with FF 5 and 5.0.1, disable solve it for now. 
Please add this info to the Mozilla Plugin Check.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0.1) Gecko/20100101 Firefox/5.0.1
Is there an update on this issue? I have tried IE 9 (which uses a separate process for plugins as well) and it works fine. I believe this issue is serious because for some reason the ICA plugin is able to interfere with other plugins in Firefox even if they are ran in different processes.
I'm definitely having this problem too with citrix ICA
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0

Citrix ICA Client

    File: npicaN.dll
    Version: 11.2.0.31560
    Citrix ICA Client Plugin (Win32)

Firefox will hang for a while and then come back to life if I do something like open a new tab (probably when they try to call flash).  It will come back to life after being "not responding" for a while but the flash still won't work without a browser restart.
Installed the latest Citrix ICA online plugin(xenapp 6.5 receiver) and same issue occurs. The version of npical.dll on the 6.5 XenApp version is 13.0.0.6685. Tested on Stable, Beta, Aurora and Nightly.
Assignee: nobody → netzen
I can visit flash sites fine, but when clicking on Add-ons Manager then Check to see if your plugins are up to date I can reproduce the browser hang.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
On a side note I am using Citrix ICA Client 1.2.1.44.1 because I couldn't find an older version. 

I can reproduce always with the "check for plugin update" hang on both FF6 and fully updated Nightly build for both release and debug builds.
The "check for plugin update" loads this plugin which will hang after the Citrix plugin is loaded.  
The "check for plugin update" loads:
"C:\Program Files (x86)\Java\jre6\bin\new_plugin\npdeployJava1.dll"

Unfortunately though as soon as I use my own builds of both Firefox-Release tip and mozilla-central tip, the problem never happens.
Here is my .mozconfig for your review bsmedberg, maybe you can see a reason why the debug/release Nightly's, and official release can reproduce but I can't with my build:
> ac_add_options --enable-application=browser
> ac_add_options --enable-debug
> ac_add_options --disable-static
> ac_add_options --disable-optimize
> ac_add_options --enable-tests
> ac_add_options --enable-logging
> ac_add_options --enable-debug-symbols

I did setup the symbol server though and after the problem reproduces I'm able to attach a debugger to get the call stack.  The below call stack is from the problem happening in FF6:

>ntdll.dll!_ZwWaitForSingleObject@12()  + 0x15 bytes	
> ntdll.dll!_ZwWaitForSingleObject@12()  + 0x15 bytes	
> kernel32.dll!_WaitForSingleObjectExImplementation@12()  + 0x43 bytes	
> kernel32.dll!_WaitForSingleObject@8()  + 0x12 bytes	
> nspr4.dll!_PR_MD_WAIT_CV(_MDCVar * cv, _MDLock * lock, unsigned int timeout)  Line 282	C
> nspr4.dll!_PR_WaitCondVar(PRThread * thread, PRCondVar * cvar, PRLock * lock, unsigned int timeout)  Line 205	C
> nspr4.dll!PR_WaitCondVar(PRCondVar * cvar, unsigned int timeout)  Line 547 + 0xb bytes	C
> xul.dll!mozilla::CondVar::Wait(unsigned int interval)  Line 103 + 0xd bytes	C++
> xul.dll!mozilla::ipc::GeckoChildProcessHost::SyncLaunch(std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > aExtraOpts, int aTimeoutMs, base::ProcessArchitecture arch)  Line 299	C++
> xul.dll!mozilla::plugins::PluginProcessParent::Launch(int timeoutMs)  Line 111 + 0x1b bytes	C++
> xul.dll!mozilla::plugins::PluginModuleParent::LoadModule(const char * aFilePath)  Line 98	C++
> xul.dll!GetNewPluginLibrary(nsPluginTag * aPluginTag)  Line 451 + 0x8 bytes	C++
> xul.dll!nsNPAPIPlugin::CreatePlugin(nsPluginTag * aPluginTag, nsNPAPIPlugin * * aResult)  Line 472 + 0x8 bytes	C++
> xul.dll!CreateNPAPIPlugin(nsPluginTag * aPluginTag, nsNPAPIPlugin * * aOutNPAPIPlugin)  Line 1708 + 0xb bytes	C++
> xul.dll!nsPluginHost::EnsurePluginLoaded(nsPluginTag * plugin)  Line 1718	C++
> xul.dll!nsPluginHost::GetPlugin(const char * aMimeType, nsNPAPIPlugin * * aPlugin)  Line 1750	C++
> xul.dll!nsPluginHost::TrySetUpPluginInstance(const char * aMimeType, nsIURI * aURL, nsIPluginInstanceOwner * aOwner)  Line 1325	C++
> xul.dll!nsPluginHost::SetUpPluginInstance(const char * aMimeType, nsIURI * aURL, nsIPluginInstanceOwner * aOwner)  Line 1251	C++
> xul.dll!nsPluginHost::InstantiateEmbeddedPlugin(const char * aMimeType, nsIURI * aURL, nsIPluginInstanceOwner * aOwner, int aAllowOpeningStreams)  Line 1084	C++
...

So I guess the hang is after the Citrix plugin is loaded, with the while loop at:
http://mxr.mozilla.org/mozilla-central/source/ipc/glue/GeckoChildProcessHost.cpp#307
and the problem is with OnChannelConnected never being called (NotificationType::CHILD_PROCESS_HOST_CONNECTED never happening).

This is the first time I work with plugin code though, so I hope there is something I can change in my .mozconfig that will allow me to debug the problem.
Some additional information:

- Only 2 plugins are loaded 
- The first plugin loaded is the Citrix one once I visit rsactx.jmfamily.com
- The second plugin loaded is the Java one once I check for updates.
- I notice in task manager that the plugin-container for the Java one disappears just as fast as it is started. 
- But the main loop is waiting for NotificationType::CHILD_PROCESS_HOST_CONNECTED on the main thread for a couple minutes completely hanging the whole browser. 

So there are 2 problems here:
1) What is the Citrix problem which causes other plugins to crash on startup.
2) I think a separate bug should be posted for the ability to check in that while loop http://mxr.mozilla.org/mozilla-central/source/ipc/glue/GeckoChildProcessHost.cpp#307 to see if the started process still exists or not.  If it doesn't exist we can break out early and not hang the browser.
I know why the 2nd process is closing right away now after the Citrix plugin is loaded.

Towards the start of plugin-container startup, inside XRE_InitChildProcess we are calling
SetRemoteExceptionHandler passing in a normal looking pipe name.

This function creates a new ExceptionHandler and stores it into a previously NULL gExceptionHandler.
But then the function returns gExceptionHandler->IsOutOfProcess().

> bool IsOutOfProcess() const { return crash_generation_client_.get() != NULL; }

But in this case crash_generation_client_ is NULL and so XRE_InitChildProcess exits prematurely with a return code of 1.

But why can't the pipe be connected? 
I assume that something in the citrix plugin is closing the pipe (some security thing?)

It seems to me that we could have similar hangs from any anti virus software that doesn't allow pipe connections. Because the plugin-container will close right away and the main firefox app will hang there for a couple minutes waiting for it to connect.
> nsresult
> XRE_InitChildProcess(int aArgc,
>                      char* aArgv[],
>                      GeckoProcessType aProcess)
> {
...
>  // on windows and mac, |crashReporterArg| is the named pipe on which the
>  // server is listening for requests, or "-" if crash reporting is
>  // disabled.
>  if (0 != strcmp("-", crashReporterArg)
>      && !XRE_SetRemoteExceptionHandler(crashReporterArg))
>    return 1; <-- we exit plugin-container after this.
I think it's probably better we don't exit the process just because crash reporting is not available by the way (not available because it can't connect to the named pipe).  So I guess that will probably be part of the fix unless there are objections.
That is fascinating, in a horrifying sort of way. I guess we've never expected that third-party code would do such horrible things to us? Disabling our crash reporting is very poor form.
For reference, the mime type for this plug-in is application/x-ica.
For reference, the change that was needed in Comment 9 was in application.ini.  
It now allows me to reproduce and debug:

...
[Crash Reporter]
Enabled=1
...

This change has the effect of passing in a named pipe parameter to plugin-container for the crash reporting that was causing the hang. 

I think the .mozconifg line which would have enabled this in my application.ini is: export MOZILLA_OFFICIAL=1
As of Comment 16 I can reproduce this on my own builds.  So I was able to verify the problem is gone now with this patch.
Attachment #557941 - Flags: review?(benjamin)
Comment on attachment 557941 [details] [diff] [review]
Fix for citrix plugin causing hangs in other plugins

I don't think you should use fprintf here. It's ok to add a NS_WARNING, or remove it entirely. r=me with that change
Attachment #557941 - Flags: review?(benjamin) → review+
OK will do, I only did by the way because the surrounding code was doing the same.
fprintf changed to NS_WARNING && minor formatting fix.
Attachment #557941 - Attachment is obsolete: true
Attachment #558667 - Flags: review?(benjamin)
Attachment #558667 - Flags: review?(benjamin) → review+
http://hg.mozilla.org/mozilla-central/rev/cab14cba2f3d
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla9
Attachment #558667 - Flags: approval-mozilla-aurora?
Attachment #558667 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
can someone give a pointer for the binary containing this bug fix ?
I think it would be best to use an aurora build or Firefox beta build once the builds are available soon if you need early access.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0

Verified on Win7/64bit using the 8b1 build and crash/browser hang didn't reproduce with the following steps:
1. Installed latest version of plugin (13.0.0.6685) from http://www.citrix.com/English/ss/downloads/details.asp?downloadId=2315305&productId=1689163&ntref=clientcenter#top
2. Navigated to rsactx.jmfamily.com
3. Opened http://www.mono-1.com/monoface/main.html and youtube video in new tabs
4. Opened Addons Manager
5. Search for plugin updates

Could not verify on Aurora and Nightly as the plugin seems to be incompatible(Citrix is not displayed in Addons Manager)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0a1) Gecko/20111004 Firefox/10.0a1
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0a2) Gecko/20111004 Firefox/9.0a2
Keywords: verified-beta
Whiteboard: [qa+]
You need to log in before you can comment on or make changes to this bug.