1468250 - Open/save file browser crashes when Solidworks PDM 2018 has been installed

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P3)

Description

[abqsteve]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
Build ID: 20180605171542

Steps to reproduce:

Repro steps:
- Install Solidworks 2018 PDM Standard.  This affects windows file explorer by adding a special type of folder which is connected to the Solidworks PDM database.
- Start firefox
- Open https://send.firefox.com/
- Click "select a file to upload"
- Firefox crashes.  All instances disappear.  No error dialog is shown.

Details:
- FireFox 60.0.2 (64-bit)
- Windows 10 Enterprise, 64-bit
- Solidworks PDM Standard 2018

The issue is reproducible.  I have seen it consistently on two computers.

Comment 1

[abqsteve]

The crash seems to happen with any open or save file dialog.

When I restart Firefox, it appears to remember my tabs correctly.

On other systems at the office which do not have Solidworks PDM installed, I do not have the problem.

Comment 2

[roxleitan]

Hi abqsteve,

Could you please paste here the crash signature from about:crashes?

Thanks

Comment 3

[abqsteve]

Roxana,

Unfortunately, about:crashes only says "No crash reports have been submitted."

Comment 4

[abqsteve]

Reproduced with:
- FireFox 62.0.3 (64-bit)
  Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0
  20181001155545
- Solidworks 2018 SP4.0

(Previous report was with Solidworks SP3.)

Comment 5

[abqsteve]

Issue still present after restarting FireFox in safemode with all add-ins disabled.

Comment 6

[abqsteve]

Issue still present with Firefox 63.0.3 (64-bit) and Solidworks 2018 SP5.0.

Comment 7

[Nils]

Issue still present with Firefox Version 60.5.2esr (32 and 64 bit) and SolidWorks 2018 SP5.0
Any solutions?

Comment 8

[SKSSF]

I can confirm this bug (FF 65.0.2) and PDM Works Pro 2018 (in my case, this is not an issue with SolidWorks itself, it's PDM Works). If I log out of the PDM Works client (a file browser that resides in the Windows explorer UI), then there are no issues with crashes. If not, then FF will crash whenever a open/save file dialog box opens in FF. Thanks!

Comment 9

[Volkmar]

I had i similiar problem with thunderbird and learned, that everything is ok, if I start Thunderbird in win7 compatible mode. I did as the same with Firefox and I believe, that I'm able to open PDM-files in Thunderbird (test with wetransfer)

Comment 10

[Volkmar]

Correction to my Text below:
I had i similiar problem with Thunderbird and learned, that everything is ok, if I start Thunderbird in win7 compatible mode. I did as the same with Firefox and I believe, that I'm able to open PDM-files in Firefox (test with wetransfer)

Comment 11

[abqsteve]

@Volkmar: I'm not certain how to start Firefox in win7 compatible mode. If you would tell me how to do that, I will try to reproduce on my system. Thanks!

Comment 12

[abqsteve]

I confirmed that when running in windows 7 compatibility mode (by setting the windows shortcut properties), Firefox no longer crashes.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:marco, could you have a look please?

Comment 15

[SKSSF]

[Bump] I'm happy to submit whatever debug information the dev team needs to address this bug. Thank you!

Comment 16

[SKSSF]

FYI...I made my SolidWorks VAR (value added reseller--all customers' link to SolidWorks corporation) aware of the issue and they advised that there is an active SPR (software performance request): #1059883. There's some info about it in this thread (note that the suggested workaround did not work for me): https://forum.solidworks.com/thread/230925

My event viewer provides the following info after FF crashes:
Faulting application name: firefox.exe, version: 67.0.4.7109, time stamp: 0x5d0adee5
Faulting module name: ntdll.dll, version: 10.0.17763.475, time stamp: 0x3230aa04
Exception code: 0xc0000374
Fault offset: 0x00000000000fb049
Faulting process id: 0x2ce0
Faulting application start time: 0x01d52bc81255e59b
Faulting application path: C:\Program Files\Mozilla Firefox\firefox.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: fe9de644-d31b-4c30-9eec-7bedb91c0db5
Faulting package full name:
Faulting package-relative application ID:

Comment 17

[Gingerbread Man]

Marking as New based on multiple users reporting this.

Comment 18

[twolaw]

Attachment: windows error reporting ff with solidworks epdm

I was curious and used Processmon while crashing, it says last actions made by firefox before crash are TCP send and receive with Solidworks EPDM vault server on port 1433 (SQL requests, EPDM uses an SQL database).
Here is the Windows Error Reporting report :

Comment 19

[pschurter71]

2019-02-12: Issue still present but workaround method of running Firefox in compatibility mode doesn't work after 73.0 update: Firefox does not run correctly in compatibility mode

Comment 20

[Gian-Carlo Pascutto [:gcp]]

Firefox does not run correctly in compatibility mode

This should have been addressed in the 73.0.1 update.

Comment 21

[JTG]

Some other useful information about this bug on this SolidWorks Thread: https://forum.solidworks.com/thread/220894

Comment 22

[Gabriele Svelto [:gsvelto]]

We have repeatedly blocked old versions of one of the shell extensions injected by SolidWorks but it didn't solve the problem as new versions are still crash-prone. After landing bug 1633052 we finally detect and report the crashes caused by this extension so I'll add the crash signatures. This should help us analyze and solve the issue.

Comment 23

[Gabriele Svelto [:gsvelto]]

There are more signatures with slightly different stacks but this make up the bulk of the crashes. If you look past the frames of the exception handler you'll notice that all the crashes originate from a call to SysFreeString. Given this is related to the file picker I wonder if a string that has been allocated with Gecko's own allocator gets passed somehow into Windows code that is then hooked by SolidWorks' injected DLL (Database.dll) which tries to free it using SysFreeString. Toshihito, can you help us figure out what's going wrong here?

Comment 24

[Toshihito Kikuchi [:toshi]]

The latest comment on the SolidWorks forum says disabling Quick Access of Windows Explorer is a workaround.

I found a screenshot showing how SolidWorks is integrated with Explorer here. And the crashing thread was running window.storage.dll's functions without involving our modules. These evidences explain why the workaround hopefully works. I guess it's a simple double free caused by SolidWorks module, but not 100% sure without a local repro.

Comment 25

[Gabriele Svelto [:gsvelto]]

After reviewing the crash signatures I've noticed that quite a few of the crashes aren't related to this particular issue. To narrow it down I've prepared this supersearch query that only filters the last week worth of crashes specifically related to SolidWorks. The volume appears to be ~15 crash reports per day which translates to at least 150 crashes per day. That's quite a lot.

Comment 26

[Toshihito Kikuchi [:toshi]]

I did some research about this crash.

First, I simulated a double-free on BSTR by simply calling SysFreeString for the same address twice. It triggered this crash with the same crash signature.

I noticed this crash did not happen in a simple process like notepad.exe. The difference is Firefox disables BSTR cache (bug 1335235). In theory, if we stop disabling BSTR cache, these SysFreeString crashes will be gone, but I don't think it's a right approach.

As a more ambitious approach, I thought of hooking SysFreeString and catching EXCEPTION_HEAP_CORRUPTION silently, but unfortunately EXCEPTION_HEAP_CORRUPTION cannot be caught by Windows SEH.

Interestingly, Chrome shows the common dialog box in a utility process (crbug 73098) to mitigate this kind of compat issues. We may consider to take this approach in the long term, but even with that, if a third-party crashes a process, the dialogbox is suddenly disappeared.

Comment 27

[Gabriele Svelto [:gsvelto]]

(In reply to Toshihito Kikuchi [:toshi] from comment #26)

As a more ambitious approach, I thought of hooking SysFreeString and catching EXCEPTION_HEAP_CORRUPTION silently, but unfortunately EXCEPTION_HEAP_CORRUPTION cannot be caught by Windows SEH.

You can catch it with a vectored exception handler like I did in bug 1633052. I'm not sure if it can be used to solve this particular problem though.

Comment 28

[Toshihito Kikuchi [:toshi]]

I discussed about this bug with my manager. Now I'm leaning towards blocking all versions of database.dll to stop the crash.

Hooking SysFreeString is apparently risky from regression perspective (and maybe security risk, too?) . Blocking database.dll will break Solidworks's functionality (e.g. uploading a file from a Solidworks-managed repo), on the other hand, but at least it unblocks users who want to upload a file from normal folders. We also noticed a report complaining the crash happened while printing a pdf. As we're actively focusing on printing experience improvement, this kind of crash should not happen even though the root cause is not on our side. All of these scenarios could give users a reason to leave Firefox.

If Chrome is not impacted by this issue, this crash may be triggered after uploading is done in the utility process and there is no actual impact on the main process even though the crash terminated the utility process.

I'll prepare a patch to block database.dll. Unfortunately I can't come up with any other action we can take. Gabriele, do you have any thoughts?

Comment 29

[Gabriele Svelto [:gsvelto]]

(In reply to Toshihito Kikuchi [:toshi] from comment #28)

I'll prepare a patch to block database.dll. Unfortunately I can't come up with any other action we can take. Gabriele, do you have any thoughts?

I'm OK with that. We already blocked older versions twice in the hope that the bug would get fixed in SolidWorks but it kept coming back. I know that the SolidWorks developers have already been informed months ago so we did everything we could to preserve its functionality. My take is that our users would rather prefer not to have the shell integration available in the file picker rather than having Firefox crash.

That being said we might also want to pull a trick like Chrome on the long run and move the file picker OOP. This is not the only shell extension we've encountered triggering crashes in Firefox.

Comment 30

[Toshihito Kikuchi [:toshi]]

Attachment: bug1468250-block-SolidWorks.txt

Comment 31

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1468250 - Block all versions of database.dll to stop the crash. r=gcp

We blocked the older versions of database.dll as Bug 1566109 in 2019,
but the same crash is still happening with the newer versions.
We decided to block all versions because crashing in the middle of printing
or file uploading is not acceptable.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fc42ab7c5189
Block all versions of database.dll to stop the crash.  r=aklotz

Comment 33

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/fc42ab7c5189