Closed Bug 1407337 Opened 2 years ago Closed 2 years ago

Crash in onepin-opensc-pkcs11.dll@0x178019 (OpenSC smart card DLL)

Categories

(External Software Affecting Firefox :: Other, defect, P2, critical)

All
Windows
defect

Tracking

(firefox-esr52 wontfix, firefox56 wontfix, firefox57 verified, firefox58 verified)

VERIFIED FIXED
Tracking Status
firefox-esr52 --- wontfix
firefox56 --- wontfix
firefox57 --- verified
firefox58 --- verified

People

(Reporter: cpeterson, Assigned: marco)

References

()

Details

(Keywords: crash)

Crash Data

Attachments

(1 file)

This bug was filed from the Socorro interface and is 
report bp-5d35d253-b38c-4099-8416-5c5aa0171010.
=============================================================

@ Marco, I checked a dozen random crash reports and they all had onepin-opensc-pkcs11.dll version 0.14.0.0 or 0.15.0.0. The latest version of OpenSC is 0.17.0, so maybe we should block older DLL versions?

This DLL has crashed in both 32- and 64-bit Firefox on all Windows versions over the last six months, but crash signature onepin-opensc-pkcs11.dll@0x178019 only affects 64-bit Firefox on Windows 7 and appears to be climbing (because 64-bit is now the default in Firefox 55+).

This DLL appears to be OpenSC smart card provider:

https://github.com/OpenSC/OpenSC
Flags: needinfo?(mcastelluccio)
David, do you have any contact info for any OpenSC developers? We see some new crash reports from 64-bit Firefox trying to load old versions of OpenSC (version 0.15.0 from 2016-02-09).

I don't see crashes from more recent versions of OpenSC (0.17.0 from 2017-07-18 and 0.16.0 from 2016-04-06). Do you think we can we safely block DLL versions less than the current 0.17.0?

I found related bug 1302175 about not being able to load opensc-pkcs11.dll on Windows 7.
Flags: needinfo?(dkeeler)
See Also: → 1302175
I've only interacted with them over github, as far as I can recall (but they're certainly responsive). I know at least 0.15.0 had a crash-causing bug ( https://github.com/OpenSC/OpenSC/issues/683 ). If I'm reading https://github.com/OpenSC/OpenSC/commit/1fb741338a010515b35f28cd9c769561f3e574a2 correctly, the fix made it into 0.16.0, so maybe we could block everything earlier than that.
Flags: needinfo?(dkeeler)
Here's a try build with the block: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152.

Can somebody check if the DLL is successfully blocked?
Flags: needinfo?(mcastelluccio)
I've verified that over the past two weeks, all 658 reports have versions of the DLL up to 0.15.0.0: https://gist.github.com/marco-c/c2055ae58a238faa80c93b0f733a6357.
Sorry, I don't know how to test OpenSC.

Graham, you commented in OpenSC bug 1302175 earlier this year. Do you mind testing whether you can reproduce the old OpenSC version 0.15.0.0 crashing in 64-bit Windows Firefox [1] and that the proposed fix (installer exe [2]) stops 0.15.0.0 from crashing?

[1] https://www.mozilla.org/en-US/firefox/all/
[2] https://queue.taskcluster.net/v1/task/QZr_GsvBSuqCRSioA32Mxg/runs/0/artifacts/public/build/setup.exe
No longer blocks: win64-migration
Flags: needinfo?(minfrin)
Graham, even if you can't reproduce the crash, it would be great to know whether the proposed fix is preventing the DLL from loading.
There are crash reports with 57. There are none with 58, but probably just because no user on Nightly uses OpenSC.
Abe, could you test that with the try build from comment 3 the onepin-opensc-pkcs11.dll DLL is blocked? A quick way to get the list of the DLLs is to force a Firefox crash and submit the crash report to socorro, then paste the link to the crash report here.
Flags: needinfo?(amasresha)
BTW, ~0.0005% release users have this DLL, so it's not a huge deal. Uplifting the patch would be really safe though, if we can confirm that it actually prevents the DLL from loading.
Hi Marco,

Here is the crash report from Windows 10 x64: https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1-37b3a0171017
Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2

Please let me know if you have comment/suggestion on this testing. Thanks
Flags: needinfo?(amasresha)
(In reply to Abe - QA (:Abe_LV) from comment #10)
> Hi Marco,
> 
> Here is the crash report from Windows 10 x64:
> https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1-
> 37b3a0171017
> Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2
> 
> Please let me know if you have comment/suggestion on this testing. Thanks

It looks like the library is not there, so maybe the block was effective. Could you repeat the same steps with a normal Nightly build? I want to make sure the library is injected there.
Crash report for latest nightly (Build ID:20171017141229): https://crash-stats.mozilla.com/report/index/bp-4109fb9c-2ef2-4e13-b946-aea610171017
It looks like the library was not injected anyway. Maybe there's something specific that needs to be done in order to get the library to be injected.

I've asked for help in a OpenSC GitHub issue: https://github.com/OpenSC/OpenSC/issues/1176.
Abe, could you repeat the two tests (try build and nightly) after you also follow the steps explained in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step?
Flags: needinfo?(amasresha)
Installed OpenSC on Windows 10 x64 and the crashes are here:

Latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e-515560171018
Trybuild crash: https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5-b16640171018
Flags: needinfo?(amasresha)
(In reply to Abe - QA (:Abe_LV) from comment #15)
> Installed OpenSC on Windows 10 x64 and the crashes are here:
> 
> Latest nightly crash:
> https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e-
> 515560171018
> Trybuild crash:
> https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5-
> b16640171018

Thanks!
In both cases onepin-opensc-pkcs11.dll is not loaded. There must be something else needed to make it load. I'll ask again on Github.
The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I am not sure what you mean by injected. Does your blacklist catch loading of pkcs11 modules?
(In reply to deengert from comment #17)
> The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I
> am not sure what you mean by injected. Does your blacklist catch loading of
> pkcs11 modules?

By "injected" I mean loaded in the Firefox process. I think the blocklist should work for pkcs11 modules too.

Abe, could you try repeating the same steps, but selecting "onepin-opensc-pkcs11.dll" instead of "opensc-pkcs11.dll"? It would be great if you could also make a screen capture of when you do it with the try build.
Flags: needinfo?(amasresha)
The try build does not load "onepin-opensc-pkcs11.dll" module. It says "Unable to add module", but the latest nightly (10-19-2017) does load the same module successfully.

Here is screen capture:https://testing-1.tinytake.com/sf/MjA0MzAxMl82NDM5Mjgw

Then I crashed the browsers, and here are the crash reports if needed.
try build crash: https://crash-stats.mozilla.com/report/index/bp-7a77665b-befa-44ae-8d0f-bbb190171019
latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-afaff5c5-a393-4013-877f-c23890171019

Let me know if additional testing is still required.
Flags: needinfo?(amasresha)
Priority: -- → P2
hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and now I am able to load it on the trybuild.
The crash report for trybuild is here: https://crash-stats.mozilla.com/report/index/bp-a9cfa706-cb89-4754-87fd-079060171019
Additional crash report for trybuild: https://crash-stats.mozilla.com/report/index/bp-464433fb-6f75-44d6-bd7c-9d0450171019
Latest Nightly crash (if needed): https://crash-stats.mozilla.com/report/index/bp-b23b1d70-4a71-4f67-abba-7dc060171019

Screen capture is here: https://testing-1.tinytake.com/sf/MjA0MzI5NF82NDQwMTYw

The steps used for testing is: 
 
1.Download and install trybuild from comment 3 
   (https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152)

2.Download and install "OpenSC-0.17.0-win64_vs12-Release.msi" and "OpenSC-0.17.0-win32_vs12-Release.msi" on Windows 10 x64 from 
       https://github.com/OpenSC/OpenSC/releases/tag/0.17.0

3.Start the trybuild from fresh profile

3.Load "onepin-opensc-pkcs11.dll" from "c:\Windows\System32\" directory. 
      Follow steps in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step4.

4.Restart the browser 

5.Crash the browser using script: 
   (Cu.import("resource://gre/modules/ctypes.jsm");
    let zero = new ctypes.intptr_t(8);
    let badptr = ctypes.cast(zero, ctypes.PointerType(ctypes.int32_t));
    badptr.contents;)

6.Submit the crash and restart the browser from the crash reporter client 
7.Copy the submitted crash report link here.
(In reply to Abe - QA (:Abe_LV) from comment #20)
> hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and
> now I am able to load it on the trybuild.

Thanks Abe, this is indeed what should happen (we only want to block up to 0.15.0.0).
Attached patch PatchSplinter Review
Given the testing revealed that the DLL can be blocked and it won't cause other problems, I think we can land the block.
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #8920416 - Flags: review?(jmathies)
Attachment #8920416 - Flags: review?(jmathies) → review+
I guess not enough crashes to justify 52 or 56, but I would uplift this to 57, given its safety.
Comment on attachment 8920416 [details] [diff] [review]
Patch

Approval Request Comment

[Feature/Bug causing the regression]: 

[User impact if declined]: Users running this old (2015) OpenSC smartcard software will continue to experience browser startup crashes. We had 187 OpenSC crashes in 56.0.x over the last seven days, affecting both 32- and 64-bit Firefox.

[Is this code covered by automated tests?]: No

[Has the fix been verified in Nightly?]: No, but Abe from QE has verified the DLL block in a Try build, but not in Nightly 58.

[Needs manual test from QE? If yes, steps to reproduce]: No unless you would like Abe to retest in Nightly 58.

[List of other uplifts needed for the feature/fix]: None

[Is the change risky?]: No

[Why is the change risky/not risky?]: This patch blocks old (2015) versions of an uncommon DLL. Affected users can upgrade to the 2017 version of the OpenSC smartcard software.

[String changes made/needed]: None
Attachment #8920416 - Flags: approval-mozilla-beta?
https://hg.mozilla.org/mozilla-central/rev/1758c184cf01
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Comment on attachment 8920416 [details] [diff] [review]
Patch

Crash fix by way of dll blocklisting, Beta57+
Attachment #8920416 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
btw, I spoke with a Firefox user who hit this openpin-opensc-pkcs11.dll crash. He didn't install the OpenSC software himself. The old DLL version (0.15.0) was packaged in the Estonian government's eID smartcard software.
QA Contact: amasresha
I further tested this on Windows 7 x86 and Windows 10 x64 with latest nightly and beta builds. Latest beta and nightly have this fix.
Unlike older Firefox builds (builds that do not have the fix), latest beta and nightly builds do not load the DLL if its version is less than or equal to 15.0. However, DLL versions greater than 15.0 load successfully.

Test cases and runs are here: https://public.etherpad-mozilla.org/p/1407337

Let me know if you have questions or feedback on this.
Status: RESOLVED → VERIFIED
Blocks: 1414773
Clearing old needinfo
Flags: needinfo?(minfrin)
You need to log in before you can comment on or make changes to this bug.