Closed Bug 1407337 Opened 2 years ago Closed 2 years ago
Crash in onepin-opensc-pkcs11
.dll@0x178019 (Open SC smart card DLL)
This bug was filed from the Socorro interface and is report bp-5d35d253-b38c-4099-8416-5c5aa0171010. ============================================================= @ Marco, I checked a dozen random crash reports and they all had onepin-opensc-pkcs11.dll version 0.14.0.0 or 0.15.0.0. The latest version of OpenSC is 0.17.0, so maybe we should block older DLL versions? This DLL has crashed in both 32- and 64-bit Firefox on all Windows versions over the last six months, but crash signature onepin-opensc-pkcs11.dll@0x178019 only affects 64-bit Firefox on Windows 7 and appears to be climbing (because 64-bit is now the default in Firefox 55+). This DLL appears to be OpenSC smart card provider: https://github.com/OpenSC/OpenSC
David, do you have any contact info for any OpenSC developers? We see some new crash reports from 64-bit Firefox trying to load old versions of OpenSC (version 0.15.0 from 2016-02-09). I don't see crashes from more recent versions of OpenSC (0.17.0 from 2017-07-18 and 0.16.0 from 2016-04-06). Do you think we can we safely block DLL versions less than the current 0.17.0? I found related bug 1302175 about not being able to load opensc-pkcs11.dll on Windows 7.
See Also: → 1302175
I've only interacted with them over github, as far as I can recall (but they're certainly responsive). I know at least 0.15.0 had a crash-causing bug ( https://github.com/OpenSC/OpenSC/issues/683 ). If I'm reading https://github.com/OpenSC/OpenSC/commit/1fb741338a010515b35f28cd9c769561f3e574a2 correctly, the fix made it into 0.16.0, so maybe we could block everything earlier than that.
Here's a try build with the block: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152. Can somebody check if the DLL is successfully blocked?
I've verified that over the past two weeks, all 658 reports have versions of the DLL up to 0.15.0.0: https://gist.github.com/marco-c/c2055ae58a238faa80c93b0f733a6357.
Sorry, I don't know how to test OpenSC. Graham, you commented in OpenSC bug 1302175 earlier this year. Do you mind testing whether you can reproduce the old OpenSC version 0.15.0.0 crashing in 64-bit Windows Firefox  and that the proposed fix (installer exe ) stops 0.15.0.0 from crashing?  https://www.mozilla.org/en-US/firefox/all/  https://queue.taskcluster.net/v1/task/QZr_GsvBSuqCRSioA32Mxg/runs/0/artifacts/public/build/setup.exe
No longer blocks: win64-migration
Graham, even if you can't reproduce the crash, it would be great to know whether the proposed fix is preventing the DLL from loading.
There are crash reports with 57. There are none with 58, but probably just because no user on Nightly uses OpenSC.
Abe, could you test that with the try build from comment 3 the onepin-opensc-pkcs11.dll DLL is blocked? A quick way to get the list of the DLLs is to force a Firefox crash and submit the crash report to socorro, then paste the link to the crash report here.
BTW, ~0.0005% release users have this DLL, so it's not a huge deal. Uplifting the patch would be really safe though, if we can confirm that it actually prevents the DLL from loading.
Hi Marco, Here is the crash report from Windows 10 x64: https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1-37b3a0171017 Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2 Please let me know if you have comment/suggestion on this testing. Thanks
(In reply to Abe - QA (:Abe_LV) from comment #10) > Hi Marco, > > Here is the crash report from Windows 10 x64: > https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1- > 37b3a0171017 > Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2 > > Please let me know if you have comment/suggestion on this testing. Thanks It looks like the library is not there, so maybe the block was effective. Could you repeat the same steps with a normal Nightly build? I want to make sure the library is injected there.
Crash report for latest nightly (Build ID:20171017141229): https://crash-stats.mozilla.com/report/index/bp-4109fb9c-2ef2-4e13-b946-aea610171017
It looks like the library was not injected anyway. Maybe there's something specific that needs to be done in order to get the library to be injected. I've asked for help in a OpenSC GitHub issue: https://github.com/OpenSC/OpenSC/issues/1176.
Abe, could you repeat the two tests (try build and nightly) after you also follow the steps explained in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step?
Installed OpenSC on Windows 10 x64 and the crashes are here: Latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e-515560171018 Trybuild crash: https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5-b16640171018
(In reply to Abe - QA (:Abe_LV) from comment #15) > Installed OpenSC on Windows 10 x64 and the crashes are here: > > Latest nightly crash: > https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e- > 515560171018 > Trybuild crash: > https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5- > b16640171018 Thanks! In both cases onepin-opensc-pkcs11.dll is not loaded. There must be something else needed to make it load. I'll ask again on Github.
The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I am not sure what you mean by injected. Does your blacklist catch loading of pkcs11 modules?
(In reply to deengert from comment #17) > The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I > am not sure what you mean by injected. Does your blacklist catch loading of > pkcs11 modules? By "injected" I mean loaded in the Firefox process. I think the blocklist should work for pkcs11 modules too. Abe, could you try repeating the same steps, but selecting "onepin-opensc-pkcs11.dll" instead of "opensc-pkcs11.dll"? It would be great if you could also make a screen capture of when you do it with the try build.
The try build does not load "onepin-opensc-pkcs11.dll" module. It says "Unable to add module", but the latest nightly (10-19-2017) does load the same module successfully. Here is screen capture:https://testing-1.tinytake.com/sf/MjA0MzAxMl82NDM5Mjgw Then I crashed the browsers, and here are the crash reports if needed. try build crash: https://crash-stats.mozilla.com/report/index/bp-7a77665b-befa-44ae-8d0f-bbb190171019 latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-afaff5c5-a393-4013-877f-c23890171019 Let me know if additional testing is still required.
hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and now I am able to load it on the trybuild. The crash report for trybuild is here: https://crash-stats.mozilla.com/report/index/bp-a9cfa706-cb89-4754-87fd-079060171019 Additional crash report for trybuild: https://crash-stats.mozilla.com/report/index/bp-464433fb-6f75-44d6-bd7c-9d0450171019 Latest Nightly crash (if needed): https://crash-stats.mozilla.com/report/index/bp-b23b1d70-4a71-4f67-abba-7dc060171019 Screen capture is here: https://testing-1.tinytake.com/sf/MjA0MzI5NF82NDQwMTYw The steps used for testing is: 1.Download and install trybuild from comment 3 (https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152) 2.Download and install "OpenSC-0.17.0-win64_vs12-Release.msi" and "OpenSC-0.17.0-win32_vs12-Release.msi" on Windows 10 x64 from https://github.com/OpenSC/OpenSC/releases/tag/0.17.0 3.Start the trybuild from fresh profile 3.Load "onepin-opensc-pkcs11.dll" from "c:\Windows\System32\" directory. Follow steps in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step4. 4.Restart the browser 5.Crash the browser using script: (Cu.import("resource://gre/modules/ctypes.jsm"); let zero = new ctypes.intptr_t(8); let badptr = ctypes.cast(zero, ctypes.PointerType(ctypes.int32_t)); badptr.contents;) 6.Submit the crash and restart the browser from the crash reporter client 7.Copy the submitted crash report link here.
(In reply to Abe - QA (:Abe_LV) from comment #20) > hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and > now I am able to load it on the trybuild. Thanks Abe, this is indeed what should happen (we only want to block up to 0.15.0.0).
Given the testing revealed that the DLL can be blocked and it won't cause other problems, I think we can land the block.
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #8920416 - Flags: review?(jmathies)
Attachment #8920416 - Flags: review?(jmathies) → review+
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/1758c184cf01 Block OpenSC < 0.16.0.0. r=jimm
I guess not enough crashes to justify 52 or 56, but I would uplift this to 57, given its safety.
Comment on attachment 8920416 [details] [diff] [review] Patch Approval Request Comment [Feature/Bug causing the regression]: [User impact if declined]: Users running this old (2015) OpenSC smartcard software will continue to experience browser startup crashes. We had 187 OpenSC crashes in 56.0.x over the last seven days, affecting both 32- and 64-bit Firefox. [Is this code covered by automated tests?]: No [Has the fix been verified in Nightly?]: No, but Abe from QE has verified the DLL block in a Try build, but not in Nightly 58. [Needs manual test from QE? If yes, steps to reproduce]: No unless you would like Abe to retest in Nightly 58. [List of other uplifts needed for the feature/fix]: None [Is the change risky?]: No [Why is the change risky/not risky?]: This patch blocks old (2015) versions of an uncommon DLL. Affected users can upgrade to the 2017 version of the OpenSC smartcard software. [String changes made/needed]: None
Attachment #8920416 - Flags: approval-mozilla-beta?
Comment on attachment 8920416 [details] [diff] [review] Patch Crash fix by way of dll blocklisting, Beta57+
Attachment #8920416 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
btw, I spoke with a Firefox user who hit this openpin-opensc-pkcs11.dll crash. He didn't install the OpenSC software himself. The old DLL version (0.15.0) was packaged in the Estonian government's eID smartcard software.
I further tested this on Windows 7 x86 and Windows 10 x64 with latest nightly and beta builds. Latest beta and nightly have this fix. Unlike older Firefox builds (builds that do not have the fix), latest beta and nightly builds do not load the DLL if its version is less than or equal to 15.0. However, DLL versions greater than 15.0 load successfully. Test cases and runs are here: https://public.etherpad-mozilla.org/p/1407337 Let me know if you have questions or feedback on this.
Clearing old needinfo
You need to log in before you can comment on or make changes to this bug.