1407337 - Crash in onepin-opensc-pkcs11.dll@0x178019 (OpenSC smart card DLL)

Status: VERIFIED FIXED

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Chris Peterson [:cpeterson]]

This bug was filed from the Socorro interface and is 
report bp-5d35d253-b38c-4099-8416-5c5aa0171010.
=============================================================

@ Marco, I checked a dozen random crash reports and they all had onepin-opensc-pkcs11.dll version 0.14.0.0 or 0.15.0.0. The latest version of OpenSC is 0.17.0, so maybe we should block older DLL versions?

This DLL has crashed in both 32- and 64-bit Firefox on all Windows versions over the last six months, but crash signature onepin-opensc-pkcs11.dll@0x178019 only affects 64-bit Firefox on Windows 7 and appears to be climbing (because 64-bit is now the default in Firefox 55+).

This DLL appears to be OpenSC smart card provider:

https://github.com/OpenSC/OpenSC

Comment 1

[Chris Peterson [:cpeterson]]

David, do you have any contact info for any OpenSC developers? We see some new crash reports from 64-bit Firefox trying to load old versions of OpenSC (version 0.15.0 from 2016-02-09).

I don't see crashes from more recent versions of OpenSC (0.17.0 from 2017-07-18 and 0.16.0 from 2016-04-06). Do you think we can we safely block DLL versions less than the current 0.17.0?

I found related bug 1302175 about not being able to load opensc-pkcs11.dll on Windows 7.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I've only interacted with them over github, as far as I can recall (but they're certainly responsive). I know at least 0.15.0 had a crash-causing bug ( https://github.com/OpenSC/OpenSC/issues/683 ). If I'm reading https://github.com/OpenSC/OpenSC/commit/1fb741338a010515b35f28cd9c769561f3e574a2 correctly, the fix made it into 0.16.0, so maybe we could block everything earlier than that.

Comment 3

[Marco Castelluccio [:marco]]

Here's a try build with the block: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152.

Can somebody check if the DLL is successfully blocked?

Comment 4

[Marco Castelluccio [:marco]]

I've verified that over the past two weeks, all 658 reports have versions of the DLL up to 0.15.0.0: https://gist.github.com/marco-c/c2055ae58a238faa80c93b0f733a6357.

Comment 5

[Chris Peterson [:cpeterson]]

Sorry, I don't know how to test OpenSC.

Graham, you commented in OpenSC bug 1302175 earlier this year. Do you mind testing whether you can reproduce the old OpenSC version 0.15.0.0 crashing in 64-bit Windows Firefox [1] and that the proposed fix (installer exe [2]) stops 0.15.0.0 from crashing?

[1] https://www.mozilla.org/en-US/firefox/all/
[2] https://queue.taskcluster.net/v1/task/QZr_GsvBSuqCRSioA32Mxg/runs/0/artifacts/public/build/setup.exe

Comment 6

[Marco Castelluccio [:marco]]

Graham, even if you can't reproduce the crash, it would be great to know whether the proposed fix is preventing the DLL from loading.

Comment 7

[Marco Castelluccio [:marco]]

There are crash reports with 57. There are none with 58, but probably just because no user on Nightly uses OpenSC.

Comment 8

[Marco Castelluccio [:marco]]

Abe, could you test that with the try build from comment 3 the onepin-opensc-pkcs11.dll DLL is blocked? A quick way to get the list of the DLLs is to force a Firefox crash and submit the crash report to socorro, then paste the link to the crash report here.

Comment 9

[Marco Castelluccio [:marco]]

BTW, ~0.0005% release users have this DLL, so it's not a huge deal. Uplifting the patch would be really safe though, if we can confirm that it actually prevents the DLL from loading.

Comment 10

[Abe - QA (:Abe_LV)]

Hi Marco,

Here is the crash report from Windows 10 x64: https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1-37b3a0171017
Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2

Please let me know if you have comment/suggestion on this testing. Thanks

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Abe - QA (:Abe_LV) from comment #10)
> Hi Marco,
> 
> Here is the crash report from Windows 10 x64:
> https://crash-stats.mozilla.com/report/index/bp-b717404c-d0db-4b6e-8cb1-
> 37b3a0171017
> Screen capture is: https://testing-1.tinytake.com/sf/MjAzNjkzNF82NDIzMTk2
> 
> Please let me know if you have comment/suggestion on this testing. Thanks

It looks like the library is not there, so maybe the block was effective. Could you repeat the same steps with a normal Nightly build? I want to make sure the library is injected there.

Comment 12

[Abe - QA (:Abe_LV)]

Crash report for latest nightly (Build ID:20171017141229): https://crash-stats.mozilla.com/report/index/bp-4109fb9c-2ef2-4e13-b946-aea610171017

Comment 13

[Marco Castelluccio [:marco]]

It looks like the library was not injected anyway. Maybe there's something specific that needs to be done in order to get the library to be injected.

I've asked for help in a OpenSC GitHub issue: https://github.com/OpenSC/OpenSC/issues/1176.

Comment 14

[Marco Castelluccio [:marco]]

Abe, could you repeat the two tests (try build and nightly) after you also follow the steps explained in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step?

Comment 15

[Abe - QA (:Abe_LV)]

Installed OpenSC on Windows 10 x64 and the crashes are here:

Latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e-515560171018
Trybuild crash: https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5-b16640171018

Comment 16

[Marco Castelluccio [:marco]]

(In reply to Abe - QA (:Abe_LV) from comment #15)
> Installed OpenSC on Windows 10 x64 and the crashes are here:
> 
> Latest nightly crash:
> https://crash-stats.mozilla.com/report/index/bp-56c58ac2-a1e5-4f40-940e-
> 515560171018
> Trybuild crash:
> https://crash-stats.mozilla.com/report/index/bp-a02428f2-427a-4215-9fc5-
> b16640171018

Thanks!
In both cases onepin-opensc-pkcs11.dll is not loaded. There must be something else needed to make it load. I'll ask again on Github.

Comment 17

[deengert]

The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I am not sure what you mean by injected. Does your blacklist catch loading of pkcs11 modules?

Comment 18

[Marco Castelluccio [:marco]]

(In reply to deengert from comment #17)
> The onepin-opensc-pkcs11.dll and opensc-pkcs11.dll are PKCS#11 modules and I
> am not sure what you mean by injected. Does your blacklist catch loading of
> pkcs11 modules?

By "injected" I mean loaded in the Firefox process. I think the blocklist should work for pkcs11 modules too.

Abe, could you try repeating the same steps, but selecting "onepin-opensc-pkcs11.dll" instead of "opensc-pkcs11.dll"? It would be great if you could also make a screen capture of when you do it with the try build.

Comment 19

[Abe - QA (:Abe_LV)]

The try build does not load "onepin-opensc-pkcs11.dll" module. It says "Unable to add module", but the latest nightly (10-19-2017) does load the same module successfully.

Here is screen capture:https://testing-1.tinytake.com/sf/MjA0MzAxMl82NDM5Mjgw

Then I crashed the browsers, and here are the crash reports if needed.
try build crash: https://crash-stats.mozilla.com/report/index/bp-7a77665b-befa-44ae-8d0f-bbb190171019
latest nightly crash: https://crash-stats.mozilla.com/report/index/bp-afaff5c5-a393-4013-877f-c23890171019

Let me know if additional testing is still required.

Comment 20

[Abe - QA (:Abe_LV)]

hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and now I am able to load it on the trybuild.
The crash report for trybuild is here: https://crash-stats.mozilla.com/report/index/bp-a9cfa706-cb89-4754-87fd-079060171019
Additional crash report for trybuild: https://crash-stats.mozilla.com/report/index/bp-464433fb-6f75-44d6-bd7c-9d0450171019
Latest Nightly crash (if needed): https://crash-stats.mozilla.com/report/index/bp-b23b1d70-4a71-4f67-abba-7dc060171019

Screen capture is here: https://testing-1.tinytake.com/sf/MjA0MzI5NF82NDQwMTYw

The steps used for testing is: 
 
1.Download and install trybuild from comment 3 
   (https://treeherder.mozilla.org/#/jobs?repo=try&revision=38b5596c0d8feffca78fa6cc64d6351aaadc7152)

2.Download and install "OpenSC-0.17.0-win64_vs12-Release.msi" and "OpenSC-0.17.0-win32_vs12-Release.msi" on Windows 10 x64 from 
       https://github.com/OpenSC/OpenSC/releases/tag/0.17.0

3.Start the trybuild from fresh profile

3.Load "onepin-opensc-pkcs11.dll" from "c:\Windows\System32\" directory. 
      Follow steps in https://github.com/OpenSC/OpenSC/wiki/Installing-OpenSC-PKCS%2311-Module-in-Firefox,-Step-by-Step4.

4.Restart the browser 

5.Crash the browser using script: 
   (Cu.import("resource://gre/modules/ctypes.jsm");
    let zero = new ctypes.intptr_t(8);
    let badptr = ctypes.cast(zero, ctypes.PointerType(ctypes.int32_t));
    badptr.contents;)

6.Submit the crash and restart the browser from the crash reporter client 
7.Copy the submitted crash report link here.

Comment 21

[Marco Castelluccio [:marco]]

(In reply to Abe - QA (:Abe_LV) from comment #20)
> hmm... I think I had an installation issue. I reinstalled openSc(v17.0)and
> now I am able to load it on the trybuild.

Thanks Abe, this is indeed what should happen (we only want to block up to 0.15.0.0).

Comment 22

[Marco Castelluccio [:marco]]

Attachment: Patch

Given the testing revealed that the DLL can be blocked and it won't cause other problems, I think we can land the block.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1758c184cf01
Block OpenSC < 0.16.0.0. r=jimm

Comment 24

[Marco Castelluccio [:marco]]

I guess not enough crashes to justify 52 or 56, but I would uplift this to 57, given its safety.

Comment 25

[Chris Peterson [:cpeterson]]

Comment on attachment 8920416 [details] [diff] [review]
Patch

Approval Request Comment

[Feature/Bug causing the regression]: 

[User impact if declined]: Users running this old (2015) OpenSC smartcard software will continue to experience browser startup crashes. We had 187 OpenSC crashes in 56.0.x over the last seven days, affecting both 32- and 64-bit Firefox.

[Is this code covered by automated tests?]: No

[Has the fix been verified in Nightly?]: No, but Abe from QE has verified the DLL block in a Try build, but not in Nightly 58.

[Needs manual test from QE? If yes, steps to reproduce]: No unless you would like Abe to retest in Nightly 58.

[List of other uplifts needed for the feature/fix]: None

[Is the change risky?]: No

[Why is the change risky/not risky?]: This patch blocks old (2015) versions of an uncommon DLL. Affected users can upgrade to the 2017 version of the OpenSC smartcard software.

[String changes made/needed]: None

Comment 26

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1758c184cf01

Comment 27

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8920416 [details] [diff] [review]
Patch

Crash fix by way of dll blocklisting, Beta57+

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d16abfdc9433

Comment 29

[Chris Peterson [:cpeterson]]

btw, I spoke with a Firefox user who hit this openpin-opensc-pkcs11.dll crash. He didn't install the OpenSC software himself. The old DLL version (0.15.0) was packaged in the Estonian government's eID smartcard software.

Comment 30

[Abe - QA (:Abe_LV)]

I further tested this on Windows 7 x86 and Windows 10 x64 with latest nightly and beta builds. Latest beta and nightly have this fix.
Unlike older Firefox builds (builds that do not have the fix), latest beta and nightly builds do not load the DLL if its version is less than or equal to 15.0. However, DLL versions greater than 15.0 load successfully.

Test cases and runs are here: https://public.etherpad-mozilla.org/p/1407337

Let me know if you have questions or feedback on this.

Comment 31

[Chris Peterson [:cpeterson]]

Clearing old needinfo