1560486 - firefox closes suddenly without crash (Estonian e-Residency: onepin-opensc-pkcs11.dll)

Status: RESOLVED INVALID

(Core :: Security: PSM, defect, P3)

Description

[leifeld]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

While surfing firefox closes randomly, maybe one time in an hour.
I did not find a way to reproduce it faster.

Webrender is enabled.

Actual results:

Firefox closes without anything in about:crashes
One time there was a dialog, that an memory error occurred.
I tried debugging firefox with windbg, but i didn't see something helpful.

Expected results:

It should not close randomly.

Comment 1

[leifeld]

I can try to build everything from source, if that would be helpful.

Comment 2

[Sotaro Ikeda [:sotaro]]

leifeld, thanks for reporting. Can you upload about:support?
Open about:support, click on "Copy text to clipboard", paste it into a text file and attach it to the report. Thanks!

Comment 3

[Sotaro Ikeda [:sotaro]]

I wonder if Bug Bug 1559681 might be related to this bug.

Comment 4

[leifeld]

Attachment: about.support.txt

The about:support output

Comment 5

[Darkspirit]

Version: 68.0b12

Compositing: WebRender

Description: Radeon(TM) RX 460 Graphics
Vendor ID: 0x1002
Device ID: 0x67ef

Qualified device by bug 1559375, therefore marking 68 as affected.

Failure Log
(#0): GP+[GFX1-]: shader-cache: Timed out before finishing loads
(#1): CP+[GFX1-]: Unexpected BufferProvider over-production.
(#2): CP+[GFX1-]: Unexpected BufferProvider over-production.

Comment 6

[Darkspirit]

Background: Graphics problems usually don't close Firefox anymore. In case of a GPU process crash the window should only flash white for a second.

Name: Firefox PKCS11 loader
Version: 1.0.1

Name: IDEMIA PKCS11 loader
Version: 2.0.6

You are using two smartcard plugins that are using the WebExtension API provided by bug 1357391.
It's not unlikely that they could cause trouble. (bug 1396030 comment 0)

Please try disabling all these addons to see if that fixes the problem. Thanks!

Comment 7

[leifeld]

I disabeld the addons and after an hour I got a new crash, this time with an call stack. It is uploaded with id
1fef7e06-3433-4002-93e1-f35fb0190625
https://crash-stats.mozilla.org/report/index/1fef7e06-3433-4002-93e1-f35fb0190625#tab-details

i guess, that it is quite common to load these two pkcs11 plugins, since it is the recocmended way to connect the estonian e residency.
see https://e-resident.gov.ee/welcome/ ⇢ install link ⇢ Firefox

Comment 8

[leifeld]

Attachment: new about.support.txt

Comment 9

[Darkspirit]

Thanks! Moving to the right component.

Comment 10

[leifeld]

The two PKCS11 loader Plugins are supposed to be enabled at once for the e residency: https://id.ee/index.php?id=34392

Strange that I got the crash, even with both plugins disabled (see second about.support.txt)

Comment 11

[Darkspirit]

The PKCS11 WebExtension API is used to install PKCS#11 modules. Afterwards, I assume, these addons are only needed for updates.

Comment 12

[leifeld]

Attachment: memory.png

Firefox being terminated by Windows

Comment 13

[leifeld]

Crashed again without anything in about:crashes
(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #11)

The PKCS11 WebExtension API is used to install PKCS#11 modules. Afterwards, I assume, these addons are only needed for updates.

Thank you for the explanation.

Comment 14

[leifeld]

Got a new crash, which could be reported 8e936a70-394f-4445-b2ac-a142f0190701

Comment 15

[Darkspirit]

(In reply to leifeld from comment #14)

Got a new crash, which could be reported bp-8e936a70-394f-4445-b2ac-a142f0190701

That one is empty :/

Comment 16

[Asif Youssuff]

80 crashes in the last week, one from reddit reported https://www.reddit.com/r/firefox/comments/foqxe8/intermittent_crashing_of_firefox_740_64bit/

New crash here: https://crash-stats.mozilla.org/report/index/499b37cc-b913-44fe-8174-294060200325

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 18

[FredMcD]

https://support.mozilla.org/en-US/questions/1283019#answer-1301119
User: I don't know if it's too early to say (been using FF for 3 hours) but removing the e-Recidency
software (onepin-opensc-pkcs11.dll) seems to have solved the problem.

Comment 19

[Dustin L. Howett]

I've seen about 50 crashes in the past week, and I finally got it under windbg.
I've got a rough, not particularly well-symbolicated backtrace:

(34b4.2ea8): C++ EH exception - code e06d7363 (first chance)
(34b4.2ea8): C++ EH exception - code e06d7363 (first chance)
(34b4.2ea8): C++ EH exception - code e06d7363 (first chance)
(34b4.2ea8): C++ EH exception - code e06d7363 (first chance)
(34b4.2ffc): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
opensc_pkcs11!C_GetTokenInfo+0x1c151d:
00007fff`ebfa6769 f3a4            rep movs byte ptr [rdi],byte ptr [rsi]

0:078> k
 # Child-SP          RetAddr           Call Site
00 000000a7`042be5f8 000001e2`75da51e0 opensc_pkcs11!C_GetTokenInfo+0x1c151d
01 000000a7`042be600 000001e2`6cc51720 0x000001e2`75da51e0
02 000000a7`042be608 00007fff`ebe7c950 0x000001e2`6cc51720
03 000000a7`042be610 00007fff`ebdea395 opensc_pkcs11!C_GetTokenInfo+0x97704
04 000000a7`042be6a0 00007fff`ebdda176 opensc_pkcs11!C_GetTokenInfo+0x5149
05 000000a7`042be6d0 00007fff`f86ad296 opensc_pkcs11!C_GetSlotList+0x7e
06 000000a7`042be720 00007fff`e12e7434 nss3!SECMOD_UpdateSlotList+0x66
07 000000a7`042be790 00007fff`de4eb496 xul!soundtouch::SoundTouch::operator=+0x13d1e34
08 000000a7`042be810 00007fff`de4eccf3 xul!workerlz4_compress+0x29ba26
09 000000a7`042beb10 00007fff`e12c0fa1 xul!workerlz4_compress+0x29d283

This may be relevant to this issue. I'll work on symbols next.

Comment 20

[Dustin L. Howett]

symbolicated.

0:000> k
 # Child-SP          RetAddr           Call Site
00 00000061`1ddfc090 00007fff`d996df9c opensc_pkcs11!_invoke_watson+0x17 [f:\dd\vctools\crt\crtw32\misc\invarg.c @ 132] 
01 00000061`1ddfc0c0 00007fff`d996dfb9 opensc_pkcs11!_invalid_parameter+0x64 [f:\dd\vctools\crt\crtw32\misc\invarg.c @ 85] 
02 00000061`1ddfc100 00007fff`d9972f65 opensc_pkcs11!_invalid_parameter_noinfo+0x19 [f:\dd\vctools\crt\crtw32\misc\invarg.c @ 97] 
03 00000061`1ddfc140 00007fff`d997ed23 opensc_pkcs11!strcpy_s+0x29 [f:\dd\vctools\crt\crtw32\h\tcscpy_s.inl @ 18] 
04 00000061`1ddfc170 00007fff`d983c31c opensc_pkcs11!_strdup+0x3f [f:\dd\vctools\crt\crtw32\string\strdup.c @ 80] 
05 00000061`1ddfc1b0 00007fff`d983c99d opensc_pkcs11!pcsc_add_reader+0x90 [c:\projects\opensc\src\libopensc\reader-pcsc.c @ 1274] 
06 00000061`1ddfc1e0 00007fff`d97aa395 opensc_pkcs11!pcsc_detect_readers+0x27d [c:\projects\opensc\src\libopensc\reader-pcsc.c @ 1403] 
07 00000061`1ddfc270 00007fff`d979a176 opensc_pkcs11!sc_ctx_detect_readers+0x39 [c:\projects\opensc\src\libopensc\ctx.c @ 733] 
08 00000061`1ddfc2a0 00007fff`f86ad296 opensc_pkcs11!C_GetSlotList+0x7e [c:\projects\opensc\src\pkcs11\pkcs11-global.c @ 407] 
09 00000061`1ddfc2f0 00007fff`e12e7434 nss3!SECMOD_UpdateSlotList+0x66 [/builds/worker/checkouts/gecko/security/nss/lib/pk11wrap/pk11util.c @ 1028] 
0a 00000061`1ddfc360 00007fff`e12f0351 xul!nsNSSComponent::CheckForSmartCardChanges+0xe4 [/builds/worker/checkouts/gecko/security/manager/ssl/nsNSSComponent.cpp @ 849] 
0b (Inline Function) --------`-------- xul!CheckForSmartCardChanges+0x3b [/builds/worker/checkouts/gecko/security/manager/ssl/nsNSSComponent.h @ 149] 
0c 00000061`1ddfc3e0 00007fff`e29313b2 xul!nsPKCS11Module::ListSlots+0x61 [/builds/worker/checkouts/gecko/security/manager/ssl/nsPKCS11Slot.cpp @ 250] 
0d 00000061`1ddfc470 00007fff`ddccf573 xul!XPTC__InvokebyIndex+0x72

from pcsc_add_reader

0:000> dv
            ctx = 0x000001a2`cefff5d0
    reader_name = 0x00000061`1ddfc228 " \i???"
reader_name_len = 0x000001a2`c680a74f
     out_reader = 0x000001a2`cedd1840

reader_name and reader_name_len appear to be bogus. This is likely an upstream issue that Firefox can't necessarily fix.

Comment 21

[J.C. Jones [:jcj] (he/him)]

Can you try:

1. Downloading Firefox Beta (75)
2. Ensure the e-residency software is not installed in Firefox Beta
3. Navigate to about:config and search for security.osclientcerts.autoload
4. Set security.osclientcerts.autoload to true
5. Try to use a website that still needs the e-residency Smart Card and see if it works?

If so, then I think the realistic resolution here is to recommend setting security.osclientcerts.autoload to true for Firefox users instead of using OpenSC. I suspect you're right that this is an upstream issue.

Comment 22

[leifeld]

With security.osclientcerts.autoload on true on 75 I can log in with the e-Residency passport!
With it on false I cannot, and I removed the dll of the software some time ago.

But on the e-residency website the they still recommend the installation of the software.

Comment 23

[bbigby64]

I'm seeing sudden crashes without a crash report on Fedora 31 Linux, too. Sometimes, I'd click in an input field and poof! Window gone! Crash! I don't know whether hardware acceleration is causing the issue, but here is my about:config hardware acceleration configuration:

layers.acceleration.force-enabled
webgl.force-enabled
layers.offmainthreadcomposition.enabled

Hardware and Software

Firefox Quantum 74.0 (64-bit)

OS: Linux 5.5.10-200.fc31.x86_64 #1 SMP Wed Mar 18 14:21:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Processor: AMD Ryzen 9 3950X 16-Core Processor

GPU: Advanced Micro Devices, Inc. 5700 XT

Memory:
total used free shared buff/cache available
Mem: 32817400 4936808 19026748 262944 8853844 27154076
Swap: 66590716 0 66590716

Comment 24

[Darkspirit]

(In reply to bbigby64 from comment #23)
Please file a new bug for that and attach your about:support to it. Thanks.

Comment 25

[bbigby64]

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #24)

(In reply to bbigby64 from comment #23)
Please file a new bug for that and attach your about:support to it. Thanks.

Okay. I shall.

Comment 26

[frankmorgner]

This looks like an OpenSC issue. Debug symbols are available here. Despite the trace above (thanks!), I cannot immediately see what's going wrong. It would be interesting to see what the value of the internal reader_buf is, which is returned from SCardListReaders. If you wish we can discuss this in the OpenSC issue.

Comment 27

[Eugene Savitsky]

I'm reporter of 1621804.

Installed an updated ID-card SW. https://id.ee/?id=30519&read=39806

Same crashes on FF78b.

Comment 28

[Darkspirit]

It's the plugin that crashes. It doesn't seem to be fixed yet:

(J.C. Jones [:jcj] (he/him) from bug 1636100 comment 9)

The upstream issue at OpenSC is https://github.com/OpenSC/OpenSC/issues/1999.

They ask affected users for debug information:
https://github.com/OpenSC/OpenSC/issues/1999#issuecomment-627297400
https://github.com/OpenSC/OpenSC/issues/1999#issuecomment-628531648

Comment 29

[Gabriele Svelto [:gsvelto] (PTO)]

FYI we've unblocked the most recent release of OpenSC upon request from the team developing the Estonian e-Residency software which explains the spike in crashes in the last month. They claimed a new version - integrating the recent OpenSC fixes - would be available soon. I'll ping them to know what's going on. We're in a really bad spot here: either our users experience crashes or we have to disable some important functionality they rely on.

Comment 31

[Eugene Savitsky]

The latest version is still crashing:
ID-software version: 20.05, DigiDoc4 client version: 4.2.5.76, released 08.06.2020

https://installer.id.ee/?lang=eng

Comment 32

[Wayne Mery (:wsmwk)]

Also crashes Thunderbird

Comment 33

[Kristjan]

Win10 64-bit Build 19042 ,FF 77.0.1 and DigiDoc4 client 4.2.5.76, 20.05 release- no crash for me. Tried with 2 different pc with same version.
Only difference is that I'm using ID-card, not e-resident one.

If anyone can reproduce the issue, please share debug information:
https://github.com/OpenSC/OpenSC/issues/1999#issuecomment-627297400
https://github.com/OpenSC/OpenSC/issues/1999#issuecomment-628531648

Comment 34

[Kristjan]

(In reply to Eugene Savitsky from comment #31)

The latest version is still crashing:
ID-software version: 20.05, DigiDoc4 client version: 4.2.5.76, released 08.06.2020

https://installer.id.ee/?lang=eng

I would recommend writing to the software developer at [e-mail] and send OpenSC logs there as well. Might speed up troubleshoot & fixes.

Comment 35

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

This isn't a bug in Firefox, so I'm going to close this. For users experiencing this, please turn on osclientcerts instead by setting security.osclientcerts.autoload to true in about:config.

Comment 36

[Gabriele Svelto [:gsvelto] (PTO)]

Note that thanks to bug 1633052 this type of crashes will never be silent again. The next time we run into something like this Firefox will generate a proper crash report which should help both identify the issue and let upstream projects develop a fix.

Comment 37

[Rasmus]

Hello,
I updated my Windows 10 to the 2004 version recently I started getting these sudden crashes (no crash report). One even took place in the middle of writing this message. I tried using Dana's solution of changing the osclientcerts to true (is false by default) but to no avail. I also tried to disable the Estonian ID plugins, but that did not help either.

Mozilla Firefox: 79.0 (64-bit)
Windows 10 Professional: Version 2004, 19041.450
Windows Feature Experience Pack: 120.2212.31.0
Estonian ID plugins installed:

PKSC11 Loader: version 1.0.5
Token signing: version 0.0.31

Please provide assistance/ideas how to prevent Firefox from crashing.

Thanks in advance,
Rasmus

Comment 38

[Rasmus]

Attachment: Report contents from Mozilla Bug Reporter

Seems that there is no way to edit my posts here so adding some extra. The issues is happening several times in an hour and is getting very annoying. Is there any way to reopen this case or direct me to a new one?

This is from the bug reporter.

Comment 39

[Rasmus]

Submitted the crash report as well and it does look that the same issue has indeed resurfaced: https://crash-stats.mozilla.org/report/index/ced71081-7b15-4bde-b44d-4f7b60200816

Please reopen this bug or provide a working workaround.

Comment 40

[Kristjan]

(In reply to Rasmus from comment #39)

Submitted the crash report as well and it does look that the same issue has indeed resurfaced: https://crash-stats.mozilla.org/report/index/ced71081-7b15-4bde-b44d-4f7b60200816

Please reopen this bug or provide a working workaround.

As Dana said, this isn't Firefox bug rather coming from OpenSC https://github.com/OpenSC/OpenSC/issues/1999#issuecomment-627297400.
Seems like the problem found a solution and needs to be implemented to releases https://github.com/OpenSC/OpenSC/projects/7.
I would suggest using alternative browsers or run FF in safe mode before OpenSC and Estonian ID release a newer version.

Comment 41

[Rasmus]

Yes decided to go for that, mostly because I do not use the actual ID software features that much. So I reinstalled the client without the Firefox plugins and will use other browsers when necessary.

Comment 42

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

If you use osclientcerts in Firefox Beta (80) (without the Estonian ID plugins), does it work for you? (as in, can you authenticate to the websites you need to authenticate to?)

Comment 43

[Rasmus]

That is a lot of testing you want me to do without clear and simple instructions how to do it. I already applied the workaround of re-installing the ID software without Firefox plugins (the only way to remove them apparently) and will use Chrome/Edge to do any ID plugin required tasks if they come up in the future. My case can be considered as closed.

Comment 44

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

1. Download and install Firefox Beta: https://www.mozilla.org/en-US/firefox/channel/desktop/
2. Open Firefox Beta
3. Navigate to about:config
4. Search for security.osclientcerts.autoload
5. Click the toggle button on the far right, which should set the value of this preference to true
6. Attempt to do something that would have required the ID plugin

Comment 45

[Giusy]

hi to you all, apologize my poor english and skills
i only if this bug with the impossibility of loading asepkcs.dll
has been fixed
i'm now running the stable last ff vers 80 but i'm still unable to load that module in order to have my Athena smart card running
thxs so much indeed for yr kind attention and patience too
cheers

Comment 46

[Dmitri]

Still happening in Firefox Nightly... removed onepin-opensc-pkcs11.dll file and all good now.

Comment 47

[Giusy]

so also in vers stable 80 latest this trick is working and permitt to load the asepkcs.dll...??

Comment 48

[Gabriele Svelto [:gsvelto] (PTO)]

(In reply to Giusy from comment #45)

hi to you all, apologize my poor english and skills
i only if this bug with the impossibility of loading asepkcs.dll
has been fixed
i'm now running the stable last ff vers 80 but i'm still unable to load that module in order to have my Athena smart card running
thxs so much indeed for yr kind attention and patience too
cheers

The module is not blocked anymore and it should load correctly. We removed the block at the request of the Estonian Information System Authority so the most recent version of the plug-in will load with Firefox 80. I do not know if the crash was also resolved in the most recent version of the Estonian ID-card software; it was resolved in upstream OpenSC though. Unfortunately when the ID-card software picks up the latest version of OpenSC is outside of our control, what you can do in the meantime is follow the instructions in comment 44.

Comment 49

[Giusy]

hi Gabriele,
first let me thank you so much for yr kind reply,
now i have finally be able to re-load the asepkcs.dll!
i have the oldest vers 6.5.0.5
otherwise the method of post 44 did not work for me...
i mean that if i've do not have the dll loaded also if i set in about config the parameter from false to true i'm really not able to navigate the site for which i've my athena smart card
about the Estonian asepkcs.dll how to have it?
cause is surely newest then mine and i'm afraid tha probably in next future ff will block again the dll probably starting from the oldest vers like mine
another user: Alcr in bugzilla n. 1629002 talks about:
"@Alcr. in yr post 3 you state to have the :

I think that I found a more recent asepkcs.dll from the Colegio de Abogados La Plata website:

http://www.calp.org.ar/download/controladores-token

(Controlador para Athena ID-Protect para Windows de 64 bits.)
asepkcs.dll version is 7.0.2.0"

...but how he did..??...if you dl the file is a zipped that require a password...:

Controlador para Athena ID-Protect en Windows

https://www.macroseguridad.net/soporte/download/detectar.php

to sum up now finally my ff works like in the past with the .dll loaded but is a very old vers 6.5.0.5 2016 so i'm afraid that this situation may vary in the very netx time..

cheers

Comment 50

[Eugene Savitsky]

The issue in Estonian ID SW still persist.

They plan to release a new version in September. But will it resolve the issue - is I understand it should be first fixed in OpenSC module:
https://github.com/OpenSC/OpenSC/projects/7
Which will happen in ver. 0.21, but it has yet to be released.

https://www.id.ee/en/article/the-next-version-of-the-id-card-software-will-introduce-several-significant-changes/