Crash in [@ asepkcs.dll | UserCallWinProcCheckWow]
Categories
(External Software Affecting Firefox :: Other, defect, P2)
Tracking
(firefox-esr68 wontfix, firefox67 wontfix, firefox68 wontfix, firefox69 wontfix, firefox74 wontfix, firefox75 fixed, firefox76 fixed)
People
(Reporter: pascalc, Assigned: toshi)
References
Details
(Keywords: crash, topcrash-thunderbird, Whiteboard: [tbird topcrash])
Crash Data
Attachments
(2 files)
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
1.86 KB,
text/plain
|
Details |
This bug is for crash report bp-805f4e51-2c64-434e-bcc7-29ee20190618.
Top 6 frames of crashing thread:
0 asepkcs.dll asepkcs.dll@0xae280
1 user32.dll UserCallWinProcCheckWow
2 user32.dll DispatchClientMessage
3 user32.dll _fnINDEVICECHANGE
4 ntdll.dll KiUserCallbackDispatch
5 win32u.dll NtUserGetMessage
Reporter | ||
Updated•5 years ago
|
Comment 1•5 years ago
|
||
This crash signature has been associated with Athena smartcard or other devices which use old code.
Comment 2•5 years ago
|
||
Strong correlation to the Athena Smart Card (as Wayne suspected in Comment 1) : (100.0% in signature vs 00.16% overall) Module "aseVCAPI.dll" = true. Moving this to a better home based on that.
Comment 3•5 years ago
|
||
The priority flag is not set for this bug.
:marcia, could you have a look please?
For more information, please visit auto_nag documentation.
Comment 4•5 years ago
|
||
Making this a P2 for now. Adam or Peter - do either of you have any contacts at Athena? Almost 19K crashes on 68 release, but not yet that visible on 69.
Comment 5•5 years ago
|
||
philipp mentioned this is likely a dupe of Bug 1458562, and that apparently the newer versions of the dlls aren't crashing anymore but no one is updating based on the crash volume.
Comment 6•5 years ago
|
||
Hi :marcia do you still need outreach to folks at Athena? I don't have active contacts there, but I'm happy to see what connections I can forge.
Comment 7•5 years ago
|
||
(In reply to Peter Saint-Andre [:stpeter] from comment #6)
Hi :marcia do you still need outreach to folks at Athena? I don't have active contacts there, but I'm happy to see what connections I can forge.
I think we probably should reach out to them, at least to find out if they can encourage their users to update to the new version.
Comment 8•5 years ago
|
||
Marcia, apologies for letting this slip. Is this still a priority?
We have contacts at Athena and have connected Wayne with them in the past. Lmk.
Comment 9•5 years ago
|
||
(In reply to Adam Stevenson [:adamopenweb] from comment #8)
Marcia, apologies for letting this slip. Is this still a priority?
We have contacts at Athena and have connected Wayne with them in the past. Lmk.
We had about 4500 crashes in the 68 cycle. In 69 so far we only have under 400. We should probably wait and see how the volume plays out in 69 and then decide if should do outreach.
Comment 11•5 years ago
|
||
about 1,500 v69 crashes in the past month. https://crash-stats.mozilla.com/signature/?product=Firefox&signature=asepkcs.dll%20%7C%20UserCallWinProcCheckWow&date=%3E%3D2019-07-06T22%3A16%3A00.000Z&date=%3C2019-10-06T22%3A16%3A00.000Z
Comment 12•5 years ago
|
||
Now a topcrash for Thunderbird 68. And Thunderbird crash rate is 5x Firefox's crash rate https://crash-stats.mozilla.com/signature/?signature=asepkcs.dll%20%7C%20UserCallWinProcCheckWow&date=%3E%3D2019-07-16T10%3A31%3A00.000Z&date=%3C2020-01-16T10%3A31%3A00.000Z#graphs
I haven't had contact with Athena folks since June 2018. I'll reach out again.
AIUI the latest drivers should be at https://www.pec.it/download-software-driver.aspx
Comment 13•5 years ago
|
||
According to my contact at Athena on 1/19/2020 - in reference to examining Thunderbird https://crash-stats.mozilla.com/report/index/57522292-20a4-4397-a8db-42d870200116#tab-modules and Firefox crash https://crash-stats.mozilla.com/report/index/b9e56e02-18ad-4302-a7ea-f20c70200116#tab-modules - "the user have uses version 6.30 that was released prior to Windows 10 release. The supported version for Windows 10 that has solved the issue is 6.44. We also have version 7 that supports Windows 10, but some users still use version 6 while other use version 7, depending on the project."
A recent example bp-0867aad5-227e-437d-afac-d03670200227 is using 6.0.0.9
So many users are not keeping up with the most recent version.
I am in favor of proceeding with blocklist per bug 1458562
Comment 14•5 years ago
|
||
Hey Toshi,
When you have a chance, could you see if you can correlate publicly available driver software with this dll? There's more information on the specific product and dll in security bug 1458562. I'll cc you into that. I'd like to determine if we think a block of the offending dll would work.
https://athena-scs.com/support/software-driver-downloads.html
Assignee | ||
Comment 15•5 years ago
|
||
Unfortunately the download page above is no longer available. When I click any of the Download links, it's redirected to Athena's homepage.
Learning from the callstacks of Athena modules, I think here's what happened.
-
First, the module
asepkcs.dll
is installed as PKCS#11 module on Firefox, and it's loaded here at startup. -
As a part of asepkcs's initialization, it loads
aseVCAPI.dll
here. -
They create a new thread in the browser process, and run asepkcs.dll's code on it.
-
After that, when a user did something (not startup),
asepkcs.dll
was suddenly unloaded even though the thread created at 3) was still running, resulting in the crash on that thread.
The module is not loaded via injection. So blocking asepkcs.dll
(probably 6.5.0.5 and older) will stop the crash. It means the device won't work either, but maybe it's better than the crash.
I also found this discussion where some people said updating ID Protect Monitor solved the crash, which supports comment #13.
Assignee | ||
Comment 16•5 years ago
|
||
Updated•5 years ago
|
Assignee | ||
Comment 18•5 years ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #17)
Is review/approval needed?
Sorry, the patch has been accepted, but its state was Changes Planned. Please hold this off now. I'll get the status updated.
Assignee | ||
Comment 19•5 years ago
|
||
Comment 20•5 years ago
|
||
Comment 21•5 years ago
|
||
bugherder |
Updated•5 years ago
|
Comment 22•5 years ago
|
||
The patch landed in nightly and beta is affected.
:toshi, is this bug important enough to require an uplift?
If not please set status_beta
to wontfix
.
For more information, please visit auto_nag documentation.
Comment 23•5 years ago
|
||
From Thunderbird perspective this is quite important enough to land on both beta and esr - #2 and #5 crash respectively
Assignee | ||
Comment 24•5 years ago
|
||
Comment on attachment 9130248 [details]
Bug 1560052 - Block Athena's PKCS11 module to avoid the crash. r=jmathies
Beta/Release Uplift Approval Request
- User impact if declined: Firefox and Thunderbird may crash if Athena's old PKCS#11 module is installed. We're getting about 60 crash reports/day from FF74, and +1000 per day from TB68.
- Is this code covered by automated tests?: No
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): The fix is to block asepkcs.dll (version<=6.5.0.5) from being loaded. We know this is loaded as PKCS#11, so the risk to break browser's functionality is low. This may break a smartcard device's functionality, but that's better than crash and a user should update the driver because the module is deprecated.
- String changes made/needed: None
Comment 26•5 years ago
|
||
Comment on attachment 9130248 [details]
Bug 1560052 - Block Athena's PKCS11 module to avoid the crash. r=jmathies
Blocklists a crash-prone DLL. Approved for 75.0b6.
Updated•5 years ago
|
Comment 27•5 years ago
|
||
bugherder uplift |
Comment 28•5 years ago
|
||
I still see reports of crashes that have dll v6.0.0.9 in builds that should have this block, like bp-f972c41c-f6c7-4b96-a29b-ef2450200409 buildid 20200330190334. Do we know why that would be?
(once we know it is working we will need this on 68 esr for Thunderbird, where dll version 6.0.0.9 is prevalent.)
Oddly, and this can't be related, Firefox (old versions like 52x) actually experienced a 100x increase in the crash signature at the time the blocklist landed https://crash-stats.mozilla.org/signature/?signature=UserCallWinProcCheckWow&date=%3E%3D2020-01-11T16%3A06%3A00.000Z&date=%3C2020-04-11T16%3A06%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#graphs
** bp-cd8382ab-1db8-4511-b7b6-800e10200326 Firefox
** bp-65eebcc1-e982-4e91-bd21-244760200326 Firefox
** bp-ea291fd6-5ba9-4eb6-bfac-d11ed0200326 Thunderbird
Assignee | ||
Comment 29•5 years ago
|
||
Thunderbird build 20200330190334 is 75.0b3. Not sure about which snapshot was picked to build TB release, but I confirmed asepkcs.dll was not included in the blocklist of mozglue.dll of 20200330190334. So that crash is expected.
Nothing was changed in ESR. Maybe some people switched to ESR or an older version because their smartcard device stopped working on a new release..?
Comment 30•5 years ago
|
||
Per graph, the thunderbird beta crash rate is greatly reduced since very late March [1]. Which is promising enough to take this block on ESR?
However, there are lingering crashes for more recent builds [2]. For example https://crash-stats.mozilla.org/report/index/2fb093ba-5853-4ef8-8870-f33770200429#tab-modules Thunderbird 76 beta shows dll v6.0.0.9
Assignee | ||
Comment 31•5 years ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #30)
Per graph, the thunderbird beta crash rate is greatly reduced since very late March [1]. Which is promising enough to take this block on ESR?
However, there are lingering crashes for more recent builds [2]. For example https://crash-stats.mozilla.org/report/index/2fb093ba-5853-4ef8-8870-f33770200429#tab-modules Thunderbird 76 beta shows dll v6.0.0.9
That module of v6.0.0.9 is aseVCAPI.dll, but the module we blocked is asepkcs.dll v6.5.0.5 or older.
We chose asepkcs.dll because it is first loaded by our process, and then aseVCAPI.dll is loaded by aseVCAPI.dll as described in comment #15.
We chose v6.5.0.5 because it's the version bundled in the downloadable Italian product, but the latest data implies asepkcs.dll newer than v6.5.0.5 is still causing the same issue.
Unfortunately when this crash happens, asepkcs.dll has been unloaded, meaning we're not sure which version is good or bad. We can move the blocking border from 6.5.0.5 to v6.6.0.0 or v7.0.0.0, but it's risky because it may block working versions.
Given that blocking v6.5.0.5 or older apparently reduced the crash numbers, I think it's good and safe to uplift this to ESR and see how it improves the number.
Comment 32•5 years ago
|
||
This fix is preventing Italian people from using their ID to access Government site.
Is there any way to manually allow the dll?
Like an about:config setting or similar?
Thank you
Assignee | ||
Comment 33•5 years ago
|
||
(In reply to lore.gnafu from comment #32)
This fix is preventing Italian people from using their ID to access Government site.
Is there any way to manually allow the dll?
Like an about:config setting or similar?
No, it's a hardcoded setting. Could you check a newer driver (probably for a smartcard reader) is available?
I have another question. If you used to hit the crash before this fix was shipped, do you know what operation triggered the crash (e.g. just starting Firefox, inserting a smartcard, or etc.)?
Comment 34•5 years ago
|
||
This fix is preventing Italian people from using their ID to access Government site.
Hi, can you indicate which website and where you downloaded the driver from? The driver has to be updated, then it'll automatically work again (and not crash all the time!), see comment 13. Maybe we can find the right person to contact.
Comment 35•5 years ago
|
||
I think the download link is:
https://www.pec.it/download-software-driver.aspx
specifically: https://ca.arubapec.it/downloads/IDP_6.44.10_Windows.zip
Comment 36•5 years ago
|
||
If you set security.osclientcerts.autoload
to true
in about:config
instead of loading this module, are you able to access the sites you're trying to access?
Comment 37•5 years ago
|
||
@toshi : I never had any crash. The driver is the one linked by @gcp
@gcp: I couldn't find a newer driver besides the one you linked. The websites that don't work are www.inps.it and agenziaentrate.gov.it
@keeler: with "osclientcerts.autoload" set to true I am able to choose the certificate from the card, but it doesn't ask me for the pin and afterward I get a SEC_ERROR_PKCS11_GENERAL_ERROR
Comment 38•5 years ago
|
||
Thanks for trying that out. Would you mind opening a new bug for that issue: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM
It would be helpful if you could run Firefox with the environment variable RUST_LOG
set to osclientcerts_static
and attach the output from that to the bug. Thanks!
I'm assuming you're on Windows, so this might help: https://developer.mozilla.org/en-US/docs/Mozilla/Command_Line_Options#Miscellaneous
Comment 39•5 years ago
•
|
||
Oof, so this means that despite https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c13 the latest available version is v6.5.0.5 which we know is crashing (also foreshadowed here https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c31 pointing out newer versions still crash).
Wayne, you had contact with the vendor. Is there any newer version than the one pointed out? If not...can they investigate the crash? We're in an impossible situation here because this product is making Firefox crash, but if we block it, users lose critical functionality. I guess improving security.osclientcerts.autoload
might be a way out if the vendor doesn't manage to fix it?
Assignee | ||
Comment 40•5 years ago
|
||
If (In reply to lore.gnafu from comment #37)
@toshi : I never had any crash. The driver is the one linked by @gcp
I see, thank you for clarifying this.
If this crash does not happen on all users and there are cases where a device works normally on Firefox, I'm leaning toward unblocking the module.
Comment 41•5 years ago
|
||
Can a blocklist entry be set to affect just Thunderbird?
(In reply to Gian-Carlo Pascutto [:gcp] from comment #39)
...
Wayne, you had contact with the vendor. Is there any newer version than the one pointed out? If not...can they investigate the crash? We're in an impossible situation here because this product is making Firefox crash, but if we block it, users lose critical functionality. I guess improvingsecurity.osclientcerts.autoload
might be a way out if the vendor doesn't manage to fix it?
I'll try to get them on board. I'll also CC you on PM I send to them.
Assignee | ||
Comment 42•5 years ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #41)
Can a blocklist entry be set to affect just Thunderbird?
There is no such option yet, but adding an option like BLOCK_THUNDERBIRD_ONLY
is not difficult.
Comment 43•4 years ago
|
||
I want to raise awareness that blocking this library (asepkcs.dll) blocks access to one of the Romanian government institution when using the Athena device. The local certificate provider (that provided the token) commented that since Firefox v74 that device is no longer compatible with Firefox and suggested to use other browsers - which I don't want to do - I might switch the token.
My asepkcs.dll version is 6.5.0.1.
Comment 44•4 years ago
|
||
Vladimir, can you try the steps outlined here: https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c36
And see if you get further? If you get stuck, see comment 38 for info that could help us debug it.
This is an impossible situation because not blocking those card readers causes thousands of Firefox crashes per day...
Comment 45•4 years ago
|
||
:gi(In reply to Gian-Carlo Pascutto [:gcp] from comment #44)
That worked! The certificate is visible in My Certificates list, it shows when I access my government site, then PIN popup shows, I use my PIN, click OK and website works fine.
Is this approach with security.osclientcerts.autoload supposed to work in the future as well ? I'm thinking to inform my certificate provider about this solution, but I want to make sure it's a good one.
Comment 46•4 years ago
|
||
solution |
I want to note that I have received another driver from the certificate provider, that contains asepkcs.dll V7.0.2.1 and with that version it works without setting the option suggested above (as expected I presume)
Comment 47•4 years ago
|
||
solution |
Going forward, using security.osclientcerts.autoload
is the preferred solution.
Comment 48•4 years ago
|
||
really NOT working for me
for me the about config workround DID really NOT FIX anything..
still having this frustrating BUG
see on mozilla forum:
https://forum.mozillaitalia.org/index.php?topic=74089.0#lastPost
ff 77.0.1 win 10 home 64 vers 2004
asepkcs.dll (version<=6.5.0.5)
Description
•