Closed Bug 1560052 Opened 5 years ago Closed 5 years ago

Crash in [@ asepkcs.dll | UserCallWinProcCheckWow]

Categories

(External Software Affecting Firefox :: Other, defect, P2)

Product:
External Software Affecting Firefox
Component:
Other
Platform:
Unspecified
Windows 10
Type:
defect

Tracking

(firefox-esr68 wontfix, firefox67 wontfix, firefox68 wontfix, firefox69 wontfix, firefox74 wontfix, firefox75 fixed, firefox76 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr68 --- wontfix
firefox67 --- wontfix
firefox68 --- wontfix
firefox69 --- wontfix
firefox74 --- wontfix
firefox75 --- fixed
firefox76 --- fixed

People

(Reporter: pascalc, Assigned: toshi)

References

Details

(Keywords: crash, topcrash-thunderbird, Whiteboard: [tbird topcrash])

Crash Data

Attachments

(2 files)

Bug 1560052 - Block Athena's PKCS11 module to avoid the crash. r=jmathies
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-beta+
Details | Review
block-asepkcs-dll.txt
1.86 KB, text/plain
Details

This bug is for crash report bp-805f4e51-2c64-434e-bcc7-29ee20190618.

Top 6 frames of crashing thread:

0 asepkcs.dll asepkcs.dll@0xae280 
1 user32.dll UserCallWinProcCheckWow 
2 user32.dll DispatchClientMessage 
3 user32.dll _fnINDEVICECHANGE 
4 ntdll.dll KiUserCallbackDispatch 
5 win32u.dll NtUserGetMessage

This crash signature has been associated with Athena smartcard or other devices which use old code.

Depends on: 1458562
Whiteboard: [tbird crash]

Strong correlation to the Athena Smart Card (as Wayne suspected in Comment 1) : (100.0% in signature vs 00.16% overall) Module "aseVCAPI.dll" = true. Moving this to a better home based on that.

Component: Untriaged → Other
Product: Firefox → External Software Affecting Firefox

The priority flag is not set for this bug.
:marcia, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(mozillamarcia.knous)

Making this a P2 for now. Adam or Peter - do either of you have any contacts at Athena? Almost 19K crashes on 68 release, but not yet that visible on 69.

status-firefox69: --- → affected
Flags: needinfo?(stpeter)
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(astevenson)
Priority: -- → P2

philipp mentioned this is likely a dupe of Bug 1458562, and that apparently the newer versions of the dlls aren't crashing anymore but no one is updating based on the crash volume.

Hi :marcia do you still need outreach to folks at Athena? I don't have active contacts there, but I'm happy to see what connections I can forge.

Flags: needinfo?(stpeter)

(In reply to Peter Saint-Andre [:stpeter] from comment #6)

Hi :marcia do you still need outreach to folks at Athena? I don't have active contacts there, but I'm happy to see what connections I can forge.

I think we probably should reach out to them, at least to find out if they can encourage their users to update to the new version.

Marcia, apologies for letting this slip. Is this still a priority?

We have contacts at Athena and have connected Wayne with them in the past. Lmk.

Flags: needinfo?(astevenson) → needinfo?(mozillamarcia.knous)

(In reply to Adam Stevenson [:adamopenweb] from comment #8)

Marcia, apologies for letting this slip. Is this still a priority?

We have contacts at Athena and have connected Wayne with them in the past. Lmk.

We had about 4500 crashes in the 68 cycle. In 69 so far we only have under 400. We should probably wait and see how the volume plays out in 69 and then decide if should do outreach.

Flags: needinfo?(mozillamarcia.knous)

about 1,500 v69 crashes in the past month. https://crash-stats.mozilla.com/signature/?product=Firefox&signature=asepkcs.dll%20%7C%20UserCallWinProcCheckWow&date=%3E%3D2019-07-06T22%3A16%3A00.000Z&date=%3C2019-10-06T22%3A16%3A00.000Z

Crash Signature: [@ asepkcs.dll | UserCallWinProcCheckWow] → [@ asepkcs.dll | UserCallWinProcCheckWow] [@ asepkcs.dll | patched_BaseThreadInitThunk ]

Now a topcrash for Thunderbird 68. And Thunderbird crash rate is 5x Firefox's crash rate https://crash-stats.mozilla.com/signature/?signature=asepkcs.dll%20%7C%20UserCallWinProcCheckWow&date=%3E%3D2019-07-16T10%3A31%3A00.000Z&date=%3C2020-01-16T10%3A31%3A00.000Z#graphs

I haven't had contact with Athena folks since June 2018. I'll reach out again.

AIUI the latest drivers should be at https://www.pec.it/download-software-driver.aspx

Keywords: topcrash-thunderbird
Whiteboard: [tbird crash] → [tbird topcrash]

According to my contact at Athena on 1/19/2020 - in reference to examining Thunderbird https://crash-stats.mozilla.com/report/index/57522292-20a4-4397-a8db-42d870200116#tab-modules and Firefox crash https://crash-stats.mozilla.com/report/index/b9e56e02-18ad-4302-a7ea-f20c70200116#tab-modules - "the user have uses version 6.30 that was released prior to Windows 10 release. The supported version for Windows 10 that has solved the issue is 6.44. We also have version 7 that supports Windows 10, but some users still use version 6 while other use version 7, depending on the project."

A recent example bp-0867aad5-227e-437d-afac-d03670200227 is using 6.0.0.9

So many users are not keeping up with the most recent version.

I am in favor of proceeding with blocklist per bug 1458562

Flags: needinfo?(astevenson)

Hey Toshi,

When you have a chance, could you see if you can correlate publicly available driver software with this dll? There's more information on the specific product and dll in security bug 1458562. I'll cc you into that. I'd like to determine if we think a block of the offending dll would work.

https://athena-scs.com/support/software-driver-downloads.html

Flags: needinfo?(astevenson) → needinfo?(tkikuchi)

Unfortunately the download page above is no longer available. When I click any of the Download links, it's redirected to Athena's homepage.

Learning from the callstacks of Athena modules, I think here's what happened.

  1. First, the module asepkcs.dll is installed as PKCS#11 module on Firefox, and it's loaded here at startup.

  2. As a part of asepkcs's initialization, it loads aseVCAPI.dll here.

  3. They create a new thread in the browser process, and run asepkcs.dll's code on it.

  4. After that, when a user did something (not startup), asepkcs.dll was suddenly unloaded even though the thread created at 3) was still running, resulting in the crash on that thread.

The module is not loaded via injection. So blocking asepkcs.dll (probably 6.5.0.5 and older) will stop the crash. It means the device won't work either, but maybe it's better than the crash.

I also found this discussion where some people said updating ID Protect Monitor solved the crash, which supports comment #13.

Flags: needinfo?(tkikuchi)
Assignee: nobody → tkikuchi
Status: NEW → ASSIGNED

Is review/approval needed?

Flags: needinfo?(tkikuchi)

(In reply to Wayne Mery (:wsmwk) from comment #17)

Is review/approval needed?

Sorry, the patch has been accepted, but its state was Changes Planned. Please hold this off now. I'll get the status updated.

Flags: needinfo?(tkikuchi)
Attached file block-asepkcs-dll.txt —
Pushed by aiakab@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0fbba48e7097 Block Athena's PKCS11 module to avoid the crash. r=jmathies

https://hg.mozilla.org/mozilla-central/rev/0fbba48e7097

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox76: --- → fixed
Resolution: --- → FIXED

The patch landed in nightly and beta is affected.
:toshi, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(tkikuchi)

From Thunderbird perspective this is quite important enough to land on both beta and esr - #2 and #5 crash respectively

Comment on attachment 9130248 [details]
Bug 1560052 - Block Athena's PKCS11 module to avoid the crash. r=jmathies

Beta/Release Uplift Approval Request

  • User impact if declined: Firefox and Thunderbird may crash if Athena's old PKCS#11 module is installed. We're getting about 60 crash reports/day from FF74, and +1000 per day from TB68.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The fix is to block asepkcs.dll (version<=6.5.0.5) from being loaded. We know this is loaded as PKCS#11, so the risk to break browser's functionality is low. This may break a smartcard device's functionality, but that's better than crash and a user should update the driver because the module is deprecated.
  • String changes made/needed: None
Flags: needinfo?(tkikuchi)
Attachment #9130248 - Flags: approval-mozilla-beta?

Comment on attachment 9130248 [details]
Bug 1560052 - Block Athena's PKCS11 module to avoid the crash. r=jmathies

Blocklists a crash-prone DLL. Approved for 75.0b6.

Attachment #9130248 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Crash Signature: [@ asepkcs.dll | UserCallWinProcCheckWow] [@ asepkcs.dll | patched_BaseThreadInitThunk ] → [@ asepkcs.dll | UserCallWinProcCheckWow] [@ asepkcs.dll | patched_BaseThreadInitThunk ] [@ UserCallWinProcCheckWow] [@ __RtlUserThreadStart | _RtlUserThreadStart] [@ patched_BaseThreadInitThunk]
See Also: → 1629002

I still see reports of crashes that have dll v6.0.0.9 in builds that should have this block, like bp-f972c41c-f6c7-4b96-a29b-ef2450200409 buildid 20200330190334. Do we know why that would be?

(once we know it is working we will need this on 68 esr for Thunderbird, where dll version 6.0.0.9 is prevalent.)

Oddly, and this can't be related, Firefox (old versions like 52x) actually experienced a 100x increase in the crash signature at the time the blocklist landed https://crash-stats.mozilla.org/signature/?signature=UserCallWinProcCheckWow&date=%3E%3D2020-01-11T16%3A06%3A00.000Z&date=%3C2020-04-11T16%3A06%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#graphs
** bp-cd8382ab-1db8-4511-b7b6-800e10200326 Firefox
** bp-65eebcc1-e982-4e91-bd21-244760200326 Firefox
** bp-ea291fd6-5ba9-4eb6-bfac-d11ed0200326 Thunderbird

Crash Signature: [@ asepkcs.dll | UserCallWinProcCheckWow] [@ asepkcs.dll | patched_BaseThreadInitThunk ] [@ UserCallWinProcCheckWow] [@ __RtlUserThreadStart | _RtlUserThreadStart] [@ patched_BaseThreadInitThunk] → [@ asepkcs.dll | UserCallWinProcCheckWow] [@ asepkcs.dll | patched_BaseThreadInitThunk ] [@ UserCallWinProcCheckWow] [@ __RtlUserThreadStart | _RtlUserThreadStart] [@ patched_BaseThreadInitThunk]
Flags: needinfo?(tkikuchi)

Thunderbird build 20200330190334 is 75.0b3. Not sure about which snapshot was picked to build TB release, but I confirmed asepkcs.dll was not included in the blocklist of mozglue.dll of 20200330190334. So that crash is expected.

Nothing was changed in ESR. Maybe some people switched to ESR or an older version because their smartcard device stopped working on a new release..?

Flags: needinfo?(tkikuchi)

(In reply to Wayne Mery (:wsmwk) from comment #30)

Per graph, the thunderbird beta crash rate is greatly reduced since very late March [1]. Which is promising enough to take this block on ESR?

However, there are lingering crashes for more recent builds [2]. For example https://crash-stats.mozilla.org/report/index/2fb093ba-5853-4ef8-8870-f33770200429#tab-modules Thunderbird 76 beta shows dll v6.0.0.9

That module of v6.0.0.9 is aseVCAPI.dll, but the module we blocked is asepkcs.dll v6.5.0.5 or older.

We chose asepkcs.dll because it is first loaded by our process, and then aseVCAPI.dll is loaded by aseVCAPI.dll as described in comment #15.

We chose v6.5.0.5 because it's the version bundled in the downloadable Italian product, but the latest data implies asepkcs.dll newer than v6.5.0.5 is still causing the same issue.

Unfortunately when this crash happens, asepkcs.dll has been unloaded, meaning we're not sure which version is good or bad. We can move the blocking border from 6.5.0.5 to v6.6.0.0 or v7.0.0.0, but it's risky because it may block working versions.

Given that blocking v6.5.0.5 or older apparently reduced the crash numbers, I think it's good and safe to uplift this to ESR and see how it improves the number.

Flags: needinfo?(tkikuchi)

This fix is preventing Italian people from using their ID to access Government site.
Is there any way to manually allow the dll?
Like an about:config setting or similar?

Thank you

(In reply to lore.gnafu from comment #32)

This fix is preventing Italian people from using their ID to access Government site.
Is there any way to manually allow the dll?
Like an about:config setting or similar?

No, it's a hardcoded setting. Could you check a newer driver (probably for a smartcard reader) is available?

I have another question. If you used to hit the crash before this fix was shipped, do you know what operation triggered the crash (e.g. just starting Firefox, inserting a smartcard, or etc.)?

Flags: needinfo?(lore.gnafu)

This fix is preventing Italian people from using their ID to access Government site.

Hi, can you indicate which website and where you downloaded the driver from? The driver has to be updated, then it'll automatically work again (and not crash all the time!), see comment 13. Maybe we can find the right person to contact.

If you set security.osclientcerts.autoload to true in about:config instead of loading this module, are you able to access the sites you're trying to access?

@toshi : I never had any crash. The driver is the one linked by @gcp

@gcp: I couldn't find a newer driver besides the one you linked. The websites that don't work are www.inps.it and agenziaentrate.gov.it

@keeler: with "osclientcerts.autoload" set to true I am able to choose the certificate from the card, but it doesn't ask me for the pin and afterward I get a SEC_ERROR_PKCS11_GENERAL_ERROR

Flags: needinfo?(lore.gnafu)

Thanks for trying that out. Would you mind opening a new bug for that issue: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM
It would be helpful if you could run Firefox with the environment variable RUST_LOG set to osclientcerts_static and attach the output from that to the bug. Thanks!
I'm assuming you're on Windows, so this might help: https://developer.mozilla.org/en-US/docs/Mozilla/Command_Line_Options#Miscellaneous

Flags: needinfo?(lore.gnafu)

Oof, so this means that despite https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c13 the latest available version is v6.5.0.5 which we know is crashing (also foreshadowed here https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c31 pointing out newer versions still crash).

Wayne, you had contact with the vendor. Is there any newer version than the one pointed out? If not...can they investigate the crash? We're in an impossible situation here because this product is making Firefox crash, but if we block it, users lose critical functionality. I guess improving security.osclientcerts.autoload might be a way out if the vendor doesn't manage to fix it?

Flags: needinfo?(vseerror)

If (In reply to lore.gnafu from comment #37)

@toshi : I never had any crash. The driver is the one linked by @gcp

I see, thank you for clarifying this.

If this crash does not happen on all users and there are cases where a device works normally on Firefox, I'm leaning toward unblocking the module.

Can a blocklist entry be set to affect just Thunderbird?

(In reply to Gian-Carlo Pascutto [:gcp] from comment #39)

...
Wayne, you had contact with the vendor. Is there any newer version than the one pointed out? If not...can they investigate the crash? We're in an impossible situation here because this product is making Firefox crash, but if we block it, users lose critical functionality. I guess improving security.osclientcerts.autoload might be a way out if the vendor doesn't manage to fix it?

I'll try to get them on board. I'll also CC you on PM I send to them.

Flags: needinfo?(vseerror) → needinfo?(tkikuchi)

(In reply to Wayne Mery (:wsmwk) from comment #41)

Can a blocklist entry be set to affect just Thunderbird?

There is no such option yet, but adding an option like BLOCK_THUNDERBIRD_ONLY is not difficult.

Flags: needinfo?(tkikuchi)

I want to raise awareness that blocking this library (asepkcs.dll) blocks access to one of the Romanian government institution when using the Athena device. The local certificate provider (that provided the token) commented that since Firefox v74 that device is no longer compatible with Firefox and suggested to use other browsers - which I don't want to do - I might switch the token.
My asepkcs.dll version is 6.5.0.1.

Vladimir, can you try the steps outlined here: https://bugzilla.mozilla.org/show_bug.cgi?id=1560052#c36
And see if you get further? If you get stuck, see comment 38 for info that could help us debug it.

This is an impossible situation because not blocking those card readers causes thousands of Firefox crashes per day...

Flags: needinfo?(red)

:gi(In reply to Gian-Carlo Pascutto [:gcp] from comment #44)
That worked! The certificate is visible in My Certificates list, it shows when I access my government site, then PIN popup shows, I use my PIN, click OK and website works fine.

Is this approach with security.osclientcerts.autoload supposed to work in the future as well ? I'm thinking to inform my certificate provider about this solution, but I want to make sure it's a good one.

Flags: needinfo?(red)

I want to note that I have received another driver from the certificate provider, that contains asepkcs.dll V7.0.2.1 and with that version it works without setting the option suggested above (as expected I presume)

Going forward, using security.osclientcerts.autoload is the preferred solution.

really NOT working for me
for me the about config workround DID really NOT FIX anything..
still having this frustrating BUG
see on mozilla forum:
https://forum.mozillaitalia.org/index.php?topic=74089.0#lastPost
ff 77.0.1 win 10 home 64 vers 2004
asepkcs.dll (version<=6.5.0.5)

See Also: → 1560486

Lore seems to be gone, so clearing needinfo.

Flags: needinfo?(lore.gnafu)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: