1458562 - Crash in UserCallWinProcCheckWow after win10 1803 update with 3rd party smartcard software

Status: RESOLVED DUPLICATE of bug 1560052

(External Software Affecting Firefox :: Other, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is
report bp-50ff65ce-0b4d-4ebd-8005-7665e0180502.
=============================================================

Top 6 frames of crashing thread:

0  @0x1800a6fd0 
1 user32.dll UserCallWinProcCheckWow 
2 user32.dll DispatchClientMessage 
3 user32.dll _fnINDEVICECHANGE 
4 ntdll.dll KiUserCallbackDispatch 
5 win32u.dll NtUserGetMessage 

=============================================================

these crashes are regressing after microsoft started rolling out their win10 april update (1803). this is accounting for 8% of browser crashes from release builds on the new platform currently.

i checked a number of crash reports and they all had the module aseVCAPI.dll 6.0.0.9 hooking into the process ("aseVCAPI Dynamic Link Library" digitally signed by "Athena Smartcard Solutions"). their homepage was athena-scs.com but they seem to have be acquired by a different company.

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late to fix in 59. The 60 release is next week.

Comment 2

[Daniel Veditz [:dveditz]]

I don't see a huge spike in April, this seems like a fairly constant crash since the Big Forgetting. Is there some subset of these crashes you can give a better query for? Sounds like there could be multiple causes under this signature.

It does look like 71% of the crashes have aseVCAPI.dll. That vendor does have a security reporting page so maybe we should take that approach. But we don't have very much in the way of STR :-(

Comment 3

[[:philipp]]

sure, this crash query should cover mainly reports what i've filed this bug for:
https://crash-stats.mozilla.com/signature/?platform_version=%3D10.0.17134&platform_version=%3D10.0.17133&signature=UserCallWinProcCheckWow&date=%3E%3D2018-03-01#reports

revisiting the reports, the crash seems to be mainly affecting users with italian builds (so this smartcard stuff might be more commonly used there).
the translated comment at bp-c2db5b05-d726-4a68-ace5-da90a0180419 might provides some clues as to how to reproduce this: "Crashes continuously. When you switch from firefox to another window the browser stops. When I copy text or even just an address from the address bar the browser freezes"

Comment 4

[Marco Castelluccio [:marco]]

It looks like the "Athena" Smart Card is distributed by one of the most popular certified email providers in Italy.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Are we resolving this bug?  (the move to core-security-release)  -- I hadn't seen the comments above as disposing of this bug yet.

Comment 6

[Daniel Veditz [:dveditz]]

It's not resolved but it doesn't have much useful information either. core-security-release lets a broader set of security-minded folks take a look, particularly our QA folks.

I don't think this is our bug though so I'm not sure it does us much good keeping it open.

Comment 7

[[:philipp]]

(In reply to Marco Castelluccio [:marco] from comment #4)
> It looks like the "Athena" Smart Card is distributed by one of the most
> popular certified email providers in Italy.

can we try to get in contact with them? (not really sure how to get this bug moving forward otherwise...)

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Note: volume of crashes is spiking hard.. Likely the 60.0 release hitting.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Actually, the spike is Thunderbird 52.7.0 crashes; Firefox crashes have held steady-ish.
(Thunderbird dominates the first and third signature, spiking in both)

Comment 10

[[:philipp]]

https://crash-stats.mozilla.com/signature/?signature=UserCallWinProcCheckWow&date=>%3D2018-04-15#graphs

Comment 11

[[:philipp]]

https://crash-stats.mozilla.com/signature/?signature=UserCallWinProcCheckWow&date=%3E%3D2018-04-15#graphs

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

On the Thunderbird crashes, user comments mention a Windows update from two days ago (March 15)

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Magnus, fyi.

Comment 14

[Magnus Melin [:mkmelin]]

Thx. I don't think I have anything to add.

Comment 15

[Daniel Veditz [:dveditz]]

I think we should unhide this and hand it off to the folks who reach out to anti-virus vendors for topcrash fixes.

Comment 16

[Marco Castelluccio [:marco]]

I can try to contact Aruba, but the actual people we should contact are the ones behind the Smart Card (so "Athena").

Adam, can you find contacts for Athena?

Comment 17

[Wayne Mery (:wsmwk)]

Is there a known workaround?

In bug 1463153 I discovered 80% of crashes are Italian locale - https://crash-stats.mozilla.com/search/?signature=~ThreadStart&product=Thunderbird&date=%3E%3D2018-05-13T19%3A38%3A12.000Z&date=%3C2018-05-22T19%3A38%3A00.000Z&_sort=user_comments&_sort=-date&_facets=signature&_facets=useragent_locale&_facets=platform_version&_columns=date&_columns=signature&_columns=user_comments&_columns=useragent_locale&_columns=platform_version#facet-useragent_locale

Comment 18

[Adam Stevenson [:adamopenweb]]

Reaching out to their CTO on LinkedIn. Also talked to support at NXP, they suggested creating a ticket on https://nxpcommunity.force.com. 

Case number: 00163891

Comment 19

[Wayne Mery (:wsmwk)]

Reporter of bp-f20a2bac-2030-4ab6-9baf-00acd0180516 tells me "I updated Athena drivers for ID Protect Monitor and now Thunderbird crashes seems are gone. ...  I downloaded from the following page the Windows driver 6.44.10 (section "ACTALIS" in the drivers table)  http://sistemats1.sanita.finanze.it/wps/content/portale_tessera_sanitaria/sts_sanita/home/il+cittadino+e+la+tessera/come+si+attiva+la+cns/elenco+driver+tscns "

Comment 22

[Adam Stevenson [:adamopenweb]]

Our support contact is trying to find the right person internally. It seems support may not be up to speed with regards to Athena yet, but they are trying to help.

Comment 23

[Wayne Mery (:wsmwk)]

This crash continues to be quite serious.

https://nxpcommunity.force.com/s/login/ is dead - no response other than some background image. Bad link?

Comment 24

[Adam Stevenson [:adamopenweb]]

Ah sorry, just made contact with them yesterday. Engaging via email now.

Comment 25

[Wayne Mery (:wsmwk)]

I don't have good insight to Firefox crashes. Affected Thunderbird users (installs) per https://crash-stats.mozilla.com/topcrashers/?product=Thunderbird&version=52.8.0&days=28 (28 days) and https://crash-stats.mozilla.com/topcrashers/?product=Thunderbird&version=60.0b7&days=7 (only 7 days)

* 5,190 _RtlUserThreadStart | _RtlUserThreadStart (~100% athena)
* 4,938 UserCallWinProcCheckWow (I doubt these are all Athena users)
* 313 _cexit
* 27 patched_BaseThreadInitThunk (beta)
* 27 UserCallWinProcCheckWow (beta)

Thunderbird is definitely disproportionately affected. Percent of crashes which *are Windows 10.0.17134 *
            _RtlUserThreadStart | 
            _RtlUserThreadStart       UserCallWinProcCheckWow 
Firefox     30% of crashes            75% of crashes 
Thunderbird 98% of crashes            99% of crashes

If vendor cannot reproduce then we will need to investigate blocklisting. Factors: unknown what percentage of Athena users are impacted, uknown how Windows 10.0.17134 factors into this.

Comment 26

[Marco Castelluccio [:marco]]

If the vendor asks for a crash dump, we could ask the reporter of bug 1471823.

Adam, any news from them?

Comment 27

[Wayne Mery (:wsmwk)]

Thanks to Adam I've been in contact with the vendor and a reporter. 

User reports 6.44.10 works from section "ACTALIS" in the drivers table for Windows http://sistemats1.sanita.finanze.it/wps/content/portale_tessera_sanitaria/sts_sanita/home/il+cittadino+e+la+tessera/come+si+attiva+la+cns/elenco+driver+tscns  "

Vendor reported the following on Thursday (I have mashed two emails together, which I hope is accurate): 

The latest version of aseVCAPI.dll that is used is 6.0.0.9. This version [of aseVCAPI.dll] has been in use from IDPC (IDProtect Client) version 6.26 and has not been changed or update since.

In IDPC version 7.16.00 it [aseVCAPI.dll] has been deprecated as there is no more use for it.

Note that clients from Italy use IDProtect Client V6 only. The IDProtect Client in Italy is mostly provided as part of the File Protector suite and not a standalone installer.

It seems that www.card.infocamere.it has bundled the FileProtector with our *old* middleware. Please be aware that IDPC version 6.26.14 is almost 5 years old! Much before Windows 10 was even announced, it is even not compatible with Windows 8.1.

We have already contacted the integrator [Actalis] and brought this to their attention. I believe they will update the bundle to include the latest version of IDProtect Client.

We attempted reproducing the reported behavior with IDPC 6.24 however have yet to succeed.
Furthermore, the clients we contacted have all upgraded to the latest IPDC and do not face any issues.

Minimal version for Win10 support is 6.40.01. 
We are not aware of any other such products [that integrate IDPC].

Comment 28

[Marco Castelluccio [:marco]]

From the reports I saw, I'm not sure it was just www.card.infocamere.it, but we'll see how the story unfolds.

Comment 29

[Wayne Mery (:wsmwk)]

I am looking for results from user update attempts before declaring a solid workaround.

I agree, I don't think it is just the Italian government.  But definitely per the following the Italian government will need to put updated software at https://www.card.infocamere.it/infocard/pub/download-software_5543.  User Giovanni, reported good results from the vendor and this info ... "IDP is installed with Actalis' Fileprotector (used to sign files using CNS card [1]) downloaded from official italian goverment site https://www.card.infocamere.it/infocard/pub/download-software_5543 [which lists File Protector ver. 6.5.3 per Windows 32/64bit].  The IDP 6.26.14 italian version [is] bundled." 
[1] CNS - National Service Card - https://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&rurl=translate.google.com&sl=auto&sp=nmt4&tl=en&u=https://www.card.infocamere.it/infocard/pub/cns-a-cosa-serve_5473&xid=17259,15700021,15700124,15700149,15700168,15700173,15700186,15700191,15700201,15700208&usg=ALkJrhhwS1t4DCvv8OvAbZobgVAa_2WSMg

Comment 30

[Marco Castelluccio [:marco]]

I think another one is Aruba, which is an Italian certified email provider.

Comment 32

[Jim Mathies [:jimm]]

Based on what I'm seeing via telemetry, this software causes instability. Why don't we just block these dlls? I need to confirm this software doesn't use injection methods we don't have a work around for, but if that looks good we can easily block. Here's some data for Firefox from our 3rd party modules data which is collected on nightly and early beta:

Org names that sign 'asepkcs.dll'

Name	Count
Athena Smartcard Solutions	1,659
NXP Semiconductors Austria GmbH	55
NXP Semiconductors	8

DLLs associated with Athena Smartcard Solutions product(s)

["asevcapi.dll","asepkcs.dll","cnstoken.dll","lasertoken.dll","asepcostoken.dll","asepindialog.dll"]

Version distribution

signed_by	file_version	total_clients
Athena Smartcard Solutions	6.5.0.1	224
Athena Smartcard Solutions	6.5.0.5	208
Athena Smartcard Solutions	5.0.3.3	29
Athena Smartcard Solutions	5.0.2.9	18
Athena Smartcard Solutions	5.0.2.1	15
Athena Smartcard Solutions	6.1.9.0	12
Athena Smartcard Solutions	6.3.0.1	11
Athena Smartcard Solutions	6.1.5.0	9
Athena Smartcard Solutions	5.0.2.2	8
Athena Smartcard Solutions	5.0.3.2	3
Athena Smartcard Solutions	5.0.1.6	1

Crash counts per version

dll_version	crash_count
6.5.0.1	1586
6.5.0.5	1182
5.0.3.3	229
6.3.0.1	221
5.0.3.2	86
6.1.9.0	85
5.0.2.1	85
7.0.2.0	42
6.1.5.0	21
7.0.2.4	20
5.0.2.2	17
5.0.1.6	5
5.0.2.6	5
5.0.2.9	5
7.0.2.1	5
6.4.0.1	2
5.0.2.5	1
5.0.0.3	1

Comment 33

[Jim Mathies [:jimm]]

I also checked aseVCAPI.dll -

version	crashes
6.0.0.9	19,062
3.9.0.0	3,143

I don't see any reason to keep this software running in Firefox, it clearly isn't well written or maintained.