Closed Bug 682839 Opened 13 years ago Closed 4 years ago

Firefox crashes with Estonian pkcs#11 module (slot->pk11slot is probably NULL)

Categories

(Core :: Security, defect)

Product:
Core
Component:
Security
Version:
6 Branch
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: karumaru, Unassigned)

References

Details

(Keywords: crash, Whiteboard: [tbird crash])

Crash Data

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0 Build ID: 20110811165603 Steps to reproduce: Randomly open links in internet. Actual results: At some point Firefox crashes. No specific web page needed. usually crash happens when you have just clicked the link or webpage is loading. Crash IDs bp-60e3ed96-ce5f-492f-b6df-7a8282110829; - in safe mode bp-d04ce781-2dd2-4441-b82a-2ce132110829; bp-1ddc3390-0fa2-4a0d-b192-92ef22110829; bp-65d47de1-33f9-437f-9fcb-7abed2110829; 10cafe05-57cd-422c-b2c2-e68ae2472a64; bp-55ee03dc-2a80-420a-80e3-0629d2110829; bp-c6d60fb5-6e43-4851-99a0-6b1032110828; bp-6f62bd5e-7415-4528-b279-f2b0e2110828; bp-f7fd0863-070e-4d69-893a-9991a2110826; bp-a879e60b-b812-439b-9672-e9dc72110826; bp-36c39aed-a966-45d2-80bc-dfa082110826; bp-5d86ad59-6187-4bc7-a326-e31152110826; bp-5b9631c2-1b5b-4120-a6e1-db2412110826; bp-a5f04b87-039a-4b1e-8b69-843c02110825; bp-1457acd5-6d35-4a48-a68b-b471e2110824; bp-ed8480ea-7b2c-4e06-8d8c-348882110824; bp-2052d295-25dd-483c-8804-87ee22110823; bp-aedc2a89-38d8-4be7-92f7-a6e9a2110823; Expected results: FF should not crash
Application Basics Name Firefox Version 6.0 User Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0 Profile Directory Open Containing Folder Enabled Plugins about:plugins Build Configuration about:buildconfig Extensions Name Version Enabled ID Adblock Plus 1.3.9 true {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} BlockSite 0.7.1.1 true {dd3d7613-0246-469d-bc65-2a3cc1668adc} Browsing Protection 1.10 true litmus-ff@f-secure.com Estonian ID Card PKCS11 module loader 3.4.3756.234 true {aa84ce40-4253-a00a-8cd6-0800200f9a66} Modified Preferences Name Value accessibility.typeaheadfind.flashBar 0 browser.places.smartBookmarksVersion 2 browser.startup.homepage_override.buildID 20110811165603 browser.startup.homepage_override.mstone rv:6.0 extensions.lastAppVersion 6.0 gfx.blacklist.webgl.angle 3 network.cookie.prefsMigrated true places.database.lastMaintenance 1314363010 places.history.expiration.transient_current_max_pages 125270 privacy.sanitize.migrateFx3Prefs true Graphics Adapter Description NVIDIA NVS 4200M Vendor ID 10de Device ID 1056 Adapter RAM 1024 Adapter Drivers nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um Driver Version 8.17.12.7533 Driver Date 5-20-2011 Direct2D Enabled true DirectWrite Enabled true (6.1.7601.17563) ClearType Parameters ClearType parameters not found WebGL Renderer NVIDIA Corporation -- NVS 4200M/PCI/SSE2 -- 4.1.0 GPU Accelerated Windows 1/1 Direct3D 10
Crash Signature: [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ]
Severity: normal → critical
Keywords: crash
What is "_etoured.dll" and "libltdl3.dll" that are injected into the firefox process ?
I think this is a dupe of Bug 679846 based on the common stack and the unversioned dlls in the module of his crash reports.
Crash Signature: [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ] → [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ] [@ PK11_IsDisabled ]
Any ideas how this issue could be solved? Basically when using FF it crasehs after every 30 minutes. I have uninstalled everything, deleted all personal info related to FF and installed clean version. Didn't import anything, no customisation but still crashes. In safe mode it crashes. Scanned computer with F-secure: no threads found. Installed FF3.6 and this is also crashing now. I am ready to provide more specific info if you will ask. I don't want to change browser as I like FF.
>I am ready to provide more specific info if you will ask. see comment #2 Those dlls shows up in your crash report.
Depends on: 716345
I got here via "PK11_IsDisabled" crash signature, the libltdl3.dll is most likely related to the Estonian ID-card software: http://systemexplorer.net/filereviews.php?fid=1798080 (Looking at the directory value; I can't find that file on my system, but I might just have updated version of the software which doesn't include that dll anymore) I've also had problems of Firefox crashing completely randomly when I upgraded to v.10 (including 10.0.2). For me it was fixed by removing "Estonain Id Card" module under security devices (There's an add-on called Estonian ID Card PKCS11 Module Loader which I guess re-registered the right module). Hopefully this will help, especially with PK11_IsDisabled crashes which seem to have hit **** Firefox 10.0.2 Estonian users (considering the foul-mouthed comments): https://crash-stats.mozilla.com/report/list?signature=PK11_IsDisabled
confirmed on the crash data for PK11_IsDisabled - also adding kai to this bug. Kai see comment 7
Status: UNCONFIRMED → NEW
Component: General → Security
Ever confirmed: true
Product: Firefox → Core
QA Contact: general → toolkit
The crash reports were produced with Firefox 6 - which is quite old already. Would you be able to please try with a more modern Firefox, which includes many fixes?
Summary: Firefox crashes at random points all the time- even in SafeMode → Firefox crashes with Estonian pkcs#11 module (slot->pk11slot is probably NULL)
So a little update to get into the current situation: On the 3rd of April version 3.5 SP1 of the ID-card software was released. The changelog is here: http://support.sk.ee/eng/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=1554 It mentions fixing of firefox component, and I haven't noticed any crashes lately. BUT the update checking isn't really often, as I got a pop-up about update being available last Friday (the 4th of May, a month after). So expect people to update the software slowly. That being said the old version of the ID-card software (3.5) does crash Firefox 12.0: https://crash-stats.mozilla.com/report/index/bp-0c6dbedf-d823-4766-a7e5-84a3a2120502 [CRASH] Okay, totally just jinx'd it. Happened while I was scrolling this very page using middle-click: https://crash-stats.mozilla.com/report/index/d85232a9-4466-4e54-bdda-ba1ab2120507
Thanks, so we still crash with modern NSS, in particular NSS 3.13.3 Bob, Wan-Teh, see https://crash-stats.mozilla.com/report/index/d85232a9-4466-4e54-bdda-ba1ab2120507 for the stack. nssTrustDomain_GetActiveSlots iterates tokens, calls PK11_IsDisabled(slot->pk11slot) for each of them, and PK11_IsDisabled crashes trying to dereference the slot. Can you imagine why slot->pk11slot might be invalid or NULL? Should nssTrustDomain_GetActiveSlots check for (slot->pk11slot == NULL) ?
Any updates on this? This is still a problem with Estonian ID Card PKCS11 module loader 3.6.0.670 and latest Firefox beta. Roughly 1000 users per week crash into this according to crash stats. Considering that http://www.id.ee/index.php?id=30011&read=36092 reports about half a million users for this software it means that on average 0,2% of users crash into it weekly. I've seen it twice in last three weeks. Interestingly unlike original reporter crashes seem to be x86 only now.
Merike, I've discussed with our pkcs#11 expert Bob Relyea, and he says my questions cannot be answered easily. Someone with pkcs#11 debugging skills and access to a smartcard would have to attempt to reproduce the issue, to find out if it's more likely to be a bug in the driver software or in the NSS code.
According to latest crashes this also affects Mac and 64bit. And also people with 3.7 series of Estonian ID Card PKCS11 module loader.
OS: Windows 7 → All
Hardware: x86_64 → All
Since it doesn't show any kind of decline whatsoever.. What are the odds to figure this one out with: * one person with pkcs#11 skills * the other with access to a smartcard, the computer that crashed once in December and once in January but not since, and basic debugging skills ?
(In reply to Merike (:merike) from comment #15) > > What are the odds to figure this one out with: > * one person with pkcs#11 skills > * the other with access to a smartcard, the computer that crashed once in > December and once in January but not since, and basic debugging skills > ? It would be necessary to have a reliable test case (crash each time with certain steps) in order to debug the issue.
(In reply to Kai Engert (:kaie) from comment #16) > It would be necessary to have a reliable test case (crash each time with > certain steps) in order to debug the issue. This assumes that it isn't too random or a rare race condition.. The comments on crash reports imply that some people crash into it (or something else) daily or even multiple times a day. Is there anything we could ask them that would help clarify the issue and help to find STR? A possible driver bug was mentioned earlier so perhaps we could ask for reader and driver details at minimum? About current crash reports, are there some recent ones with email addresses from users whose crash-rate is high? Marcia, maybe you know how to find that out?
> perhaps we could ask for reader and driver details at minimum? Yes. It would be good to know, if only one reader model crashes, and maybe even some driver versions crash. Collecting this information will be helpful to either set up a test environment, or tell people to stop using outdated drivers. Are all crash reports on Windows? > Is there anything we > could ask them that would help clarify the issue and help to find STR? It would be best if you can find someone who can never complete a specific activity and always crashes. That would allow the user to repeat the test as often as necessary. If you cannot find someone with such STR, then the crash could be random, and I don't know what we could do. I don't know if the driver for the smartcard is open source. If it were, and if the smartcard works on Linux, too, then you could ask a developer with access to a smartcard to use the "valgrind" memory debugging tool, to check if the smartcard driver has bugs.
(In reply to Kai Engert (:kaie) from comment #18) > Are all crash reports on Windows? No, but linux crashes are extremely rare (none in last 4 weeks) and Mac is also rare (only 2 in last 4 weeks). > I don't know if the driver for the smartcard is open source. If it were, and > if the smartcard works on Linux, too, then you could ask a developer with > access to a smartcard to use the "valgrind" memory debugging tool, to check > if the smartcard driver has bugs. Smartcard is usable on linux. I don't understand the whole stack much but the userspace software that provides Firefox plugin is open source. It needs pcscd and opensc to work and in my case I use binary driver for Omnikey 4040 PCMCIA reader. Source for that is available so debugging should be possible. But I only crashed twice and both times while using rdp to a windows computer (which also has a reader attached) so I'm probably not a good case for reproducing this bug.
I was able to contact one of the frequent crashers. He was also experiencing flash crashes and hangs but also crashes with this signature couple of times a week if using smartcard. The reader was Omnikey 1021 USB-reader which was given out by SEB (one of the two most popular banks here) and is very common. This means it either affects very different readers although from the same manufacturer or isn't reader specific at all. The latest crash happened on outlook.com when clicking reply button. It doesn't reproduce with the same steps for him. Without knowing security code I'm suspecting (especially in the light of bug 550258 not being fixed) it happens when Firefox initiates new ssl connection that accesses smartcard info and something goes wrong with multiple threads assuming different things. But exact trigger conditions aren't any clearer than before. By the way, any ideas why foreign ssl sites would trigger a read on the reader? I'm sometimes seeing it blink when loading some (no recognizable pattern) site which is either ssl itself or references ssl sources. I don't believe any of them actually access card info on purpose.
Thank Merike, The real issue isn't the reader, per se. It's most like the PKCS #11 driver that's in use. PKCS #11 drivers are 'foriegn code' (as in code that's not part of NSS or mozilla, not necessarily code that was developed in another country) the runs as a pluggin. That PKCS #11 module could be doing all sorts of weird things. Basically without a copy of the module and a card, there's very little the NSS team can do, so it's not surprising the problem isn't fixed. bob
(In reply to Robert Relyea from comment #21) It's most like the PKCS #11 driver > that's in use. PKCS #11 drivers are 'foriegn code' (as in code that's not > part of NSS or mozilla, not necessarily code that was developed in another > country) the runs as a pluggin. That PKCS #11 module could be doing all sorts of weird things. You might have made this a bit clearer for me :) By plugin do you mean code like https://svn.eesti.ee/projektid/idkaart_public/trunk/esteid-plugin/firefox-win/? This part is developed in Estonia and if it was possible to point out something they do in a wrong way it may be possible to make them fix it. I'm afraid that getting a test card to someone in NSS team would be more difficult. (standard process goes like https://www.sk.ee/en/services/testcard) Although if you're sure the issue lies in code referenced above then again it might be possible.
Just crashed into this with Thunderbird: https://crash-stats.mozilla.com/report/index/2e4a1057-1d8d-490b-bbd4-739272130901 Right after restarting the following connections were still present: Proto VvJrk SaatJrk Kohalik aadress Väline aadress Olek User Inode PID/Program name tcp 0 0 Mary-Jo.local:38918 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38941 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38945 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38944 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38924 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38929 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:38942 OCSP.SFO1.VERISIGN:http TIME_WAIT root 0 - tcp 0 0 Mary-Jo.local:54647 OCSP.LAX2.VERISIGN:http TIME_WAIT root 0 - I had been using ID-card earlier with Firefox but the card was not inserted during the crash. Actually even pcsd was stopped (I only run it while using the card actively). Admittedly this is not what a regular user does, they have it running all the time unless it crashes? Still, it seems that revocation checking is one way to crash into it. Can anyone clarify when revocation checks are triggered? This might help to figure out STR.
~#200 crash for Thunderbird 24.0.1 for the PK11_IsDisabled signature
Whiteboard: [tbird crash]
This is statistically less of a problem than it used to be: 990 crashes in November and 885 in December. Sadly, still without STR. Might slightly affect Mozilla's income though: https://crash-stats.mozilla.com/report/index/6d8dbd12-b365-4773-a66d-5f27c2141231 has user comment "Maybe 11 time in this week. And it's start hapining when I donate money to Mozilla, strage things :)"
Having Estonian ID software seems to not be a strict requirement to crash, recently a Fennec 36 crash https://crash-stats.mozilla.com/report/index/6e50a939-4bf3-40e9-bda0-f3d542150219 was logged.
I don't know if i am allowed, but adding 1 more crash report on the list. https://crash-stats.mozilla.com/report/index/edfcd04e-1ccc-474c-a73e-3713c2160805 I can tell that time it was happened my ID-card reader wasn't even plugged in into the usb port. OS: windows 7 64 bit Firefox version : 48.0
Merike, do you have any additional testcase users? bp-2f14ace8-ec14-47f4-a927-5be120170830 is a user who also sees signature SmartCardMonitoringThread::~SmartCardMonitoringThread as in bp-48cd083b-9220-4f2a-a2ec-2e9340170806
Flags: needinfo?(merikes.lists)
(In reply to martin raud from comment #27) > I don't know if i am allowed, but adding 1 more crash report on the list. Martin, certainly you are. Can you still reproduce this crash?
Flags: needinfo?(martinraud)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #28) > Merike, do you have any additional testcase users? > > bp-2f14ace8-ec14-47f4-a927-5be120170830 is a user who also sees signature > SmartCardMonitoringThread::~SmartCardMonitoringThread as in > bp-48cd083b-9220-4f2a-a2ec-2e9340170806 No. I think this bug was mostly about [@ PK11_IsDisabled ] though, which is almost non-existent compared to 5y ago. I cannot find the details atm but supposedly there was a fix in opensc which should be responsible for that. Also, most Estonian ID-card users who haven't disabled it on purpose should have {aa84ce40-4253-a00a-8cd6-0800200f9a67} extension in their crashes too.
Flags: needinfo?(merikes.lists)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #28) Forgot to say, if you have any extra cycles for smartcard bugs, then please look into signature onepin-opensc-pkcs11.dll%400x130938 :)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #29) > (In reply to martin raud from comment #27) > > I don't know if i am allowed, but adding 1 more crash report on the list. > > Martin, certainly you are. Can you still reproduce this crash? No i can't. I guess it fixed now.
Flags: needinfo?(martinraud)

Following the reporter's steps I am able to confirm that the issues doesn't happen anymore on Windows 10 on any of the current versions of Firefox Nightly 87.0a1 (2021-02-16), beta 86.0 and release 85.0.2.
Also the last comment fro 4 years ago might suggest it's fixed as well.

Closing this issue as Resolved > Worksforme.
Feel free to re-open or file a new bug if this issue reoccurs again.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.