Firefox crashes with Estonian pkcs#11 module (slot->pk11slot is probably NULL)

NEW
Unassigned

Status

()

--
critical
8 years ago
2 years ago

People

(Reporter: karumaru, Unassigned)

Tracking

({crash})

6 Branch
crash
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [tbird crash], crash signature)

(Reporter)

Description

8 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110811165603

Steps to reproduce:

Randomly open links in internet.


Actual results:

At some point Firefox crashes. No specific web page needed. usually crash happens when you have just clicked the link or webpage is loading.

Crash IDs

bp-60e3ed96-ce5f-492f-b6df-7a8282110829; - in safe mode
bp-d04ce781-2dd2-4441-b82a-2ce132110829;
bp-1ddc3390-0fa2-4a0d-b192-92ef22110829;
bp-65d47de1-33f9-437f-9fcb-7abed2110829;
10cafe05-57cd-422c-b2c2-e68ae2472a64;
bp-55ee03dc-2a80-420a-80e3-0629d2110829;
bp-c6d60fb5-6e43-4851-99a0-6b1032110828;
bp-6f62bd5e-7415-4528-b279-f2b0e2110828;
bp-f7fd0863-070e-4d69-893a-9991a2110826;
bp-a879e60b-b812-439b-9672-e9dc72110826;
bp-36c39aed-a966-45d2-80bc-dfa082110826;
bp-5d86ad59-6187-4bc7-a326-e31152110826;
bp-5b9631c2-1b5b-4120-a6e1-db2412110826;
bp-a5f04b87-039a-4b1e-8b69-843c02110825;
bp-1457acd5-6d35-4a48-a68b-b471e2110824;
bp-ed8480ea-7b2c-4e06-8d8c-348882110824;
bp-2052d295-25dd-483c-8804-87ee22110823;
bp-aedc2a89-38d8-4be7-92f7-a6e9a2110823;


Expected results:

FF should not crash
(Reporter)

Comment 1

8 years ago

  Application Basics

        Name
        Firefox

        Version
        6.0

        User Agent
        Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0

        Profile Directory

          Open Containing Folder

        Enabled Plugins

          about:plugins

        Build Configuration

          about:buildconfig

  Extensions

        Name

        Version

        Enabled

        ID

        Adblock Plus
        1.3.9
        true
        {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}

        BlockSite
        0.7.1.1
        true
        {dd3d7613-0246-469d-bc65-2a3cc1668adc}

        Browsing Protection
        1.10
        true
        litmus-ff@f-secure.com

        Estonian ID Card PKCS11 module loader
        3.4.3756.234
        true
        {aa84ce40-4253-a00a-8cd6-0800200f9a66}

  Modified Preferences

      Name

      Value

        accessibility.typeaheadfind.flashBar
        0

        browser.places.smartBookmarksVersion
        2

        browser.startup.homepage_override.buildID
        20110811165603

        browser.startup.homepage_override.mstone
        rv:6.0

        extensions.lastAppVersion
        6.0

        gfx.blacklist.webgl.angle
        3

        network.cookie.prefsMigrated
        true

        places.database.lastMaintenance
        1314363010

        places.history.expiration.transient_current_max_pages
        125270

        privacy.sanitize.migrateFx3Prefs
        true

  Graphics

        Adapter Description
        NVIDIA NVS 4200M

        Vendor ID
        10de

        Device ID
        1056

        Adapter RAM
        1024

        Adapter Drivers
        nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um

        Driver Version
        8.17.12.7533

        Driver Date
        5-20-2011

        Direct2D Enabled
        true

        DirectWrite Enabled
        true (6.1.7601.17563)

        ClearType Parameters
        ClearType parameters not found

        WebGL Renderer
        NVIDIA Corporation -- NVS 4200M/PCI/SSE2 -- 4.1.0

        GPU Accelerated Windows
        1/1 Direct3D 10

Updated

8 years ago
Crash Signature: [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ]

Updated

8 years ago
Severity: normal → critical
Keywords: crash
What is "_etoured.dll" and "libltdl3.dll" that are injected into the firefox process ?
I think this is a dupe of Bug 679846 based on the common stack and the unversioned dlls in the module of his crash reports.

Updated

8 years ago
Crash Signature: [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ] → [@ _SEH_epilog4 ] [@ hash_access ] [@ GetItem ] [@ CreatePenIndirect ] [@ SdbInitDatabase ] [@ PL_DHashTableOperate ] [@ PK11_IsDisabled ]
(Reporter)

Comment 5

8 years ago
Any ideas how this issue could be solved?

Basically when using FF it crasehs after every 30 minutes.
I have uninstalled everything, deleted all personal info related to FF and installed clean version. Didn't import anything, no customisation but still crashes.
In safe mode it crashes.

Scanned computer with F-secure: no threads found. 
Installed FF3.6 and this is also crashing now.

I am ready to provide more specific info if you will ask.

I don't want to change browser as I like FF.
>I am ready to provide more specific info if you will ask.
see comment #2
Those dlls shows up in your crash report.

Updated

7 years ago
Depends on: 716345

Comment 7

7 years ago
I got here via "PK11_IsDisabled" crash signature, the libltdl3.dll is most likely related to the Estonian ID-card software: http://systemexplorer.net/filereviews.php?fid=1798080 (Looking at the directory value; I can't find that file on my system, but I might just have updated version of the software which doesn't include that dll anymore)

I've also had problems of Firefox crashing completely randomly when I upgraded to v.10 (including 10.0.2). For me it was fixed by removing "Estonain Id Card" module under security devices (There's an add-on called Estonian ID Card PKCS11 Module Loader which I guess re-registered the right module).

Hopefully this will help, especially with PK11_IsDisabled crashes which seem to have hit **** Firefox 10.0.2 Estonian users (considering the foul-mouthed comments): https://crash-stats.mozilla.com/report/list?signature=PK11_IsDisabled
confirmed on the crash data for PK11_IsDisabled - also adding kai to this bug. Kai see comment 7
Status: UNCONFIRMED → NEW
Component: General → Security
Ever confirmed: true
Product: Firefox → Core
QA Contact: general → toolkit
The crash reports were produced with Firefox 6 - which is quite old already.

Would you be able to please try with a more modern Firefox, which includes many fixes?

Updated

7 years ago
Summary: Firefox crashes at random points all the time- even in SafeMode → Firefox crashes with Estonian pkcs#11 module (slot->pk11slot is probably NULL)
So a little update to get into the current situation:

On the 3rd of April version 3.5 SP1 of the ID-card software was released. The changelog is here: http://support.sk.ee/eng/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=1554 It mentions fixing of firefox component, and I haven't noticed any crashes lately.

BUT the update checking isn't really often, as I got a pop-up about update being available last Friday (the 4th of May, a month after). So expect people to update the software slowly.

That being said the old version of the ID-card software (3.5) does crash Firefox 12.0: https://crash-stats.mozilla.com/report/index/bp-0c6dbedf-d823-4766-a7e5-84a3a2120502

[CRASH]

Okay, totally just jinx'd it. Happened while I was scrolling this very page using middle-click: https://crash-stats.mozilla.com/report/index/d85232a9-4466-4e54-bdda-ba1ab2120507
Thanks, so we still crash with modern NSS, in particular NSS 3.13.3

Bob, Wan-Teh, see 
https://crash-stats.mozilla.com/report/index/d85232a9-4466-4e54-bdda-ba1ab2120507
for the stack.

nssTrustDomain_GetActiveSlots iterates tokens, 
calls PK11_IsDisabled(slot->pk11slot) for each of them,
and PK11_IsDisabled crashes trying to dereference the slot.

Can you imagine why slot->pk11slot might be invalid or NULL?

Should nssTrustDomain_GetActiveSlots check for (slot->pk11slot == NULL) ?
Any updates on this?

This is still a problem with Estonian ID Card PKCS11 module loader 3.6.0.670 and latest Firefox beta. Roughly 1000 users per week crash into this according to crash stats. Considering that http://www.id.ee/index.php?id=30011&read=36092 reports about half a million users for this software it means that on average 0,2% of users crash into it weekly. I've seen it twice in last three weeks.

Interestingly unlike original reporter crashes seem to be x86 only now.
Merike, I've discussed with our pkcs#11 expert Bob Relyea,
and he says my questions cannot be answered easily.

Someone with pkcs#11 debugging skills and access to a smartcard would have to attempt to reproduce the issue, to find out if it's more likely to be a bug in the driver software or in the NSS code.
According to latest crashes this also affects Mac and 64bit. And also people with 3.7 series of Estonian ID Card PKCS11 module loader.
OS: Windows 7 → All
Hardware: x86_64 → All
Since it doesn't show any kind of decline whatsoever..

What are the odds to figure this one out with:
 * one person with pkcs#11 skills
 * the other with access to a smartcard, the computer that crashed once in December and once in January but not since, and basic debugging skills
?
(In reply to Merike (:merike) from comment #15)
> 
> What are the odds to figure this one out with:
>  * one person with pkcs#11 skills
>  * the other with access to a smartcard, the computer that crashed once in
> December and once in January but not since, and basic debugging skills
> ?

It would be necessary to have a reliable test case (crash each time with certain steps) in order to debug the issue.
(In reply to Kai Engert (:kaie) from comment #16)
> It would be necessary to have a reliable test case (crash each time with
> certain steps) in order to debug the issue.
This assumes that it isn't too random or a rare race condition..

The comments on crash reports imply that some people crash into it (or something else) daily or even multiple times a day. Is there anything we could ask them that would help clarify the issue and help to find STR? A possible driver bug was mentioned earlier so perhaps we could ask for reader and driver details at minimum?

About current crash reports, are there some recent ones with email addresses from users whose crash-rate is high? Marcia, maybe you know how to find that out?
> perhaps we could ask for reader and driver details at minimum?

Yes. It would be good to know, if only one reader model crashes, and maybe even some driver versions crash. Collecting this information will be helpful to either set up a test environment, or tell people to stop using outdated drivers.

Are all crash reports on Windows?


> Is there anything we
> could ask them that would help clarify the issue and help to find STR?

It would be best if you can find someone who can never complete a specific activity and always crashes. That would allow the user to repeat the test as often as necessary.

If you cannot find someone with such STR, then the crash could be random, and I don't know what we could do.

I don't know if the driver for the smartcard is open source. If it were, and if the smartcard works on Linux, too, then you could ask a developer with access to a smartcard to use the "valgrind" memory debugging tool, to check if the smartcard driver has bugs.
(In reply to Kai Engert (:kaie) from comment #18)
> Are all crash reports on Windows?
No, but linux crashes are extremely rare (none in last 4 weeks) and Mac is also rare (only 2 in last 4 weeks).

> I don't know if the driver for the smartcard is open source. If it were, and
> if the smartcard works on Linux, too, then you could ask a developer with
> access to a smartcard to use the "valgrind" memory debugging tool, to check
> if the smartcard driver has bugs.
Smartcard is usable on linux. I don't understand the whole stack much but the userspace software that provides Firefox plugin is open source. It needs pcscd and opensc to work and in my case I use binary driver for Omnikey 4040 PCMCIA reader. Source for that is available so debugging should be possible. But I only crashed twice and both times while using rdp to a windows computer (which also has a reader attached) so I'm probably not a good case for reproducing this bug.
I was able to contact one of the frequent crashers. He was also experiencing flash crashes and hangs but also crashes with this signature couple of times a week if using smartcard.

The reader was Omnikey 1021 USB-reader which was given out by SEB (one of the two most popular banks here) and is very common. This means it either affects very different readers although from the same manufacturer or isn't reader specific at all.

The latest crash happened on outlook.com when clicking reply button. It doesn't reproduce with the same steps for him. Without knowing security code I'm suspecting (especially in the light of bug 550258 not being fixed) it happens when Firefox initiates new ssl connection that accesses smartcard info and something goes wrong with multiple threads assuming different things. But exact trigger conditions aren't any clearer than before.

By the way, any ideas why foreign ssl sites would trigger a read on the reader? I'm sometimes seeing it blink when loading some (no recognizable pattern) site which is either ssl itself or references ssl sources. I don't believe any of them actually access card info on purpose.

Comment 21

6 years ago
Thank Merike,

The real issue isn't the reader, per se. It's most like the PKCS #11 driver that's in use. PKCS #11 drivers are 'foriegn code' (as in code that's not part of NSS or mozilla, not necessarily code that was developed in another country) the runs as a pluggin. That PKCS #11 module could be doing all sorts of weird things.

Basically without a copy of the module and a card, there's very little the NSS team can do, so it's not surprising the problem isn't fixed.

bob
(In reply to Robert Relyea from comment #21)
It's most like the PKCS #11 driver
> that's in use. PKCS #11 drivers are 'foriegn code' (as in code that's not
> part of NSS or mozilla, not necessarily code that was developed in another
> country) the runs as a pluggin. That PKCS #11 module could be doing all sorts of weird things.
You might have made this a bit clearer for me :) By plugin do you mean code like https://svn.eesti.ee/projektid/idkaart_public/trunk/esteid-plugin/firefox-win/? This part is developed in Estonia and if it was possible to point out something they do in a wrong way it may be possible to make them fix it.

I'm afraid that getting a test card to someone in NSS team would be more difficult. (standard process goes like https://www.sk.ee/en/services/testcard) Although if you're sure the issue lies in code referenced above then again it might be possible.
Just crashed into this with Thunderbird: https://crash-stats.mozilla.com/report/index/2e4a1057-1d8d-490b-bbd4-739272130901

Right after restarting the following connections were still present:
Proto  VvJrk SaatJrk Kohalik aadress        Väline aadress          Olek       User       Inode       PID/Program name
tcp        0      0 Mary-Jo.local:38918     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38941     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38945     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38944     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38924     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38929     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:38942     OCSP.SFO1.VERISIGN:http TIME_WAIT   root       0           -               
tcp        0      0 Mary-Jo.local:54647     OCSP.LAX2.VERISIGN:http TIME_WAIT   root       0           -               

I had been using ID-card earlier with Firefox but the card was not inserted during the crash. Actually even pcsd was stopped (I only run it while using the card actively). Admittedly this is not what a regular user does, they have it running all the time unless it crashes?

Still, it seems that revocation checking is one way to crash into it. Can anyone clarify when revocation checks are triggered? This might help to figure out STR.
~#200 crash for Thunderbird 24.0.1 for the PK11_IsDisabled signature
Whiteboard: [tbird crash]
This is statistically less of a problem than it used to be: 990 crashes in November and 885 in December. Sadly, still without STR. Might slightly affect Mozilla's income though:
https://crash-stats.mozilla.com/report/index/6d8dbd12-b365-4773-a66d-5f27c2141231 has user comment "Maybe 11 time in this week. And it's start hapining when I donate money to Mozilla, strage things :)"
Having Estonian ID software seems to not be a strict requirement to crash, recently a Fennec 36 crash https://crash-stats.mozilla.com/report/index/6e50a939-4bf3-40e9-bda0-f3d542150219 was logged.

Comment 27

3 years ago
I don't know if i am allowed, but adding 1 more crash report on the list.

https://crash-stats.mozilla.com/report/index/edfcd04e-1ccc-474c-a73e-3713c2160805

I can tell that time it was happened my ID-card reader wasn't even plugged in into the usb port.

OS: windows 7 64 bit
Firefox version : 48.0
Merike, do you have any additional testcase users?

bp-2f14ace8-ec14-47f4-a927-5be120170830 is a user who also sees signature SmartCardMonitoringThread::~SmartCardMonitoringThread as in bp-48cd083b-9220-4f2a-a2ec-2e9340170806
Flags: needinfo?(merikes.lists)
(In reply to martin raud from comment #27)
> I don't know if i am allowed, but adding 1 more crash report on the list.

Martin, certainly you are.  Can you still reproduce this crash?
Flags: needinfo?(martinraud)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #28)
> Merike, do you have any additional testcase users?
> 
> bp-2f14ace8-ec14-47f4-a927-5be120170830 is a user who also sees signature
> SmartCardMonitoringThread::~SmartCardMonitoringThread as in
> bp-48cd083b-9220-4f2a-a2ec-2e9340170806

No.

I think this bug was mostly about [@ PK11_IsDisabled ] though, which is almost non-existent compared to 5y ago. I cannot find the details atm but supposedly there was a fix in opensc which should be responsible for that. Also, most Estonian ID-card users who haven't disabled it on purpose should have {aa84ce40-4253-a00a-8cd6-0800200f9a67} extension in their crashes too.
Flags: needinfo?(merikes.lists)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #28)
Forgot to say, if you have any extra cycles for smartcard bugs, then please look into signature onepin-opensc-pkcs11.dll%400x130938 :)

Comment 32

2 years ago
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #29)
> (In reply to martin raud from comment #27)
> > I don't know if i am allowed, but adding 1 more crash report on the list.
> 
> Martin, certainly you are.  Can you still reproduce this crash?

No i can't. I guess it fixed now.
Flags: needinfo?(martinraud)
You need to log in before you can comment on or make changes to this bug.