1631749 - Firefox crashes every time I click to show or copy a password in about:logins due to BLEtokenCredentialProvider.dll

Status: RESOLVED FIXED

(Firefox :: about:logins, defect, P1)

Description

[jimstag999]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

click profile icon
click logins and passwords
go to any password and click copy password or click the show password icon

Actual results:

a windows security popup come up requesting my computers login pin/password
whether i close it or input the password firefox closes/crashes without the crash reporter opening
when i next launch firefox it resumes the previous session and reloads the tabs

Expected results:

password would have been copied or shown

Comment 1

[Matthew N. [:MattN]]

Did you submit a crash report? Please load about:crashes and give a link or crash report ID from the time when it crashed.

Comment 2

[jimstag999]

that's part of the problem, no crash report is generated

Comment 3

[Matthew N. [:MattN]]

Sorry, I didn't read comment 0 close enough to see that.

Can you find an entry in the Windows Event Viewer when you get the crash? I imagine they would be under the "Application" section. There are some instructions at https://www.partitionwizard.com/partitionmagic/windows-10-crash-log.html which may be useful. If you can see the crash there, please attach/copy the contents to this bug. Thank you.

Comment 4

[jimstag999]

i think this is what you want

Log Name: Application
Source: Application Error
Date: 4/22/2020 11:05:12 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: i
Description:
Faulting application name: firefox.exe, version: 76.0.0.7415, time stamp: 0x5e9d24ef
Faulting module name: ntdll.dll, version: 10.0.17763.1158, time stamp: 0x54d5d94b
Exception code: 0xc0000374
Fault offset: 0x00000000000fb0b9
Faulting process id: 0x29e8
Faulting application start time: 0x01d6187b2906d08b
Faulting application path: C:\Program Files\Mozilla Firefox\firefox.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: a58736de-e88d-4dc6-a440-00b52022f846
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-04-22T08:05:12.001998900Z" />
<EventRecordID>44056</EventRecordID>
<Channel>Application</Channel>
<Computer>i</Computer>
<Security />
</System>
<EventData>
<Data>firefox.exe</Data>
<Data>76.0.0.7415</Data>
<Data>5e9d24ef</Data>
<Data>ntdll.dll</Data>
<Data>10.0.17763.1158</Data>
<Data>54d5d94b</Data>
<Data>c0000374</Data>
<Data>00000000000fb0b9</Data>
<Data>29e8</Data>
<Data>01d6187b2906d08b</Data>
<Data>C:\Program Files\Mozilla Firefox\firefox.exe</Data>
<Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
<Data>a58736de-e88d-4dc6-a440-00b52022f846</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>

Comment 5

[Matthew N. [:MattN]]

Thank you. According to https://stackoverflow.com/a/13413054 0xc0000374 means heap corruption and suggests faulty RAM or a buffer overrun as the likely cause. Can you run Windows Memory Diagnostic to see if it finds problems with your RAM?

Jimm, is comment 4 enough info to find submitted Windows Error Reports of this problem? I assume you have access to the WinQual replacement but if not then please redirect. We need to know how common this is to know whether we can ship this feature from Beta 76. Maybe the report would also have more info to point to a cause.

Comment 6

[Matthew N. [:MattN]]

Since we can't really test this native Windows UI in automation we aren't running tests against it and therefore ASAN builds aren't running this code (it sounds like ASAN builds could catch this if it's a buffer overrun but I could be wrong). Jared, can you try running a Windows ASAN build and trying to trigger the issue?

jimstag999, maybe you can also try run the address sanitizer beta build from here (extract the zip and run firefox/firefox.exe which will use a new Firefox profile). Our documentation suggests that [it will output a message before a crash if it finds an issue] so you may need to launch it from a command prompt in order to see that output but I'm not an expert in this.

Comment 7

[Jared Wein [:jaws] (please needinfo? me)]

I have tested an ASAN-opt and ASAN-debug build from https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=win%2Casan&revision=d8eecc663784c8463af1d2bc3f91f8078c7e1940&selectedJob=298793232 without any luck reproducing this.

Comment 8

[Jared Wein [:jaws] (please needinfo? me)]

Thank you for your bug report jimstag999.

Can you describe the characteristics of your Windows accounts? Are they assigned to a domain? How many accounts do you have? What permission level is the current user account (administrator / guest / etc)? Do you have a PIN or other non-password authentication methods enabled? Is your account a local account or a Microsoft account?

Also, do you have any 3rd party (non-Microsoft) security software installed on your computer?

Comment 9

[Jim Mathies [:jimm]]

(In reply to Matthew N. [:MattN] (PM me if request are blocking you) from comment #5)

Jimm, is comment 4 enough info to find submitted Windows Error Reports of this problem? I assume you have access to the WinQual replacement but if not then please redirect. We need to know how common this is to know whether we can ship this feature from Beta 76. Maybe the report would also have more info to point to a cause.

No unfortunately not. All we have there is top frame info, I didn't see anything in nt.dll. I'm surprised the crash reporter doesn't trigger here but maybe that's due to the thread being wrapped up in authentication related calls. Some suggestions -

• Have the user trigger some other crash so you can get a look at generic crash data for the device. (Like injected modules)
• Identify the device and buy one, see if you can reproduce.
• Try to figure out which winapi call triggers the crash.

Comment 10

[Gabriele Svelto [:gsvelto]]

We can investigate further by trying to get WER to create a local dump of the crash. This can be done like so:

1. In the Windows registry create the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps key and leave it empty
2. Use Firefox until it crashes
3. Once Firefox has crashed dismiss the debugger window you're presented and go into %LOCALAPPDATA%\CrashDumps (e.g. C:\Users\<name>\AppData\Local|CrashDumps) you should find a file called firefox.exe.<number>.dmp there
4. Send me the .dmp file via e-mail, do not attach it to the bug since it might contain sensitive information

If something went wrong and no .dmp file was generated try again but this time launch firefox manually via cmd.exe with the MOZ_CRASHREPORTER_DISABLE=1 variable set, e.g.:

set MOZ_CRASHREPORTER_DISABLE=1
cd "C:\Program Files\Mozilla Firefox" # Or the correct path to Firefox installation directory
firefox.exe

Once you're done you can delete the LocalDumps registry key you created, this will prevent WER from gathering more minidumps. See this page for more information on this:

https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

Comment 11

[Matthew N. [:MattN]]

jimstagg999, it would also be good to know what your Windows 10 build number is? (e.g. 1809 vs. 1909) It seems like some related issues have been fixed in 1909.

(In reply to Jim Mathies [:jimm] from comment #9)

(In reply to Matthew N. [:MattN] (PM me if request are blocking you) from comment #5)

Jimm, is comment 4 enough info to find submitted Windows Error Reports of this problem? I assume you have access to the WinQual replacement but if not then please redirect. We need to know how common this is to know whether we can ship this feature from Beta 76. Maybe the report would also have more info to point to a cause.

No unfortunately not.

Are you saying you have access to this data but this bug doesn't have enough info to find the reports? I guess you can see if any top frames mention windows.ui.cred*.dll?

All we have there is top frame info, I didn't see anything in nt.dll. I'm surprised the crash reporter doesn't trigger here but maybe that's due to the thread being wrapped up in authentication related calls. Some suggestions -

• Have the user trigger some other crash so you can get a look at generic crash data for the device. (Like injected modules)

Good idea. jimstag999, can you follow all but the last step of the Windows instructions at https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox#Submit_a_crash_report_for_a_hung_Firefox to trigger an unrelated crash of Firefox so we can see details about your system and Firefox code.

• Identify the device and buy one, see if you can reproduce.
• Try to figure out which winapi call triggers the crash.

https://searchfox.org/mozilla-central/rev/41c3ea3ee8eab9ce7b82932257cb80b703cbba67/security/manager/ssl/OSReauthenticator.cpp#28-231 is the relevant code so MOZ_LOG can help narrow this down…

jimstag999, can you please record some logs for us…

1. Load about:networking#logging in the address bar
2. Set Log Modules to timestamp,sync,credentialmanagersecret:5 by filling the field with this and hitting the "Set Log Modules" button.
3. Click "Start Logging"
4. Open about:logins and reveal a password to trigger the crash.
5. Attach log.txt-child.NNNN.moz_log (not the log.txt-main* one) from the directory shown in the log file textbox on about:networking#logging to the bug. It should be in C:\Users\*your_username*\AppData\Local\Temp\log.txt-child.*NNNN*.moz_log It shouldn't contain any sensitive info but you can open it in Notepad to confirm.

Thanks for the ideas JimM.

Comment 12

[jimstag999]

Matthew N
I followed the steps to provide you with logs
i did it twice, generating 4 files each time
3 of 4 files each time are 0KB, i checked with notepad++, they are empty
1 of 4 files had this line only in them:
[GPU 7788: Main Thread]: I/Logger Flushing old log files
[GPU 7700: Main Thread]: I/Logger Flushing old log files

Comment 13

[Matthew N. [:MattN]]

OK, so I guess you are crashing before our first message is logged which is still helpful to know.

Hopefully you can also provide answers for comment 5, comment 6, comment 8, comment 10 and comment 11. Sorry there are so many questions but we are running out of time to fix things in Firefox 76 Beta so we are trying to figure out the cause of the crash and understanding how widespread the problem is.

Comment 14

[jimstag999]

here's my win10 info
edition: win10 home
version: 1809 (currently downloading some update)
osbuild: 17763.1158
the version is quite old installed on 3/22/2019, it has since the applied many updates idk why it shows such an old date

Comment 15

[Matthew N. [:MattN]]

Hi jimstaff999, were you able to run the memory test? Does the problem still happen after restarting your computer? I know we're asking for a lot but if you have answers to some of the other questions it would really help us know whether we can ship the current code in 76 very soon. We are running out of time to address any issues but so far you are the only one reporting this issue.

Comment 16

[jimstag999]

I run memtest86, no errors, updated to firefox 76.0b8, it still crashes the same way
as for me being the only one reporting this issue, i AM incredibly unlucky, no joke
maybe something is botched on my computer from microsofts side of things

Comment 17

[Matthew N. [:MattN]]

Thanks. Very interesting.

If you can provide the info from comment 10 we would have a better idea if it is a Windows issues, Firefox, or some third-party software (e.g. A/V or anti-malware or malware).

Let me know if you're not interesting in debugging this anymore. I don't want to keep bothering you for the info we asked about earlier.

Comment 18

[Gabriele Svelto [:gsvelto]]

Yep, my bet is on an injected DLL. We've seen so many crashing Firefox as of late.

Comment 19

[jimstag999]

ok i have the .dmp and i'm sending it to the email address on your profile
[e-mail]

Comment 20

[jimstag999]

also sending it to [e-mail]

Comment 21

[Matthew N. [:MattN]]

Thank you very much! Gabriele is looking at the dump. Assigning to him for now. Feel free to unassign and move components as needed.

Comment 22

[Gabriele Svelto [:gsvelto]]

I've tracked down the crash to the BLEtokenCredentialProvider.dll DLL. This seems to be part of the drivers from a Bluetooth dongle from Cambridge Silicon Radio. The driver might be quite old since the company was bought by Qualcomm at some point a few years ago. The DLL seems to be offering a credential service provider and it's crashing Windows code trying to call it. Since the crash is happening on the logins/passwords page my guess is that we're using Windows CSP services there (for stuff like WebAuthn) and those reach the DLL and ultimately trip over it.

Can you check if you have that DLL installed? Save a copy of it somewhere and then delete it from wherever you found it (or uninstall the entire driver if it appears in your list of installed programs) then check if the crash goes away. If it does send me a copy of the DLL and we'll block it to save other users from running into the same problem.

Comment 23

[Matthew N. [:MattN]]

That's great that you figured out the cause. The relevant code is https://searchfox.org/mozilla-central/rev/b8fbb6ead517720daf0b0211115f407b4b951c74/security/manager/ssl/OSReauthenticator.cpp#85,152 btw. We are requesting that the Windows Auth. dialog appear to protect logins from snooping (like Chrome) but this only works on Windows/macOS. We can request only certain auth packages are used but idk if that would prevent the crash and being able to use third-party auth providers may be convenient.

Comment 24

[Matthew N. [:MattN]]

Is there any way our crash reporter could catch this?

Comment 25

[Gabriele Svelto [:gsvelto]]

(In reply to Matthew N. [:MattN] (PM me if request are blocking you) from comment #23)

That's great that you figured out the cause. The relevant code is https://searchfox.org/mozilla-central/rev/b8fbb6ead517720daf0b0211115f407b4b951c74/security/manager/ssl/OSReauthenticator.cpp#85,152 btw. We are requesting that the Windows Auth. dialog appear to protect logins from snooping (like Chrome) but this only works on Windows/macOS. We can request only certain auth packages are used but idk if that would prevent the crash and being able to use third-party auth providers may be convenient.

Thanks for the pointer, that's a part of the codebase I'm not familar with.

(In reply to Matthew N. [:MattN] (PM me if request are blocking you) from comment #24)

Is there any way our crash reporter could catch this?

It's tricky. The crash reason here is STATUS_HEAP_CORRUPTION which by default on Windows terminates the process w/o invoking the structured exception handler. I've looked around and the only way to catch this is via a vectored exception handler which we don't use in crash reporting code - and neither Chrome does AFAIK. I have a vague memory that we decided against using VEH for "reasons" but I can't quite remember the details. Anyway since I'm rewriting the whole thing in bug 1588530 I'll file a bug to see if we can use it to catch more crashes.

Comment 26

[Gabriele Svelto [:gsvelto]]

I just remembered why we're not catching STATUS_HEAP_CORRUPTION errors: we're using HeapSetInformation() with the HeapEnableTerminationOnCorruption option to mitigate heap corruption issues. That causes Windows to terminate the affected process immediately w/o doing exception handling.

Comment 27

[Gabriele Svelto [:gsvelto]]

I filed bug 1633052 to investigate how to catch those exceptions and grab crash reports when they happen.

MattN edit: fixed bug number

Comment 28

[jimstag999]

I found the .dll, here's a firefox send link, i can't send it in an email
https://send.firefox.com/download/d597987e4afa9992/#MzXTrfX3Rqo7ZLuxZuFmUg

Comment 29

[Gabriele Svelto [:gsvelto]]

Thanks! Does removing it solve your crash?

Comment 30

[jimstag999]

Indeed it does, thanks to everyone that took part in this discussion in these hard times, much better response than any other company/community
where i encountered a problem or a bug

Comment 31

[Matthew N. [:MattN]]

You mentioned earlier about adding this DLL to the blocklist. Will that work in this case where I presume the DLL is loaded by Windows due to OS re-auth prompt appearing?

Comment 32

[Ryan VanderMeulen [:RyanVM]]

Downgrading the severity of the bug based on the investigation that's been done. Happy to take a DLL blocklist update still, but it'd be in either an RC respin or dot release ride-along at this point.

Comment 33

[Gabriele Svelto [:gsvelto]]

(In reply to Matthew N. [:MattN] (PM me if request are blocking you) from comment #31)

You mentioned earlier about adding this DLL to the blocklist. Will that work in this case where I presume the DLL is loaded by Windows due to OS re-auth prompt appearing?

Possibly not, it needs to be analyzed. We've got a wiki page describing how it must be done and more specifically there's a questionnaire that needs to be filled up before we do and it basically walks you through the process.

Here's the info I extracted from the DLL which will help you fill the questionnaire:

• name - BLEtokenCredentialProvider.dll
• version - 2.1.63.0
• signed by - Cambridge Silicon Radio Ltd.
• date of signature - 2012-03-22, this stuff is positively ancient

One of the questions you'll have to answer is if there are more up-to-date versions of this DLL that do not have the same problem. I could not find any versions of this DLL more recent than this one and the company that wrote it has been gone for years, acquired by Qualcomm at some point so my guess is that there aren't.

Comment 34

[Matthew N. [:MattN]]

Thanks Gabriele,

Jared, is this something you can do?

Looks we have some data on the prevalence via the third-party-modules ping: https://sql.telemetry.mozilla.org/queries/67485?p_token_67485=cambridge => 0.00622%

Comment 35

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: DLL Blocklist request

Comment 36

[Gabriele Svelto [:gsvelto]]

Thanks for picking this up Jared.

Comment 37

[Jared Wein [:jaws] (please needinfo? me)]

Status update: This work is being tracked in bug 1634538. I don't expect a patch to be supplied in this bug. I am planning to close this when bug 1634538 is fixed.

Comment 38

[Cosmin Muntean [:cmuntean], Ecosystem QA]

In order to verify this issue first I tried to reproduce it on Firefox 76.0.1 release:

• I have searched on the Internet and I have found the CSR-Harmony-Wireless-Software-Stack 2.1.63.0 driver. I have installed it on my personal machine with Windows 10 Pro version 1903 build 18362.836. Unfortunately, I haven't managed to reproduce the issue, the show/copy/edit password worked as expected after I entered the OS credentials. Tested on Firefox 76.01 release and Nightly 78.0a1.
• Since @jimstag999 mentioned in comment 14 that he encountered this issue on Windows 10 Home version 1809, I have installed this version of Windows 10 on a virtual machine. Unfortunately the issue is still not reproducible on my end even on this version of Windows. Tested on Firefox 76.01 release and Nightly 78.0a1.

Indeed, I can find the BLEtokenCredentialProvider.dll file on the driver path on my computer, but I cannot reproduce the issue. I'm still trying to figure out if I cannot reproduce it because I don't have a Mini USB Bluetooth dongle and never used one on the computer to connect Bluetooth devices or because I'm using a virtual machine. From what I read, the Adaptor Bluetooth Orico BTA-403, v4.0, and the Mini Bluetooth CSR 4.0 USB 2.0 CSR4.0 Dongle Adapter uses the CSR-Harmony-Wireless-Software-Stack driver.
However, I will try to acquire a USB Bluetooth dongle and try to use it and see if I can reproduce the issue so I can verify the fix.
@jimstag999 did you used any Mini USB Bluetooth and had any Bluetooth devices connected on your computer while you encountered this issue?

@Gabriele, @Matt please let me know if you have any thoughts or scenarios that would help to reproduce the issue.

Comment 39

[Gabriele Svelto [:gsvelto]]

I suspect the installing the driver is not sufficient to cause the BLEtokenCredentialProvider.dll token provider to be injected. It's possible that the DLL was registered for injection only after the hardware was installed and triggered the actual driver registration. You might try with a tool such as MemJect to see if you can forcibly inject the DLL into Firefox.

Comment 40

[jimstag999]

The issue happened without the dongle plugged in, since i uninstalled the driver everything works fine.
I suggest closing this thread if you can't reproduce the bug and no other user is affected.

Comment 41

[Cosmin Muntean [:cmuntean], Ecosystem QA]

@Gabriele thank you for your suggestion. Unfortunately I didn't manage to inject the dll file into the Firefox process with the MemJect tool. I have also tried with a different one but didn't work. I got some errors that I cannot inject the dll file into Firefox.

The remaining scenario is to use a USB Bluetooth dongle. I will come back with a comment when I manage to test this.

Comment 42

[Matthew N. [:MattN]]

My only idea would be if the reporter still has a copy of the installer that they could send it on https://send.firefox.com/ as maybe the version you found wasn't the same.

Comment 43

[jimstag999]

here is the .iso ripped from the install disc
https://send.firefox.com/download/a073b1de3ec597af/#LBAko93JNdEV3J7JikSlGA

Comment 44

[Cosmin Muntean [:cmuntean], Ecosystem QA]

@jimstag999 thank you for sending the installer.

I have installed the driver on both Windows 10 versions (1809 and 1903), but no luck. I wasn't able to reproduce the issue on Firefox 76 release. However, I will retest this in a few days using a USB Bluetooth dongle and I will come back with the results.

Comment 45

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have managed to connect a USB Bluetooth dongle to my PC. The "CSR-Harmony-Wireless-Software-Stack" driver recognized it and I have managed to connect a device via Bluetooth. Unfortunately, I still cannot reproduce the issue on Firefox 76. I tried on two Windows 10 versions (1809 and 1909).

I have looked over the driver's settings and there is a feature available "BLUETOOTH SECURITY". Probably this feature inject the .dll file, but I didn't manage to use it. Not sure if I need any specific device or probably I need a laptop that has a Bluetooth incorporated.

@jimstag999 did you had any CSR SECURITY Token set, or the Bluetooth device required any login? Also, did you reproduced the issue on a laptop?

Comment 46

[jimstag999]

i don't know what CSR SECURITY Token set is but the bluetooth device didn't have anything to do with any login of any sort, also the bluetooth dongle was not required for the crash to happen
i didn't reproduce the issue on any device, since all my other devices run linux
Also, i have uninstalled the driver and the issue has disappeared
I suggest closing this thread if you can't reproduce the bug and no other user is affected, not to waste anyones time

Comment 47

[Matthew N. [:MattN]]

Cosmin, did you try installing the software from comment 43? That would probably allow you to reproduce the issue.

Jim, sorry if these conversations are annoying but we think it's still useful to know if our DLL blocklist entry is working in this case.

Edit: sorry, I see comment 44 now.

Comment 48

[Matthew N. [:MattN]]

Since this is quite old I wonder if it only works for 32-bit systems or older versions of Windows 10?

Comment 49

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have installed on a VM the same Windows version on which Jim encountered the issue (Windows 10 Home version 1809 x64) and installed the driver provided in comment 43. I have also tried to verify this on Windows 10 version 1903 on my personal PC without any VM, but no luck.
Probably I need a laptop that has a Bluetooth incorporated and somehow to enable the "BLUETOOTH SECURITY" feature to trigger the issue.

However, I can give it a try and install an x32 Windows version or an older one.

Comment 50

[Matthew N. [:MattN]]

If you already matched what Jim has then I think you tried enough and we can just make this qe-verify-. Thanks for trying!